---
name: roche-portkey-llm
description: Add an LLM feature to a backend via the Roche Portkey / Galileo AI gateway (OpenAI-compatible, serves Claude etc.). Use when a project needs to call an LLM at Roche, wants the Portkey config, a structured-JSON-output pattern, or a stateless analysis endpoint.
---

# Roche LLM via Portkey / Galileo gateway

Call an LLM (Claude, etc.) through Roche's **Galileo** AI gateway, which speaks the
**Portkey / OpenAI-compatible** chat-completions API. Requires the Roche network / VPN.

## Config (env-driven; the API key is a secret)
| Var | Value |
| --- | --- |
| `PORTKEY_BASE_URL` | `https://eu.aigw.galileo.roche.com/v1` |
| `PORTKEY_API_KEY` | your gateway key (never hard-code / commit) |
| `PORTKEY_MODEL` | e.g. `@org-aws-general-eu-west-3/eu.anthropic.claude-haiku-4-5-20251001-v1:0` |
| `PORTKEY_MAX_TOKENS` | output budget — **4000+** (see truncation gotcha) |

## Python call
```python
pip install portkey-ai

from portkey_ai import Portkey   # import lazily so the app boots without it / offline
client = Portkey(base_url=PORTKEY_BASE_URL, api_key=PORTKEY_API_KEY)
resp = client.chat.completions.create(
    model=PORTKEY_MODEL,
    max_tokens=PORTKEY_MAX_TOKENS,
    messages=[{"role": "system", "content": SYSTEM},
              {"role": "user", "content": user}],
)
content = resp.choices[0].message.content or ""
```
(The JS SDK `import Portkey from 'portkey-ai'` uses the identical `baseURL`/`apiKey`/`model`.)

## Structured JSON output (robust pattern)
1. In the **system prompt**, demand *strict JSON only* and show the exact shape/schema.
   State the output language explicitly ("write all text in English").
2. Parse tolerantly — models sometimes wrap JSON in prose/fences:
```python
import json
def extract_json(text):
    text = text.strip()
    try: return json.loads(text)
    except json.JSONDecodeError: pass
    s, e = text.find("{"), text.rfind("}")
    if s != -1 and e > s: return json.loads(text[s:e+1])
    raise ValueError("LLM did not return valid JSON")
```
3. Validate the dict with a Pydantic model (`Model.model_validate(data)`).

## Error handling that matters
- **Truncated JSON** (`Expecting ',' delimiter: line N`) = you hit `max_tokens` mid-output.
  Raise `max_tokens`; keep titles/labels short in the prompt.
- Distinguish causes: bad **input** / not configured → **422**; parse or gateway failure →
  **502**. A `json.JSONDecodeError` is a subclass of `ValueError`, so don't let it fall into
  your "bad input → 422" branch — wrap parsing and re-raise as `RuntimeError` for 502.
- Log the raw model output (first ~2000 chars) on parse failure for debugging.

## Design pattern: stateless analysis endpoint
If the frontend already holds the data (e.g. loaded rows/objects), POST it straight to the
LLM endpoint (`POST /analyze {items:[...]}`). No DB re-query, no cache coupling, trivially
testable with a `curl -d @payload.json`.

## Reference implementation
`bar-dashboard/backend/app/services/llm_service.py` + `app/routers/analyze.py`
(frontend selects items → POST → drawer renders the structured result).