Roche LLM via Portkey / Galileo gateway

Call an LLM (Claude, etc.) through Roche’s Galileo AI gateway, which speaks the Portkey / OpenAI-compatible chat-completions API. Requires the Roche network / VPN.

Config (env-driven; the API key is a secret)

Python call

pip install portkey-ai

from portkey_ai import Portkey   # import lazily so the app boots without it / offline
client = Portkey(base_url=PORTKEY_BASE_URL, api_key=PORTKEY_API_KEY)
resp = client.chat.completions.create(
    model=PORTKEY_MODEL,
    max_tokens=PORTKEY_MAX_TOKENS,
    messages=[{"role": "system", "content": SYSTEM},
              {"role": "user", "content": user}],
)
content = resp.choices[0].message.content or ""

(The JS SDK import Portkey from 'portkey-ai' uses the identical baseURL/apiKey/model.)

Structured JSON output (robust pattern)

1. In the system prompt, demand strict JSON only and show the exact shape/schema. State the output language explicitly (“write all text in English”).
2. Parse tolerantly — models sometimes wrap JSON in prose/fences:

import json
def extract_json(text):
    text = text.strip()
    try: return json.loads(text)
    except json.JSONDecodeError: pass
    s, e = text.find("{"), text.rfind("}")
    if s != -1 and e > s: return json.loads(text[s:e+1])
    raise ValueError("LLM did not return valid JSON")

3. Validate the dict with a Pydantic model (Model.model_validate(data)).

Error handling that matters

• Truncated JSON (Expecting ',' delimiter: line N) = you hit max_tokens mid-output. Raise max_tokens; keep titles/labels short in the prompt.
• Distinguish causes: bad input / not configured → 422; parse or gateway failure → 502. A json.JSONDecodeError is a subclass of ValueError, so don’t let it fall into your “bad input → 422” branch — wrap parsing and re-raise as RuntimeError for 502.
• Log the raw model output (first ~2000 chars) on parse failure for debugging.

Design pattern: stateless analysis endpoint

If the frontend already holds the data (e.g. loaded rows/objects), POST it straight to the LLM endpoint (POST /analyze {items:[...]}). No DB re-query, no cache coupling, trivially testable with a curl -d @payload.json.

Reference implementation

bar-dashboard/backend/app/services/llm_service.py + app/routers/analyze.py (frontend selects items → POST → drawer renders the structured result).