Use Opus and environment variables for model selection

[hanna-paasivirta]

Short Description

Fable was retired, which broke every chat service that pointed at it. This moves the main chat model off Fable and makes it configurable. Model selection now lives in one place, and optional env vars let us change the live model without a redeploy.

Fixes #533 and #534

Implementation Details

• services/models.py owns the whole model story: a default (Opus), a per-service map, and preferred_chat_model(service).
• Each service resolves its model from its own env var if set, otherwise its code default, otherwise the global default. There is one env var per service, no catch-all.
• workflow_chat defaults to Sonnet, not Opus. It forces JSON/YAML output through structured outputs, and Opus handles that worse than Sonnet right now. The other services default to Opus.
• The service yamls no longer set a model. They point a comment at models.py.
• Optional env vars, one per agent: APOLLO_GLOBAL_CHAT_MODEL (planner), APOLLO_WORKFLOW_CHAT_MODEL, APOLLO_JOB_CHAT_MODEL. Unset by default. doc_agent has no var and runs on the default.
• Tests for models.py live in a new services/tests/unit/ directory, since models.py is a shared root module not owned by any one service.

AI Usage

Please disclose how you've used AI in this work (it's cool, we just want to know!):

• Code generation (copilot but not intellisense)
• Learning or fact checking
• Strategy / design
• Optimisation / refactoring
• Translation / spellchecking / doc gen
• Other
• I have not used AI

You can read more details in our Responsible AI Policy

[hanna-paasivirta]

I'm seeing the garbled model outputs triggered by structured outputs in workflow_chat. I've never seen these before in that service with Sonnet, but saw them in job_chat and fixed them by adding a code edit tool and limiting the use of structured outputs to that only. In order to switch to Opus safely, I may need to make a similar architecture change to workflow_chat. To keep workflow_chat on Sonnet, I'll make the model setting more fine-grained and set it by service instead.

[josephjclark]

Hi @hanna-paasivirta - I just wanted to post some initial impressions. Unfortunately this project really doesn't sit well with me, and seeing the implementation makes me very nervous.

I'm only half-way through the review and need to give it some more time, but I wanted to post where I'd got to so far.

Also - despite very heavy AI commenting there's no readme documentation about how this works (just some very spurious looking env var defaults?). We should correct that because the relationship between envs and config is murky

[josephjclark]

this comment doesn't make sense in isolation: it only makes sense if you know that the model used to be set in config. It's confusing and we should remove it. Same for the other models.

[josephjclark]

I don't actually love it 😬 Intuitively it feels that this config should be defaults for all values, and env vars can be used to override it(somehow, it's not entirely practical!)

The split now of some things being envs and some things being config values feels confusing, rigid and arbitrary

[josephjclark]

these comments are so so verbose. I think I need to start pushing back on them. The lightning codebase is probably more comment than code now.

Anyway this second sentence I don't like. It's repetitive, plus the "we can switch models without redeploying" thing is misleading. To change an env var you have to configure kubernetes and then restart the service.

It would be more accurate to say you can update it without a rebuild. But I wouldn't even say that at this level.

[josephjclark]

These sample env vars don't make sense do that? What does job_chat resolve to?

[josephjclark]

@hanna-paasivirta what I think makes more sense here is:

a) to hard-code each service to a particular model, as we do on main
b) to use env vars to drive the version of each model

So you'd have an env var OPUS_VERSION with a value of 4-8.

Then models.py would have a function like getModelVersion(name) or something. And job_chat calls getModelVersion('opus'), which returns claude-opus-4-8 (where the version suffix comes from env or a default).

The model name can still come from the config.yaml file for each service, which is where any service specific stuff lives. But it would only have a model name, not a version.

Basically this means that the env only drives the version number, not the model itself. Otherwise the code is much as it is on main right now, where the service itself makes the big decisions about which model to use, and the env just bumps the version to keep it modern.

As I've said on slack: this architecture would not have helped us with the fable switch-off: fable support needed more than just a version string, and a dynamic downgrade likely needs more to. If we want to be robust to models disappearing overnight (a very worrying precedent) we should put some thought into be better rollback solutions (I'm aware there's another PR open for this)

[hanna-paasivirta]

As I've said on slack: this architecture would not have helped us with the fable switch-off: fable support needed more than just a version string, and a dynamic downgrade likely needs more to. If we want to be robust to models disappearing overnight (a very worrying precedent) we should put some thought into be better rollback solutions (I'm aware there's another PR open for this)

We needed more because we were upgrading the model to a new one. The added protections work ok for existing models. But there's little guarantee that we can always downgrade the model easily in the future like we can now with fable/opus/sonnet

[hanna-paasivirta]

Basically this means that the env only drives the version number, not the model itself.

I think there has never been a situation where the pointer to a model version (without specifying a snapshot) would stop work working and require an intervention on our side. But to be fair a model family had never been taken down before either.