LLM5 min

Escaping provider lock-in with a unified LLM interface

UnderOcean Team

Provider lock-in happens when your agent logic is written against one vendor's SDK — its message format, its function-calling schema, its streaming protocol — so that switching models means rewriting code instead of changing a config value. Every team that builds on a single LLM provider eventually hits the same wall: a pricing change, a rate limit during a traffic spike, a model deprecation notice, or simply a better, cheaper model from a competitor. If swapping models requires touching application code, you've built a dependency you didn't choose on purpose.

Why this happens even to careful teams

It rarely happens on purpose. A team picks OpenAI's SDK because it's the fastest way to ship a prototype, wires openai.ChatCompletion calls directly into agent logic, and six months later the same shape is duplicated across a dozen call sites — the agent executor, a background summarization job, an embedding pipeline, a few one-off scripts. By the time a second provider becomes attractive (better reasoning for one workload, lower cost for another, a self-hosted model for compliance reasons), untangling those call sites is a multi-week project, not a config change.

How UnderOcean avoids it: one interface, one config model

UnderOcean routes every model call — agent execution, a flow's llm_node, RAG embedding generation, GraphRAG entity extraction and community summarization — through a single unified interface built on LiteLLM, which speaks the same OpenAI-compatible request/response shape regardless of the underlying provider:

await litellm.acompletion(
    model=f"{config.provider_type}/{config.model_name}",
    messages=[...],
    api_base=config.base_url or None,   # set only for self-hosted/custom endpoints
    api_key=decrypt_api_key(config.api_key_enc),
    stream=True,
)

That single call shape works identically whether config.provider_type is openai, anthropic, gemini, azure, bedrock, or ollama — LiteLLM normalizes over 100 providers' APIs into one interface, including streaming, function/tool calling, and token usage accounting.

Underneath that call, every provider connection is the same ProviderConfig record — one schema reused for LLM, embedding, TTS, and STT connections alike, scoped to either an organization (available to every project) or a single project (overrides the org default):

class ProviderConfig:
    scope_type: ScopeType        # org | project
    scope_id: UUID
    usage_type: UsageType         # llm | embedding | tts | stt
    provider_type: ProviderType   # openai | anthropic | ollama | azure | bedrock | ...
    model_name: str
    base_url: str | None          # set only for self-hosted / custom endpoints
    api_key_enc: str               # encrypted at rest, never returned in any API response
    is_default: bool

What switching actually looks like

Because every agent, flow, and RAG pipeline references a ProviderConfig by ID rather than hardcoding a vendor SDK, switching the model behind an agent is a dropdown selection in the UnderOcean UI, not a code change: pick a different provider config, save, done. The same provider-selector component is reused for both LLM and embedding selection across the entire app — one schema, one encryption path, one UI pattern, so there's exactly one place this logic can drift.

This unlocks a few things teams actually do in production:

  • Cost/latency experiments. Point the same agent at two provider configs in a staging project and compare cost-per-execution and p95 latency before committing to one in production.
  • Graceful degradation during an outage. If a provider has a bad day, an org admin swaps the default provider config for that usage type — agents keep running against a fallback model without a deploy.
  • Self-hosted and custom endpoints. A self-hosted vLLM or Ollama deployment is configured the same way as any hosted provider: set base_url to the endpoint and pick the matching provider_type. No separate integration path.
  • Per-project overrides. A project with stricter data-residency requirements can point at a different provider than the org default, without affecting any other project.

Credentials never leave the vault

None of this works safely if switching providers means passing API keys around in plaintext. ProviderConfig.api_key_enc is envelope-encrypted at rest (AES-256-GCM) and deliberately excluded from every ProviderOut response schema — not even an org owner's browser ever receives the raw key back from the API, only a masked placeholder. POST /providers/{id}/test makes a real, server-side LiteLLM call with the stored credentials so you can verify a connection works before anything in production depends on it.

FAQ

Does switching providers change my agent's behavior? It can — different models reason differently — but nothing about your flow, prompt structure, or tool definitions has to change. You're swapping the model behind a fixed interface, not re-architecting the agent.

Can different agents in the same project use different providers? Yes. Provider selection is per-agent (and per-node, for flows), not global — a support agent can run on a fast, cheap model while a research agent runs on a stronger reasoning model, side by side in the same project.

What happens if a provider goes down mid-conversation? In-flight executions against the failed provider will error like any upstream API failure; new executions can be routed to a fallback provider config as soon as an admin flips the default — no redeploy required.

Is a self-hosted model treated as a second-class citizen? No — a self-hosted OpenAI-compatible endpoint (vLLM, Ollama, etc.) uses the exact same ProviderConfig shape and the exact same LiteLLM call path as any hosted provider.