Skip to main content
Shipfastai’s LLM abstraction layer lets you swap providers — OpenAI, Anthropic, or Google Gemini — by changing a single provider field in your API request. Your application code never needs to know which underlying SDK is in use. The get_llm_provider factory in app/packages/ai/llm.py instantiates the right client, passes the correct API key from your environment, and returns a consistent ChatResponse regardless of which model generated it.

​
Supported providers

Shipfastai ships with three first-class provider implementations inside products/pro/backend/app/packages/ai/llm.py:
Providerprovider valueDefault modelEnv var
OpenAIopenaigpt-4oOPENAI_API_KEY
Anthropicanthropicclaude-sonnet-4-20250514ANTHROPIC_API_KEY
Google Geminigeminigemini-2.0-flashGOOGLE_API_KEY
All three implement the same abstract LLMProvider interface with chat() and stream_chat() methods, so switching providers requires no changes to your route handlers.

​
Configuring OpenAI

1

Set your API key

Add your OpenAI API key to your backend .env file:
.env
OPENAI_API_KEY=sk-...
2

Send requests using the openai provider

Pass "provider": "openai" in your request body. You can optionally specify a model; if you omit it the default gpt-4o is used.
POST /api/ai/chat
{
  "provider": "openai",
  "model": "gpt-4o",
  "messages": [
    { "role": "user", "content": "Explain RAG in one sentence." }
  ],
  "temperature": 0.7,
  "max_tokens": 256
}

​
Configuring Anthropic

1

Set your API key

.env
ANTHROPIC_API_KEY=sk-ant-...
2

Send requests using the anthropic provider

POST /api/ai/chat
{
  "provider": "anthropic",
  "model": "claude-opus-4-5",
  "messages": [
    { "role": "system", "content": "You are a helpful assistant." },
    { "role": "user", "content": "Explain RAG in one sentence." }
  ],
  "temperature": 0.7,
  "max_tokens": 256
}
The AnthropicProvider automatically extracts system-role messages and passes them to Anthropic’s system parameter, so your request format is identical across providers.

​
Configuring Google Gemini

1

Set your API key

.env
GOOGLE_API_KEY=AIza...
2

Send requests using the gemini provider

POST /api/ai/chat
{
  "provider": "gemini",
  "model": "gemini-2.0-flash",
  "messages": [
    { "role": "user", "content": "Explain RAG in one sentence." }
  ],
  "temperature": 0.7,
  "max_tokens": 256
}

​
Switching providers at runtime

Because the provider and model fields are part of each request body, you can switch providers on a per-request basis without redeploying. This is useful for A/B testing models or falling back to a cheaper provider under load.
POST /api/ai/chat
{
  "provider": "openai",
  "model": "gpt-4o",
  "messages": [{ "role": "user", "content": "Hello!" }]
}
You can also enable streaming for any provider by adding "stream": true to the request body. The endpoint returns a text/event-stream response where each event is a JSON object { "token": "..." }, terminated by data: [DONE].
POST /api/ai/chat (streaming)
{
  "provider": "openai",
  "model": "gpt-4o",
  "messages": [{ "role": "user", "content": "Write a haiku about Python." }],
  "stream": true
}

​
Extending with a new provider

All providers inherit from the abstract base class LLMProvider defined in products/pro/backend/app/packages/ai/llm.py. To add a new provider, you implement two async methods and register the provider in the factory function.
1

Create your provider class

Add a new class that extends LLMProvider and implements chat() and stream_chat():
products/pro/backend/app/packages/ai/llm.py
class GroqProvider(LLMProvider):
    """Groq LLM provider."""

    def __init__(self, api_key: Optional[str] = None, model: str = "llama-3.3-70b-versatile"):
        from groq import AsyncGroq
        self.client = AsyncGroq(api_key=api_key or os.getenv("GROQ_API_KEY"))
        self.model = model

    async def chat(
        self,
        messages: list[Message],
        temperature: float = 0.7,
        max_tokens: int = 1000,
    ) -> ChatResponse:
        response = await self.client.chat.completions.create(
            model=self.model,
            messages=[m.model_dump() for m in messages],
            temperature=temperature,
            max_tokens=max_tokens,
        )
        return ChatResponse(
            content=response.choices[0].message.content or "",
            model=response.model,
            usage={
                "prompt_tokens": response.usage.prompt_tokens,
                "completion_tokens": response.usage.completion_tokens,
                "total_tokens": response.usage.total_tokens,
            },
            finish_reason=response.choices[0].finish_reason,
        )

    async def stream_chat(
        self,
        messages: list[Message],
        temperature: float = 0.7,
        max_tokens: int = 1000,
    ) -> AsyncGenerator[str, None]:
        stream = await self.client.chat.completions.create(
            model=self.model,
            messages=[m.model_dump() for m in messages],
            temperature=temperature,
            max_tokens=max_tokens,
            stream=True,
        )
        async for chunk in stream:
            if chunk.choices[0].delta.content:
                yield chunk.choices[0].delta.content
2

Register the provider in the factory

Update get_llm_provider() to handle your new provider string:
products/pro/backend/app/packages/ai/llm.py
def get_llm_provider(
    provider: Literal["openai", "anthropic", "gemini", "groq"] = "openai",
    model: Optional[str] = None,
) -> LLMProvider:
    if provider == "openai":
        return OpenAIProvider(model=model or "gpt-4o")
    elif provider == "anthropic":
        return AnthropicProvider(model=model or "claude-sonnet-4-20250514")
    elif provider == "gemini":
        return GeminiProvider(model=model or "gemini-2.0-flash")
    elif provider == "groq":
        return GroqProvider(model=model or "llama-3.3-70b-versatile")
    else:
        raise ValueError(f"Unknown provider: {provider}")
3

Export the class and add the env var

Add GroqProvider to the __all__ list in packages/ai/__init__.py, then add GROQ_API_KEY to your .env file.
Because the chat endpoint in app/api/ai/chat.py delegates entirely to get_llm_provider(), your new provider is immediately available to all routes — including streaming completions — without any further changes.