Add a new LLM provider to your AI app

Shipfastai’s LLM abstraction layer lets you swap providers — OpenAI, Anthropic, or Google Gemini — by changing a single provider field in your API request. Your application code never needs to know which underlying SDK is in use. The get_llm_provider factory in app/packages/ai/llm.py instantiates the right client, passes the correct API key from your environment, and returns a consistent ChatResponse regardless of which model generated it.

Supported providers

Shipfastai ships with three first-class provider implementations inside products/pro/backend/app/packages/ai/llm.py:

Provider	`provider` value	Default model	Env var
OpenAI	`openai`	`gpt-4o`	`OPENAI_API_KEY`
Anthropic	`anthropic`	`claude-sonnet-4-20250514`	`ANTHROPIC_API_KEY`
Google Gemini	`gemini`	`gemini-2.0-flash`	`GOOGLE_API_KEY`

All three implement the same abstract LLMProvider interface with chat() and stream_chat() methods, so switching providers requires no changes to your route handlers.

Configuring OpenAI

Set your API key

Add your OpenAI API key to your backend .env file:

.env

OPENAI_API_KEY=sk-...

Send requests using the openai provider

Pass "provider": "openai" in your request body. You can optionally specify a model; if you omit it the default gpt-4o is used.

POST /api/ai/chat

{
  "provider": "openai",
  "model": "gpt-4o",
  "messages": [
    { "role": "user", "content": "Explain RAG in one sentence." }
  ],
  "temperature": 0.7,
  "max_tokens": 256
}

Configuring Anthropic

Set your API key

.env

ANTHROPIC_API_KEY=sk-ant-...

Send requests using the anthropic provider

POST /api/ai/chat

{
  "provider": "anthropic",
  "model": "claude-opus-4-5",
  "messages": [
    { "role": "system", "content": "You are a helpful assistant." },
    { "role": "user", "content": "Explain RAG in one sentence." }
  ],
  "temperature": 0.7,
  "max_tokens": 256
}

The AnthropicProvider automatically extracts system-role messages and passes them to Anthropic’s system parameter, so your request format is identical across providers.

Configuring Google Gemini

Set your API key

.env

GOOGLE_API_KEY=AIza...

Send requests using the gemini provider

POST /api/ai/chat

{
  "provider": "gemini",
  "model": "gemini-2.0-flash",
  "messages": [
    { "role": "user", "content": "Explain RAG in one sentence." }
  ],
  "temperature": 0.7,
  "max_tokens": 256
}

Switching providers at runtime

Because the provider and model fields are part of each request body, you can switch providers on a per-request basis without redeploying. This is useful for A/B testing models or falling back to a cheaper provider under load.

OpenAI GPT-4o
Anthropic Claude
Google Gemini

POST /api/ai/chat

{
  "provider": "openai",
  "model": "gpt-4o",
  "messages": [{ "role": "user", "content": "Hello!" }]
}

POST /api/ai/chat

{
  "provider": "anthropic",
  "model": "claude-sonnet-4-20250514",
  "messages": [{ "role": "user", "content": "Hello!" }]
}

POST /api/ai/chat

{
  "provider": "gemini",
  "model": "gemini-2.0-flash",
  "messages": [{ "role": "user", "content": "Hello!" }]
}

You can also enable streaming for any provider by adding "stream": true to the request body. The endpoint returns a text/event-stream response where each event is a JSON object { "token": "..." }, terminated by data: [DONE].

POST /api/ai/chat (streaming)

{
  "provider": "openai",
  "model": "gpt-4o",
  "messages": [{ "role": "user", "content": "Write a haiku about Python." }],
  "stream": true
}

Extending with a new provider

All providers inherit from the abstract base class LLMProvider defined in products/pro/backend/app/packages/ai/llm.py. To add a new provider, you implement two async methods and register the provider in the factory function.

Create your provider class

Add a new class that extends LLMProvider and implements chat() and stream_chat():

products/pro/backend/app/packages/ai/llm.py

class GroqProvider(LLMProvider):
    """Groq LLM provider."""

    def __init__(self, api_key: Optional[str] = None, model: str = "llama-3.3-70b-versatile"):
        from groq import AsyncGroq
        self.client = AsyncGroq(api_key=api_key or os.getenv("GROQ_API_KEY"))
        self.model = model

    async def chat(
        self,
        messages: list[Message],
        temperature: float = 0.7,
        max_tokens: int = 1000,
    ) -> ChatResponse:
        response = await self.client.chat.completions.create(
            model=self.model,
            messages=[m.model_dump() for m in messages],
            temperature=temperature,
            max_tokens=max_tokens,
        )
        return ChatResponse(
            content=response.choices[0].message.content or "",
            model=response.model,
            usage={
                "prompt_tokens": response.usage.prompt_tokens,
                "completion_tokens": response.usage.completion_tokens,
                "total_tokens": response.usage.total_tokens,
            },
            finish_reason=response.choices[0].finish_reason,
        )

    async def stream_chat(
        self,
        messages: list[Message],
        temperature: float = 0.7,
        max_tokens: int = 1000,
    ) -> AsyncGenerator[str, None]:
        stream = await self.client.chat.completions.create(
            model=self.model,
            messages=[m.model_dump() for m in messages],
            temperature=temperature,
            max_tokens=max_tokens,
            stream=True,
        )
        async for chunk in stream:
            if chunk.choices[0].delta.content:
                yield chunk.choices[0].delta.content

Update get_llm_provider() to handle your new provider string:

products/pro/backend/app/packages/ai/llm.py

def get_llm_provider(
    provider: Literal["openai", "anthropic", "gemini", "groq"] = "openai",
    model: Optional[str] = None,
) -> LLMProvider:
    if provider == "openai":
        return OpenAIProvider(model=model or "gpt-4o")
    elif provider == "anthropic":
        return AnthropicProvider(model=model or "claude-sonnet-4-20250514")
    elif provider == "gemini":
        return GeminiProvider(model=model or "gemini-2.0-flash")
    elif provider == "groq":
        return GroqProvider(model=model or "llama-3.3-70b-versatile")
    else:
        raise ValueError(f"Unknown provider: {provider}")

Export the class and add the env var

Add GroqProvider to the __all__ list in packages/ai/__init__.py, then add GROQ_API_KEY to your .env file.

Because the chat endpoint in app/api/ai/chat.py delegates entirely to get_llm_provider(), your new provider is immediately available to all routes — including streaming completions — without any further changes.

Get Started

Core Features

Configuration

Deployment

Guides

Add a new LLM provider to your AI app

Supported providers

Configuring OpenAI

Configuring Anthropic

Configuring Google Gemini

Switching providers at runtime

Extending with a new provider

​Supported providers

​Configuring OpenAI

​Configuring Anthropic

​Configuring Google Gemini

​Switching providers at runtime

​Extending with a new provider

Supported providers

Configuring OpenAI

Configuring Anthropic

Configuring Google Gemini

Switching providers at runtime

Extending with a new provider