> ## Documentation Index
> Fetch the complete documentation index at: https://docs.shipfastai.dev/llms.txt
> Use this file to discover all available pages before exploring further.
# Add a new LLM provider to your AI app
> Configure OpenAI, Anthropic, or Google Gemini in Shipfastai, switch providers per request, and extend the LLM abstraction to add a custom provider.
Shipfastai's LLM abstraction layer lets you swap providers — OpenAI, Anthropic, or Google Gemini — by changing a single `provider` field in your API request. Your application code never needs to know which underlying SDK is in use. The `get_llm_provider` factory in `app/packages/ai/llm.py` instantiates the right client, passes the correct API key from your environment, and returns a consistent `ChatResponse` regardless of which model generated it.
## Supported providers
Shipfastai ships with three first-class provider implementations inside `products/pro/backend/app/packages/ai/llm.py`:
| Provider | `provider` value | Default model | Env var |
| ------------- | ---------------- | -------------------------- | ------------------- |
| OpenAI | `openai` | `gpt-4o` | `OPENAI_API_KEY` |
| Anthropic | `anthropic` | `claude-sonnet-4-20250514` | `ANTHROPIC_API_KEY` |
| Google Gemini | `gemini` | `gemini-2.0-flash` | `GOOGLE_API_KEY` |
All three implement the same abstract `LLMProvider` interface with `chat()` and `stream_chat()` methods, so switching providers requires no changes to your route handlers.
## Configuring OpenAI
Add your OpenAI API key to your backend `.env` file:
```bash .env theme={null}
OPENAI_API_KEY=sk-...
```
Pass `"provider": "openai"` in your request body. You can optionally specify a `model`; if you omit it the default `gpt-4o` is used.
```json POST /api/ai/chat theme={null}
{
"provider": "openai",
"model": "gpt-4o",
"messages": [
{ "role": "user", "content": "Explain RAG in one sentence." }
],
"temperature": 0.7,
"max_tokens": 256
}
```
## Configuring Anthropic
```bash .env theme={null}
ANTHROPIC_API_KEY=sk-ant-...
```
```json POST /api/ai/chat theme={null}
{
"provider": "anthropic",
"model": "claude-opus-4-5",
"messages": [
{ "role": "system", "content": "You are a helpful assistant." },
{ "role": "user", "content": "Explain RAG in one sentence." }
],
"temperature": 0.7,
"max_tokens": 256
}
```
The `AnthropicProvider` automatically extracts `system`-role messages and passes them to Anthropic's `system` parameter, so your request format is identical across providers.
## Configuring Google Gemini
```bash .env theme={null}
GOOGLE_API_KEY=AIza...
```
```json POST /api/ai/chat theme={null}
{
"provider": "gemini",
"model": "gemini-2.0-flash",
"messages": [
{ "role": "user", "content": "Explain RAG in one sentence." }
],
"temperature": 0.7,
"max_tokens": 256
}
```
## Switching providers at runtime
Because the `provider` and `model` fields are part of each request body, you can switch providers on a per-request basis without redeploying. This is useful for A/B testing models or falling back to a cheaper provider under load.
```json POST /api/ai/chat theme={null}
{
"provider": "openai",
"model": "gpt-4o",
"messages": [{ "role": "user", "content": "Hello!" }]
}
```
```json POST /api/ai/chat theme={null}
{
"provider": "anthropic",
"model": "claude-sonnet-4-20250514",
"messages": [{ "role": "user", "content": "Hello!" }]
}
```
```json POST /api/ai/chat theme={null}
{
"provider": "gemini",
"model": "gemini-2.0-flash",
"messages": [{ "role": "user", "content": "Hello!" }]
}
```
You can also enable streaming for any provider by adding `"stream": true` to the request body. The endpoint returns a `text/event-stream` response where each event is a JSON object `{ "token": "..." }`, terminated by `data: [DONE]`.
```json POST /api/ai/chat (streaming) theme={null}
{
"provider": "openai",
"model": "gpt-4o",
"messages": [{ "role": "user", "content": "Write a haiku about Python." }],
"stream": true
}
```
## Extending with a new provider
All providers inherit from the abstract base class `LLMProvider` defined in `products/pro/backend/app/packages/ai/llm.py`. To add a new provider, you implement two async methods and register the provider in the factory function.
Add a new class that extends `LLMProvider` and implements `chat()` and `stream_chat()`:
```python products/pro/backend/app/packages/ai/llm.py theme={null}
class GroqProvider(LLMProvider):
"""Groq LLM provider."""
def __init__(self, api_key: Optional[str] = None, model: str = "llama-3.3-70b-versatile"):
from groq import AsyncGroq
self.client = AsyncGroq(api_key=api_key or os.getenv("GROQ_API_KEY"))
self.model = model
async def chat(
self,
messages: list[Message],
temperature: float = 0.7,
max_tokens: int = 1000,
) -> ChatResponse:
response = await self.client.chat.completions.create(
model=self.model,
messages=[m.model_dump() for m in messages],
temperature=temperature,
max_tokens=max_tokens,
)
return ChatResponse(
content=response.choices[0].message.content or "",
model=response.model,
usage={
"prompt_tokens": response.usage.prompt_tokens,
"completion_tokens": response.usage.completion_tokens,
"total_tokens": response.usage.total_tokens,
},
finish_reason=response.choices[0].finish_reason,
)
async def stream_chat(
self,
messages: list[Message],
temperature: float = 0.7,
max_tokens: int = 1000,
) -> AsyncGenerator[str, None]:
stream = await self.client.chat.completions.create(
model=self.model,
messages=[m.model_dump() for m in messages],
temperature=temperature,
max_tokens=max_tokens,
stream=True,
)
async for chunk in stream:
if chunk.choices[0].delta.content:
yield chunk.choices[0].delta.content
```
Update `get_llm_provider()` to handle your new provider string:
```python products/pro/backend/app/packages/ai/llm.py theme={null}
def get_llm_provider(
provider: Literal["openai", "anthropic", "gemini", "groq"] = "openai",
model: Optional[str] = None,
) -> LLMProvider:
if provider == "openai":
return OpenAIProvider(model=model or "gpt-4o")
elif provider == "anthropic":
return AnthropicProvider(model=model or "claude-sonnet-4-20250514")
elif provider == "gemini":
return GeminiProvider(model=model or "gemini-2.0-flash")
elif provider == "groq":
return GroqProvider(model=model or "llama-3.3-70b-versatile")
else:
raise ValueError(f"Unknown provider: {provider}")
```
Add `GroqProvider` to the `__all__` list in `packages/ai/__init__.py`, then add `GROQ_API_KEY` to your `.env` file.
Because the chat endpoint in `app/api/ai/chat.py` delegates entirely to `get_llm_provider()`, your new provider is immediately available to all routes — including streaming completions — without any further changes.