Documentation Index
Fetch the complete documentation index at: https://docs.shipfastai.dev/llms.txt
Use this file to discover all available pages before exploring further.
Shipfastai’s LLM abstraction layer lets you swap providers — OpenAI, Anthropic, or Google Gemini — by changing a single provider field in your API request. Your application code never needs to know which underlying SDK is in use. The get_llm_provider factory in app/packages/ai/llm.py instantiates the right client, passes the correct API key from your environment, and returns a consistent ChatResponse regardless of which model generated it.
Supported providers
Shipfastai ships with three first-class provider implementations inside products/pro/backend/app/packages/ai/llm.py:
| Provider | provider value | Default model | Env var |
|---|
| OpenAI | openai | gpt-4o | OPENAI_API_KEY |
| Anthropic | anthropic | claude-sonnet-4-20250514 | ANTHROPIC_API_KEY |
| Google Gemini | gemini | gemini-2.0-flash | GOOGLE_API_KEY |
LLMProvider interface with chat() and stream_chat() methods, so switching providers requires no changes to your route handlers.
Configuring OpenAI
Set your API key
Add your OpenAI API key to your backend .env file: Send requests using the openai provider
Pass "provider": "openai" in your request body. You can optionally specify a model; if you omit it the default gpt-4o is used.{
"provider": "openai",
"model": "gpt-4o",
"messages": [
{ "role": "user", "content": "Explain RAG in one sentence." }
],
"temperature": 0.7,
"max_tokens": 256
}
Configuring Anthropic
Set your API key
ANTHROPIC_API_KEY=sk-ant-...
Send requests using the anthropic provider
{
"provider": "anthropic",
"model": "claude-opus-4-5",
"messages": [
{ "role": "system", "content": "You are a helpful assistant." },
{ "role": "user", "content": "Explain RAG in one sentence." }
],
"temperature": 0.7,
"max_tokens": 256
}
The AnthropicProvider automatically extracts system-role messages and passes them to Anthropic’s system parameter, so your request format is identical across providers.
Configuring Google Gemini
Send requests using the gemini provider
{
"provider": "gemini",
"model": "gemini-2.0-flash",
"messages": [
{ "role": "user", "content": "Explain RAG in one sentence." }
],
"temperature": 0.7,
"max_tokens": 256
}
Switching providers at runtime
Because the provider and model fields are part of each request body, you can switch providers on a per-request basis without redeploying. This is useful for A/B testing models or falling back to a cheaper provider under load.
OpenAI GPT-4o
Anthropic Claude
Google Gemini
{
"provider": "openai",
"model": "gpt-4o",
"messages": [{ "role": "user", "content": "Hello!" }]
}
{
"provider": "anthropic",
"model": "claude-sonnet-4-20250514",
"messages": [{ "role": "user", "content": "Hello!" }]
}
{
"provider": "gemini",
"model": "gemini-2.0-flash",
"messages": [{ "role": "user", "content": "Hello!" }]
}
"stream": true to the request body. The endpoint returns a text/event-stream response where each event is a JSON object { "token": "..." }, terminated by data: [DONE].
POST /api/ai/chat (streaming)
{
"provider": "openai",
"model": "gpt-4o",
"messages": [{ "role": "user", "content": "Write a haiku about Python." }],
"stream": true
}
Extending with a new provider
All providers inherit from the abstract base class LLMProvider defined in products/pro/backend/app/packages/ai/llm.py. To add a new provider, you implement two async methods and register the provider in the factory function.
Create your provider class
Add a new class that extends LLMProvider and implements chat() and stream_chat():products/pro/backend/app/packages/ai/llm.py
class GroqProvider(LLMProvider):
"""Groq LLM provider."""
def __init__(self, api_key: Optional[str] = None, model: str = "llama-3.3-70b-versatile"):
from groq import AsyncGroq
self.client = AsyncGroq(api_key=api_key or os.getenv("GROQ_API_KEY"))
self.model = model
async def chat(
self,
messages: list[Message],
temperature: float = 0.7,
max_tokens: int = 1000,
) -> ChatResponse:
response = await self.client.chat.completions.create(
model=self.model,
messages=[m.model_dump() for m in messages],
temperature=temperature,
max_tokens=max_tokens,
)
return ChatResponse(
content=response.choices[0].message.content or "",
model=response.model,
usage={
"prompt_tokens": response.usage.prompt_tokens,
"completion_tokens": response.usage.completion_tokens,
"total_tokens": response.usage.total_tokens,
},
finish_reason=response.choices[0].finish_reason,
)
async def stream_chat(
self,
messages: list[Message],
temperature: float = 0.7,
max_tokens: int = 1000,
) -> AsyncGenerator[str, None]:
stream = await self.client.chat.completions.create(
model=self.model,
messages=[m.model_dump() for m in messages],
temperature=temperature,
max_tokens=max_tokens,
stream=True,
)
async for chunk in stream:
if chunk.choices[0].delta.content:
yield chunk.choices[0].delta.content
Register the provider in the factory
Update get_llm_provider() to handle your new provider string:products/pro/backend/app/packages/ai/llm.py
def get_llm_provider(
provider: Literal["openai", "anthropic", "gemini", "groq"] = "openai",
model: Optional[str] = None,
) -> LLMProvider:
if provider == "openai":
return OpenAIProvider(model=model or "gpt-4o")
elif provider == "anthropic":
return AnthropicProvider(model=model or "claude-sonnet-4-20250514")
elif provider == "gemini":
return GeminiProvider(model=model or "gemini-2.0-flash")
elif provider == "groq":
return GroqProvider(model=model or "llama-3.3-70b-versatile")
else:
raise ValueError(f"Unknown provider: {provider}")
Export the class and add the env var
Add GroqProvider to the __all__ list in packages/ai/__init__.py, then add GROQ_API_KEY to your .env file.
app/api/ai/chat.py delegates entirely to get_llm_provider(), your new provider is immediately available to all routes — including streaming completions — without any further changes.