AI Provider Configuration

Step-by-step guide to configure your chosen AI provider for natural language to SQL queries.

🌟 Powered by LiteLLM

Actyze supports 100+ AI providers through LiteLLM integration.

📚 Choosing a Provider?

AI Providers Feature Page - See all 100+ providers, compare options, get recommendations
LiteLLM Providers Docs - Official provider documentation

This page shows: How to configure the most popular providers step-by-step.

🚀 Quick Start

All providers follow this simple pattern:

# 1. Set provider's API key
PROVIDER_API_KEY=your-key

# 2. Set model name
EXTERNAL_LLM_MODEL=provider/model-name

That's it! LiteLLM handles the rest automatically.

Configuration Examples

Anthropic Claude (Recommended)

Best for: Most users - excellent SQL accuracy and complex query understanding

Get API Key: https://console.anthropic.com/settings/keys

Docker (.env):

ANTHROPIC_API_KEY=sk-ant-xxxxx
EXTERNAL_LLM_MODEL=claude-sonnet-4-20250514

Helm (values.yaml + values-secrets.yaml):

# values.yaml
modelStrategy:
  externalLLM:
    enabled: true
    model: "claude-sonnet-4-20250514"

# values-secrets.yaml
secrets:
  externalLLM:
    apiKey: "sk-ant-xxxxx"

Available Models:

claude-sonnet-4-20250514 (recommended)
claude-opus-4-20250514 (most powerful)
claude-3-5-sonnet-20241022 (fast)

Pricing: $3/1M input tokens, $15/1M output tokens

OpenAI GPT-4

Best for: General purpose, widely supported

Get API Key: https://platform.openai.com/api-keys

Docker (.env):

OPENAI_API_KEY=sk-xxxxx
EXTERNAL_LLM_MODEL=gpt-4o

Helm (values.yaml + values-secrets.yaml):

# values.yaml
modelStrategy:
  externalLLM:
    enabled: true
    model: "gpt-4o"

# values-secrets.yaml
secrets:
  externalLLM:
    apiKey: "sk-xxxxx"

Available Models:

gpt-4o (recommended)
gpt-4-turbo
gpt-3.5-turbo (faster, cheaper)

Pricing: $2.5/1M input tokens, $10/1M output tokens

Google Gemini

Best for: Cost-conscious users, fast responses

Get API Key: https://makersuite.google.com/app/apikey

Docker (.env):

GEMINI_API_KEY=your-key
EXTERNAL_LLM_MODEL=gemini/gemini-pro

Helm (values.yaml + values-secrets.yaml):

# values.yaml
modelStrategy:
  externalLLM:
    enabled: true
    model: "gemini/gemini-pro"

# values-secrets.yaml
secrets:
  externalLLM:
    apiKey: "your-key"

Available Models:

gemini/gemini-pro (recommended)
gemini/gemini-2.0-flash-exp (fastest)

Pricing: $0.50/1M input tokens, $1.50/1M output tokens (cheapest!)

Azure OpenAI

Best for: Microsoft Azure customers with compliance requirements

Get API Key: Azure Portal → OpenAI Service → Keys

Docker (.env):

AZURE_API_KEY=your-azure-key
AZURE_API_BASE=https://YOUR-RESOURCE.openai.azure.com
AZURE_API_VERSION=2023-05-15
EXTERNAL_LLM_MODEL=azure/your-deployment-name

Helm (values.yaml + values-secrets.yaml):

# values.yaml
modelStrategy:
  externalLLM:
    enabled: true
    model: "azure/your-deployment-name"
    baseUrl: "https://YOUR-RESOURCE.openai.azure.com"

# values-secrets.yaml
secrets:
  externalLLM:
    apiKey: "your-azure-key"

Note: Deployment name is what you named it in Azure, not the model name.

Need Another Provider?

We show the top 4 providers above, but 100+ more are supported!

See the complete list:

AI Providers Page - All providers categorized by use case
- Fast & Free (Groq, Together AI)
- Enterprise (AWS Bedrock, Vertex AI)
- Open Source & Local (Ollama, LM Studio, vLLM)
- And 90+ more!

Configuration is the same for all:

# Pattern for ANY provider
PROVIDER_API_KEY=your-key
EXTERNAL_LLM_MODEL=provider/model-name

Check the AI Providers page for specific provider details.

Enterprise Gateway (Advanced)

For organizations with custom LLM wrappers, internal gateways, or compliance requirements.

When to Use Enterprise Mode

Your company has an internal LLM gateway/proxy
You need centralized cost management/tracking
You require additional security/compliance layers
You want custom authentication systems

Requirements

Your enterprise gateway must:

Accept OpenAI-compatible request format (/v1/chat/completions)
Return OpenAI-compatible response format
Support HTTP authentication

Configuration

Docker (.env):

# Gateway mode (bypasses LiteLLM, uses OpenAI format)
EXTERNAL_LLM_MODE=openai-compatible
EXTERNAL_LLM_BASE_URL=https://llm-gateway.yourcompany.com/v1/chat/completions
EXTERNAL_LLM_API_KEY=your-enterprise-token
EXTERNAL_LLM_MODEL=your-internal-model-name

# Optional: Custom headers for tracking/compliance
EXTERNAL_LLM_AUTH_TYPE=bearer
EXTERNAL_LLM_EXTRA_HEADERS={"X-Department": "engineering", "X-Cost-Center": "analytics"}

Helm (values.yaml + values-secrets.yaml):

# values.yaml
modelStrategy:
  externalLLM:
    enabled: true
    mode: "openai-compatible"
    model: "your-internal-model"
    baseUrl: "https://llm-gateway.yourcompany.com/v1/chat/completions"
    authType: "bearer"
    extraHeaders: '{"X-Department": "engineering"}'

# values-secrets.yaml
secrets:
  externalLLM:
    apiKey: "your-enterprise-token"

Enterprise Benefits

Centralized Billing: Track all LLM costs in one place
Access Control: Manage who can use which models
Compliance: Add audit logging, data residency controls
Multi-Model: Route requests across multiple providers with fallbacks

Configuration Reference

Required Variables

Variable	Description	Example
`EXTERNAL_LLM_MODEL`	Model name	`claude-sonnet-4-20250514`
`{PROVIDER}_API_KEY`	Provider's API key	`ANTHROPIC_API_KEY=sk-ant-xxxxx`

Optional Variables

Variable	Default	Description
`EXTERNAL_LLM_MODE`	`auto`	Mode: `auto`, `standard`, or `openai-compatible`
`EXTERNAL_LLM_MAX_TOKENS`	`4096`	Maximum tokens in response
`EXTERNAL_LLM_TEMPERATURE`	`0.1`	Temperature (0.0-1.0, lower = more deterministic)
`EXTERNAL_LLM_BASE_URL`	Auto	Custom base URL (for enterprise/local)

See: Environment Variables Reference for complete list

Testing Configuration

Docker

# Start services
./start.sh

# Check logs for API connection
docker-compose logs nexus | grep -i "llm\|error"

# Verify environment variables
docker-compose exec nexus env | grep -i "api_key\|llm"

# Test a query
curl -X POST http://localhost:8000/api/generate-sql \
  -H "Content-Type: application/json" \
  -d '{"query": "show me all tables"}'

Helm/Kubernetes

# Check pod logs
kubectl logs -n actyze deployment/dashboard-nexus | grep -i "llm\|error"

# Verify environment variables
kubectl exec -n actyze deployment/dashboard-nexus -- env | grep EXTERNAL_LLM

# Check secret
kubectl get secret dashboard-secrets -n actyze -o jsonpath='{.data.EXTERNAL_LLM_API_KEY}' | base64 -d

Troubleshooting

Invalid API Key

Error: 401 Unauthorized or 403 Forbidden

Solutions:

Verify API key is correct: echo $ANTHROPIC_API_KEY
Check for leading/trailing spaces
Ensure key has not expired
Test key directly with provider's API

Model Not Found

Error: 404 Model not found or Invalid model

Common Issues:

Wrong model name format (should be provider/model-name for most)
Model not available in your region
Model requires special access (e.g., Claude Opus)

Fix: Check correct model names in AI Providers page

Examples of correct format:

# Anthropic - no prefix needed
EXTERNAL_LLM_MODEL=claude-sonnet-4-20250514

# Google - needs prefix
EXTERNAL_LLM_MODEL=gemini/gemini-pro

# Groq - needs prefix
EXTERNAL_LLM_MODEL=groq/llama-3.3-70b-versatile

Slow Responses

Symptoms: Queries take > 10 seconds

Solutions:

Use faster models:
- Gemini Flash (fastest)
- Groq (very fast, free tier)
- GPT-3.5-turbo (faster than GPT-4)

Lower max_tokens:

EXTERNAL_LLM_MAX_TOKENS=2048  # Default is 4096

Enable caching (enabled by default):
- Repeated queries return instantly from cache

Rate Limit Exceeded

Error: 429 Too Many Requests

Solutions:

Wait and retry - Rate limits reset after a time period
Upgrade your plan with the provider
Use different provider for development/testing:
- Groq has generous free tier
- Ollama is unlimited (self-hosted)

Best Practices

Production Deployments

Use Enterprise Providers for SLAs:
- Azure OpenAI
- AWS Bedrock
- Vertex AI
Set Reasonable Timeouts:
```
EXTERNAL_LLM_TIMEOUT=30  # seconds
```
Monitor Costs via provider dashboards
Configure Retries:
```
EXTERNAL_LLM_MAX_RETRIES=3
```

Development & Testing

Use Free Tiers:
- Groq (fast, free)
- Ollama (local, no API costs)
- Gemini (generous free tier)

Switch Easily:

# Test with Groq
GROQ_API_KEY=gsk_xxxxx
EXTERNAL_LLM_MODEL=groq/llama-3.3-70b-versatile

# Then production with Claude
ANTHROPIC_API_KEY=sk-ant-xxxxx
EXTERNAL_LLM_MODEL=claude-sonnet-4-20250514

Temperature Settings

For SQL generation, use low temperature (0.0-0.2):

EXTERNAL_LLM_TEMPERATURE=0.1  # Recommended for SQL

0.0-0.2: Deterministic, consistent SQL (recommended)
0.3-0.5: More creative, may vary results
0.6-1.0: Very creative, not recommended for SQL

Token Limits

Set appropriate limits based on your queries:

# Simple queries
EXTERNAL_LLM_MAX_TOKENS=2048

# Complex queries with large schemas
EXTERNAL_LLM_MAX_TOKENS=4096  # Default

# Very complex queries
EXTERNAL_LLM_MAX_TOKENS=8192

Note: Higher limits = higher costs

Cost Optimization

Start with cheaper models: Gemini Pro ($0.50/1M tokens)
Use local models for dev: Ollama (free)
Enable caching (default): Repeated queries cost nothing
Monitor usage via provider dashboards

Security Considerations

API Key Management

Never commit API keys to git:

# ✅ Use environment variables
ANTHROPIC_API_KEY=sk-ant-xxxxx

# ✅ Use secret management
# - Docker: .env file (gitignored)
# - Kubernetes: Sealed Secrets or External Secrets Operator
# - Cloud: AWS Secrets Manager, Azure Key Vault

Network Security

For production deployments:

Use TLS/SSL for all API connections (enabled by default)
Restrict network access to LLM APIs via firewall rules
Use VPC/Private endpoints when available (AWS PrivateLink, Azure Private Link)
Enable logging for audit trails

Next Steps

AI Providers - See all 100+ supported providers
Environment Variables - Complete variable reference
Quick Start Guide - Start using Actyze
Troubleshooting - Common deployment issues

Additional Resources

LiteLLM Documentation - Official LiteLLM guide
LiteLLM Providers - All provider configs
Model Pricing Comparison - Compare costs
Actyze GitHub - Source code

Support

Need help?

GitHub Issues - Report problems
Documentation - Complete docs
AI Providers Page - Provider troubleshooting

🌟 Powered by LiteLLM​

🚀 Quick Start​

Configuration Examples​

Anthropic Claude (Recommended)​

OpenAI GPT-4​

Google Gemini​

Azure OpenAI​

Need Another Provider?​

Enterprise Gateway (Advanced)​

When to Use Enterprise Mode​

Requirements​

Configuration​

Enterprise Benefits​

Configuration Reference​

Required Variables​

Optional Variables​

Testing Configuration​

Docker​

Helm/Kubernetes​

Troubleshooting​

Invalid API Key​

Model Not Found​

Slow Responses​

Rate Limit Exceeded​

Best Practices​

Production Deployments​

Development & Testing​

Temperature Settings​

Token Limits​

Cost Optimization​

Security Considerations​

API Key Management​

Network Security​

Next Steps​

Additional Resources​

Support​

🌟 Powered by LiteLLM

🚀 Quick Start

Configuration Examples

Anthropic Claude (Recommended)

OpenAI GPT-4

Google Gemini

Azure OpenAI

Need Another Provider?

Enterprise Gateway (Advanced)

When to Use Enterprise Mode

Requirements

Configuration

Enterprise Benefits

Configuration Reference

Required Variables

Optional Variables

Testing Configuration

Docker

Helm/Kubernetes

Troubleshooting

Invalid API Key

Model Not Found

Slow Responses

Rate Limit Exceeded

Best Practices

Production Deployments

Development & Testing

Temperature Settings

Token Limits

Cost Optimization

Security Considerations

API Key Management

Network Security

Next Steps

Additional Resources

Support