GPT-4 in Production: Real Implementation Experience

Project Context

For the past 6 months, I've been working on an automatic document processing system using GPT-4. The system processes 50K+ documents daily, extracts structured data, and generates reports. Here's what I learned during this time.

Integration Architecture

The main rule — never call OpenAI API directly from the main processing flow.

type AIService struct {
    client     *openai.Client
    cache      Cache
    rateLimiter *rate.Limiter
    circuitBreaker *gobreaker.CircuitBreaker
}
func (s *AIService) ProcessWithRetry(ctx context.Context, prompt string) (string, error) {
    // Check cache
    if cached, ok := s.cache.Get(prompt); ok {
        return cached, nil
    }
    // Rate limiting
    if err := s.rateLimiter.Wait(ctx); err != nil {
        return "", err
    }
    // Circuit breaker for API failure protection
    result, err := s.circuitBreaker.Execute(func() (interface{}, error) {
        return s.callOpenAI(ctx, prompt)
    })
    if err != nil {
        return "", err
    }
    response := result.(string)
    s.cache.Set(prompt, response)
    return response, nil
}

Rate Limits and Quotas

OpenAI has a complex limit system:

• RPM (Requests Per Minute) — usually 200-500 for GPT-4
• TPM (Tokens Per Minute) — 40,000-90,000 tokens
• RPD (Requests Per Day) — depends on account tier

Adaptive rate limiting implementation:

func NewAdaptiveRateLimiter() *AdaptiveRateLimiter {
    return &AdaptiveRateLimiter{
        baseRate: 200, // RPM
        current:  200,
        minRate:  50,
        maxRate:  500,
    }
}
func (r *AdaptiveRateLimiter) Adjust(responseHeaders http.Header) {
    remaining := responseHeaders.Get("x-ratelimit-remaining-requests")
    resetTime := responseHeaders.Get("x-ratelimit-reset-requests")
    if remaining != "" {
        rem, _ := strconv.Atoi(remaining)
        if rem < 10 {
            r.current = int(float64(r.current) * 0.7)
        } else if rem > 100 {
            r.current = int(float64(r.current) * 1.2)
        }
    }
}

Cost Optimization

GPT-4 is expensive. Here's how we reduced costs by 60%:

1. Smart Caching

type SemanticCache struct {
    embeddings *EmbeddingService
    vectorDB   *VectorDB
    threshold  float64
}
func (c *SemanticCache) Get(prompt string) (string, bool) {
    embedding := c.embeddings.Encode(prompt)
    similar := c.vectorDB.SearchSimilar(embedding, c.threshold)
    if len(similar) > 0 {
        return similar[0].Response, true
    }
    return "", false
}

2. Model Cascade

Using cheaper models for simple tasks:

func SelectModel(task Task) Model {
    switch task.Complexity {
    case Low:
        return GPT35Turbo  // $0.002 per 1K tokens
    case Medium:
        return GPT4Turbo   // $0.01 per 1K tokens
    case High:
        return GPT4        // $0.03 per 1K tokens
    }
}

3. Prompt Optimization

Reducing prompts without quality loss:

// Before: 500 tokens
oldPrompt := `You are an AI assistant helping with document processing.
Your task is to extract information from the following document.
Please be accurate and thorough in your extraction...`
// After: 50 tokens
newPrompt := `Extract: [name, date, amount] from:`

Error Handling

OpenAI API can fail. Be prepared for it:

func (s *AIService) callWithFallback(ctx context.Context, prompt string) (string, error) {
    // Main call
    result, err := s.callOpenAI(ctx, prompt)
    if err == nil {
        return result, nil
    }
    // Analyze error
    switch {
    case isRateLimitError(err):
        // Wait and retry
        time.Sleep(time.Second * 10)
        return s.callOpenAI(ctx, prompt)
    case isServerError(err):
        // Switch to fallback model
        return s.callAnthropic(ctx, prompt)
    case isTimeoutError(err):
        // Simplify request
        simplified := simplifyPrompt(prompt)
        return s.callOpenAI(ctx, simplified)
    default:
        return "", err
    }
}

Monitoring and Metrics

Critical metrics:

• Latency P50/P95/P99 — usually 2s/5s/15s
• Error rate — keep below 1%
• Token usage — for cost control
• Cache hit rate — target 40%+

func (s *AIService) recordMetrics(start time.Time, tokens int, err error) {
    duration := time.Since(start)
    metrics.RecordLatency("openai.request.duration", duration)
    metrics.IncrementCounter("openai.tokens.used", tokens)
    if err != nil {
        metrics.IncrementCounter("openai.errors", 1)
    }
    if duration > 10*time.Second {
        log.Warn("Slow OpenAI request", "duration", duration)
    }
}

Problems and Solutions

Problem 1: Response Instability

GPT-4 can give different answers to the same prompt. Solution — structured output:

response := openai.ChatCompletionRequest{
    Model: "gpt-4-1106-preview",
    Messages: []Message{{Role: "user", Content: prompt}},
    ResponseFormat: &ResponseFormat{
        Type: "json_object",
    },
    Temperature: 0.1, // Reduce variability
}

Problem 2: Token Limits

Implemented chunking for large documents:

func ProcessLargeDocument(doc string) []Result {
    chunks := splitIntoChunks(doc, 3000) // tokens per chunk
    results := make([]Result, 0)
    for _, chunk := range chunks {
        result := processChunk(chunk)
        results = append(results, result)
    }
    return mergeResults(results)
}

Problem 3: Compliance and Security

Filtering sensitive data before sending:

func sanitizePrompt(prompt string) string {
    // Remove PII
    prompt = regexp.MustCompile(`\b\d{3}-\d{2}-\d{4}\b`).ReplaceAllString(prompt, "[SSN]")
    prompt = regexp.MustCompile(`\b[A-Z0-9._%+-]+@[A-Z0-9.-]+\.[A-Z]{2,}\b`).ReplaceAllString(prompt, "[EMAIL]")
    return prompt
}

Results

After 6 months in production:

• ✅ Processed 9M+ documents
• ✅ Data extraction accuracy: 94%
• ✅ Average processing time: 3.2 seconds
• ✅ Cost per document: $0.008
• ✅ Uptime: 99.95%

Tips for Beginners

1. Start with playground — test prompts before coding
2. Use prompt versioning — they change more often than code
3. Plan fallback strategy — OpenAI will be unavailable
4. Monitor expenses — set alerts for budget overruns
5. Cache aggressively — it's the easiest way to save money

Conclusion

GPT-4 in production is not just an API call. It's a complex engineering challenge with many pitfalls. But with the right approach, LLMs can fundamentally transform your product's capabilities. The key is to treat them as an unreliable but powerful tool, and build your system accordingly.