Kubernetes for Go Developers: Just Enough to Be Dangerous

Why Should Go Developers Care About Kubernetes?

Let me tell you why I was forced to learn Kubernetes after avoiding it for years:

The Problems That Made Me Learn K8s

• Deployment Hell: SSH into server, git pull, go build, systemctl restart. Pray nothing breaks. Do this for 10 services.
• The 3 AM Call: "Server is down!" SSH in, service crashed, manually restart. No idea why it died.
• Scaling Nightmare: Black Friday traffic spike. Manually spin up VMs, deploy code, update load balancer. Takes 2 hours.
• Version Chaos: "Which version is in production?" Nobody knows. No rollback plan.
• Resource Waste: 20 VMs running at 5% CPU because we're scared to consolidate.

What Kubernetes Actually Gives You

• Auto-restart when your app crashes - No more 3 AM calls
• Zero-downtime deployments - Deploy at 2 PM on Tuesday, not 2 AM on Sunday
• Auto-scaling - Handle traffic spikes without touching anything
• One-command rollback - Fucked up? kubectl rollout undo
• Resource efficiency - Pack multiple services on fewer machines, save 60% on hosting
• Self-healing - Node dies? K8s moves your pods to healthy nodes
• Service discovery - Services find each other by name, no hardcoded IPs

Real Numbers from Production

Before Kubernetes:
- Deployment time: 45 minutes (manual, scary)
- Rollback time: 2 hours (if we're lucky)
- Server costs: $8,000/month (lots of idle VMs)
- On-call incidents: 15/month
- Time to scale: 2-3 hours
After Kubernetes:
- Deployment time: 5 minutes (automated, boring)
- Rollback time: 30 seconds
- Server costs: $3,200/month (60% reduction!)
- On-call incidents: 2/month
- Time to scale: 30 seconds (automatic)

When You DON'T Need Kubernetes

Let's be honest - K8s is overkill if:

• You have 1-2 services that rarely change
• Your traffic is predictable and flat
• You're fine with 10-minute downtime during deploys
• You enjoy SSH-ing into servers

But if you're managing 5+ services, dealing with variable traffic, or tired of manual deployments, Kubernetes will change your life.

The Only 5 Kubernetes Concepts You Need

Now that you know WHY, here's WHAT. Forget the 50+ resource types. You'll use these 5 for 95% of your work:

1. Pod - Your Go app running in a container
2. Deployment - Manages multiple pods, handles updates
3. Service - Makes pods accessible (like a load balancer)
4. ConfigMap - Environment variables and config files
5. Secret - Passwords, API keys, certificates

That's it. Master these and you can deploy anything.

Your First Go App on Kubernetes

Kubernetes-ready Go apps need three critical endpoints: main handler, health check, and readiness probe. These enable Kubernetes to manage your app's lifecycle intelligently.

Health checks tell Kubernetes if your app is alive. If health checks fail, Kubernetes restarts the pod. This automatic recovery handles crashes, deadlocks, and unrecoverable errors without manual intervention.

Readiness probes determine if your app can handle traffic. During startup, apps might need to load caches, establish database connections, or warm up. Readiness checks prevent requests from hitting unprepared instances.

The hostname in responses helps debug load balancing. When multiple pods serve requests, seeing which pod responded helps diagnose sticky session issues or uneven distribution.

Step 1: The Go App

func main() {
    port := os.Getenv("PORT")
    if port == "" {
        port = "8080"
    }
    http.HandleFunc("/", handler)
    http.HandleFunc("/health", health)  // Liveness probe
    http.HandleFunc("/ready", ready)    // Readiness probe
    log.Fatal(http.ListenAndServe(":"+port, nil))
}
func health(w http.ResponseWriter, r *http.Request) {
    w.WriteHeader(http.StatusOK)
}
func ready(w http.ResponseWriter, r *http.Request) {
    // Check dependencies
    if !dbConnected() || !cacheWarmed() {
        w.WriteHeader(http.StatusServiceUnavailable)
        return
    }
    w.WriteHeader(http.StatusOK)
}

Step 2: The Deployment

apiVersion: apps/v1
kind: Deployment
metadata:
  name: go-app
  labels:
    app: go-app
spec:
  replicas: 3  # Run 3 instances
  selector:
    matchLabels:
      app: go-app
  template:
    metadata:
      labels:
        app: go-app
    spec:
      containers:
      - name: app
        image: myapp:latest
        ports:
        - containerPort: 8080
        env:
        - name: PORT
          value: "8080"
        - name: DB_HOST
          value: postgres-service
        resources:
          requests:
            memory: "64Mi"
            cpu: "250m"
          limits:
            memory: "128Mi"
            cpu: "500m"
        livenessProbe:
          httpGet:
            path: /health
            port: 8080
          initialDelaySeconds: 30
          periodSeconds: 10
        readinessProbe:
          httpGet:
            path: /ready
            port: 8080
          initialDelaySeconds: 5
          periodSeconds: 5

Step 3: The Service

apiVersion: v1
kind: Service
metadata:
  name: go-app-service
spec:
  selector:
    app: go-app
  ports:
  - port: 80
    targetPort: 8080
  type: LoadBalancer  # Or ClusterIP for internal only

Step 4: Deploy It

# Apply configurations
kubectl apply -f deployment.yaml
kubectl apply -f service.yaml
# Check status
kubectl get pods
kubectl get svc
# See logs
kubectl logs -f deployment/go-app
# Get inside a pod (for debugging)
kubectl exec -it go-app-xxxxx -- sh

Configuration That Actually Works

ConfigMap for Environment Variables

apiVersion: v1
kind: ConfigMap
metadata:
  name: app-config
data:
  LOG_LEVEL: "info"
  CACHE_TTL: "3600"
  API_TIMEOUT: "30s"
---
apiVersion: v1
kind: Secret
metadata:
  name: app-secrets
type: Opaque
data:
  # echo -n 'mypassword' | base64
  DB_PASSWORD: bXlwYXNzd29yZA==
  API_KEY: c2VjcmV0a2V5MTIz

Using Config in Deployment

spec:
  containers:
  - name: app
    envFrom:
    - configMapRef:
        name: app-config
    - secretRef:
        name: app-secrets
    # Or individual values
    env:
    - name: DB_PASSWORD
      valueFrom:
        secretKeyRef:
          name: app-secrets
          key: DB_PASSWORD

Go Code to Read Config

type Config struct {
    Port        string `env:"PORT" envDefault:"8080"`
    LogLevel    string `env:"LOG_LEVEL" envDefault:"info"`
    DBHost      string `env:"DB_HOST" envDefault:"localhost"`
    DBPassword  string `env:"DB_PASSWORD" required:"true"`
    CacheTTL    time.Duration `env:"CACHE_TTL" envDefault:"1h"`
}
func LoadConfig() (*Config, error) {
    cfg := &Config{}
    if err := env.Parse(cfg); err != nil {
        return nil, fmt.Errorf("parsing config: %w", err)
    }
    return cfg, nil
}

Resource Limits: Don't Let K8s Kill Your App

Kubernetes will kill your app if it uses too much memory. Here's how to set limits properly:

Finding the Right Values

# Run your app locally and measure
go build -o app
/usr/bin/time -v ./app
# Look for "Maximum resident set size"
# Add 20% buffer for production

Common Go App Settings

resources:
  requests:
    memory: "128Mi"  # Minimum guaranteed
    cpu: "100m"      # 0.1 CPU core
  limits:
    memory: "256Mi"  # Killed if exceeded
    cpu: "1000m"     # Throttled if exceeded
# GOMAXPROCS auto-detection fix
env:
- name: GOMAXPROCS
  value: "1"  # Match CPU limit

The GOMAXPROCS Problem (That Cost Us $70K)

Before Go 1.25, the runtime doesn't see container CPU limits - it sees all host CPUs. On a 64-core node with 0.5 CPU limit, Go spawns 64 OS threads. Result: massive context switching, 10x slower performance.

🎉 Great news in Go 1.25! The Go runtime is now container-aware by default. It automatically detects CPU limits from cgroups and sets GOMAXPROCS accordingly. No more third-party libraries needed!

// Go 1.25+ (Released February 2025) - Works automatically!
// GOMAXPROCS will be set to your container's CPU limit
// No code changes needed!
// For Go < 1.25 - Use automaxprocs
import _ "go.uber.org/automaxprocs"
// Automatically sets GOMAXPROCS to match container CPU limit
// Or manual calculation for older versions
import "k8s.io/apimachinery/pkg/api/resource"
func init() {
    // Only needed for Go < 1.25
    if cpuLimit := os.Getenv("CPU_LIMIT"); cpuLimit != "" {
        if cpu, err := resource.ParseQuantity(cpuLimit); err == nil {
            gomaxprocs := int(cpu.MilliValue() / 1000)
            if gomaxprocs < 1 {
                gomaxprocs = 1
            }
            runtime.GOMAXPROCS(gomaxprocs)
        }
    }
}

How Go 1.25 Container Awareness Works

• Automatic Detection: Go reads cgroup v1/v2 CPU quota and period to determine limits
• Dynamic Updates: GOMAXPROCS adjusts automatically if container limits change at runtime
• Fractional CPUs: A limit of 1.5 CPUs rounds up to GOMAXPROCS=2
• Manual Override: Setting GOMAXPROCS env var still works as before
• Important: Go uses CPU limits, not CPU requests from Kubernetes

# Kubernetes configuration
resources:
  limits:
    cpu: "2"      # Go 1.25 sets GOMAXPROCS=2
  requests:
    cpu: "1"      # Go ignores this value

For Go 1.25+, you don't need automaxprocs anymore - it just works! For older versions, import automaxprocs and forget about it. We cover this disaster in detail in our GOMAXPROCS article.

Health Checks That Save Your Ass

Kubernetes needs to know when your app is alive and ready. Get this wrong and you'll have random downtime.

Go Implementation

type HealthChecker struct {
    db    *sql.DB
    redis *redis.Client
}
// Liveness: Is the app running?
func (h *HealthChecker) Live(w http.ResponseWriter, r *http.Request) {
    // Don't check external dependencies here!
    // This is just "is my process alive"
    w.WriteHeader(http.StatusOK)
    json.NewEncoder(w).Encode(map[string]string{
        "status": "alive",
    })
}
// Readiness: Can the app serve traffic?
func (h *HealthChecker) Ready(w http.ResponseWriter, r *http.Request) {
    checks := []struct {
        name string
        fn   func() error
    }{
        {"database", h.checkDB},
        {"redis", h.checkRedis},
    }
    for _, check := range checks {
        if err := check.fn(); err != nil {
            w.WriteHeader(http.StatusServiceUnavailable)
            json.NewEncoder(w).Encode(map[string]string{
                "status": "not ready",
                "failed": check.name,
                "error":  err.Error(),
            })
            return
        }
    }
    w.WriteHeader(http.StatusOK)
    json.NewEncoder(w).Encode(map[string]string{
        "status": "ready",
    })
}
func (h *HealthChecker) checkDB() error {
    ctx, cancel := context.WithTimeout(context.Background(), 1*time.Second)
    defer cancel()
    return h.db.PingContext(ctx)
}

Kubernetes Configuration

livenessProbe:
  httpGet:
    path: /health/live
    port: 8080
  initialDelaySeconds: 30  # Time to start
  periodSeconds: 10
  timeoutSeconds: 1
  failureThreshold: 3      # Restart after 3 failures
readinessProbe:
  httpGet:
    path: /health/ready
    port: 8080
  initialDelaySeconds: 5   # Quick initial check
  periodSeconds: 5
  timeoutSeconds: 1
  failureThreshold: 1      # Remove from load balancer immediately

Graceful Shutdown (Or Your Data Will Be Corrupted)

Kubernetes gives you 30 seconds to shut down gracefully. Use them wisely:

func main() {
    srv := &http.Server{Addr: ":8080"}
    // Start server
    go func() {
        if err := srv.ListenAndServe(); err != http.ErrServerClosed {
            log.Fatalf("Server failed: %v", err)
        }
    }()
    // Wait for interrupt signal
    quit := make(chan os.Signal, 1)
    signal.Notify(quit, syscall.SIGINT, syscall.SIGTERM)
    <-quit
    log.Println("Shutting down server...")
    // Give active requests 30 seconds to complete
    ctx, cancel := context.WithTimeout(context.Background(), 30*time.Second)
    defer cancel()
    // Stop accepting new requests and wait for active ones
    if err := srv.Shutdown(ctx); err != nil {
        log.Printf("Server forced to shutdown: %v", err)
    }
    // Clean up resources
    closeDatabase()
    flushMetrics()
    log.Println("Server exited")
}

Debugging in Production

Get Into a Running Pod

# Execute shell in pod
kubectl exec -it go-app-xxxxx -- sh
# If no shell, use kubectl debug
kubectl debug -it go-app-xxxxx --image=busybox --target=app
# Copy files from pod
kubectl cp go-app-xxxxx:/tmp/heap.prof ./heap.prof

Debug Crashed Pods

# See why it crashed
kubectl describe pod go-app-xxxxx
kubectl logs go-app-xxxxx --previous
# Common issues to look for:
# - "OOMKilled" = out of memory
# - "CrashLoopBackOff" = app keeps crashing
# - "ImagePullBackOff" = can't pull Docker image

Performance Profiling in K8s

import _ "net/http/pprof"
func main() {
    // pprof server (don't expose publicly!)
    go func() {
        log.Println(http.ListenAndServe("localhost:6060", nil))
    }()
    // Your main server
    http.ListenAndServe(":8080", nil)
}
// Access via port-forward
// kubectl port-forward go-app-xxxxx 6060:6060
// go tool pprof http://localhost:6060/debug/pprof/heap

Scaling Patterns

Horizontal Pod Autoscaler

apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: go-app-hpa
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: go-app
  minReplicas: 2
  maxReplicas: 10
  metrics:
  - type: Resource
    resource:
      name: cpu
      target:
        type: Utilization
        averageUtilization: 70
  - type: Resource
    resource:
      name: memory
      target:
        type: Utilization
        averageUtilization: 80

Manual Scaling

# Scale to 5 replicas
kubectl scale deployment go-app --replicas=5
# Rolling update with new image
kubectl set image deployment/go-app app=myapp:v2
# Watch the rollout
kubectl rollout status deployment/go-app
# Undo if something goes wrong
kubectl rollout undo deployment/go-app

Secrets Management

Don't Put Secrets in Your Code!

# Bad: Secret in deployment.yaml
env:
- name: API_KEY
  value: "secret123"  # NO!
# Good: Reference a Secret
env:
- name: API_KEY
  valueFrom:
    secretKeyRef:
      name: api-secrets
      key: API_KEY

Creating Secrets Properly

# From literal values
kubectl create secret generic api-secrets \
  --from-literal=API_KEY=secret123 \
  --from-literal=DB_PASSWORD=pass456
# From files
echo -n 'secret123' > ./api-key
kubectl create secret generic api-secrets --from-file=./api-key
# From .env file
kubectl create secret generic api-secrets --from-env-file=.env

Common Gotchas and Solutions

1. DNS Not Working

// Service DNS format:  . .svc.cluster.local
dbHost := "postgres-service.default.svc.cluster.local"
// Or just use service name if in same namespace
dbHost := "postgres-service"  

2. Container Keeps Restarting

# Add startup probe for slow-starting apps
startupProbe:
  httpGet:
    path: /health
    port: 8080
  failureThreshold: 30  # 30 * 10 = 5 minutes to start
  periodSeconds: 10

3. Can't Pull Private Images

# Create image pull secret
kubectl create secret docker-registry regcred \
  --docker-server=https://index.docker.io/v1/ \
  --docker-username=  \
  --docker-password=
# Use in deployment
spec:
  imagePullSecrets:
  - name: regcred  

4. Lost Logs After Pod Restart

// Log to stdout/stderr for kubectl logs
log.SetOutput(os.Stdout)
// Or use structured logging
logger := slog.New(slog.NewJSONHandler(os.Stdout, nil))
logger.Info("server started", "port", port)

Production Checklist

Before deploying to production, make sure you have:

• ✅ Health checks (liveness and readiness)
• ✅ Resource limits set
• ✅ Graceful shutdown implemented
• ✅ Secrets in Secret resources (not in code)
• ✅ Logging to stdout/stderr
• ✅ Metrics exposed for monitoring
• ✅ Multiple replicas (at least 2)
• ✅ Pod Disruption Budget configured
• ✅ Network policies (if required)
• ✅ Autoscaling configured

Useful Commands Cheat Sheet

# Debugging
kubectl logs -f deployment/go-app --tail=100
kubectl describe pod go-app-xxxxx
kubectl get events --sort-by='.lastTimestamp'
kubectl top pods
kubectl top nodes
# Quick fixes
kubectl delete pod go-app-xxxxx  # Force restart
kubectl rollout restart deployment/go-app
kubectl scale deployment go-app --replicas=0  # Stop all
kubectl scale deployment go-app --replicas=3  # Start again
# Port forwarding for debugging
kubectl port-forward service/go-app 8080:80
kubectl port-forward pod/go-app-xxxxx 6060:6060  # pprof
# Get YAML for existing resource
kubectl get deployment go-app -o yaml > deployment.yaml
# Dry run to validate YAML
kubectl apply -f deployment.yaml --dry-run=client

The Bottom Line

Kubernetes is complex, but as a Go developer, you don't need to know everything. Focus on:

1. Getting your app running (Deployment + Service)
2. Making it observable (logs, metrics, health checks)
3. Making it resilient (resource limits, replicas, graceful shutdown)
4. Debugging when it breaks (kubectl logs, exec, describe)

Start with the examples above. Deploy something simple. Break it. Fix it. That's how you really learn Kubernetes.

And remember: if your app works in Docker, it'll work in Kubernetes. The rest is just YAML.

Production-Grade Additions

Monitoring and Metrics

Without metrics, you're blind in production. When users complain about slowness, you need to identify the cause in minutes.

Mandatory minimum:

• Prometheus for metrics collection (built-in K8s support)
• RED metrics: Request rate, Errors, Duration per endpoint
• System metrics: CPU, memory, goroutines
• Alerts on anomalies (response time > 1s, error rate > 1%)

Don't add 100 metrics day one. Start with 10 key metrics, add as needed.

Database Migrations with Init Containers

Problem: Deploy new version, 5 pods start simultaneously, all try to run migrations. Result: deadlock or corrupted schema.

Solution: Init containers run before the main app, apply migrations once. If migration fails, deployment rolls back, old version keeps running.

spec:
  initContainers:
  - name: migrate
    image: migrate/migrate
    command: ['migrate', '-path=/migrations', '-database=postgres://...', 'up']
  containers:
  - name: app
    image: myapp:latest

StatefulSets for Stateful Services

If your service stores state (databases, caches, queues), regular Deployments won't work. StatefulSets provide:

• Stable pod names (redis-0, redis-1)
• Persistent storage surviving restarts
• Guaranteed startup/shutdown order

Network Policies for Security

By default in K8s, all pods can talk to each other. Security hole. Network Policies are firewalls inside your cluster. Frontend can only talk to backend, backend only to DB. Compromise one service doesn't give access to everything.

CI/CD Automation

Manual deploys are error-prone. Proper pipeline:

1. Push to main → automatic tests (coverage > 85%)
2. Build Docker image (multi-stage, < 20MB for Go)
3. Auto-deploy to staging
4. Smoke tests
5. Deploy to production via button or git tag

Rollback = change version in git. ArgoCD or Flux handle the rest.

Practical Tips

1. Start with one service - Don't try to migrate everything at once
2. Use managed K8s - EKS, GKE, AKS. Don't run your own cluster
3. Learn as needed - You don't need all 50+ resource types
4. Monitoring from day one - Otherwise you won't understand what's happening
5. Staging = production copy - Same limits, versions, settings
6. Use namespace separation - dev/staging/prod in different namespaces
7. Set up kubectl aliases - alias k=kubectl will save thousands of keystrokes

The Real Bottom Line

Kubernetes is an investment. The first two weeks will be YAML hell and cryptic errors. But then you'll get a system that:

• Self-heals after crashes
• Auto-scales based on load
• Deploys without downtime
• Rolls back with one command
• Saves 60% on infrastructure costs

Focus on the basics: Deployments, Services, ConfigMaps. Add advanced features as you grow. And remember: you don't need to know all of Kubernetes, just 20% solves 80% of problems.

Start simple. Deploy something. Break it. Fix it. That's how you really learn Kubernetes.