Lesson 3 — Memory Systems

This lesson adds production conversational memory: sliding-window short-term storage, a TTL semantic cache, and optional Redis. Agents that forget waste tokens; agents that remember forever raise cost and serve stale answers. You implement bounded session history, cache hit/miss paths, TTL eviction, and a FastAPI dashboard for retained, reused, and discarded state.

• Sliding-window short-term memory with turn and token budgets.

• Exact-match semantic cache with LRU and TTL eviction.

• Redis persistence with in-memory fallback.

• Live dashboard for hits, misses, evictions, Redis ops, and latency.

• Docker Compose using official Python and Redis images.

• ShortTermMemory and SemanticCache core stores.

• RedisStore and MemoryService coordinating persistence and metrics.

• FastAPI: /health, /metrics, /memory/*, /cache/*, /demo, /dashboard.

• CLI demo and pytest coverage for memory and API behavior.

• Scripts: start, demo, test, stop, cleanup.

Lesson 2 delivered a permission-gated tool boundary. Lesson 3 sits above it: memory decides what context survives and which answers can be reused without another model call.

Bounded memory enables rate limiting and cost control. Hit rates and token estimates feed Day 4 spend throttling.

• Where this component sits — Between the agent turn loop and durable storage; after tool/model results, before next prompt assembly.

• Why it exists — Context and reuse must be explicit policies, not unbounded lists.

• Problem solved — Prevents context overflow, duplicate LLM calls, and lost session state across restarts when Redis is available.

ShortTermMemory and SemanticCache own local state; RedisStore wraps persistence; MemoryService orchestrates both and feeds METRICS; app.py exposes HTTP and the UI.

• memory/short_term.py, semantic_cache.py, redis_store.py, service.py, metrics.py

• app.py API/dashboard; main.py CLI; Compose and lifecycle scripts

• Context discipline, cache economics, persistence, observability, deployability.

Short-term memory keeps a per-session deque capped by turns and estimated tokens, dropping oldest pairs when budgets break—the tradeoff between recall and prompt cost. Semantic cache keys normalized queries; hits avoid recomputation, but TTL keeps answers from outliving truth. Redis is the shared durable layer for multi-instance deployments; when unreachable, the service degrades to in-memory so demos and tests still run. Metrics treat store, lookup, eviction, and Redis ops as first-class events because memory bugs rarely crash—they quietly raise cost or serve stale data.

• Memory is a cost and correctness control, not a convenience buffer.

• Unbounded history becomes a production incident.

• Cache without TTL creates silent staleness; TTL without metrics hides thrash.

• In-process metrics suit single-worker lessons; replicas need shared stores.

• Fail open on Redis for availability; surface connection state on /health.

Production agents need a memory plane that bounds context, caches expensive answers, and survives restarts. This lesson is that plane for later rate limiters and multi-agent workflows.

• request flow — API/CLI calls MemoryService for store, history, cache lookup/set, or demo.

• execution flow — Update local structures, optionally mirror to Redis, record latency and outcomes.

• data flow — Turns and cache values move through service methods; dashboard polls /metrics.

• state changes — Deques grow/trim; cache entries expire; Redis keys set with TTL; counters update.

Request → service method → local update → optional Redis → metrics → response.

• production architecture fit — Shared library beside the agent runtime; Redis as multi-pod fabric.

• enterprise deployment patterns — Compose/K8s, official images, env REDIS_URL, health probes.

• scalability — App replicas require Redis for shared sessions/cache.

• observability — Hit rate, evictions, Redis ops, latency, event timelines.

• security considerations — No secrets in repo; private Redis; sanitize session IDs and payloads.

Store/lookup APIs accept turns and cache values; metrics snapshots drive the dashboard; demo exercises window trim, hit/miss, and TTL eviction.

Turn, ShortTermMemory, SemanticCache, RedisStore, MemoryService, MetricsStore.

store(), get(), cache_lookup(), cache_store(), run_demo(), record(), snapshot().

https://github.com/sysdr/production-ai-engineering/tree/main/lesson3/aiam-day03

Bounded session memory → TTL cache → Redis with safe fallback → MemoryService + metrics → FastAPI/dashboard → Compose packaging. Validate with unit tests, /demo non-zero metrics, and Redis-connected health. Redis failures degrade without crashing the API.

Token budget trimming prevents prompt blowups before the model is called.

TTL-checked hits are the economic win; expired keys become misses and evictions.

Normalized metrics make memory behavior operable, not anecdotal.

• scalability: shared Redis across replicas.

• security: private Redis, no API keys, bounded payloads.

• monitoring: hit rate, evictions, Redis connectivity, latency.

• logging: structured op labels without secrets.

• testing: window limits, hit/miss, TTL, demo metric assertions.

• failure handling: Redis offline → local memory; health reports status.

• edge cases: empty sessions, case-normalized keys, LRU at max size.

• verification: pytest; Compose health; /demo smoke checks.

• testing strategy: unit tests without Redis; integration with Redis connected.

• success criteria: start.sh / demo.sh / run_tests.sh work from the child project alone; post-demo metrics non-zero.

• expected outputs: hits/misses, TTL evictions, Redis ops when connected, live /dashboard events.

• benchmarks: hit rate and latency_ms_avg after demo > 0.

• production checklist: official images, .gitignore, cleanup script, no parent-setup dependency.

Support bots keep the last N turns per ticket in Redis so pod handoffs stay coherent, while FAQ answers cache for minutes to cut duplicate LLM spend. Coding assistants cache identical build-error explanations with short TTL so outdated remediation is not served after dependency changes.