Week 25-26: Natural Language Processing (NLP)

Most NLP tutorials stop at isolated notebooks: one file for tokenization, another for sentiment, a third for intent, none wired to HTTP, persistence, or observability. week_25_26_aiml_integrated_project is a self-contained capstone that consolidates classical text processing, transformer tokenization, embeddings, sequence modeling, sentiment analysis, intent routing, and response selection into one runnable system:

• An installable Python package (week2526_python) with a core engine, learning lab wrappers, and a product NLP pipeline

• A single FastAPI backend exposing a lab path for component exploration and a product path for analyze, predict, and experiment training

• A React dashboard with live telemetry, a support-assistant workflow, a pipeline studio, and an operations console

• Postgres-backed experiment metadata, PyTorch artifacts on disk, and production hardening (timeouts, structured logging, graceful ML degradation)

The engineer payoff is operational clarity: one /analyze call runs preprocess → intent → sentiment → response; training jobs become Run rows you poll from the dashboard; missing models fall back to lexicon and keyword rules instead of crashing the API.

Each block below is a distinct capability inside this repository—not an external dependency.

week2526_python.core.text_pipeline provides NLTK-backed normalization, tokenization, stemming, lemmatization, POS tagging, and TF-IDF feature extraction. nltk_bootstrap.py lazily downloads punkt and stopwords only when classical processing runs, keeping cold starts fast.

core.tokenization.distilbert and core.tokenization.subword wrap Hugging Face tokenizers (DistilBERT, BERT, GPT-2). Lab endpoints expose side-by-side token comparisons without loading full models on every health check.

core.embeddings.glove loads GloVe vectors or falls back to a deterministic synthetic vocabulary for offline demos. glove_bootstrap.py caches vector files under GLOVE_CACHE_DIR on startup when WEEK2526_BOOTSTRAP_MISSING=true.

core.sequence_models.char_lstm implements a character-level LSTM for text generation. Checkpoints live under artifacts/sequence/char_lstm.pt after a lab train call.

The sentiment module spans vocabulary building (vocabulary.py), Bi-LSTM architecture (model.py), training (training.py), lexicon scoring (lexicon.py), and inference (inference.py). Artifacts: artifacts/sentiment/best_model.pt, vocab.json, and metrics.json.

core.intent.classifier trains a DistilBERT head over six support intents. core.intent.lightweight provides keyword-based routing when no checkpoint exists or when WEEK2526_LIGHTWEIGHT_MODE=true. Checkpoint path: artifacts/intent/intent_classifier.pt.

core.response_selection.selector maps intent + sentiment score to template replies, optionally reranking with embedding similarity when ML artifacts are present.

core.nlp_engine.pipeline implements analyze_text, predict_sentiment_product, system_status, and bootstrap_missing_assets. This is the orchestration layer the product API calls through app.services.model_loader.

backend/app/services/model_loader.py lazy-loads sentiment and intent models behind a thread-safe registry, exposes readiness(), and never raises on missing artifacts—lexicon and lightweight intent are used instead.

SQLAlchemy models (Experiment, Run, Artifact) plus async repositories manage product training. nlp_train_job.py schedules background jobs with asyncio.create_task and supports synchronous training via ?sync=true.

backend/app/main.py defines the only FastAPI() instance. Middleware includes request timeouts (30s default, 600s for long train routes), upload size limits (WEEK2526_MAX_UPLOAD_MB), JSON request logging with request_id and latency_ms, and environment-aware CORS.

The frontend (frontend/src/ui/) ships four views—Overview, Support Assistant, Pipeline Studio, and Operations—with sidebar navigation, live metric polling (4s global, 3s during active training runs), animated pipeline progress on analyze, and module cards grouped by topic (text processing, embeddings, sentiment, intent and response).

http://localhost:3000

OperatorsMulti-panel NLP console proxied through Nginx

The diagram below shows how the dashboard, API layer, Python package tiers, and persistence volumes connect. External caches (NLTK, Hugging Face) feed the core package without blocking the product path in lightweight mode.