Days 173–176: Build a Sentiment Analyzer

• Wire together everything from Days 169–172 into a single end-to-end pipeline

• Build a production-style sentiment classifier on the IMDB movie reviews dataset

• Train, evaluate, and serve the model through a lightweight REST inference endpoint

• Measure real metrics — accuracy, F1 score, and confusion matrix — not just training loss

Every product that handles user-generated text has a sentiment layer somewhere. Amazon uses it to surface trending complaints before they become PR crises. Airbnb uses it to flag toxic reviews in real time. Spotify uses it on social data to figure out which artists are gaining buzz before the charts reflect it. The pattern is always the same: preprocess text, encode meaning, classify intent, act on the signal. You have all three of those pieces now — tokenization from Day 169, embeddings from Day 170, and sequence modeling from Day 172. This project stitches them into a deployable system and teaches you the engineering discipline of closing the loop from raw data to a live prediction endpoint.

Up to now you have built each stage in isolation. A sentiment analyzer forces you to reason about the pipeline as a single contract: garbage in one stage poisons every stage after it. The input to your model is not text — it is a fixed-length integer sequence produced by a specific tokenizer trained on a specific vocabulary. That tokenizer must be saved alongside the model weights or the model is useless at inference time. This is the single most common mistake junior engineers make when deploying NLP systems: they checkpoint the weights and throw away the tokenizer.

Think of it like a lock-and-key pair. The model is the lock; the tokenizer is the key. Ship them as one artifact.

A standard LSTM reads a sentence left to right. But the word “not” near the beginning of a sentence can completely invert the meaning of a word that appears twelve tokens later. A Bidirectional LSTM runs two passes — one forward, one backward — and concatenates the hidden states at each timestep. The model can now “see” what comes after a word before it decides how to weight that word. For sentiment tasks, Bi-LSTMs consistently outperform unidirectional ones by three to five percentage points on benchmark datasets with no extra training data.

You can initialize your embedding layer with random weights and let the model learn them from scratch. That works fine if you have hundreds of thousands of training examples. For smaller datasets — under fifty thousand examples — you are better off seeding the embedding layer with pretrained GloVe vectors (100d or 200d). GloVe was trained on billions of tokens; your model gets that knowledge for free and only needs to fine-tune it. The practical difference in validation accuracy on IMDB is typically three to eight points, and training converges two to three times faster.

Binary sentiment on IMDB is balanced (50% positive, 50% negative), so accuracy is a fair metric here. The moment you move to a real production dataset — say, customer support tickets where 80% are neutral — accuracy becomes misleading. A model that always predicts “neutral” gets 80% accuracy while being completely useless. F1 score, which balances precision and recall, is the metric that actually tells you whether the model is doing something useful. Wire in scikit-learn’s classification_report from day one so you develop the habit of looking at the full picture.

A trained model sitting in a .pt file does nothing for a product team. What they need is an inference function with a clean, stable interface: string in, label and confidence score out. In production this function gets wrapped in a FastAPI endpoint, containerized, and deployed behind a load balancer. Building that wrapper is as much a part of the project as training the model. The function needs to handle edge cases — empty strings, non-ASCII characters, extremely long inputs — without crashing.