Meta released Llama 4, the next version of their open-weight language models. It introduces native multimodality, longer context windows, and a more efficient architecture. This post breaks down what’s new, what’s useful, and where to get it.
The standout feature is the 10 million token context window—but there’s a tradeoff: none of the current models are lightweight. The smallest one, Llama 4 Scout, still requires at least an Nvidia H100, so they’re not viable for most local or consumer-grade GPU setups.
Key Features
- Multimodal: Works with both text and image inputs
- Mixture of Experts (MoE): More efficient model execution
- Context Window: Up to 10 million tokens
- Open Weights: Download and run locally, or use via API
Variants
| Model | Active / Total Params | Context Limit | Use Case |
| Llama 4 Scout | 17B / 109B | 10M tokens | Long-context tasks (e.g. docs, code) |
| Llama 4 Maverick | 17B / 400B | 1M tokens | General-purpose + image understanding |
| Llama 4 Behemoth | 288B / 2T (in training) | TBD | Large-scale reasoning / STEM |
Architecture: Mixture of Experts
Llama 4 is Meta’s first model family to adopt a Mixture of Experts (MoE) architecture. Unlike dense models—where every parameter is used for every token—MoE models activate only a fraction of the total parameters per token.
This architectural change increases compute efficiency for both training and inference. According to Meta, MoE allows Llama 4 to deliver higher output quality for the same amount of compute (FLOPs) compared to dense models.
Why It Matters
- Training efficiency: More parameters can be trained without increasing the compute budget.
- Inference efficiency: Only a subset of the model is active at runtime, reducing memory and latency requirements.
- Scalability: Enables significantly larger models (like Behemoth) to be trained and deployed efficiently.
Multimodal by Design
Llama 4 was trained natively on both text and image data. This isn’t a bolted-on capability—multimodal support was built into the training pipeline from the start. The models are designed to handle image and text inputs jointly, enabling tasks like image captioning, visual Q&A, and context-aware generation using both modalities.
Long Context Windows
Llama 4 Scout supports up to 10 million tokens. The model is both pre-trained and post-trained with a 256K context length, which helps it generalize well to long inputs.
A key architectural change is the introduction of iRoPE (interleaved Rotary Position Embeddings). Instead of using standard positional encodings, Scout uses interleaved attention layers without fixed positional embeddings, improving its ability to generalize to longer contexts. Meta also applies inference-time temperature scaling of attention weights to further enhance performance on long sequences.
This makes it practical for:
- Retrieval-augmented generation (RAG)
- Summarizing entire codebases or large document collections
- Long-form reasoning and multi-step workflows
- Processing legal, financial, or research archives
Language Support
Llama 4 handles 12 languages well, including:
- English, German, Spanish, French, Chinese, Japanese, Korean, Portuguese, Italian, Dutch, Russian, Arabic
The tokenizer and training data were optimized for multilingual tasks. For other languages not officially supported, the model may still perform reasonably well for simpler tasks like keyword extraction, classification, or summarization—though results may vary.
Access and Licensing
Run It Yourself
Meta released the weights, subject to a usage restriction:
If your product serves over 700M monthly active users, you need a separate commercial license.
Due to privacy and regulatory concerns—particularly under the EU AI Act and GDPR—the Llama 4 models are not officially available for download or use in EU member states. The license explicitly restricts access to individuals and organizations domiciled in the EU.
Everyone else can:
- Download models directly
- Use with libraries like transformers or llama.cpp
Cloud Providers
Llama 4 is already available on:
- AWS (SageMaker, Bedrock)
- Azure AI
- Databricks
- Cloudflare Workers AI
What’s Next?
As of now, two models are publicly released: Llama 4 Scout and Llama 4 Maverick. Both are available with open weights and can be used via APIs or self-hosted. Meta is currently training one additional model:
- Llama 4 Behemoth: A high-capacity model with 288B active parameters (2T total), designed for complex tasks like reasoning and STEM-heavy workloads.
This upcoming model is expected to expand the Llama 4 family’s capabilities later this year.
That’s the current state of Llama 4. We’ll revisit once the next wave of models becomes available.
