Recent content on Hugo Cisneros
tags Reinforcement learning, Reinforcement learning with human feedback, Reward shaping Recovering an unknown reward function from expert demonstrations such that the demonstrated behavior is optimal under it. It is a classical setting extended in modern post-training via RLHF, adversarial IRL, and demonstration-conditioned implicit rewards.
tags Continual learning, Catastrophic forgetting, Distillation, In-context learning, Large language models source (Shenfeld et al. 2026) Summary This paper introduces Self-Distillation Fine-Tuning (SDFT), an on-policy alternative to supervised fine-tuning (SFT) for continual learning from expert demonstrations. The motivation is a known asymmetry in post-training: on-policy reinforcement learning…
tags Distillation, Supervised fine-tuning, Large language models Distillation regime where the student is trained on a fixed dataset of trajectories generated by a different distribution. typically the teacher, an earlier checkpoint, or a static expert corpus — rather than from the student’s own current policy. Standard knowledge distillation and most supervised fine-tuning recipes are…
tags Distillation, Reinforcement learning, Large language models, Continual learning Distillation regime where the student samples its own trajectories and minimizes divergence to a teacher evaluated on those samples. It combines the credit-assignment denseness of distillation with the distribution-matching guarantees of on-policy learning. Contrast with off-policy distillation, where the student…
tags Reinforcement learning, Distillation, Large language models, In-context learning source (Hübotter et al. 2026) Summary This paper introduces Self-Distillation Policy Optimization (SDPO), a new algorithm for post-training large language models with reinforcement learning. Current methods for RL with verifiable rewards (RLVR), such as GRPO, learn only from sparse scalar outcome rewards (e.g.,…
tags Vision Language Models, Grounding, Synthetic training data, Spatial Reasoning, Foundation models source (Clark et al. 2026) Summary Molmo2 is a family of fully open Vision Language Models (4B, 8B built on Qwen3, and a 7B built on OLMo) trained without distillation from proprietary systems. The work closes the open-source gap for video-capable VLMs with a core focus on Grounding — producing…
tags LLM, Vision transformer Vision language models (VLMs) are generative AI models trained on both text and images. They can be effective tools for Image classification. CLIP is an early and influential example of contrastive vision-language pre-training. VLMs are a class of Foundation models. Remote Sensing VLMs VLMs for remote sensing are maturing rapidly (as of early 2026), approaching the…
tags Vision Language Models, Reinforcement learning, GRPO, Tool calling, Grounding, Reinforcement learning with verifiable rewards source (Zheng et al. 2025) Summary DeepEyes is a Vision-Language Model that learns to “think with images” — interleaving textual chain-of-thought with self-initiated image zoom-ins during reasoning — and is trained purely with end-to-end Reinforcement…
tags Vision Language Models, Geospatial AI, Tool calling, GRPO, Reinforcement learning source (Wang et al. 2026) Summary GeoEyes addresses visual question answering (VQA) on ultra-high-resolution (UHR) remote sensing imagery — scenes where task-relevant cues occupy only tiny fractions of the full image. The authors target the “thinking-with-images” paradigm, in which Vision Language…
tags LLM, Continual learning, Catastrophic forgetting, Transfer learning, Language modeling source (Biderman et al. 2024) Summary This paper provides a rigorous head-to-head comparison of Low-Rank Adaptation (LoRA) against full finetuning of Llama-2-7B on two challenging target domains (code and math) under two training regimes: continued pretraining (CPT, ~20B unlabeled tokens) and instruction…
tags Large language models, Foundation models, Catastrophic forgetting, Continual learning Off-policy adaptation of a pretrained model by training on (input, target) pairs from expert demonstrations using cross-entropy loss. It is the dominant post-training recipe for skill and knowledge injection but prone to catastrophic forgetting due to its off-policy nature
tags Transformers, LLM, Scaling laws, Attention, Residual neural networks, Memory in neural networks source (Mak, Flanigan 2025) Summary Standard Transformers use a residual stream of dimension \(D\) as a “memory bus” where every layer reads and writes features (Elhage et al., 2021). Resizing this bus also resizes every weight matrix, so the bandwidth of the residual stream is…
Notes by Topic ALife 2020 Talk: Alife 2020 keynote Lee Cronin - A Top Down Chemically Embodied Artificial Life Computation (9 links, 1 backlinks) Talk: Alife 2020 keynote Michael Levin - Robot Cancer (4 links, 1 backlinks) Talk: Alife 2020 keynote Luis Zaman - New Frontiers in Alife: What was old is new again (14 links, 1 backlinks) ALife Conference ALife 2020 (24 links, 9 backlinks) Agent…
tags Object recognition, Vision Language Models, Grounding, Computer vision Object detection where the class vocabulary is unbounded and specified at inference via natural-language phrases or visual exemplars. It requires vision-language alignment and calibrated rejection of hard negatives.
tags Computer vision, Image processing, Object recognition, Foundation models Predicting per-pixel labels (instance, semantic, panoptic) for an image, including the open-vocabulary regime where target classes are specified by text or visual exemplars at inference time.
tags Reinforcement learning, Reinforcement learning with verifiable rewards, GRPO Pathologies where agents exploit literal reward structure (e.g., spamming tool use without accuracy gains).
tags Reinforcement learning, Reinforcement learning with verifiable rewards, GRPO Patterns for designing RL rewards that produce intended behaviors (e.g., conditional bonuses that trigger only on correct outcomes).
tags LLM, Test-time compute, Reinforcement learning, Token-level credit assignment in reasoning traces Prompting and training paradigm where models emit intermediate reasoning steps before a final answer, improving multi-step problem solving and enabling RL on verifiable outcomes.
tags Machine learning, LLM, Agent, Computer vision, Retrieval augmented generation The application of AI and LLMs to geospatial data like satellite imagery, GIS databases, mapping, and Earth observation. LLMs have become production-grade tools across the geospatial industry in 2024–2026. Enterprise GIS + LLMs The dominant deployment pattern (as of early 2026) is agentic orchestration: LLMs serve…
tags Vision Language Models, Spatial Reasoning, Grounding Task of answering natural-language questions grounded in image content, spanning global scene understanding to fine-grained perception and compositional reasoning
tags Machine learning, Program synthesis, Agent A LLM agent (based on a generic of specialized model) dedicated to programming. One popular way of training effective coding agents is to do agentic reinforcement learning
tags Machine learning, Optimization, Meta-learning, Program synthesis source (Lee et al. 2026) Summary This paper introduces Meta-Harness, an outer-loop system for automatically optimizing the “harness” of LLM applications — the code that determines what information to store, retrieve, and present to the model at each step. The key insight is that harness design matters as much…
tags Reinforcement learning, Reinforcement learning with verifiable rewards, GRPO, Tool calling RL post-training for LLM/VLM agents that decide when and how to invoke tools during reasoning, with rewards shaped around tool-use policy (necessity, efficiency, trajectory geometry) rather than just final-answer correctness
tags Vision Language Models, Tool calling, Spatial Reasoning, Grounding Multimodal reasoning paradigm where VLMs interleave textual chain-of-thought with active visual operations (zoom, crop, search) to acquire evidence on demand.
tags Machine learning, Self-supervised learning Big models trained on a large amount of domain data to learn useful patterns and serve as a basis for other downstream machine learning applications. These models usually operate at the frontier fo scaling laws of compute+data vs. performance. These models can serve as bases for Transfer learning. Some common classes of foundation models are: LLM…
tags Transformers, T5, NLP paper (Fedus et al. 2022) Architecture This model increases the parameter count of T5-like architecture while allowing efficient routing through different experts in a mixture of experts. Parameter count 1T Bibliography William Fedus, Barret Zoph, Noam Shazeer. June 16, 2022. "Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity".…
tags Vision Language Models, Computer vision, CLIP, Contrastive learning, Vision transformer, Foundation models source (Bolya et al. 2025) Summary This paper introduces Perception Encoder (PE), a family of vision encoders from Meta FAIR trained with a purely global CLIP-style contrastive vision-language objective that nonetheless produces state-of-the-art features for tasks as diverse as zero-shot…
tags Self-supervised learning, Vision transformer, Foundation models, Robotics source (Mur-Labadia et al. 2026) Summary V-JEPA 2.1 is a family of self-supervised video models (ViT-g/G, 1B/2B, plus distilled ViT-L/B variants) from FAIR at Meta that extends the Joint-Embedding Predictive Architecture (JEPA) line to produce representations that are simultaneously strong on dense spatio-temporal tasks…
tags Transformers, Attention, Computer vision, Object recognition, Positional encoding source (Carion et al. 2020) Summary DETR (DEtection TRansformer) reframes object detection as a direct set prediction problem, eliminating hand-designed components that traditional detectors rely on: anchor generation, non-maximum suppression (NMS), and coordinate-regression heuristics against proposals. The…
tags Computer vision, Foundation models, Object recognition, Vision Language Models, Grounding, Synthetic training data source (Carion et al. 2025) Summary SAM 3 (Segment Anything Model 3) is Meta’s third installment of the SAM family of Foundation models for Computer vision. The headline contribution is a new task — Promptable Concept Segmentation (PCS) — which generalizes the SAM 1/2…
tags Neural networks, Transformers Implementation Self-attention is a weighted average of all input elements from a sequence, with a weight proportional to a similarity score between representations. The input \(x \in \mathbb{R}^{L \times F}\) is projected by matrices \(W_Q \in \mathbb{R}^{F \times D}\), \(W_K \in \mathbb{R}^{F\times D}\) and \(W_V \in \mathbb{R}^{F\times M}\) to representations…
tags LLM, Diffusion models, Language modeling, Transformers Language model architecture that use diffusion instead of autoregression. They generate text by iteratively denoising masked or noised tokens, which enables parallel decoding.
tags LLM, Diffusion models, Transformers, Test-time compute source (Chen et al. 2026) Summary DFlash is a speculative decoding framework that replaces the usual small autoregressive draft model with a lightweight block diffusion draft model. The draft model generates a whole block of tokens in a single forward pass, which are then verified in parallel by the target LLM. The authors argue that the…
tags Machine learning
tags LLM, Optimization, Test-time compute, Language modeling The principle of speculative decoding for LLMs is based on the asymmetry of Transformer-based architecture between: decoding tokens one by one, resulting in individual full passes through the model verifying multiple tokens at once, which results in only one full pass on a slightly longer sequence In speculative decoding, a large…
tags Machine learning, LLM, Agent, Evaluating NLP
tags Agent, Coding agent, Multi-agent collaboration An open protocol (originally from Anthropic) that standardizes how AI agents connect to external tools and data sources. MCP defines a client-server architecture where agents (clients) invoke capabilities exposed by tool servers through a standard interface, similar to how USB-C standardizes device connectivity. It is closely related to Tool…
tags Logic
tags Reinforcement learning, Distillation, Self-training Three early 2026 papers (MiniMax-M1 (CISPO), Zhang et al. (SSD), and Hübotter et al. (SDPO)) converge on a shared structural observation: not all tokens in a reasoning trace are equally important for learning, and naive uniform treatment of tokens is a core failure mode of current training methods. The fork/filler distinction All three…
tags Reinforcement learning, Algorithm, Machine learning
tags Reinforcement learning, Algorithm, Machine learning
tags Foundation models, Reinforcement learning, Transformers, Scaling laws source (MiniMax 2025) Summary MiniMax-M1 is the first open-weight, large-scale reasoning model built on a hybrid attention architecture combining Transformers with lightning attention (a linear attention variant). The model uses a Mixture-of-Experts (MoE) design with 456 billion total parameters (45.9B activated per token)…
tags Transformers, LLM, Scaling laws, Attention, Residual neural networks source (Chen et al. 2026) Summary Standard residual connections in modern LLMs accumulate all layer outputs with fixed unit weights via PreNorm, causing uncontrolled hidden-state growth with depth and progressively diluting each layer’s contribution. This paper proposes Attention Residuals (AttnRes), which replaces…
tags Attention, Transformers, Machine learning, Applied maths Attention variants that replace softmax with linear kernels, reducing complexity from quadratic to linear in sequence lengt
tags Transformers, RNN source (Katharopoulos et al. 2020) Summary Transformers have traditionally been described as different models from RNNs. This is because instead of processing the sequence one token at a time, Transformers use attention to process all elements simultaneously. The paper introduces an interesting new formulation, replacing the softmax attention with a feature map-based dot…
tags Transformers, LLM, Machine learning, Scaling laws Sparse neural network architecture that routes inputs to a subset of expert subnetworks, enabling parameter scaling without proportional compute increase.
tags Distillation, Language modeling, Program synthesis, Large language models source (Zhang et al. 2026) Summary This paper introduces simple self-distillation (SSD), a method where an LLM improves its own code generation by sampling solutions from itself with specific temperature and truncation settings, then fine-tuning on those raw, unverified samples using standard supervised fine-tuning…
tags Machine learning, LLM, Reinforcement learning
tags Machine learning, Reinforcement learning, LLM This is related to RLHF, but instead of relying on human scoring of outputs, it uses programmatically verifiable outcomes (such as unit tests for code, math proofs, etc.).
tags Machine learning, Distillation, Language modeling, LLM Implications for open-ended evolution Looking at self training through the lense of Open-ended Evolution, it feels like pure self training (only a model with itself) cannot lead to radical improvement or novel behavior since it is fundamentally limited by the original distribution that the model is capable of modeling. Only external…