Recent content on Artificial Intelligence and Machine Learning Research
Abstract Link to heading Let’s talk a bit about Model-Based Reinforcement Learning. The idea is that our RL Agent not just learns a policy to follow or/and a value function (Q function) but also tries to model the environment it is in. This is done by learning the transition dynamics $p(s’|s,a)$ (also known as World Model) and a Reward function $\hat{R}(s,a)$. Once we have our world…
Abstract Link to heading A while ago we looked into Policy Gradient and Reinforce. Policy gradient is versatile and under mild conditions it is guaranteed to converge to a local minimum (if we choose the correct policy and step size). This is already a huge step up when compared to Q Learning, which may just diverge. However, we may still want stronger guarantees like monotonic improvement at each…
Abstract Link to heading Till now we have considered only learning the Value or Q function and estimating the policy from those. In the next few posts, we are going to look into directly learning the policy. Why directly learn the policy? First, Q learning has a lot of issues involving the Deadly Triad; second, if we have continuous actions we cannot really use it; and lastly, Q learning always…
Abstract Link to heading We already know how to learn the Value function, however we also know that the Value function by itself is not enough since it averages over all possible actions, instead of taking into consideration specific actions the agent should take. We can derive the Q function from the Value function, however we can also try to directly learn it. Especially directly approximating…
Source Link to heading Paper link: https://openreview.net/pdf?id=R8SolCx62K Source Code: https://github.com/hedongxiao-tju/SGRL Abstract Link to heading Contrastive Learning (CL) is one of my favorite techniques, it is a self-supervised approach for learning latent representations with a special property: Similar elements have representations that are closer together and elements that are…
Abstract Link to heading In the last RL-Bite I wrote about Bellman’s Equations and the Value Function and now we will figure out how we actually apply these equations to compute the Value Function! Known World Model Link to heading Let’s start with the simple case, and make an assumption that the underlying World Model of the Markov Decision Process is known, and we have finite…
Source Link to heading Paper link: https://arxiv.org/abs/2405.14742 Source Code: https://github.com/JonathanGXu/HC-GAE Abstract Link to heading Graph Representation Learning is an essential topic in Graph ML, and it is all about compressing a whole Graph (arbitrarily large) into a fixed representation. Usually these techniques leverage Graph Auto Encoders, which are trained in a self-supervised…
Value Based Reinforced Learning Link to heading In value based Reinforced Learning we learn a Value Function: $$ V_{\pi}(s) = E_{\pi}[G_0|s_0 = s] = E_{\pi}[\sum_{t=0}^T \gamma^t r_t|s_0 = s] $$ $G_t$ is the Total Return at time t, this is just the sum of Rewards an Agent gets walking the trajectory T (fancy name but this is just a sequence of actions the agent takes) $\gamma^t$ is the Discount…
Source Link to heading Paper link: https://arxiv.org/abs/2406.05391 Source Code: https://github.com/alipay/DUPLEX Abstract Link to heading I am a huge fan of Graph Machine Learning, it has a lot of cool applications, and I am particularly interested in Source Code understanding and Vulnerability Detection, where Graph Neural Networks (GNN) are unambiguous. One of the obvious downsides of general…
Explore vs Exploit Link to heading In reinforcement learning, we have an agent that has to take actions, for which it receives a reward. Here we have a dilemma: do we choose an action that gives the biggest reward or do we explore new actions that may lead to regions with even higher payouts? Greedy Policy Link to heading This is a simple one, we always take the option that gives us the highest…
Abstract Link to heading I am a huge fan of Graph Neural Networks (GNNs), and I am (a bit less) a fan of Large Language Models (LLMs), however they are hard to ignore. Both have different strengths, while GNNs excel when it comes to problems that have an inherent structure, LLMs thrive in cases where we treat everything as a sequence of tokens (maybe Bytes in the future). A natural question…
Abstract Link to heading State space models are really close to my heart, I even have a dedicated page about them. But when it comes to Language Models they lack some performance and that gave rise to SSM-Attention Hybrids. Until now it was conventional that in hybrid layers you sequentially combine Mamba(2) with Attention layers. Nvidia introduced Hymba - this paper changes the game by using…
Abstract Link to heading In the last two years there has been a surge of Large Language Models. This is understandable, since LLMs are amazing at generating text, and a lot of things can be viewed as text. However, generation is not always all we need - sometimes we want to have semantically rich representations. In general, LLMs are not the best tool to get semantically rich representations, and…
Intro Link to heading Programming languages come and go and during my 20 years of coding I have used many of them to at least some degree (more than just hello world). Thanks to ThePrimeagen I decided to take Zig for a spin. And boy I really like it! Just look at the mascot: Come on, a crocodile with a jetpack? How cool is that? Why Zig Link to heading Sure I would not invest a lot of time into…
Abstract Link to heading It is notoriously expensive to train a Language Model from scratch, making independent research impossible and trying out new architectures extremely risky. Because of these costs, Transformer++ models like LLaMa, based on Rotary Embedding, SwiGLU, MLP, RMSNorm, without linear bias, sometimes with grouped query attention and/or sliding window attention, are the de facto…
Abstract Link to heading Last time we looked into the weak points of State Space Models (Mamba, Mamba2), especially when compared with Attention-Based models (LLama, GPT-like). They lack in terms of in-context learning. To alleviate this, we focused on SSM-Transformer Hybrids and introduced multiple models that do this differently. Here we look into the expressivity of State Space Models from two…
Abstract Link to heading We have already looked into Mamba and Mamba2. In terms of efficiency, with their linear complexity and the absence of Key-Value cache, they are a significant improvement over Attention-based models in terms of throughput and memory usage. However, not everything is perfect. Transformers have a certain advantage when it comes to in-context learning. In-context learning is…
This series will cover a bunch of posts about State Space Models, their extensions and applications. Basics Link to heading Mamba, Mamba2 Bidirectional Link to heading Hydra Attention Hybrids Link to heading SSM-Transformer Hybrids covers: An Empirical Study of Mamba-based Language Models SAMBA Simple Hybrid State Space Models for Efficient Unlimited Context Language Modeling Jamba A Hybrid…
Abstract Link to heading State Space Models are awesome, models like Mamba and Mamba2 boast unparalleled performance especially when it comes to long sequences. The only downside is that they are causal, which means they model one token at a time, looking only at past tokens. Bidirectional models like Bert, CodeBERT and GraphCodeBERT have been shown to excel when it comes to code understanding.…
Abstract Link to heading This is not my first gig where I write about State Space Models. I already mentioned them here and here. Now what is the deal with this Mamba(2) thing? They are proving to be an alternative to the strong Transformer++ architecture (Transformer++ models like LLaMa are based on Rotary Embedding, SwiGLU, MLP, RMSNorm, without linear bias, sometimes with grouped query…
Abstract Link to heading I have been working on a project of creating my own Large Language Model, as I am huge fan of T5, or to be more concrete I recognize the added value of having an Encoder-Decoder architecture. The biggest challenge, at least in my opinion, in training an LLM is the sheer computational costs required to do so. I was originally planning to take the Encoder introduced by ColT5…
Abstract Link to heading For a while now I have a new passion and that is binary reverse engineering and vulnerability exploitation. This interest has led me to create CodeBreakers a platform dedicated to applying machine learning to reverse engineering, vulnerability detection, exploitation, and other cybersecurity-related applications. I found two notable research papers where T5 has been…
Abstract Link to heading In a previous post, T5 the Old New Thing, we briefly touched upon CodeT5 and CodeT5+. Now, we aim to dive deeper into these topics. I have previously explored CodeBERT and GraphCodeBERT. These models, based on BERT and RoBERTa architectures, excel at code understanding and retrieval tasks. However, they fall short when it comes to code generation tasks. It’s worth…
Here is a bunch of my posts about a super cool model called T5 its extensions and applications. Basics Link to heading T5, FLAN-T5, UL2 LongT5, CoLT5 Coding T5 Link to heading CodeT5, CodeT5+ AST-T5 (Coming Soon) CodeFusion (Commit Soon) there is a bit of confusion about this research paper, there are some citation issues, and the overall reviews are not that great, however it still sounds like an…
Why does T5 need a longer context? Link to heading In my previous post T5 the Old New Thing we already explored why T5 is awesome. But one downside is its limited context length of 512 tokens. However, it does have a limitation - its context length is restricted to 512 tokens. This can’t be directly compared to the context length of a decoder-only model, as T5 is an encoder-decoder model.
Why T5 Link to heading A couple of weeks ago I run into the following paper Tiny Titans. It compares multiple smallish (up to 1B parameters) open source LLMs with bigger proprietary ones on meeting summarization. TLDR; the small models tend to perform worse in zero-shot setting as well after fine-tunnig than big ones. Except for FLAN-T5-Large which after finetuning performs way beyond its league,…
Lately, I’ve embarked on a side project: CodeBreakers. It’s nothing too fancy, just a website where I plan to release videos and articles about my latest passions—Reverse Engineering and Binary Exploitation. Creating a website isn’t all that complex, and I’ve done it a couple of times before. The main challenge was deciding where and how to host it. Initially, I considered…
High level overview Link to heading By combining State Space Models (SSMs) with Attention, we get a model that generates text more efficiently, with a speed increase of approximately 1.6 times. Additionally, this approach requires less paremters, enabling the development of larger models on existing hardware. Language modeling requirements Link to heading The Transformer architecture, which forms…
The problem it solves Link to heading Imagine you have an irregullary sampled time series, where at various time points we perform interventions. These interventions may influence the dynamics of the timeseries. The question we want to answer is: If I perform a hypothetical sequence of interventions how will my time series evolve? An example and some details Link to heading Example Link to heading…
Probabilistic Matrix factorization is a simple but useful model for matrix imputation. The main idea is to decompose a tall and wide matrix into a product of two matrices, one tall and thin and one short and wide. $$ R_{n\times m} = U_{m \times d} \cdot V_{d \times n} $$ If you are a Bayesian, you can express this model as: $$ R_{ij} \sim \mathcal{N}(u_i \cdot v_j^T, \sigma) $$ $$ u_i \sim…
About Link to heading I’m Marek Barak, I started my journey into computer science when I was 15 by coding in C++, without having access to the internet and in good old fashioned Bloodshed C++. Since then nearly 20 years have passed, and I managed to get a Master degree in Computer Science and Statistics from the University of Economics in Bratislava. I had the opportunity to do machine…
Contact Link to heading Feel free to contact me via email mrk.barak@gmail.com or socials.
Projects Link to heading Study notes Link to heading This is my compilation of notes from various subjects. This is work in progress, where I continuously migrate, rewrite, simplify the notes I have. During the years I wrote thousands of pages worth of notes, not always concise. Right now I prefer taking notes that are somewhat similar to the Zettelkasten method. Thus keeping them short, easy to…