Alchemist's Notes on Deep Learning
I recently compiled a more cleaned-up set of explanations and implementations on topics relating to deep learning and generative modelling. Check them out at https://notes.kvfrans.com/
kevin frans blog
kevin frans blog
Latest posts
Small-Research: Tanh Activations with DDPG
When implementing DDPG-style policy extraction, we often use a tanh normalizer to bound the action space. That way the policy does not attempt to output something OOD when the Q-function is only trained on actions in [-1, 1]. dist = agent.actor(batch['observations']) normalized_actions = jnp.tanh(dist.loc) q
Small-Research: Policy Extraction in IQL
Insight : IQL-AWR agents on 'maze2d' converge to a standard deviation of 0.4, which is much higher than expected. Hypothesis : The KL constraint of AWR prevents it from learning Gaussian policies that are centered around the optimal action. Conclusion : In data-heavy settings, use DDPG or discretized AWR for policy extraction.
Successor Representations Explained
Note: We're going to conflate terminology for value-functions and Q-functions. In general the Q-function Q ( s , a ) Q(s,a) Q ( s , a ) is just an action-conditioned version of the value function V ( s ) V(s) V ( s ) . Successor Representations Let's consider a classic temporal difference method -- estimating the
Variational Information Bottleneck Explained
Let's take a look at neural networks from the perspective of information theory. We'll be following along with the paper Deep Variational Information Bottleneck (Alemi et al. 2016). Given a dataset of inputs X X X and outputs Y Y Y , let's define some intermediate representation Z Z Z . A
A Brief Examination of Generative Models
Let's explore some generative models. There's a nice variety of them around. How well can the various methods model a given dataset? What are their flaws and strengths? I'm interested in how well generative models handle mode coverage . Let's say my dataset has images of five cat breeds. How is
Deriving the KL divergence loss in variational autoencoders
Let's derive some things related to variational auto-encoders (VAEs). Evidence Lower Bound (ELBO) First, we'll state some assumptions. We have a dataset of images, x x x . We'll assume that each image is generated from some unseen latent code z z z , and there's an underlying distribution of latents p
For AGI, we need better tasks. For better tasks, we need open-endedness. (ALOE 2022 Notes)
Open-endedness is the idea of a system that "endlessly creates increasingly interesting designs". That's a simple concept with big implications. Biological life on earth is diverse and beautiful and the result of open-ended evolution. Human culture is open-ended; we've created a lot of art and technology without any real plan.
A Mathematical Definition of Interestingness
In open-endedness research, we care about building systems that continously generate interesting designs. So what makes something interesting? Previously, I argued that interestestness depends on a viewer's perspective. But this answer isn't very satisfying. Ideally, we would want some mathematically-definable way to measure how interesting something is. A naive approach
To extract information from language models, optimize for causal response
One way to view AI progress is that we can measure increasingly abstract concepts. A decade ago, it was really hard to take a photo and measure “how much does this image look like a dog?”. Nowadays, just pick your favorite image classification model and run the image through. Similarly,
Data digesters, ML^2, Interestingness
This post is a bunch of random thoughts about things I've read recently. It's organized into sections, and the sections don't really relate to each other, but here they are. Neural Networks are Data Digesters A common philosophy around my head is that "representation learning is everything". Need to label
CLIPDraw: Exploring Text-to-Drawing Synthesis
CLIPDraw synthesizes novel drawings from text. Play with the codebase yourself. AI-assissted art has always been intriguing to me. To me, visual art is a very human thing -- I can’t imagine a way that a computer rediscovers our own cultural concepts without some kind of experience living in
StampCA: Growing Emoji with Conditional Neural Cellular Automata
StampCA growing emoji. Play with the codebase yourself. When a baby is born, it doesn’t just appear out of nowhere -- it starts as a single cell. This seed cell contains all the information needed to replicate and grow into a full adult. In biology, we call this process
Quality Diversity: Evolving Ocean Creatures
If you've ever folded a paper airplane, chances are you folded The Dart. It's fast and simple, and kids all around the world have launched it through the air. Soon, however, they will move on to other designs – airplanes that glide, make loops, or twirl in flight. There are thousands
Open-Endedness 3: Multicell World
How many shapes can be made from six 2x4 LEGO bricks? The answer is 915,103,765. By combining only a few simple parts, it’s surprisingly easy to achieve high levels of complexity. A Lego Counting Problem: http://web.math.ku.dk/~eilers/lego.html If you look at