Distill

Understanding Convolutions on Graphs

Understanding the building blocks and design choices of graph neural networks.

A Gentle Introduction to Graph Neural Networks

What components are needed for building learning algorithms that leverage the structure and properties of graphs?

Distill Hiatus

After five years, Distill will be taking a break.

Adversarial Reprogramming of Neural Cellular Automata

Reprogramming Neural CA to exhibit novel behaviour, using adversarial attacks.

Weight Banding

Weights in the final layer of common visual models appear as horizontal bands. We investigate how and why.

Branch Specialization

When a neural network layer is divided into multiple branches, neurons self-organize into coherent groupings.

Multimodal Neurons in Artificial Neural Networks

We report the existence of multimodal neurons in artificial neural networks, similar to those found in the human brain.

Self-Organising Textures

Neural Cellular Automata learn to generate textures, exhibiting surprising properties.

Visualizing Weights

We present techniques for visualizing, contextualizing, and understanding neural network weights.

Curve Circuits

Reverse engineering the curve detection algorithm from InceptionV1 and reimplementing it from scratch.

High-Low Frequency Detectors

A family of early-vision neurons reacting to directional transitions from high to low spatial frequency.

Naturally Occurring Equivariance in Neural Networks

Neural networks naturally learn many transformed copies of the same feature, connected by symmetric weights.

Understanding RL Vision

With diverse environments, we can analyze, diagnose and edit deep reinforcement learning models using attribution.

Communicating with Interactive Articles

Examining the design of interactive articles by synthesizing theory from disciplines such as education, journalism, and visualization.

Thread: Differentiable Self-organizing Systems

A collection of articles and comments with the goal of understanding how to design robust and general purpose self-organizing systems.

Self-classifying MNIST Digits

Training an end-to-end differentiable, self-organising cellular automata for classifying MNIST digits.

Curve Detectors

Part one of a three part deep dive into the curve neuron family.

Exploring Bayesian Optimization

How to tune hyperparameters for your machine learning model using Bayesian optimization.

An Overview of Early Vision in InceptionV1

An overview of all the neurons in the first five layers of InceptionV1, organized into a taxonomy of 'neuron groups.'

Visualizing Neural Networks with the Grand Tour

By focusing on linear dimensionality reduction, we show how to visualize many dynamic phenomena in neural networks.

Thread: Circuits

What can we learn if we invest heavily in reverse engineering a single neural network?

Zoom In: An Introduction to Circuits

By studying the connections between neurons, we can find meaningful algorithms in the weights of neural networks.

Growing Neural Cellular Automata

Training an end-to-end differentiable, self-organising cellular automata model of morphogenesis, able to both grow and regenerate specific patterns.

Visualizing the Impact of Feature Attribution Baselines

Exploring the baseline input hyperparameter, and how it impacts interpretations of neural network behavior.

Computing Receptive Fields of Convolutional Neural Networks

Detailed derivations and open-source code to analyze the receptive fields of convnets.

The Paths Perspective on Value Learning

A closer look at how Temporal Difference Learning merges paths of experience for greater statistical efficiency

A Discussion of 'Adversarial Examples Are Not Bugs, They Are Features'

Six comments from the community and responses from the original authors

A Discussion of 'Adversarial Examples Are Not Bugs, They Are Features': Adversarial Example Researchers Need to Expand What is Meant by 'Robustness'

The main hypothesis in Ilyas et al. (2019) happens to be a special case of a more general principle that is commonly accepted in the robustness to distributional shift literature

A Discussion of 'Adversarial Examples Are Not Bugs, They Are Features': Robust Feature Leakage

An example project using webpack and svelte-loader and ejs to inline SVGs

A Discussion of 'Adversarial Examples Are Not Bugs, They Are Features': Two Examples of Useful, Non-Robust Features

An example project using webpack and svelte-loader and ejs to inline SVGs

A Discussion of 'Adversarial Examples Are Not Bugs, They Are Features': Adversarially Robust Neural Style Transfer

An experiment showing adversarial robustness makes neural style transfer work on a non-VGG architecture

A Discussion of 'Adversarial Examples Are Not Bugs, They Are Features': Adversarial Examples are Just Bugs, Too

Refining the source of adversarial examples

A Discussion of 'Adversarial Examples Are Not Bugs, They Are Features': Learning from Incorrectly Labeled Data

Section 3.2 of Ilyas et al. (2019) shows that training a model on only adversarial errors leads to non-trivial generalization on the original test set. We show that these experiments are a specific case of learning from errors.

A Discussion of 'Adversarial Examples Are Not Bugs, They Are Features': Discussion and Author Responses

Open Questions about Generative Adversarial Networks

What we'd like to find out about GANs that we don't know yet.

A Visual Exploration of Gaussian Processes

How to turn a collection of small building blocks into a versatile tool for solving regression problems.

Visualizing memorization in RNNs

Inspecting gradient magnitudes in context can be a powerful tool to see when recurrent units use short-term or long-term contextual understanding.

Activation Atlas

By using feature inversion to visualize millions of activations from an image classification network, we create an explorable activation atlas of features the network has learned and what concepts it typically represents.

AI Safety Needs Social Scientists

If we want to train AI to do what humans want, we need to study humans.

Distill Update 2018

An Update from the Editorial Team

Differentiable Image Parameterizations

A powerful, under-explored tool for neural network visualizations and art.

Feature-wise transformations

A simple and surprisingly effective family of conditioning mechanisms.

The Building Blocks of Interpretability

Interpretability techniques are normally studied in isolation. We explore the powerful interfaces that arise when you combine them -- and the rich structure of this combinatorial space.

Using Artificial Intelligence to Augment Human Intelligence

By creating user interfaces which let us work with the representations inside machine learning models, we can give people new tools for reasoning.

Sequence Modeling with CTC

A visual guide to Connectionist Temporal Classification, an algorithm used to train deep neural networks in speech recognition, handwriting recognition and other sequence problems.

Feature Visualization

How neural networks build up their understanding of images

Why Momentum Really Works

We often think of optimization with momentum as a ball rolling down a hill. This isn't wrong, but there is much more to the story.

Research Debt

Science is a human activity. When we fail to distill and explain research, we accumulate a kind of debt...

Experiments in Handwriting with a Neural Network

Several interactive visualizations of a generative model of handwriting. Some are fun, some are serious.

Deconvolution and Checkerboard Artifacts

When we look very closely at images generated by neural networks, we often see a strange checkerboard pattern of artifacts.