Neural Texture Compression in Three.js

A physically-based material is rarely one texture. Base color, normal, roughness, metalness, transmission, emissive: a fully-specified PBR material can be six, eight, ten separate texture maps, each with its own resolution, its own mip chain, its own memory footprint. And those maps are correlated. The edges in a normal map line up with the edges in a roughness map; a metal flake shows up as a coordinated bump in metalness and a highlight in albedo. Ordinary texture compression treats each map as its own unrelated image and throws that correlation away.

Neural Texture Compression (NTC), the idea behind NVIDIA's 2023 paper of the same name, fits every channel of a material jointly, with one small shared representation, and lets a neural network exploit the correlation ordinary compression ignores. I built an implementation for Three.js: a compact .ntc asset format, a loader, and a decoder that runs directly inside a node material's shader graph. The five materials in the live example each decode from that representation. This post covers what that representation is, how a .ntc file stores it, how the decoder is implemented, and how the per-texel cost scales.

Some of the terminology below (MLP, ReLU, latent vector) is covered in more depth in a companion primer on neural network basics.

What's actually stored#

An NTC asset is two things: a small multiresolution latent feature grid, and a tiny MLP decoder shared by every channel.

The feature grid is a stack of a few low-resolution textures — three levels in my implementation, each coarser than the last, the same coarse-to-fine idea behind instant-ngp's multiresolution hash encoding. Query it at a UV coordinate, bilinearly filtered and summed across levels, and you get a compact latent vector: a learned feature for that surface point.

The MLP takes that latent vector and decodes it into every channel the material needs, in one forward pass. Each channel gets its own named slice of the output and its own activation function, chosen to match its physical range: sigmoid for bounded [0,1] reflectance-like values (albedo, roughness, transmission), tanh for signed [-1,1] vectors (a tangent-space normal offset), softplus for unbounded non-negative HDR values (emission). A material with ten active channels — say a glass material with albedo, roughness, metalness, transmission, IOR, thickness, attenuation color and distance, and dispersion — trains all ten through that one shared grid and decoder. Those five sample materials run 25.0–31.5KB each for this entire representation: grid plus decoder, every active channel included.

A shared latent representation encodes that correlation once. Independent per-texture compression stores it in every map.

That joint fit works best when every channel shares the same UV set. One query coordinate produces one latent, and that latent has to describe the same surface point in every map. Separate UV layouts, or a stack that is rotated and scaled relative to the mesh UVs, force the grid to represent several mappings at once. A shared UV transform, learned at training time and applied to the query before the grid lookup, would cover the common case where the whole stack is transformed together. I have not added that yet.

The quality win is largest on the maps that look worst under block compression. Normals and vector displacement are sensitive to quantization: a block artifact in a normal map shows up as a faceted highlight, and a block artifact in displacement shows up as a stairstep in the silhouette. NTC trains against reconstruction error, so those structured artifacts are what the optimizer is minimizing.

Each asset names its own channels#

Each channel is a small descriptor: a key (roughness, normal, dispersion, ...), a width (1 for a scalar, 2 for a tangent-space offset, 3 for a color), an activation function, and a function that knows how to apply a decoded slice onto a target material's matching property. Loading an asset walks its list of active channels and applies each one in turn.

NTC decodes to whatever named, activated channels this asset was trained against. Adding a new channel type is a new descriptor. The decoder stays the same. I demonstrate the format on a THREE.MeshPhysicalNodeMaterial, the material with the widest channel vocabulary in Three.js. The same asset and evaluator can assign decoded channels onto any other NodeMaterial that exposes matching node properties.

The .ntc file#

A .ntc file is JSON, tagged format: "three-ntc", currently version 1. It holds the quantized latent pyramid, the packed MLP, and the channel list that slices the decoder's output into named PBR properties. The gold asset in the demo looks like this, with the binary blobs cut:

{
  "format": "three-ntc",
  "version": 1,
  "name": "Gold",
  "latents": {
    "channelsPerLevel": 4,
    "wrap": "repeat",
    "levels": [
      { "width": 16, "height": 16, "channels": 4, "dtype": "uint8", "min": -0.69, "max": 0.65, "dataBase64": "…" },
      { "width": 32, "height": 32, "channels": 4, "dtype": "uint8", "min": -0.73, "max": 0.69, "dataBase64": "…" },
      { "width": 64, "height": 64, "channels": 4, "dtype": "uint8", "min": -0.69, "max": 0.75, "dataBase64": "…" }
    ]
  },
  "outputChannels": 9,
  "mlp": {
    "dtype": "float16",
    "layout": [
      { "rows": 12, "cols": 8, "kind": "weight", "activation": "relu" },
      { "rows": 1, "cols": 8, "kind": "bias" },
      { "rows": 8, "cols": 8, "kind": "weight", "activation": "relu" },
      { "rows": 1, "cols": 8, "kind": "bias" },
      { "rows": 8, "cols": 9, "kind": "weight", "activation": "linear" },
      { "rows": 1, "cols": 9, "kind": "bias" }
    ],
    "dataBase64": "…"
  },
  "renderFlags": { "side": 0, "transparent": false },
  "channels": {
    "activeKeys": [ "albedo", "roughness", "metalness", "specularIntensity", "specularColor" ],
    "constantValues": { "opacity": 1, "ior": 1.5 }
  }
}

Each latent level is a uint8 image with its own min / max for dequantization. The MLP is one float16 blob; mlp.layout says how to slice it into per-layer weight and bias matrices (rows is the input size, cols the output size). channels.activeKeys are the slices the network produces. Everything else in the PBR vocabulary lives in channels.constantValues and never enters the decoder: a uniform IOR, a zero clearcoat, a default opacity. The loader applies those as TSL literals. If a constant equals the material's own default, it is skipped entirely, so an unused clearcoat does not enable MeshPhysicalNodeMaterial's extra shading branch.

Neural displacement#

A vector displacement channel fits in the same MLP as albedo and normals. The vertex shader queries it and offsets the original vertices, and the subdivision vertices, so the mesh silhouette picks up high-frequency shape that a normal map can only fake in the interior of a triangle. Wang et al. (ICLR 2022) use the same split on implicit surfaces, offsetting a coarse SDF along its normals.

A traditional displacement map is a vec3 per texel, a large extra texture. Displacement shares edges and features with the normal and albedo, so in NTC it is another named slice of a decoder you were already running. The extra cost is a few more output neurons and a vertex-stage evaluate; the grid and most of the network are already paid for. That is also why NTC's savings are largest on normals and displacement: those are the maps where block-compression artifacts are most visible, and the maps whose spatial structure is already in the shared latent.

Decoding fast: fp16 storage buffers and mat4-packed evaluation#

None of this is useful if decoding a material costs more than sampling a texture would. Two things make the decoder fast.

Half precision, where the hardware has it. I added f16 support to TSL and to storage buffers at the same time as building this — half, hvec2/3/4, hmat2/3/4 types that behave like their fp32 counterparts everywhere in TSL, compiling to real WGSL f16 on WebGPU with the shader-f16 feature, and falling back to fp32 everywhere else. The decoder's weight matrices live in a real fp16 storage buffer whenever that feature is available:

const weights = instancedArray( matrixCount, 'hmat4' ); // real fp16 storage buffer

Half precision is a well-documented ~2x on every major mobile GPU architecture. Apple's own Metal optimization guidance states that half runs at double rate versus float for vectorized code on Apple GPUs. Arm's GPU Best Practices Guide gives the same ~2x figure for Mali's mediump versus highp across ALU throughput, varying interpolation, and texture sampling. Qualcomm's Adreno Mobile Best Practices docs report the same ~2x for Adreno, down to the hardware detail: Adreno 640 pairs 64-wide FP32 ALUs that need two cycles per wave with 128-wide FP16 ALUs that finish the same wave in one. Three completely different GPU architectures, the same number, for the same underlying reason — halving a value's width roughly doubles how many of it a fixed-width ALU can push through per cycle, and halves the memory traffic for every weight the decoder reads.

Packed mat4 × vec4 evaluation. Each linear layer's weights are packed into mat4 blocks — four output neurons by four input features per block — and evaluated with a single mat4 * vec4 multiply. Four independent dot(vec4, vec4) calls, one per output neuron, do the same arithmetic as four separate instructions; the packed multiply issues effectively one. That's an architectural ceiling of up to 4x fewer instructions for the multiply. The realized speedup depends on register pressure, occupancy, and memory bandwidth, so I'd treat 4x as a ceiling this technique can approach. Directionally it is the same trick as the fp16 win: fewer, wider operations.

Between the two, decoding one texel costs on the order of 500 FLOPs, cheap enough on an average consumer mobile GPU.

Cost per texel#

A decode is bilinear grid samples plus one MLP forward pass. Grid resolution does not change the arithmetic: a level and a level are both one hardware texture fetch. Resolution shows up in memory and in cache behavior.

Write for channels per level, for the concatenated latent, for the number of hidden layers, for hidden width, for the output width (the sum of active channel sizes), and for mip samples per level. Today . A filtered mip pyramid that interpolates two adjacent levels uses .

Texture fetches:

MLP multiply-adds, counting a fused multiply-add as two FLOPs:

The three terms are the input layer, the hidden-to-hidden layers, and the output layer. Bias and ReLU add a handful of ops per neuron and do not change the scaling.

The gold material in the demo is , so , with , , :

Plus biases and ReLUs, about 500 FLOPs per texel.

Hidden width is the expensive knob. appears linearly in the input and output layers and as in every hidden-to-hidden layer. Doubling from 8 to 16 more than doubles the work when . An extra hidden layer of the same width only adds . The decoder is two layers of 8 because of that, and because in my experiments the grid's resolution and level count did more for reconstruction quality than MLP width. A wider network spends its extra capacity re-deriving spatial detail the grid should already be carrying. Grid lookups are hardware-filtered texture samples; every extra unit of is more matrix-vector work per texel.

Output width is the other term you can cut. Any channel that is spatially constant should not occupy a decoder slot. Detect those, drop them from , and store the value in channels.constantValues. The gold asset's MLP emits 9 numbers; the rest of MeshPhysicalNodeMaterial's vocabulary is constants.

Grid storage, uint8, no mips, with the spatial size of level :

A full mip chain on each level multiplies that by , the usual series. The MLP is float16, two bytes per weight and bias:

The gold grids are at 4 channels: 21,504 bytes. The MLP is a few hundred float16s. That is why a whole material lands in the 25–32 KB range. A mip pyramid is the next storage cost worth paying: one third more grid bytes for alias-free minification, with decode cost unchanged if you pick a single mip, or fetches if you filter between two.

What's not here yet#

Two things worth flagging:

No mip pyramid yet. The current implementation samples the latent grid at a single resolution regardless of view distance. A mip chain, the same idea ordinary textures use, reduces aliasing when the object is far away or seen at a glancing angle. Multiresolution NTC in the literature already establishes how to do this; I have not added it yet.

Where the assets come from. Every asset here was trained. Fitting a grid and decoder to a set of source channels is itself a nontrivial GPU training problem, with its own optimizer, quantization, and convergence questions. That's a substantial enough topic for its own writeup. I'll cover the training side in a follow-up.

PR: (link pending)