GitHub

@@ -54,7 +54,6 @@ tags: [hide-output]

5454

We will use the following imports.

55555656

```{code-cell} ipython3

57-

import random

5857

from functools import partial

59586059

import numpy as np

@@ -483,18 +482,67 @@ Notice that the second run is significantly faster after JIT compilation complet

483482484483

Numba's compilation is typically quite fast, and the resulting code performance is excellent for sequential operations like this one.

485484485+486486

### JAX Version

487487488-

Now let's create a JAX version using `lax.scan`:

488+

Now let's create a JAX version using `at[t].set` style syntax, which, as

489+

{ref}`discussed in the JAX lecture <jax_at_workaround>`, provides a workaround for immutable arrays.

489490490-

(We'll hold `n` static because it affects array size and hence JAX wants to

491-

specialize on its value in the compiled code.)

491+

We'll apply a `lax.fori_loop`, which is a version of a for loop that can be compiled by XLA.

492492493493

```{code-cell} ipython3

494494

cpu = jax.devices("cpu")[0]

495495496-

@partial(jax.jit, static_argnames=('n',), device=cpu)

497-

def qm_jax(x0, n, α=4.0):

496+

@partial(jax.jit, static_argnames=("n",), device=cpu)

497+

def qm_jax_fori(x0, n, α=4.0):

498+499+

x = jnp.empty(n + 1).at[0].set(x0)

500+501+

def update(t, x):

502+

return x.at[t + 1].set(α * x[t] * (1 - x[t]))

503+504+

x = lax.fori_loop(0, n, update, x)

505+

return x

506+507+

```

508+509+

* We hold `n` static because it affects array size and hence JAX wants to specialize on its value in the compiled code.

510+

* We pin to the CPU via `device=cpu` because this sequential workload consists of many small operations, leaving little opportunity for GPU parallelism.

511+512+

Although `at[t].set` appears to create a new array at each step, inside a JIT-compiled function the compiler detects that the old array is no longer needed and performs the update in place.

513+514+

Let's time it with the same parameters:

515+516+

```{code-cell} ipython3

517+

with qe.Timer():

518+

# First run

519+

x_jax = qm_jax_fori(0.1, n)

520+

# Hold interpreter

521+

x_jax.block_until_ready()

522+

```

523+524+

Let's run it again to eliminate compilation overhead:

525+526+

```{code-cell} ipython3

527+

with qe.Timer():

528+

# Second run

529+

x_jax = qm_jax_fori(0.1, n)

530+

# Hold interpreter

531+

x_jax.block_until_ready()

532+

```

533+534+

JAX is also quite efficient for this sequential operation.

535+536+537+

There's another way we can implement the loop that uses `lax.scan`.

538+539+

This alternative is arguably more in line with JAX's functional approach ---

540+

although the syntax is difficult to remember.

541+542+543+

```{code-cell} ipython3

544+

@partial(jax.jit, static_argnames=("n",), device=cpu)

545+

def qm_jax_scan(x0, n, α=4.0):

498546

def update(x, t):

499547

x_new = α * x * (1 - x)

500548

return x_new, x_new

@@ -505,20 +553,12 @@ def qm_jax(x0, n, α=4.0):

505553506554

This code is not easy to read but, in essence, `lax.scan` repeatedly calls `update` and accumulates the returns `x_new` into an array.

507555508-

```{note}

509-

We specify `device=cpu` in the `jax.jit` decorator because this computation

510-

consists of many small sequential operations, leaving little opportunity for the

511-

GPU to exploit parallelism. As a result, kernel-launch overhead tends to

512-

dominate on the GPU, making the CPU a better

513-

fit.

514-

```

515-516556

Let's time it with the same parameters:

517557518558

```{code-cell} ipython3

519559

with qe.Timer():

520560

# First run

521-

x_jax = qm_jax(0.1, n)

561+

x_jax = qm_jax_scan(0.1, n)

522562

# Hold interpreter

523563

x_jax.block_until_ready()

524564

```

@@ -528,13 +568,11 @@ Let's run it again to eliminate compilation overhead:

528568

```{code-cell} ipython3

529569

with qe.Timer():

530570

# Second run

531-

x_jax = qm_jax(0.1, n)

571+

x_jax = qm_jax_scan(0.1, n)

532572

# Hold interpreter

533573

x_jax.block_until_ready()

534574

```

535575536-

JAX is also quite efficient for this sequential operation.

537-538576

Both JAX and Numba deliver strong performance after compilation.

539577540578

@@ -547,9 +585,11 @@ array and fill it element by element using a standard Python loop.

547585548586

This is exactly how most programmers think about the algorithm.

549587550-

The JAX version, on the other hand, requires using `lax.scan`, which is significantly less intuitive.

588+

The JAX versions, on the other hand, require either `lax.fori_loop` or

589+

`lax.scan`, both of which are less intuitive than a standard Python loop.

551590552-

Additionally, JAX's immutable arrays mean we cannot simply update array elements in place, making it hard to directly replicate the algorithm used by Numba.

591+

While JAX's `at[t].set` syntax does allow element-wise updates, the overall code

592+

remains harder to read than the Numba equivalent.

553593554594

For this type of sequential operation, Numba is the clear winner in terms of

555595

code clarity and ease of implementation.

@@ -575,12 +615,12 @@ For **sequential operations**, Numba has clear advantages.

575615

The code is natural and readable --- just a Python loop with a decorator ---

576616

and performance is excellent.

577617578-

JAX can handle sequential problems via `lax.scan`, but the syntax is less

579-

intuitive.

618+

JAX can handle sequential problems via `lax.fori_loop` or `lax.scan`, but

619+

the syntax is less intuitive.

580620581621

```{note}

582-

One important advantage of `lax.scan` is that it supports automatic

583-

differentiation through the loop, which Numba cannot do.

622+

One important advantage of `lax.fori_loop` and `lax.scan` is that they

623+

support automatic differentiation through the loop, which Numba cannot do.

584624

If you need to differentiate through a sequential computation (e.g., computing

585625

sensitivities of a trajectory to model parameters), JAX is the better choice

586626

despite the less natural syntax.

Read the original on github.com ↗