Guest Posts: Fake Clocks, Real Guarantees: Inside Go's synctest

The first time I reached for testing/synctest, I assumed it was a sleep helper with better manners. The public API is small enough to support that assumption:

synctest.Test(t, func(t *testing.T) { ... })
synctest.Wait()
synctest.Sleep(d) // new in Go 1.27

Three functions, one obvious guess: fake time, wrapped in test scaffolding. That guess is wrong. It isn’t a sleep helper, a goroutine wrapper, or a mocking library. It’s a runtime feature wearing a testing package as a disguise.

Here’s the short version, so you have somewhere to hang the details that follow.

When a test starts, the runtime gathers it into what the source calls a “bubble”: the test’s goroutines, its channels, its timers, all tracked as one isolated group, with a clock of their own that only the runtime moves. (Isolated doesn’t mean sandboxed: the test can still reach global state and the outside world, and later sections cover what happens when it does.) Inside that group, the runtime can tell something apart that’s invisible everywhere else: whether a blocked goroutine is waiting on something inside the bubble, or something outside it. A goroutine blocked in a way that only another bubble member can undo is called durably blocked, and durable blocks are safe to trust: nothing outside can sneak in and change the picture. So when every goroutine in the bubble is either finished or durably blocked, the runtime knows, not guesses, that the test has gone as far as it can on its own. That state is called settled, and it’s the moment synctest.Wait can return, or the fake clock can jump to the next timer. The rest of this article is how the runtime pulls that off.

If you already understand goroutines, channels, timers, and testing.T, but haven’t explored this corner of the Go runtime, this article is for you.

This walks the runtime source as it stands for Go 1.27; the package has been stable since Go 1.25, and I’ll link each file on GitHub as it comes up. The package first shipped behind GOEXPERIMENT=synctest in Go 1.24, and Go 1.27 adds synctest.Sleep, which we’ll get to.

One more pointer before we descend. This article is about the runtime machinery underneath, not the release notes; if you want a user-level look at what Go 1.27 changed, including synctest.Sleep, VictoriaMetrics’ Go 1.27 interactive tour has runnable examples you can edit in the browser.

Enough setup. Let’s start with the problem synctest actually solves.

The core idea

A normal concurrent test leans on real time, and quietly hopes:

go doSomething()
time.Sleep(10 * time.Millisecond) // hope this is long enough
if !done {
 t.Fatal("not done")
}

That’s a guess dressed up as a test. The goroutine may have run, or the scheduler may not have reached it yet. Making the sleep longer only buys you a slower guess.

synctest replaces that guess with a question the runtime can actually answer:

Has every goroutine in this test either finished or reached a block point
that only another goroutine in this same test can unblock?

If yes, the bubble is settled. Everything from here on is how the runtime answers that question honestly instead of guessing.

That question gets answered deep in the scheduler, but the story starts at the top, in the public package.

Public API: a small overlay

The public package lives in src/testing/synctest/synctest.go , and it’s thinner than you’d expect. Let’s look inside synctest.Test and see how it works:

func Test(t *testing.T, f func(*testing.T)) {
 var ok bool
 synctest.Run(func() { // internal/synctest.Run creates the runtime bubble
  ok = testingSynctestTest(t, f) // a testing-package hook runs f inside it
 })
 if !ok {
  t.FailNow()
 }
}

That’s the whole entry point: internal/synctest.Run asks the runtime to create the bubble, and a hook in the testing package runs your test function inside it. The hook is reached through go:linkname, a linker trick the standard library uses to call another package’s unexported function without importing it. Why the trick? testing can’t import testing/synctest back without a cycle, and exposing its test-running internals publicly would be worse. So the two packages shake hands behind the linker’s back. That’s plumbing, though, not mechanics. The interesting part is what the hook does.

Adapting testing.T

The hook lands in src/testing/testing.go , and its job is adaptation: your test function needs a *testing.T that behaves inside a bubble. So it wraps the original t in a child *testing.T that looks like an ordinary subtest, with one difference: the new T is tagged isSynctest: true.

That one flag carries weight. Inside a bubble, t.Run, t.Parallel, and t.Deadline all panic, and cleanup reporting is adjusted so the bubble is not treated like a normal nested subtest.

Forbidding Deadline looks arbitrary until you notice what it would return: the outer test’s real wall-clock deadline. Inside the bubble, time.Now says it’s the year 2000, so any arithmetic between the two is nonsense, exactly the kind that surprises someone at 2am. The source is explicit that it could return “no timeout” instead, but panicking makes the mistake impossible to miss.

The test function still runs through tRunner, the same internal function every ordinary test runs through, so the rest of testing keeps working: failures, fatal exits, cleanup, logs, panic handling, and race checks all pass through the ordinary machinery.

That’s the last of the adapter layer. Everything said so far, the bubble, the settling condition, has been building toward one object that hasn’t actually appeared yet: the bubble itself. Time to open it up.

The Bubble

The real machinery starts in src/runtime/synctest.go on the struct synctestBubble:

type synctestBubble struct {
 mu      mutex
 timers  timers // the bubble's own timer heap
 id      uint64
 now     int64  // the fake clock
 root    *g
 waiter  *g
 main    *g
 waiting bool
 done    bool

 total   int
 running int // goroutines not yet proven durably blocked
 active  int // other activity the bubble must wait for (see below)
}

mu is just a mutex guarding the rest of the struct, and timers, id, waiting, done are bookkeeping the article will explain as each one comes up. Three fields are worth pausing on right now, because everything that follows leans on them.

now is the fake clock: nanoseconds since the Unix epoch, moved by the runtime instead of by the OS.

running counts goroutines the bubble can’t yet prove are stuck waiting on something internal. That phrasing matters more than it looks: running > 0 doesn’t mean “CPU work is happening” anywhere. It means “at least one goroutine is in a state the bubble can’t certify as durably blocked”, which is not the same thing.

active exists to patch a hole running alone would leave open. First, a word the rest of this article leans on: when a goroutine blocks, on a channel, a sleep, whatever, the scheduler parks it, taking it off the CPU and marking it as waiting. Parking a goroutine isn’t one atomic step: the scheduler first records it as blocked with its wait reason, which is the exact moment running can drop to zero, and only then runs the commit function, a small callback that gets the last word on whether the park actually sticks. A park can still be vetoed at that point, and the goroutine resumes as if nothing happened. Picture the bubble checking running == 0 inside that gap, without active to lean on: it would see zero and declare the test settled while a goroutine is half-parked and might yet come back. active is what stops that false reading: the runtime bumps it before the status change and brings it back down only once the park decision is resolved, so the bubble always has a nonzero counter to check during the gap.

Put those two together and the whole design hangs off one invariant:

    running > 0 || active > 0

Which answers the one question the bubble ever needs to ask: is anything still going on? Everything else in this article is bookkeeping in service of keeping that invariant honest.

Creating a bubble

With the shape of the struct settled, the next question is who fills it in. synctest.Run is implemented by runtime.synctestRun, in the same file.

It opens by refusing to nest:

gp := getg()
if gp.bubble != nil {
 panic("synctest.Run called from within a synctest bubble")
}

Then it creates the bubble:

bubble := &synctestBubble{
 id:      bubbleGen.Add(1),
 total:   1,
 running: 1,
 root:    gp,
}

const synctestBaseTime = 946684800000000000 // 2000-01-01 00:00:00 UTC
bubble.now = synctestBaseTime

946684800000000000 is midnight UTC on 2000-01-01, in nanoseconds since the Unix epoch. A round, memorable starting point for a clock nobody outside the bubble will ever read.

Then the root goroutine attaches:

gp.bubble = bubble
defer func() {
 gp.bubble = nil
}()

The function passed to Run does not run directly on the root goroutine. Instead, the runtime starts a new goroutine for it and records that goroutine as bubble.main. (For synctest.Test there’s one more layer: bubble.main runs the testing adapter, which spawns the tRunner goroutine that runs your actual test body. Both are bubble members, so the distinction won’t matter again.)

So the root goroutine becomes the controller, while the main goroutine runs the test code. That split raises an obvious question: when the test spawns more goroutines of its own, how do they know they’re inside a bubble at all?

Bubble membership

The answer is a field every goroutine carries, bubble or not:

type g struct {
 ...
 bubble *synctestBubble
 ...
}

That field gets set the moment a goroutine is born, inside newproc1 in src/runtime/proc.go , the runtime function that creates every new goroutine (this is the code behind every go f() statement, whether it’s user code or the runtime’s own plumbing):

if isSystemGoroutine(newg, false) {
 sched.ngsys.Add(1)
} else {
 newg.bubble = callergp.bubble // the child inherits its parent's bubble
}

callergp is the goroutine doing the spawning, newg is the one being created. If newg isn’t a runtime-internal goroutine, it simply copies whatever bubble its parent belongs to.

That single line carries the isolation guarantee, precisely because it’s inherited rather than assigned once. Goroutines started by the test join the bubble. Goroutines started by those goroutines also join, since they copy the bubble field from their own parent, which already has it set. Runtime system goroutines never inherit membership, because the isSystemGoroutine branch never touches bubble at all. (Never inherited isn’t quite never involved, though: as we’ll see later, the timer machinery briefly lends a system goroutine bubble membership to run a fake timer’s callback, then takes it right back.)

That is exactly what we want. GC, finalizers, the network poller, and friends have no business inside the test’s deterministic world.

Membership answers “who’s in the bubble.” It doesn’t yet answer the harder question the running counter depends on: for a goroutine that’s blocked, how does the runtime decide whether that block is the safe kind, one only the bubble itself can resolve?

Durable blocking is a wait reason

The public docs define that safe kind of blocking, “durable blocking”, in human terms:

blocked and can only be unblocked by another goroutine in the same bubble.

The runtime implements that as “wait reasons”.

When a goroutine parks, the runtime records a waitReason: a small enum naming what it’s waiting for (a channel receive, a sleep, a mutex). synctest treats some wait reasons as idle and others as not.

In src/runtime/runtime2.go :

var isIdleInSynctest = [len(waitReasonStrings)]bool{
 waitReasonChanReceiveNilChan:    true,
 waitReasonChanSendNilChan:       true,
 waitReasonSelectNoCases:         true,
 waitReasonSleep:                 true, // time.Sleep
 waitReasonSyncCondWait:          true, // sync.Cond.Wait
 waitReasonSynctestWaitGroupWait: true, // WaitGroup.Wait, when associated
 waitReasonCoroutine:             true,
 waitReasonSynctestRun:           true,
 waitReasonSynctestWait:          true, // synctest.Wait itself
 waitReasonSynctestChanReceive:   true, // receive on a bubbled channel
 waitReasonSynctestChanSend:      true, // send on a bubbled channel
 waitReasonSynctestSelect:        true, // select where every case is bubbled
}

So at the lowest level, durable blocking comes down to a two-part check: the goroutine is _Gwaiting (the runtime’s status code for “blocked, waiting on something”; every goroutine carries one of a handful of these _G-prefixed states), and goroutine.waitreason is marked idle in synctest.

The list is conservative on purpose. Take sync.Mutex.Lock: it isn’t durable, because a goroutine blocked on a mutex could be unblocked by a goroutine outside the bubble. Ordinary mutexes belong to no bubble, so the runtime can’t prove the block is internal, and it refuses to guess.

A few entries at the top are durable for a bleaker reason: a send or receive on a nil channel, or an empty select {}, can never be unblocked by anyone, inside the bubble or out. Counting them as idle is safe, and if nothing else can end the test, they surface as a deadlock panic instead of a silent hang.

That two-part check, _Gwaiting plus wait reason, is wired directly into the code path that updates bubble.running every time a goroutine’s status changes.

Scheduler accounting

Nearly every goroutine status transition, running to waiting, waiting to runnable, and so on, runs through casgstatus in proc.go , “compare-and-swap g status”, the one function that atomically flips a goroutine from one _G state to another. When the goroutine belongs to a bubble, that function also calls:

gp.bubble.changegstatus(gp, oldval, newval)

changegstatus is what actually updates running on each transition. The diagram below traces the decision:

Inside it, the classification that matters most is this one:

case _Gwaiting:
 if gp.waitreason.isIdleInSynctest() {
  isRunning = false // stops counting toward bubble.running
 }

That’s the exact line where a blocked goroutine stops counting as running, the moment the two-part check from the previous section gets applied for real.

Status transitions cover the clean case. But remember the mid-park window active was invented for? Here’s where the runtime actually closes it.

Why active exists

In park_m, before the goroutine transitions to _Gwaiting, the runtime does:

bubble := gp.bubble
if bubble != nil {
 bubble.incActive() // "someone is mid-park, don't trust running == 0 yet"
}

The token is held until the park decision is resolved, and released on both outcomes, park confirmed or park aborted:

if bubble != nil {
 bubble.decActive() // park decided, counters are honest again
}

Skip this step, and the bubble could catch a glimpse of running == 0 and active == 0 while a goroutine is halfway through parking, and wrongly conclude it had settled.

active, then, isn’t user activity in any sense, just scheduler bookkeeping that guards the exact window where the counters would otherwise lie. Parking isn’t its only client, either: the wake path and the root goroutine hold active tokens of their own, and both show up in the next two sections.

Two counters, one invariant, and a scattering of increments across the scheduler don’t check themselves, though. Something has to actually watch running > 0 || active > 0 and act on it. That something is the root goroutine, sitting in a loop of its own.

The bubble event loop

After starting the main goroutine, the root goroutine takes an active token for itself, bubble.active++, and enters that loop:

bubble.active++ // the root's own token: "I'm still working"
for {
 bubble.timers.check(bubble.now, bubble) // run fake timers due right now
 gopark(synctestidle_c, nil, waitReasonSynctestRun, ...) // try to park

 next := bubble.timers.wakeTime() // earliest pending fake timer
 if next == 0 {
  break // no timers left: nothing can ever happen again
 }
 if bubble.done {
  break // the test function has returned
 }
 bubble.now = next // jump the fake clock forward
}

(gopark is the runtime call that does the parking from earlier: everything from time.Sleep to a channel receive bottoms out in it, and its first argument is the commit function.)

Read that loop as three steps repeating: run any fake timers that are due right now, try to park, then check whether there’s a next timer to jump to or whether it’s time to stop. “Try” to park, because when the bubble has settled, the root doesn’t stay down. Parking hands back the root’s active token, and if that leaves both counters at zero, the wake logic below immediately re-readies the root, so it comes straight back to advance the clock. One more wrinkle: the root’s commit function, synctestidle_c, holds a veto. In the narrow case where parking would leave the root as the only activity left, it aborts the park outright: no point going to sleep when you’re the only one who could wake you. Only while the rest of the bubble is genuinely busy does the root actually sleep, waiting to be woken.

And who wakes it? That decision lives elsewhere, triggered every time the bubble’s state changes.

maybeWakeLocked

When a goroutine blocks, exits, or finishes an active transition, the runtime calls maybeWakeLocked, the function that decides whether any of that just made the bubble worth waking up for.

Its decision tree:

That last branch earns its keep. If no Wait is pending, the root still has to wake so it can either advance fake time or notice a deadlock.

Notice the middle branch: “a goroutine is waiting in synctest.Wait”. That’s the other half of this mechanism, the one your test code actually calls.

synctest.Wait

synctest.Wait parks the caller until every other goroutine in the bubble is durably blocked. It’s the piece that turns “the bubble settled” from an internal runtime fact into something your test can act on.

Runtime outline:

func synctestWait() {
 gp := getg()
 if gp.bubble == nil {
  panic("goroutine is not in a bubble")
 }
 if gp.bubble.waiting {
  panic("wait already in progress") // one waiter per bubble, no more
 }
 gp.bubble.waiting = true
 gopark(synctestwait_c, nil, waitReasonSynctestWait, ...)
 ...
}

One waiter per bubble, no more. The bubble.waiting flag is what catches a second concurrent wait.

When the waiter returns, the race detector picks up a happens-before edge, its unit of “these two goroutines properly synchronized here,” the same guarantee a mutex handoff or a channel send establishes. Without it, reads after Wait of data written by the now-blocked goroutines would look like races:

raceacquireg(gp, gp.bubble.raceaddr())

Matching release operations happen when goroutines durably block, when bubbled goroutines exit, and when fake timer callbacks finish. That’s what makes synctest.Wait mean something to the race detector, not just to your own reasoning about the code.

And since Go 1.27 there’s a convenience built on top of Wait: synctest.Sleep(d) is literally time.Sleep(d) followed by Wait(). It reads as “advance the fake clock by d, then let everything that woke up finish reacting”, which is what most tests actually mean when they sleep. One caveat carries over from real Go: goroutines whose timers expire at the same instant still run in unspecified order; the trailing Wait is what makes that safe to ignore.

Everything up to here has been about proving a bubble is settled. What actually happens once it is, that’s fake time, and it’s the part of synctest I find most interesting, having written my share of flaky clock-dependent tests over the years.

Fake time: time.Now

The time package reaches the runtime through linknamed functions in src/runtime/time.go , the same linker trick from earlier.

Inside a bubble:

func time_runtimeNow() (sec int64, nsec int32, mono int64) {
 if bubble := getg().bubble; bubble != nil {
  sec = bubble.now / 1e9
  nsec = int32(bubble.now % 1e9)
  return sec, nsec, 0 // monotonic reading deliberately zero
 }
 return time_now() // outside a bubble: the real clock, as usual
}

Quick refresher, because this bit assumes it: a time.Time normally carries two readings: the wall clock, which can jump around when NTP adjusts it, and a monotonic clock that only ever counts forward, which is what makes time.Since trustworthy.

Notice the monotonic value comes back zero.

That’s intentional.

Returning a fake monotonic clock would make arithmetic between inside-bubble and outside-bubble times confusing.

Returning the real monotonic clock would make arithmetic between two inside-bubble times confusing.

So synctest drops monotonic time inside bubbles and lives with the one downside instead of two: time.Since(start) inside a bubble falls back to wall-clock subtraction, which is fine, because the bubble’s wall clock is runtime-controlled and never jumps unpredictably anyway.

Knowing what time it is inside a bubble is only half the story. The other half is how that clock actually moves forward, starting with the simplest way code asks for time to pass.

Fake time: time.Sleep

time.Sleep is implemented by runtime.timeSleep. For a bubbled goroutine, the timer it creates is tagged as fake, and the goroutine parks on it with waitReasonSleep, one of the wait reasons marked idle:

if gp.bubble != nil {
 t.isFake = true // this timer lives on the bubble's clock
}

...

if t.isFake {
 resetForSleep(gp, nil)
 gopark(nil, nil, waitReasonSleep, ...) // durably blocked
}

So a sleeping goroutine costs no real time; it just parks on a fake timer and counts as durably blocked. The clock moves when the root goroutine decides nothing else can happen. Watch it play out with two sleepers:

synctest.Test(t, func(t *testing.T) {
 go func() {
  time.Sleep(3 * time.Second) // G2: timer due at fake 00:00:03
  fmt.Println("G2 woke")
 }()
 time.Sleep(10 * time.Second) // main: timer due at fake 00:00:10
})

The fake clock starts at midnight, 2000-01-01. Both goroutines park on their fake timers almost immediately, so running drops to zero: nothing in the bubble can make progress on its own. That wakes the root goroutine, which looks at its timer heap, finds the earliest deadline, and jumps the clock straight to 00:00:03. No waiting, just an assignment to bubble.now. G2 wakes, prints, and exits. Now the bubble is idle again, the only pending timer is main’s, and the root jumps the clock to 00:00:10. Main wakes, the test function returns, and the whole thing, ten seconds of fake time (the two sleeps overlap), finished with no wall-clock waiting at all.

That covers time.Sleep, but the same fake-clock trick has to work for every other way Go code waits on time. time.NewTimer, time.NewTicker, time.After, and time.AfterFunc all bottom out in the same runtime timer code, so they all need the same bubble-awareness time.Sleep just got.

Fake timers and bubble timer heaps

When a timer is created inside a bubble, it gets the same tag:

if bubble := getg().bubble; bubble != nil {
 t.isFake = true
}

And when the timer is added to a heap (a “timer heap” being nothing fancier than a priority queue of timers ordered by deadline), the tag decides which heap:

if t.isFake {
 ts = &bubble.timers // the bubble's private heap
} else {
 ts = &mp.p.ptr().timers // a normal per-P timer heap
}

So fake timers never touch a normal P timer heap. Each P (the runtime’s logical processor, the thing that owns a run queue and schedules goroutines onto OS threads) keeps its own heap of real timers driven by the OS clock; a bubble keeps one more heap of its own, driven entirely by bubble.now. The two worlds never mix.

When a fake timer fires, the runtime temporarily makes the executing goroutine belong to the bubble:

gp.bubble = bubble // borrow bubble membership for the callback
bubble.changegstatus(gp, _Gdead, _Grunning)

f(arg, seq, delay) // the timer callback runs as bubble activity

bubble.changegstatus(gp, _Grunning, _Gdead)
gp.bubble = nil

This matters for time.AfterFunc: the callback has to count as bubble activity, and any goroutines the callback starts have to inherit the bubble. Don’t read those _Gdead/_Grunning arguments as real scheduler transitions, though: they’re synthetic inputs to the bubble’s accounting, fed to changegstatus so the callback registers as activity while it runs on a borrowed system goroutine.

Timers show a pattern worth carrying forward: an object gets tagged at creation, and that tag decides how the runtime treats it later. For a timer, the tag is just the isFake flag; channels take the pattern further and record the owning bubble itself.

Channels: bubble association in hchan

The runtime channel type in src/runtime/chan.go carries a bubble pointer:

type hchan struct {
 ...
 bubble *synctestBubble
 ...
}

makechan sets it:

if b := getg().bubble; b != nil {
 c.bubble = b // set once at creation, never changes
}

So a channel created in a bubble is married to that bubble for life.

Operations from outside are fatal:

if c.bubble != nil && getg().bubble != c.bubble {
 fatal("send on synctest channel from outside bubble")
}

Receive, close, and select get the same treatment.

Fatal, not merely non-durable, and the reason is the isolation boundary. A bubbled channel is part of that boundary. Letting an outside goroutine poke it would break the runtime’s proof that a channel block is internally controlled, so the runtime refuses outright instead of degrading quietly.

So hchan.bubble decides who’s even allowed to touch the channel. The next question is what happens once a goroutine that is allowed to touch it blocks on a send or receive.

Channels: durable send and receive

When a send on a bubbled channel blocks, chansend swaps the wait reason:

reason := waitReasonChanSend // not durable: an outside goroutine could complete it
if c.bubble != nil {
 reason = waitReasonSynctestChanSend // durable: only the bubble can
}
gopark(..., reason, ...)

Receives do the mirror image with waitReasonChanReceive and waitReasonSynctestChanReceive.

Only the synctest-specific wait reasons are marked idle. So the same blocked receive ends up in one of two buckets, durable or not, and the only thing that decides which is whether the channel itself carries a bubble tag, one field set at makechan time. You’ve already seen this fork in the durable-blocking flowchart earlier: “receive on bubbled channel” lands on the durable side, “receive on unbubbled channel” doesn’t.

A single channel is one thing; select waits on several at once, so the durability check has to run over the whole set.

Select

A select is durable only when every case with a non-nil channel is a bubbled channel in the current bubble (nil-channel cases can never fire, so selectgo skips them before the check). In selectgo, in src/runtime/select.go :

allSynctest := true
for each case:
 if cas.c.bubble != nil {
  if getg().bubble != cas.c.bubble {
   fatal("select on synctest channel from outside bubble")
  }
 } else {
  allSynctest = false // one unbubbled case spoils durability
 }

waitReason := waitReasonSelect
if gp.bubble != nil && allSynctest {
 waitReason = waitReasonSynctestSelect // durable only if every case is bubbled
}

Conservative, and it has to be. One outside channel in the mix means an outside goroutine could make the select proceed, and the runtime loses its claim that the block is internally controlled. One leak is enough to spoil the proof.

Channels and select cover the two ways goroutines talk to each other directly. sync has its own coordination primitives, WaitGroup, Cond, Mutex, and each needs its own answer to the same durability question.

WaitGroup association

sync.WaitGroup carries explicit synctest support.

The WaitGroup state packs everything into one word:

high 32 bits: counter
bit 31:       synctest bubble membership flag (0x8000_0000)
low 31 bits:  waiter count

On Add inside a bubble, sync.WaitGroup calls:

synctest.Associate(wg)

The runtime records that association as a “heap special”: a side note attached to the heap memory the WaitGroup lives in, saying which bubble owns it. (Specials are an existing runtime mechanism; finalizers ride on the same one. synctest is reusing plumbing, not inventing it.) The note isn’t permanent, either: once the counter drains to zero and the runtime notices, either by the Add that releases blocked waiters or by a later Wait observing the drained state, the association is removed and the WaitGroup can be reused, even in a different bubble.

That word “heap” is doing real work, and it explains a limitation buried in the public docs. A package-level variable like

var wg sync.WaitGroup

isn’t allocated on the heap at all: the linker places it directly into the binary’s data segment, the region where package-level variables live, at a fixed address that exists before the program even starts running. There’s no heap memory to attach a side note to, so the runtime has nowhere to record “this WaitGroup belongs to bubble X.”

But:

var wg = new(sync.WaitGroup)

does allocate on the heap, so it has a home the runtime can attach that note to, and association works. Same type, same API, and the difference between them is invisible in the code you write, yet decisive in whether your bubble ever settles, exactly the kind of thing that costs you an afternoon if you don’t know it.

Association is the setup. It’s what Wait actually checks when it has to decide whether blocking here is safe to call durable.

WaitGroup.Wait as durable semaphore wait

When WaitGroup.Wait has to block, it works out whether the wait is durable:

synctestDurable := false
if state&waitGroupBubbleFlag != 0 && synctest.IsInBubble() { // Add'ed in a bubble, waiting in a bubble
 if synctest.IsAssociated(wg) { // and it's the same bubble, via the heap special
  synctestDurable = true
 }
}
runtime_SemacquireWaitGroup(&wg.sema, synctestDurable)

The runtime semaphore code, in src/runtime/sema.go , maps that boolean to a wait reason:

reason := waitReasonSyncWaitGroupWait
if synctestDurable {
 reason = waitReasonSynctestWaitGroupWait // the only durable one of the pair
}
semacquire1(addr, false, semaBlockProfile, 0, reason)

So a WaitGroup wait is durable only when the Add happened inside the current bubble, the Wait is called inside that same bubble, and the runtime managed to associate the WaitGroup object with the bubble. Miss any one of the three and the wait stops counting as idle, and the bubble can’t settle for as long as that wait stays blocked.

sync.Cond needs the same kind of answer, but it gets there without a bubble field of its own.

sync.Cond

sync.Cond.Wait runs through runtime notify lists, the runtime’s internal ticket queue of goroutines waiting to be signaled, one per Cond:

t := runtime_notifyListAdd(&c.notify)
c.L.Unlock()
runtime_notifyListWait(&c.notify, t) // parks with waitReasonSyncCondWait: durable
c.L.Lock()

waitReasonSyncCondWait is marked idle for synctest.

Unlike channels and WaitGroups, the Cond object doesn’t carry a bubble field, and it doesn’t need one. The bubble rides on the parked goroutine instead: every waiter in the notify list is a parked g, and every g carries its bubble field, so the information is already sitting there in the queue. When notifyListNotifyOne and notifyListNotifyAll (the runtime behind Signal and Broadcast) go to wake a waiter, they check the waiter’s bubble against the caller’s. A wake from outside the waiter’s bubble, or from another bubble, is fatal: fatal("semaphore wake of synctest goroutine from outside bubble"). So a stray Signal from the wrong bubble takes down the whole process, not just the failing test.

Channels, WaitGroups, and Cond each get a path to durability: a tag, an association, or a bubble check that lets the runtime prove the block is internal. sync.Mutex conspicuously doesn’t, and that absence is the point of this next section.

Why mutex is not durable

This one tripped me up, and I’ve watched it trip up people who know the runtime deeply.

Blocking on sync.Mutex.Lock doesn’t count as durable, and neither does sync.RWMutex. A mutex has no bubble ownership, so an outside goroutine might be the one to unlock it, and the runtime can’t prove the block is internal to the bubble.

So this never settles:

var mu sync.Mutex // outside or unassociated

synctest.Test(t, func(t *testing.T) {
 mu.Lock()
 go func() {
  mu.Lock() // blocks, but not durably: waitReasonSyncMutexLock isn't idle
 }()
 synctest.Wait() // never returns
})

The bubble has no way to tell whether the blocked goroutine is waiting on an internal event or an outside one, so it refuses to call the test settled. In that example, synctest.Wait simply never returns, and the test hangs.

That raises the obvious follow-up: hangs forever, or does the runtime eventually give up and say something useful? That’s what the root loop’s exit path is for.

Deadlock detection

Recall the root goroutine’s loop from earlier: it wakes up, checks for a due timer, and either advances fake time or stops. When it decides to stop, that’s the moment it checks for a deadlock:

if total != 1 { // someone besides the root is still in the bubble
 if bubble.done {
  reason = "deadlock: main bubble goroutine has exited but blocked goroutines remain"
 } else {
  reason = "deadlock: all goroutines in bubble are blocked"
 }
 panic(synctestDeadlockError{...})
}

Two distinct failures hide behind that. “All goroutines in bubble are blocked” means every goroutine, main included, is durably blocked with no fake timer left to jump to: everyone stuck on a select {}, say, or on a bubbled channel nobody will ever send on. Notice the mutex example above does not end here. A mutex wait isn’t durable, so running never reaches zero, the root never wakes to run this check, and the test simply hangs until go test’s own timeout kills it. The panic is reserved for the case the runtime can prove. “Main bubble goroutine has exited but blocked goroutines remain” means the test function returned but leaked bubbled goroutines that are stuck forever.

That split is worth committing to memory, because it’s your debugging fork. Durable blocks with nowhere left to go get you a loud, named panic. Non-durable blocks get you silence. So when a synctest test hangs instead of panicking, don’t stare at the deadlock detector; go looking for a mutex or an unbubbled channel.

This is stricter than an ordinary test. A leaked goroutine blocked forever on a bubbled channel doesn’t get silently ignored. It gets you a panic with a name on it.

Channels, timers, and WaitGroups each got their own section explaining how they associate with a bubble. Laid out together, the pattern across all of them is the same rule wearing different clothes.

Isolation rules

Objects created inside a bubble tend to become associated with it:

created inside bubble:

channel       hchan.bubble = bubble
timer         timer.isFake = true, uses bubble.timers
ticker        same timer machinery
WaitGroup     associated on Add/Go, if it's heap-allocated (see the `new(sync.WaitGroup)` note above)

Moving those objects outside the bubble is usually fatal:

send on bubbled channel from outside        fatal
receive on bubbled channel from outside     fatal
close bubbled channel from outside          fatal
reset fake timer from outside               fatal
stop fake timer from outside                fatal
WaitGroup Add from outside its bubble       fatal
WaitGroup Add from multiple bubbles         fatal

Objects created outside a bubble can be used inside, but waits on them aren’t durable. Only sync.WaitGroup can join a bubble after the fact, through Add; a channel or timer is bubbled at creation or never. (sync.Cond sidesteps the question entirely: as we saw, its durability rides on the parked goroutine, not on the object.) That’s the deliberate compromise: tests can still touch the outside world, while the runtime refuses to treat those operations as deterministic settling points.

That’s every piece: the struct, membership, wait reasons, the event loop, and now the isolation rules tying object lifetimes to bubble lifetimes. Here’s all of it in one place.

The mental model

A synctest bubble is a miniature scheduler domain: its own membership rule, its own idea of durable blocking, its own clock.

Settling happens when every goroutine in the bubble is dead or _Gwaiting for a synctest-idle wait reason (so running is zero), and the active counter, covering mid-park windows, wake tokens, and the root’s own loop, is also zero. Once that holds, the diagram’s bottom box takes over: the root dispatches any due timer, a pending Wait is woken, the loop stops once the main goroutine is done (finishing cleanly only if no other goroutines remain), fake time advances to the next timer, or, with none of those available, the runtime declares a deadlock.

The model is easier to trust once you’ve watched it run a real test, so let’s do exactly that.

One test, from start to finish

Here’s a small but realistic test: a worker waits for a message with a timeout, and the test delivers the message after three fake seconds.

synctest.Test(t, func(t *testing.T) {
 ch := make(chan string) // bubbled channel: hchan.bubble is set

 go func() {
  select {
  case msg := <-ch:
   t.Log("got", msg)
  case <-time.After(5 * time.Second): // fake timer, due at 00:00:05
   t.Error("timed out")
  }
 }()

 time.Sleep(3 * time.Second) // fake timer, due at 00:00:03
 ch <- "hello"
})

Now the same test, narrated from the runtime’s side:

1. synctest.Test calls into the runtime. A synctestBubble is created, its clock set to fake midnight 2000, and the calling goroutine becomes the root, the controller that never runs test code.
2. The runtime spawns the main goroutine for the test function, and it inherits the bubble through newproc1.
3. The test creates ch. makechan sees the goroutine’s bubble and stamps it into hchan.bubble: the channel now belongs to the bubble.
4. The go func() starts the worker, which copies bubble from its parent. Membership by inheritance, again.
5. The worker reaches the select. Both cases are receives on bubbled channels: ch by its tag, and the channel time.After returns, which was also created inside the bubble. So the worker parks as waitReasonSynctestSelect. Durable. running drops by one. (The timer behind time.After being fake matters separately: it’s what puts the 5-second deadline on the bubble’s clock.)
6. The main goroutine hits time.Sleep(3 * time.Second) and parks on a fake timer with waitReasonSleep. Also durable. running is now zero, active is zero: the bubble has settled.
7. maybeWakeLocked wakes the root. No timer is due yet at 00:00:00 and nobody is in synctest.Wait, so the root advances bubble.now straight to 00:00:03, the earliest deadline, and fires main’s sleep timer.
8. Main wakes and sends "hello". The send hands the value to the parked worker and makes it runnable; the select picks the receive case, and the worker logs and exits. The 5-second timeout never fires; as the select completes, its cleanup unregisters the blocked receiver and marks the now-useless timer for removal from bubble.timers.
9. Main returns. The bubble is marked done, the root’s loop breaks, and the exit check finds only the root left: no deadlock, test over.

Thirteen lines of test, three fake seconds, zero real waiting, and at no point did anyone guess whether “the goroutine has probably run by now.”

Why this design works

The runtime doesn’t try to make all concurrency deterministic. It makes one isolated bubble deterministic enough to answer a precise question:

Is this test's internal concurrent work settled?

It can answer because it owns the low-level facts a library never sees:

• which goroutine belongs to which bubble;
• why each goroutine is blocked;
• whether a channel was created inside the bubble;
• whether a timer uses fake time;
• whether a WaitGroup was associated with this bubble;
• when fake timers are due;
• when the main bubble goroutine has exited.

None of that is reachable from user space (you!).

That’s what I take away from reading the source: synctest is cooperation between testing, sync, time, channel operations, the timer heap, the semaphore implementation, and the scheduler.

The tiny public API is the only part that was ever meant to be small. Everything this article walked through, the bubble struct, the wait reasons, the durability rules for every sync primitive, is what that smallness is quietly standing on.

Practical reading tips

If that’s whetted your appetite to read the source yourself rather than take my word for any of it, this order follows the same path this article did, public API down to the scheduler. You don’t need to absorb every file in one sitting or recognize every name below on first read, each stage maps straight back to a section you’ve already read here, so you can flip up to that section if a name doesn’t ring a bell.

Bubble lifecycle, from public call to running scheduler domain:

1. testing/synctest.Test, the public entry point.
2. testingSynctestTest in testing.go, where *testing.T gets adapted for a bubble.
3. runtime.synctestRun, where the bubble itself gets created.
4. synctestBubble.changegstatus, the scheduler hook that keeps running accurate.
5. runtime2.go’s wait reasons, the table that decides what counts as durable blocking.

Fake time, the clock and its timers:

1. time_runtimeNow, timeSleep, newTimer, and timer.maybeAdd.

Synchronization primitives, how each one earns (or fails to earn) durability:

1. makechan, chansend, chanrecv, and selectgo, for channels and select.
2. sync.WaitGroup.Add and Wait, for the heap-association trick.
3. runtime.sync_runtime_SemacquireWaitGroup, where the durable flag becomes a wait reason, and notifyListWait, the Cond parking path.
4. internal/synctest/synctest_test.go, for the edge cases the tests themselves worry about.

That sequence is also the one I’d hand to anyone who tells me synctest is “just a sleep helper”.

Recap

If you keep nothing else, keep these:

• A bubble is an isolated scheduler domain the runtime builds for one test: its own goroutines, channels, timers, and clock.
• Membership is inherited: every goroutine spawned inside the bubble joins it automatically; runtime system goroutines never inherit it.
• Durable blocking is implemented as a list of wait reasons. A goroutine blocked with an idle-in-synctest reason is provably waiting on the bubble itself; anything else, a mutex, an unbubbled channel, keeps the bubble unsettled.
• Two counters, running and active, uphold one invariant. When both hit zero, the bubble has settled: nothing inside can make progress on its own.
• The root goroutine runs an event loop over that invariant: dispatch due fake timers, let Wait return, advance the fake clock to the next timer, stop once the main goroutine is done (a clean exit only if no goroutines were leaked), or panic with a deadlock when none of those apply.
• Fake time never waits once the bubble settles. bubble.now jumps from deadline to deadline in bubble-private timer heaps, so ten fake minutes cost no real waiting.
• Objects created inside a bubble belong to it: channels by a field, timers by a flag, WaitGroups by heap association on Add. Touching them from outside is usually fatal, and that strictness is what keeps the settling proof honest.