Chip-8 Dev Challenge: Sound Revisited

Introduction

Last year I wrote about generating XO-Chip sound and the challenges it creates. Specifically, taking a 128 bit pattern of 1’s and 0’s and turning that into a sound wave. However, that post only went into detail of the audio pattern. Recently I was updating Chipped-8 and optimizing sound was one of my areas of focus.

In addition to optimization I wanted to fix a few bugs and annoyances that were present.

Audio Playback Reliability

Startup Pop and Silence Workaround

Background: Startup Pop

When starting the audio there is an audible pop. It’s especially evident with 3dvipermaze.ch8. Oddly, it even occurs when the speakers are muted. It always happens when the ROM’s audio starts.

My guess is, the DAC in my laptop is powering up due to the audio sink I’m using having a signal. Even though the volume is muted, there is still sound being sent through.

Writing Silence at Startup

To prevent the pop I’m always writing audio when the ROM starts. However, I write silence at start and when there is no audio to play. This successfully prevents the popping. That said, it does increase CPU usage on ROMs without sound since audio is always being processed.

Frame Timing and Drift Issues

Frame-Based Playback Timing

Previously, I was writing audio to the sink as long as the internal audio callback was triggered indicating audio should be played. For some ROMs, 3dvipermaze.ch8 and xomusicplayer2.ch8, audio plays every frame.

In an ideal world, an audio pattern is written to the audio sink exactly when the previous frame’s audio has finished playing. This assumes perfect frame timing and that the sink can play the pattern immediately

While audio is always exactly one frame of data the emulator doesn’t have perfect frame timing. Chipped-8 is pretty good at getting really close and the overall time averages out to be more or less 60 fps. Being close enough and averaging out is typically fine.

Popping Between Frames

The audio sink has an internal buffer so if we’re early it will buffer the next frames of data and start playing when it should. A slightly slow frame could still have data in the audio buffer, so there won’t be any gaps in playback.

“Could” in that last sentence is key. While it could have data that was previously buffered due to a frame running fast, it’s possible a few frames in a row run slowly. This creates a gap in playback where there is a brief moment of no audio playing before the audio starts again. This can introduce a slight pop.

Audio Drift

There basically isn’t an audio buffer with how audio is continuously streamed. This can result in two things. First, if the frame are always a little faster than 60 fps, for example 60.5 fps the audio will start to back up and drift. Eventually the audio buffer will become full and start discarding. While discarding is proper, we could be significantly behind in the audio because Qt’s audio sink buffer is quite large. Depending on the system it’s possible to have a 1 second delay between audio playing and what’s happening on screen. If there is music and sound effects (3dvipermaze.ch8) the audio can become disconnected from the video.

Buffered Audio System

I’m handling buffering of audio data myself and using a buffer that holds 4 frames of bytes. The buffer discards the oldest audio when new audio is added.

The audio callback that triggers when audio should play adds the audio to the buffer. A separate thread flushes the buffer to the audio sink and if there isn’t enough data in the buffer, it writes silence to keep the audio sink from becoming empty.

The buffer means we’re usually about 2-3 frame of audio behind but that’s such a small amount of time it’s imperceptible. This solves the popping by always having something for the sink to play and limits drift by more aggressively dropping backed up data.

Further, I’m tracking how many samples should have played since the last time data was pushed into the audio sink. If we start to get behind, more data from the buffer can be written. This is one of the reasons we’re a few frames behind.

If there isn’t any or enough data in the buffer, then, silence will be written. This prevents the DAC from powering down and popping.

Audio Quality Enhancements

Harsh Audio and Filtering Attempts

The sound was very harsh because of how I was constructing the square wave represented by the 128 bit pattern. I wasn’t centering it and was using 0 and 0x15 as the values for the wave. I tried smoothing it with an IIR filter that created a very basic sine wave from the square wave.

However, the low and uncentered range caused the sound to be noisy and a bit distorted. The IIR filter helped but not a whole lot. Mainly because the sound was very loud even with such a low value. Likely because it wasn’t centered. The low range gave the IIR filter very little to smooth.

Centering and Output Adjustments

Centering

I changed it to center on 128 and use 121 and 134 to represent the 0 and 1 bits in the pattern. This led to a far cleaner tone. Though there is still some buzz I’d like to clear up in the future. I also removed the filter and left it as a square wave which actually sounds really good without any filtering. The volume is also lower with this version, making it more in line with the rest of the system audio.

Phase Tracking

Phase tracking was also added to ensure smooth transitions between frames. The audio pattern often remain the same for several frames without changing. However, 128 bits isn’t enough for a full frame and the pitch can stretch or compress the waveform across samples.

If there isn’t enough to create an entire frame’s worth of samples, we loop back to the start of the pattern and repeat. Both repeating and not finishing the entire pattern puts us somewhere in the middle of the pattern. By returning the phase, we can pass it back in and start the sequence where we left off. This produces a continuous stream without a harsh reset between frames. It both reduces popping and smooths out the audio.

The phase works great if it’s the same pattern across multiple frames. However, patterns change. Instead of resetting the phase to 0 if a different pattern is specified, the difference between the previous and current pattern is computed. Only when the difference is greater than the threshold will the phase be reset to 0.

The same is done with pitch but it’s a bit less lenient. I’m only allowing a small vibrato. Pitch can only change by 1 to retain the current phase.

Sample Generation Performance

Old Inefficient Code

The code that generates the audio samples for the frame worked but was horribly inefficient and slow. The biggest optimization was switching to NumPy but doing so basically dealt with all the other issues.

NumPy-Based Optimization

For example, the original version converted the pattern to a string of 0 and 1s. Then string indexing was used to check for a '0' or '1'. This is the worst way I could have done it, but it was easy and quick. I really should have taken the time to properly look at the bits within the byte pattern in the first place. The new version does this right and is probably the single biggest optimization.

In both versions it’s known how many samples need to be generated. In the original I was creating an empty list and then appending. This causes memory allocations as the list grows. The new version preallocates the sample buffer to the output size and fills it in.

Finally, loops were used with the original, but the new version leans on NumPy’s vector math magic for yet another big improvement.

Conclusion

Overall, the audio system in Chipped-8 is in a much better place now. It’s cleaner, smoother, and sounds more accurate to what the XO-Chip is trying to express. Fixing the pops, handling buffering myself, tracking phase, and optimizing the sample generation made a big difference. There’s still a little bit of buzz I’d like to clean up in the future, but for now, I’m happy with how it’s performing and sounding.