Using AI with erings

Introduction

As I’ve been working on my Sega Saturn emulator erings, I started thinking about how exactly I’m using AI to help with development. Especially with wanting to keep it as close to a clean room implementation as I can. I’m making sure everything is either documentation based or from tracing I’ve done and never letting it autonomously read source code from other emulators.

I’ve been using AI in a lot of ways but there are a handful of common use cases that I’ve found most helpful. Basically, I’m having the AI handle tedious tasks that would take me a long time to get through and limiting how much code it writes on its own. And when it does write code it’s writing to my design and I don’t blindly accept what it wrote.

I’m not going to go into the ethics of AI using existing knowledge. I’m only going to go over the most impactful uses I’ve found while developing erings. The best way to look at this is by going through a few examples.

Fixing a Complex Bug

During the opening gameplay sequence of “Albert Odyssey - Legend of Eldean”, there was garbage across most of the bottom of the screen. It was about 1/5 of the screen height. So obviously not something that was legitimately there that overscan would have hidden. Additionally, during the sequence there are characters walking around and their shadow would switch from being under to on top of them. Definitely not intended behavior.

While on the screen I paused emulation and took a screenshot. Then I captured a state dump of the emulator. This way I’d have everything the game had in memory, was executing, and plenty of information to dig into it.

Then I gave the AI a description of the bug, the screenshot, and pointed it to the state data. With all that ready we got started.

Where and What?

The first thing I had the AI do is determine which layer the corruption was appearing on and it was able to determine it was VDP1. It did this by rendering every layer from the state dump into scratch images to isolate which layer was the problem.

Now that I know it’s VDP1, figuring out what the data was and why it was getting put there came next. Thankfully, VDP1 has a command list which handily was in the state dump. The AI was able to walk the command list and put together a small app that uses the emulator renderer and rendered the data to multiple pictures. Each picture was the data but with different CRAM pallets applied and one greyscale.

I wanted to see if the garbage was actual garbage, an off screen element that was being drawn in the wrong place, an artifact from a previous screen, or something valid that was leaking into the VDP1 command list.

Setting up this temporary renderer and going through the layers is easy enough to do but the AI just made it so much faster. It was able to do this within a minute. It’s very nice for putting together temporary tooling that targets a specific needed.

So, what was it? It was the tile map for an enemy. The enemy shows later as the screen pans so it’s something from the scene. However, it’s a tile map so it’s not an off screen sprite that was getting drawn at the wrong time and in the wrong place.

Why was it there?

Next up was a close look at the command list itself. Specifically, what it was doing. While looking at it, I had the AI provide a report of every command and what it does so I could look though. As well as tell me where it was going wrong. Meaning where in the list did the garbage start.

The command list was fine until a certain point. That’s where an unknown command was seen and after that command is when the garbage started. This lead to figuring why this was there at all when it’s invalid to begin with. My mind went to something overflowing, something not being calculated right, something being written to the wrong place.

The command was being treated as a no-op and processing would continue. As a temporary measure I had the AI change it to an end draw event. That fixed it. However, this doesn’t necessary mean it’s the right fix. I needed to determine if that was a stop gap or work around of a real modeling issue.

Figuring this out was a multi step process. First using the state dump the AI was able to disassemble what was executing and find the command list builder. A deep dive into the disassembly if the routine lead to the conclusion that the game was properly executing and building the command list. So, the game put it there.

Timing?

It’s put there but the VDP1 trails the SH-2 by a small margin because the VDP1 has to read what the SH-2 puts into the command list. To keep the VDP1 (and VDP2) behind the SH-2, erings is 1 scan line behind with rendering. It does some latching and things to make this work. However, the fact is that erings has a larger delay between writing and reading the VDP1 command list than is likely hardware accurate.

My theory was, maybe the VDP1 was too far behind and the SH-2 would have overwritten the bad command with something valid. The AI was very useful for making timing tweaks and just having it rework things to force certain timings. The code it made was not very good quality but it was no worse than what I’d hack in for testing a theory.

Untimely, this went nowhere. Back to it’s really there and supposed to be there.

Is the Game Doing Something Wrong?

This is where it became clear, the game was doing the right thing and erings timing wasn’t at fault. Which means how erings was handling the command had to be the problem. Back to an invalid command shouldn’t be a no-op continue. Once again the AI saved time because I pointed it at the VDP1 documentation and told it to find the command list, read all commands, and what the documentation says about any invalid commands. Maybe I had missed something previously when the command list handling was created.

The AI was able to locate and read this faster than I could. The manual just said it’s invalid and shouldn’t be used. However, it doesn’t say anything about what should happen if invalid commands are seen. So, no-op is a valid way to handle an invalid command but so is treating it as an end draw. Basically, an error condition.

Now I had a real solution.

Was it Really Right?

Since the documentation doesn’t actually say what to do, both previous and new behavior can be interpreted as correct. I wanted a second and third opinion. This is where I had the AI look at the code for other Saturn emulators. I used a different model, in a new session, and I told it not to save anything to memory. I pointed it to the other emulators code, and gave it clear instructions. I told it to look at the VDP1 command processing and tell me what the other emulators do when they encounter an undocumented or invalid command.

The AI came back with an answer, functionally end of draw.

This made me think about how I want to and have been writing erings as a clean room without looking at the code from other emulators except in very limited circumstances like this verification of a guess. This means I’ve never looked at the code for other Saturn emulators myself. Only the AI has and, assuming it actually listens to me, it’s not retaining what it read in it’s memory. This means I’m insulated from the code of the other emulators and not writing something I think I read on the manual but actually read from one of the other emulator’s code.

Granted the LLM likely had the code for at least one of the other emulators in it’s training data… but let’s not go into that because it’s not the point of this post.

So, it’s fixed in erings.

Adding Save State Support

It took quite a while for me to add save state support. I knew there was a lot of hardware that needed to be modeled and I wanted the bulk of it in place before starting on save states. The more changes the more the save state format needs to change. But at a certain point I just had to do it because they’re just so nice.

For save states I knew how I wanted to build them. I bounced it off of the AI to be sure I wasn’t missing anything and it did suggest some tweaks that I incorporated. It also suggested quite a few things that I ignored.

File Format

For example, I told it I wanted to use a tagged format. It’s recommendation was a flat system with no hierarchy and it wanted to use a unsigned 32 bit integer for the tags. This makes sense from the stand point of efficiency but I didn’t like this idea.

I wanted to use a two level hierarchy mainly because there are two SH-2 chips. In a flat system each chip’s component need a different tag. Whereas with a hierarchy I have a top level tag for each one and then the internal tags are all the same. This makes the writer that much simpler.

I also didn’t like using a number for the tag. It’s efficient but I wanted to use a named tag. While it’s unlikely anyone would look at this file it’s easier to understand with text tags. Also, in the code it makes it a bit clearer when a constant is a clear string vs a number. The AI wasn’t wrong here but the optimization just wasn’t needed.

Compression

Another thing I wanted was compression because of the GUIs rewind feature. It uses the save state system and takes a save state every few frames. The smaller the save state the less memory and the more time the rewind can support. The solution is compression.

Since the save state is used for rewind how long it takes to create and load the state is also important. I asked the AI what compression has a good ratio of compression speed, decompression speed, and size. It said deflate and it’s no dependency because it’s part of the standard lib. That was good enough to at least get started.

Letting the AI Write the Code

This project I let the AI write the majority of the code at my direction. With the design in place I had the AI inventory the emulator and determine what was accessible and what wasn’t for state operations.

The big one that needed changes was the SH-2. This was designed as a module (even though it’s internal to erings) so the emulator can’t reach into it. It needed public getters and setters. However, the AI made a very interesting suggestion. It suggested a state struct so there would only be two public functions. This was a good suggestion and made for a very clean implementation.

Most of the code from this point was pretty similar and copy paste. Take a value from a component pass the value to a serialer function for the data type, pass it plus the tag to a formatter function, and add it to the save state block of other data. I let the AI handle all of this. First pass, it worked and the code was pretty decent.

I was delighted because at this point I had working save states.

Changing the Compression

While they were working the feature wasn’t done. The whole point of compression was to allow rewind to work smoothly. And while rewind did work with this first pass it caused game play to stutter. Basically, the compression and decompression time was too slow and took more than a frame to run.

Parallelization

The AI had it’s second good suggestion for speeding this up. It wanted to split the save state blob into sections and compress / decompress sections in parallel using multiple go routines. Super easy and quick to implement with minimal changes to the design. Using multiple go routines requires the machine to have multiple cores to get the speed benefit. However, the emulator really needs to be running on a multi-core system anyway so utilization all cores makes sense here.

This make the both save and load of save states faster but it wasn’t enough. It took less than a full frame but it was still less than the headroom left over after a frame runs.

New Algorithm?

This is where the AI stopped being as helpful. I asked if a faster compression algorithm could be used. It said yes with klauspost/compress but warned it’s a third party module I’d have to include. The warning is good. I do like that it said that and didn’t start pulling in random modules. However, it’s already being used by erings and already in the go.mod. So it’s warning about a module that’s already being used for CHD support.

The new algorithm it suggested was deflate. But with the module it’s a faster implementation. Later I’ll get to why this was a bad suggestion and not what I asked it.

Testing rewind with deflate from the module was faster and I didn’t get any stutter. Problem solved but not really. I could have left it here but I was still worried about the performance of deflate.

Actually a New Algorithm

Even though the module’s deflate is faster, I asked the AI for a faster algorithm not a different version of deflate. So it didn’t actually answer my question. klauspost/compress includes multiple algorithms. I asked the AI for a review of algorithms and a comparison of them previously. I asked the AI again from what the module provides and for information about each algorithm. Once again it didn’t actually answer my question and instead said to switch to zstandard.

I tried a different track and told the AI I need real numbers for compression and decompression. There are already multiple dumps of real games at different places that can be used. I needed a test harness that compresses the data, then decompresses the data and captures how long it takes. With and without parallelization.

The AI built this and I finally could see the difference across multiple algorithms. The numbers showed S2 as the best performance across all of them. So, the compression was changed over to that.

Still no issues seen with the save states and rewind. At this point save and load are pretty much as fast as I can make them without dropping compression entirely.

MPEG Card

The AI helped immensely with adding support for the MPEG card. I used a three pronged approach for reverse engineering this one. Going into this I knew the MPEG card interface is through the CD Block system and that I needed to support the MPEG commands here. I also knew that it’s MPEG 1 video as well as how audio is handled. Additionally, I knew which games support or require MPEG. Oh, and I already had an MPEG decoder in mind for this.

Parsing Disasm

I took the CD Block firmware and ran it through the disasm program that is part of the emulator development stack. This gave me the firmware in an ASM text format. I then gave it to the AI to identify all of the MPEG commands. This gave me a base documentation of which commands I need. However, there were huge gaps. I have the commands and some values but no meaning.

Next I gave the AI the MPEG library from the official SDK. Same process disasm and have the AI analyse. This filled in a number of gaps and the documentation became more complete.

Documentation and Implementation Loop

Once I added stubs for every command, I used Lunar to start getting traces. Lunar MPEG version requires MPEG video so I’ll easily know if something isn’t working. Other games could just fallback to the non-MPEG path.

Lunar allowed me to get real game tracing of what commands and values it’s sending to the MPEG card. More documentation gets updated. Cross reference with the SDK and CD Block disasm as well as the other documentation. That plus a bit of trial and error and it gets more complete with each iteration.

Every time the documentation got updated with more info, the code would get updated. Which allowed this loop of, implement, gather new data, update docs, and implement more. Through this loop I was able to get Lunar working.

However, Lunar only used a subset of MPEG commands and features. So I repeated this with game after game filling in gaps as I went. AI would make sense of the data, document it and then I’d have something concrete to work from. Using real games and game traces was essential because even small things like being 1 pixel off from the correct placement was visible and correctable.

Eventually, through this process, I was able to get MPEG support added to the emulator and working and validate many games that support MPEG.

What About Coding?

I touched on this a little bit and I do use it to write code but not always. When it does write code I do review just like I’d do with something written by a Junior developer. The exception is tests, those don’t need to be good, they just have to get the correct result.

AI, today, still has the issue of focusing on the task and not the boarder state and shape of the project. Often it will get something working but not account for existing convention, or tooling. It also doesn’t take into account introducing regressions. Get X working is what it’s focused on and it don’t care about what it could break.

That’s not to say I don’t introduce regressions but I keep that in mind and try to minimize the potential of a regression. The AI doesn’t. Hence, needing to always review the AI’s work. I wouldn’t say it’s often but I do need to reject the AI’s work outright from time to time. More so, I have to make corrections pretty frequently. Probably 85% of the time. That said, for coding tasks I give the AI, even with corrections and rewriting it’s still faster overall working with the AI than without, but it really comes down to the coding task.

Conclusion

In all three tasks the AI was sometimes frustrating but overall it was helpful and a time save. It took care of the tedious and copy paste tasks. It also gave me some good feedback and threw together some helper tools for quick testing. Also, it was able to quickly process disassembled code and parse documentation.

That said, these tasks did take a lot of oversight. The AI did things wrong or missed things, a lot. Even so, these took far less time to accomplish with the AI than if I had done it all myself.

MPEG went very smoothly. More so than the other two. Likely because the AI use was targeted to having it understand and document. It had a focused task and it’s a task AI is well suited for. It wasn’t making decisions nor was it doing anything subjective. It was purely, read this, write a summary.

Overall using an AI coding tool has been very helpful as an assistant. I want to stress assistant. It allows me to focus on the fun things while taking care of the not fun things for me. Like analysing a disassembly, and writing documentation.

Something I use it for that I didn’t really talk about here is writing comments. I do read and edit what it writes because it’s rough draft quality. But it helps a lot with explain non-obvious code and documenting where things like magic values came from. Another tedious and time consuming task.

Oh, and I didn’t use any AI on this post. Like I said, the AI is great for the boring things. Writing posts like this are far from boring.