GitHub

Progress

North America

The US and JP versions are now 100% decompiled! This includes all game functions, data and rodata within the game's main code section, and all library functions, as well as DSL for the cutscenes and dialogue that compiles to the game's custom bytecode systems. All text segments are also automatically extracted and transpiled back to the custom text format during the build process.

The current build system also supports shiftability. For modding, it's highly recommended to use the dev or dev-qol branch in this repo as your starting point. The bare minumum to get shiftability working on the master branch requires:

  • Removal of common_bss.ld. To preserve matching, config/us/common_bss.ld is used to match up game variables with their original addresses. For modding, this file should be empty or removed to allow the linker to reallocate memory addresses without conflict.
  • Remove bssPadding.o from the CODE_OBJECTS list in the Makefile. This file contains an empty array as a hack to match the original BSS size when using hardcoded memory addresses for BSS symbols

Japan

To build a JP ROM, place your JP ROM in the root directory as baserom.jp.z64 and simply pass in VERSION=jp to the normal make commands

Setting up

  1. Install WSL2
  2. WSL: sudo apt-get update && sudo apt install -y python3 pip binutils-mips-linux-gnu gcc-mips-linux-gnu build-essential wget

WSL Workflow

  1. git clone --recursive https://github.com/harvestwhisperer/hm64-decomp.git
  2. cd hm64-decomp
  3. sudo chmod +x tools/setup.sh
  4. sudo tools/setup.sh. This will fetch needed system packages not already installed, Splat, GCC 2.7.2, and the necessary GCC binutils for compiling and linking.
  5. Copy baserom.us.z64 (or baserom.jp.z64 if building JP) to the project's root directory (must provide your own and must be in big endian/z64 format).
  6. Run make setup && make or simply make fresh (with VERBOSE=1 and -j[n] optional). For building JP, run make fresh VERSION=jp, etc.

Docker Workflow (Non-Debian Linux)

Docker is also supported, and can be useful on Linux and non-Debian-based distributions where Python packaging and MIPS toolchain packages differ from the WSL setup assumptions.

Prerequisite: install Docker.

  1. git clone --recursive https://github.com/harvestwhisperer/hm64-decomp.git
  2. cd hm64-decomp
  3. Copy baserom.us.z64 (or baserom.jp.z64 if building JP) to the project's root directory (must provide your own and must be in big endian/z64 format).
  4. Run bash tools/docker.sh make fresh (with VERBOSE=1 optional). For building JP, run bash tools/docker.sh make fresh VERSION=jp, etc.

For an interactive container shell, run bash tools/docker.sh. The image sets PYTHON, KMC_PATH, and MODERN_GCC=0, so normal build commands work inside the shell.

The Docker helper defaults to linux/amd64 because the KMC toolchain release is a Linux binary toolchain. Override HM64_DOCKER_IMAGE or HM64_DOCKER_PLATFORM if you need a custom image tag or platform.

Docker Workflow (Windows)

Using Docker via Windows is currently not supported in favor of WSL.

Asset extraction

Sprites

Sprites can be extracted and converted to png files using the extract.py script in the tools directory (make extract-sprites or cd tools && python3 extract_sprites.py). Extracted assets will be put in ./assets/sprites folder. The locations for all used sprite assets are listed in the sprite_addresses.csv file in the tools directory.

There are two sprite asset formats used. Both types have a spritesheet at the beginning and an asset lookup table that follows it. Type 1 (used with character and map object sprites) includes a separate spritesheet index at the end of the asset, while type 2 (for smaller sprites, such as overlay icons) has the spritesheet index at the beginning of the actual spritesheet. Thus, the format for sprite_addresses.csv is:

  • type 1: spritesheet start, asset lookup table start, spritesheet index start, asset end, and asset name
  • type 2: spritesheet start, asset lookup table start, asset end, and asset name.

Animations

The game contains two layers of animation data: animation scripts compiled in the data section and another set of metadata stored on the sprite asset in the binary section of the rom. The binary data contains metadata for the animation sequence (total frame count, duration in game ticks, contains audio trigger) and metadata for the final rendered bitmap. The scripts are bitpacked arrays that contain an index into the animation metadata table, an animation type, and a flag for whether to flip horizontally.

Animation Scripts

These animation scripts are automatically generated as .inc.c files by Splat (via an extension), but they can be manually generated as well via make extract-animation-scripts. Running this script will generate a c file in src/data/animations for each specified asset in the animation_addresses.csv file.

Gifs

There's now support for converting animations to gifs. These seem to be working with all sprites, but there are issues with sprite anchoring for frames with layered sprites.

To create gifs, simply run make extract-animations, which will fetch the animation metadata, necessary sprites, and then run the gif conversion script, with all the generated assets in the /assets/animations directory.

Text

Text data (including decoding of control characters that encode character avatar information, new lines, etc.) can now be dumped via make extract-texts. Currently, only English characters are supported (there appears to be at least one text bank in Japanese included in the US version of the game).

The text system has two key components: special opcode characters (0-10), which system/message.c functions parse and use to modify message boxes and printing, and a 1-byte bitfield number that appears every 8 characters. The bitfield encodes which characters in an 8-character stream are 1 or 2 bytes. I.e., 0x03 (b0000 0011) has bits 1 and 2 set, which means the first and second characters are 2-bytes long, while the rest are 1 byte.

The character values themselves are used by the game code as offsets into the font texture array, adjusted for the opcode characters by subtracting 11.

Dialogue Bytecode

Dialogue bytecode can be both compiled (from DSL) and disassembled into text and json via make extract-dialogues (only a handful of dialogue bytecode segments are listed out currently). The dialogue bytecode is a somewhat sophisticated system that allows for branching to different texts within a text bank based on various conditions (game variable values, like NPC affection; random values; special dialogue status bits) and also setting game variable values and special dialogue bits. The bytecode also supports spawning sub-dialogue boxes (such as with choice menus).

Decompiled dialogue bytecode can be found in src/bytecode/dialogues.

Maps

Map tiles and "core map objects" can now be extracted via make extract-map-sprites, and maps can now be rendered in Blender via the ./tools/modding/map/blender_import.py script (follow the README in that directory for more information).

Map assets consist of 10 sub-assets:

  • 1.) The map grid information, which includes basic map metadata and a mapping between tile number and grid position.
  • 2.) Mesh data, which is essentially tile vertex information. This consists of two parts:
    • a.) Individual vertex information, which contains a pointer to its coordinate data, how many vertices to use per tile, y offest information, etc.
    • b.) Drawing information, which includes flags, coloring data, and bitfields that encapsulate vertex order for triangle drawing commands
  • 3.) Terrain quads used for certain entity height calculations
  • 4.) Grid to level interaction indicies (i.e., which squares in the grid have a specific flag the game can use for game interaction logic, such as the home TV, signs, the vineyard wine barrel, etc.)
  • 5.) Core map objects data (coordinates and flags)
  • 6.) Tile textures
  • 7.) Tile palettes
  • 8.) Core map object textures
  • 9.) Core map object palettes
  • 10.) "Map addtions" metadata (new geometry added to the map once they're unlocked in the game, such as the house extensions)

The "core map objects" are sprites that are always rendered on top of the geometry, independent of game status, which includes things like trees and fences.

The map code manages two separate display lists for the tiles and ground objects (foragable items) and appends other sprites to the scene graph indirectly via calling sprite.c functions (i.e., weather sprites, map objects set up by the game, map additions, and the core map objects).

Cutscenes

The repo now supports parsing cutscene bytecode and building the rom with matching DSL (see src/bytecode/cutscenes for matched segments).

The bytecode system works by having variable length messages where the first two bits correspond to an index in a cutscene handler function table defined in cutscene.c.

As an example, opcode 88 corresponds to cutsceneHandlerSetSong, which is the 88th function in the cutscene function table (starting count from 0). Each opcode is variable length, where the rest of the data after the first two bits are used by the handlng function (these data are usually parameters for other function calls). In the case of cutsceneHandlerSetSong, the parameters are song index (16 bits), a pointer to the rom address for the song start (32 bits), and a pointer to the rom address for the song end (32 bits). The handler function then calls setSong(songIndex, songStart, songEnd); from audio.c and increments the bytecode pointer for the next message.

Compared to the dialogue bytecode, the cutscene bytecode is much more sophisticated and involves multiple bytecode executors. I.e., the initial executor can spawn child executors that execute different segments of the bytecode and can spawn their own child executors as well. To handle this, the default make extract-cutscenes command generates 3 parses of each bytecode segment: a linear scan, a graph-based scan that tracks spawns, branches, and subroutine jumps, and a disassembled assembly file.

Audio

The standard build automatically extracts the original instrument sample data and N64 Wavetable into the assets directory and repackages for the build. To extract a general midi soundfont easily, run make extract-soundfont. For more customization, run the Python module directly: python -m libhm64.audio.wavetable.sf2_writer --layout <gm|game> --root-transpose <semitones> (the game order preserves the original instrument order in the soundfont).

Modding

See the dev and dev-qol branches for more information.

Music

See the README at tools/modding/music for more details for converting and inserting MIDI into existing song slots.

Contributing

Contributions are much welcome. There are a few areas of work still do in the project:

  • Cleaning up fake/forced matches (searchable under FIXME)
  • Research into function, struct member, flag, and variable purposes and making accurate labels
  • Improving labeling and macro usage in bytecode files for readability

Read the original on github.com ↗