Build Pipeline

Build orchestration validates a checked byte budget before the builder turns registered PostgreSQL tables into an Engine. The builder avoids retaining all raw edges in Rust memory by using PostgreSQL temporary spool tables and bounded SPI cursor batches.

PostgreSQL Source Boundary

Before reading graph registrations, the build takes SHARE ... NOWAIT locks on the graph catalog in a fixed order. It then resolves registered relation OIDs to partition roots, locks each root, discovers descendants while attachment is blocked, and locks descendants in OID order. Caller-owned write locks are rejected in production builds so an uncommitted source change cannot escape through a filesystem artifact.

After sync-trigger reconciliation, the exclusive writer barrier defines watermark W0. Catalog identity, source schema, caller privileges, and the maximum sync-log ID are checked again before persistence and installation; a build is publishable only when the catalog fingerprint is unchanged and W1 == W0.

Inputs

The build receives:

The SQL facade and catalog modules validate registrations before they are stored. The builder still keeps defensive checks for malformed or drifting inputs.

Pipeline

Governed Build Runs

The persisted-build run layer uses a manual little-endian record format with a 128-byte checksummed header. Every run is bound to one graph ID, build ID, catalog fingerprint, and data kind: nodes, outbound relationships, inbound relationships, filters, filter dictionaries, or resolution entries. Readers verify the header, exact file length, complete payload checksum, record bounds, and sorted order before returning the first record.

Collectors charge rows, bytes, disk, files, elapsed time, and interrupt checks. Runs are merged with a fixed fanout and one decoded record per input. Merge outputs are validated and fsynced before inputs are released. Workspaces use mode 0700, run files use 0600, partial files are removed on failure, and abandoned workspace cleanup accepts only a well-formed build directory name while the graph build lock is held.

Build Scan Mode

graph.build_scan_mode supports:

Do not document or implement caller-facing COPY behavior until the code owns a safe, tested server-side COPY reader path.

Node Loading

For each registered table:

1. Resolve table OID.
2. Build a primary-key SQL expression.
3. Include registered filter columns and tenant column in the SELECT list.
4. Open an SPI cursor.
5. Fetch up to graph.build_batch_size rows per batch.
6. Append each row to NodeStore.
7. Buffer (table_oid, primary_key, node_idx) rows into pg_temp.graph_build_nodes.
8. Add resolution entry.
9. Collect filter values for later FilterIndex initialization.
10. Add tenant membership if the table is tenanted.

Composite primary keys use:

jsonb_build_array(col1::text, col2::text, ...)::text

Filter Loading

Filter columns are registered after all nodes are loaded because storage choice depends on node_count and populated count. Values collected during node loading are then encoded and written to FilterIndex.

Encoding examples:

Edge Loading

For each registered edge:

1. Register static label or prepare dynamic label_column.
2. Determine endpoint expression style.
3. Read endpoint keys, optional weight, optional dynamic label.
4. Resolve endpoints through pg_temp.graph_build_nodes.
5. Add one directed edge, and a reverse directed edge when bidirectional.
6. Flush to pg_temp.graph_build_edges in bounded batches.

Then load_edge_store_from_spool() streams:

SELECT source, target, type_id, weight
FROM pg_temp.graph_build_edges
ORDER BY source, target, type_id

into SortedEdgeStoreBuilder.

Relationship multiplicity

CSR construction retains every registered source relationship row. Two rows with the same endpoints and label remain separate relationships; applications that want a set of projected results can request DISTINCT in their query. The projection does not use endpoint/type topology as a substitute for a source-row identity.

Build Completion

At the end of build_graph():

Persistence is orchestrated outside builder.rs by sql_build.rs.

The build governor remains alive through optional persistence and reload. Persistent engine growth and transient node, edge, endpoint-resolution, and serialization work use separate RAII leases. The spool batches retain the configured row ceiling but flush earlier when their measured strings and array elements reach the adaptive byte target. Major Rust vectors call try_reserve after the policy lease is accepted so allocator failure is reported as PG001 instead of panicking.

Failure Modes

Skipping unresolved edge rows is intentional: source relationships that point to unregistered or missing nodes cannot become graph edges.