GitHub

v0.1.53

@levkk levkk released this

13 Aug 20:09

New parser

New parser is now enabled by default. Please test this release before launching in production. You should see 6x-8x speed improvements resulting in lower latency for query cache misses, and up to 90% memory usage reduction overall. This is not a typo! We fixed quite a few things here.

This also makes the query_parser_engine setting redundant and obsolete. It will now be ignored.

By @sgrif

Bug fixes

  • Fix negative interval text format parsing in cross-shard aggregates @bryanmehall
  • Connection pools were giving clients closed server connections, causing intermittent errors @nsavvide
  • In role = "auto" mode, set all hosts to role = "replica" until proven otherwise; this unblocks reads faster during configuration changes @murex971
  • [New parser] Fix GRANT and CREATE STATISTICS DDL handling @levkk
  • When using schema-based sharding, ignore the inconsistent omnisharded tables write check @levkk
  • Include sv_idle_xact calculation into the example Datadog dashboard @zacharywelch
  • Fix crash when a cross-shard avg() calculation was done on empty tables @sgrif
  • RESET PREPARED admin command wasn't doing anything @IgorOhrimenko
  • Fix Mac OS build for the new parser @jkaczman
  • Fix pipelined queries using Postgres portals; specifically affected the Node JS Postgres drivers @nsavvide
  • Sending portal close messages could potentially close prepared statements that were identically named @meskill
  • Schema synchronization during resharding now restores REPLICA IDENTITY indexes in the post-data phase @rlittlefield
  • Cross-shard aggregates with passthrough columns now work correctly @jkaczman
  • PREPARE was leaking memory in the prepared statements cache @meskill
  • Fix incorrect handling of errors in mixed simple/extended protocol exchange @jkaczman

Performance

  • Lower the number of memory allocations in prepared statements cache @meskill
  • Remove unnecessary statement allocation in the parser (50% allocation reduction in the hot path) @sgrif

Features

  • Track selected, inserted and deleted rows, as reported by server connections @murex971
  • Support prepared statements cache TTL (time to live), automatically re-planning statements periodically; this sidesteps the stale execution plan problem @meskill

Misc

  • Clippy pass over new parser code @sgrif
  • Bump Mac OS runner to macos-26 (macos-14 is about to be deprecated) @jkaczman
  • Docker compose demo wasn't saving Postgres data correctly @andyatkinson
  • Small refactor of pool health tracking @sgrif

New Contributors

Full Changelog: v0.1.52...v0.1.53

v0.1.52

@levkk levkk released this

06 Aug 19:31

Bug fixes

  • Use one timestamp for OTEL metric batches, fixing Prometheus + OTEL integration @KennanHunter
  • Direct-to-shard queries were incorrectly flagged as omnisharded when using schema sharding function @levkk
  • Pub/sub channels were not scoped by database, causing misrouted messages with multiple databases using LISTEN/NOTIFY in transaction mode @4thel00z
  • Normalize sharding tables identifiers to support Postgres-like format (quotes preserve capitalization) @nsavvide

Features

  • Handle different OIDs for custom types in sharded deployments @sgrif
  • New lookup sharding function for manual sharding using omnisharded tables @rlittlefield
  • Support turning on/off dry run mode through an admin database command @levkk
  • Allow per-role pool configuration in [[users]] and [[databases]] config sections @levkk

Performance

  • Binary COPY tuple was encoding column data one byte at a time, causing allocator churn @ygxio

Misc

New Contributors

Full Changelog: v0.1.51...v0.1.52

v0.1.51

@levkk levkk released this

30 Jul 21:56

Performance

  • Implement async processing during resharding, improving performance @ygxio
  • Remove an allocation in the load balancer hot path @sgrif
  • Don't reset/set parameters that haven't changed during parameter sync in transaction mode @levkk
  • Prevent thundering herd with multiple users in users.toml when loading schema at startup @levkk

Stability

  • Fix timers/loops stop firing after 12 days of uptime breaking connection pools @meskill
  • Fix crash in new parser handling partitioned table DDL @levkk
  • Enable jemalloc background_thread by default, giving memory back to OS @IgorOhrimenko

New features

  • Support mTLS between PgDog and Postgres servers @andy-paine
  • Re-implement two-phase commit WAL @levkk
  • Optionally allow password users to bypass mTLS @bhargavtheertham-cb
  • Add locked clients OTEL metric to track count of clients locked to a specific server @KennanHunter

Code quality

  • Remove dead types, simplify construction of other types in tests @sgrif
  • Fix flaky load balancer test @KrishnaSindhur
  • Fix flaky test test_reload_connection_count_stable @sgrif

New Contributors

Full Changelog: v0.1.50...v0.1.51

v0.1.50

@levkk levkk released this

23 Jul 17:59

What's Changed

  • fix: validate message length before widening it to an unsigned integer @frangz
  • fix: hot mutex in client stats @levkk
  • fix: switch to alternative Tokio timer for >2 workers (10-20% CPU reduction with 16+ workers) @levkk
  • fix: fully support the extended protocol for BEGIN/ROLLBACK commands intercepted by the proxy @Kerem-Kurt
  • fix: avoid heap allocations when encoding integer and float parameters in sharded queries @crodas
  • fix(config): idempotent random admin password generation @meskill
  • fix: make pgdog-config doc comments pass rustdoc lints @crodas
  • fix: +lse (atomics) optimizations for ARM64 (5-10% CPU reduction) @levkk
  • fix(config): improve validation messages for sharding config @meskill
  • feat: add process name to 2pc transaction gid @levkk
  • feat: add correct SIGTERM handler

New Contributors

  • @Kerem-Kurt made their first contribution in #1231
  • @crodas made their first contribution in #1240

Full Changelog: v0.1.49...v0.1.50

v0.1.49

@levkk levkk released this

16 Jul 21:52

Highlights

New query parser

We rewrote the query parser to use our in-house library: https://github.com/pgdogdev/pg_raw_parse/. It's based on pg_query but maps Rust structs directly to the arena-allocated MemoryContext allocated by the Postgres parser. This brings our (de)ser cost for parsing queries to 0.

Early metrics:

  1. 2x improvement in parsing speed vs query_parser_engine = "pg_query_raw"
  2. 8x improvement in parsing speed vs query_parser_engine = "pg_query_protobuf" (used by default, so most users will see this)
  3. 90% (6x) reduction in memory usage (eliminates the OOM problems we've been having with large queries e.g., #854 #1061)

It's entirely generated using Rust build scripts directly from Postgres C code and allows us to do major version upgrades in one, 5 line PR. More on this soon on our blog.

How to use

The new parser is experimental and disabled by default. We are building a separate Docker image for each release / commit in main. To use it, just append -new-parser prefix to the tag, for example:

image:
  tag: v0.1.49-new-parser

by @sgrif

New plugin interface

The plugins system was rewritten to use safer primitives and simpler API. This is a breaking change. We ported all in-tree plugins to the new interface, so no impact on any plugins we currently ship with PgDog in the Docker image. If you wrote your own plugins, reach out to us and we'll help you fix them. New documentation coming soon.

by @sgrif

Changelog

  • fix: skip multi-statement SET validation in session mode by @zefixlluja
  • fix: add default to engine in [[query_parsers]] by @levkk
  • fix: include_primary_if_replica_banned ignored primary if it was the only candidate by @levkk @kzemek
  • fix: role detector deadlocked when new databases with role = "auto" were added at runtime by @levkk
  • feat: allow read_write_split when parser is disabled to direct traffic by default @levkk
  • fix: query parser ran regex twice by @levkk
  • test: disable copy_data retry_tests temporarily by @meskill
  • fix(replication): fix wait_for_replication logic for unrelated changes by @meskill
  • fix: LSN detector can now correctly identify replica-only deployments by @levkk
  • fix: support psycopg2 (handle SET commands) in admin DB by @levkk
  • feat(breaking): use a less horrendously unsafe FFI interface for plugins by @sgrif
  • chore(config): save file content for ConfigAndUsers for diff handling in EE by @meskill
  • refactor: remove fingerprinting from our codebase by @sgrif
  • feat: update the parser to support Postgres 18 by @sgrif

New Contributors

Full Changelog: v0.1.48...v0.1.49

v0.1.48

@levkk levkk released this

09 Jul 14:49

What's Changed

  • feat: manual connection pinning using SET pgdog.pin TO true; -- false / RESET to unpin; useful to temporarily use session-level features in transaction mode
  • feat: Hashicorp Vault static role credentials @larrywax
  • feat: per-database query parser setting
  • fix(sharding): correctly handle pipelined cross-shard queries @sgrif
  • fix(replication): better cancellation handling during replication streaming @meskill
  • fix: don't re-create connections when password changes, allowing for faster reloads
  • chore: normalize configs for nextest and rust integration tests @meskill
  • refactor: ported several functions to new, much faster parser @sgrif

Full Changelog: v0.1.47...v0.1.48

v0.1.47

@levkk levkk released this

02 Jul 22:16

What's Changed

  • feat(lb): speed up failover & add standalone connection fallback for LSN check
  • feat(sharding): add support for two-phase commit to COPY used for resharding @yogesh1801
  • refactor(resharding): rework of async tasks used in resharding to handle cancellation better @meskill
  • fix(sharding): prevent premature cutover due to lag miscalculation @meskill
  • fix(sharding): update the processed LSN on keep-alive messages @meskill
  • fix(lb): replica lag evaluation in load balancer used for banning replicas
  • fix(lb): fix no routing error with prefer_primary when all replicas are banned @stewart-glabs
  • refactor(sharding): more parser code is moving to pg_parse_raw @sgrif

New Contributors

  • @yogesh1801 made their first contribution in #1088

Full Changelog: v0.1.46...v0.1.47

v0.1.46

@levkk levkk released this

25 Jun 22:23

Breaking change

PgDog now blocks UPDATE queries that mutate a sharding key in a table that has foreign key references to other tables which automatically mutate the referenced tables using:

  • CASCADE
  • SET NULL

Both of these were previously breaking referential integrity, since we were not moving the referenced rows. RESTRICT (which is typically default) is not affected because Postgres is able to validate ref integrity before we move the row.

Changelog

  • feat(sharding): rewrite [[sharded_mappings]] config to make it easier to use @meskill (not documented yet, old config format still works)
  • feat: add query_size_limit setting to block/warn on very large queries @meskill
  • feat: RELOAD keeps connections and paused state if adding or removing connection databases or users @murex971
  • feat(load balancer): add prefer_primary read/write separation mode to allow manual opt into load balancing @stewart-glabs
  • feat: log warning when lsn_check_delay is misconfigured
  • refactor: start porting the codebase to use the new, much faster SQL parser @sgrif
  • feat(sharding): block cross-shard sharding key updates on tables that have foreign key constraints @murex971
  • feat(auth): Hashicorp Vault server authentication integration @larrywax

New Contributors

Full Changelog: v0.1.45...v0.1.46

v0.1.45

@levkk levkk released this

18 Jun 19:12

What's Changed

Distribution

Pre-compiled binaries now ship with every release in GitHub:

  • Linux x86/ARM
  • Mac (Apple Silicon)
  • Debian package (amd64/arm64).

Breaking change

SET LOCAL pgdog.shard and SET LOCAL pgdog.sharding_key now assume the entire transaction will be sent to that one shard only. Any query that attempts to talk to a different shard, incl. omnisharded writes, will be blocked. This is not really a breaking change, but more of a bug fix, but still worth calling this out since this changes the behavior of the sharding logic a bit.

If using sharding, please double check this release before deploying to prod just to make sure the new shard checker doesn't accidentally (and incorrectly) block your query from execution.

Changelog

  • feat: check that tuples in multi-tuple INSERT statement belong to the same shard before rewriting @murex971
  • feat: handle the special case of SELECT set_config(...), which we didn't track previously; this fixes the "pg_dump broke my pooler" issue @sgrif
  • feat: enable parser for LISTEN/NOTIFY/UNLISTEN in pooler-only mode (`query_parser = "on" not required, makes PgDog much faster)
  • feat: add LISTEN/NOTIFY stats
  • fix(sharding): COPY with rows exceeding one CopyData message size were incorrectly routed to all shards @LnL7
  • fix: Elixir driver sends Close for one-off statements, which we incorrectly rejected with an error
  • fix: major performance improvements in pg_query, affecting long queries @sgrif
  • fix: Windows build by @imor
  • fix: race condition in LISTEN when registering for a channel for the first time
  • fix: UnexpectedEof error log when passthrough auth is enabled
  • fix: use interval instead of sleep in stats loop which may have caused our stats to stop working after 12 days of uptime (potential Tokio bug, not confirmed)

New Contributors

Full Changelog: v0.1.44...v0.1.45

v0.1.44

@levkk levkk released this

11 Jun 18:55

What's Changed

  • fix(security): use constant_eq for password checks @meskill
  • fix(security): admin password not checked when using passthrough auth
  • fix(resharding): fix issues with copy data @meskill
  • feat(resharding): use binary protocol for replication (slightly faster than text) @meskill
  • fix(sharding): don't track columns independently, allowing duplicate expression rewrites in cross-shard queries @sgrif
  • feat(sharding): error when encountering an unsupported aggregate function in cross-shard queries @sgrif
  • feat(sharding): give a more meaningful error message when a helper column is missing in cross-shard queries @sgrif
  • chore: update to Rust 2024 @sgrif
  • fix: refactor and improve the functionality of avg() @sgrif
  • refactor: aggregate functions cleanup @sgrif
  • fix: clippy fixes @imor
  • fix: more clippy errors and CI enforcement @meskill
  • fix: handle comments in SET in regex detector

Also introducing our company mascot, Von Count.

image

New Contributors

Full Changelog: v0.1.43...v0.1.44

Read the original on github.com ↗