Declarative Audit Log Collection from HTTP APIs

Introducing Helr: a Rust-based generic HTTP API log collector that turns YAML config into a resilient log pipeline

You have a growing list of SaaS services: Okta for identity, Google Workspace for email and docs, GitHub for code, Slack for communication, 1Password for secrets, Tailscale for networking and possibly many others. Each one exposes an audit log API. You need those logs for compliance, for threat detection, or just for incident response, to answer “what happened?” when something goes wrong. Some log collectors ship vendor-specific integrations for the most popular APIs. For example, Vector has an Okta source, CrowdStrike has a Falcon LogScale collector, and so on. Some vendors support pushes. But each integration, regardless of the ingestion type, is a one-off: if your vendor or integration isn’t on the list, you’re back to writing a custom polling script that handles auth, pagination, rate limits, state, and retries on its own. Five vendors without pre-built support, five scripts, five slightly different implementations of exponential backoff.

That pattern led me to build Helr, pronounced Heler. If you’ve used the Grafana okta-logs-collector, think of Helr as that concept taken to its logical conclusion: a single binary that can poll any HTTP API that returns JSON, handle pagination loops, persist cursors across restarts, and emit clean NDJSON to standard output (stdout) for whatever downstream collector you already run.

I’ve previously written about the other half of this problem: Pattern Detection and Correlation in JSON Logs introduced RSigma, a tool that evaluates Sigma detection rules against JSON events. But detection rules need events to evaluate against, and somebody has to collect them first. Helr is that somebody. Ingestion goes before detection.

The Problem

Before writing Helr, I surveyed over 50+ SaaS and infrastructure audit APIs across identity providers, cloud platforms, collaboration tools, DevOps services, security products, and more. The pattern was consistent: every vendor exposes events over HTTP, but no two implementations agree on the details.

Auth alone spans bearer tokens with vendor-specific prefixes (Okta uses SSWS, not Bearer), API keys in headers or query params, OAuth2 with refresh tokens or client credentials (sometimes requiring DPoP or private_key_jwt), HMAC request signing (Duo), AWS Signature V4 (CloudTrail), and Google service accounts with domain-wide delegation. Pagination is equally fragmented: Okta and GitHub use Link headers with rel="next", Google Workspace and Slack use cursor tokens in the response body (with different field names: nextPageToken vs response_metadata.next_cursor), 1Password sends cursors in POST bodies, Terraform Cloud uses page numbers, and Tailscale uses time windows with no pagination at all. Rate limit headers might be X-RateLimit-Remaining, Retry-After, or something entirely vendor-specific. And you always need to remember where you left off so restarts don’t cause gaps or duplicate ingestion. You get the idea.

Of those 50+ APIs, about 7 were an excellent fit for a generic HTTP poller (standard REST, well-documented pagination), another 12 were a good fit with minor configuration, and roughly 35 were moderate fits requiring some combination of hooks or custom mapping. Only 6 were truly incompatible (push-only, CSV export, or SQL-based). That means the vast majority of audit APIs follow patterns that can be captured declaratively, if the collector supports enough strategies.

These are all cross-cutting concerns. If you solve them in a Python script for Okta, you’ll solve them again for Google Workspace, again for Slack, and again for whatever vendor you onboard next quarter. Each script carries its own retry logic, its own backoff strategy, its own state file format, and its own way of failing at 2 AM. The total maintenance cost grows linearly with the number of APIs. Helr absorbs all of it into a single YAML file per source and a single binary that handles the rest.

Why Not Vector, Alloy, Fluent Bit, or Logstash?

These are excellent tools, and Helr is designed to work alongside them, not replace them. But they solve a different problem. Most log collectors excel at receiving logs (from files, stdout, push endpoints) and routing, enriching, and shipping them. Generic “poll an HTTP API, handle pagination, persist state” is not their core competency.

Vector provides an http_client source that periodically calls an endpoint, but it is explicitly stateless (stated in the docs header) and has no pagination loop. It fires one request per interval. A Vector contributor confirmed this limitation directly in issue #22967: “The http_client source has no way of updating the URL to use, and runs on an interval (i.e, it won’t send two requests in succession until the next interval) so it’s not suited for the task.” Vector’s answer was a dedicated okta source (merged August 2025), but it too is marked stateless and best-effort delivery, and each new vendor API would need another purpose-built source.

Grafana Alloy is a strong pipeline and shipper, especially for Loki, but it is not a polling client. loki.source.api is an HTTP server that receives pushed log entries. loki.source.file tails local files. Both are ideal downstream of Helr, not a replacement for the polling layer.

Fluent Bit’s http input is also a push receiver. The docs say it “allows Fluent Bit to open an HTTP port for receiving data dynamically”: you configure listen and port, and it accepts incoming POSTs. There is no built-in input for polling external HTTP APIs.

Logstash comes closest with http_poller, which can call endpoints on a schedule. But it has no pagination and no state. GitHub issue #51, open since April 2016 with 26 upvotes and still unresolved, requests exactly this. Logstash creator Jordan Sissel commented on that issue: “Having the http_poller input be aware of how to paginate is something we’ve discussed internally. I personally feel it’s not something we can achieve because of the ways that things present pagination.” His recommendation: write a new dedicated input plugin per API.

Helr fills the missing upstream layer: stateful HTTP API polling → normalized NDJSON → stdout or file, designed to feed into these existing collectors rather than compete with them.

Helr at a Glance

Helr is written in Rust, ships as a single binary with no runtime dependencies beyond a config file and secrets, and runs as a long-lived process (or one-shot with --once). The data flow is:

helr.yaml ──▶ Config (validate, expand env vars)
                │
                ▼
        Poll loop (per-source, concurrent via tokio::spawn)
                │
                ├── Auth (Bearer/OAuth2/GCP SA/DPoP)
                ├── Pagination (link_header/cursor/page_offset/single_page)
                ├── Retry + Circuit Breaker
                ├── Streaming JSON parse (events one at a time)
                └── Deduplicate (LRU event ID cache)
                │
                ▼
        EventSink (stdout/file/backpressure wrapper)
                │
                ▼
        NDJSON ──▶ downstream (Grafana Alloy, Vector, Fluent Bit, Loki)

Two traits define the pluggable boundaries: StateStore (SQLite, Redis, Postgres, or memory) for persisting cursors and watermarks, and EventSink (stdout, file, or a backpressure wrapper around either) for output.

Beyond the core polling loop, Helr includes the production features that you’d otherwise skip in a “quick script” and then regret:

• Resilience: Retries with exponential backoff and jitter. A per-source circuit breaker (closed/open/half-open state machine) stops hammering a failing API. Rate limit header parsing (X-RateLimit-Remaining, Retry-After) combined with client-side RPS caps and adaptive throttling.
• Backpressure: When the downstream consumer can’t keep up, a bounded queue enforces block, disk-buffer, or drop strategies with Prometheus metrics for each.
• Deduplication: An LRU cache of event IDs prevents emitting the same event twice when paginating through overlapping windows.
• Signals: SIGHUP triggers live config reload without dropping state; SIGUSR1 dumps current state to the log; SIGTERM/SIGINT initiates a graceful shutdown that checkpoints all sources before exiting.
• REST API: /api/v1/sources lists configured sources; /api/v1/sources/:id/state inspects persisted cursors; POST /api/v1/sources/:id/poll triggers an immediate poll; POST /api/v1/reload reloads config.
• Session replay: --record-dir saves raw API responses to disk;--replay-dir replays them for deterministic testing without hitting production
• Audit trail: Logs credential access events (which source, which auth method, when) without ever logging the secret values themselves.
• Observability: Prometheus metrics endpoint (/metrics) exposes poll counts, latencies, error rates, circuit breaker state, and queue depth.

Configuring a source looks like this:

okta-audit:
  url: "https://${OKTA_DOMAIN}/api/v1/logs"
  auth:
    type: bearer
    token_env: OKTA_API_TOKEN
    prefix: SSWS
  pagination:
    strategy: link_header
    rel: next
    max_pages: 20
  schedule:
    interval_secs: 60
    jitter_secs: 10

That YAML block replaces a custom Okta polling script. Compare it with a cursor-paginated POST source like 1Password:

1password-audit:
  url: "https://events.1password.com/api/v2/auditevents"
  method: post
  body: {}
  auth:
    type: bearer
    token_env: ONEPASSWORD_EVENTS_TOKEN
  pagination:
    strategy: cursor
    cursor_param: cursor
    cursor_path: cursor
    max_pages: 50
  schedule:
    interval_secs: 300
    jitter_secs: 30

Same binary, same config structure, completely different API. The cursor_param and cursor_path tell Helr where to find the next-page token in the response and where to inject it into the next request body. No code, just declarations.

Auth types span bearer (with custom prefix), API key, basic, OAuth2 (refresh token or client credentials, with optional private_key_jwt and DPoP per RFC 9449), and Google service accounts with domain-wide delegation. All secrets come from environment variables or files; ${VAR} placeholders are expanded at load time and validated before the first request. helr validate catches misconfigurations at startup, not at 3 AM.

Pagination Strategies

Pagination is the core problem that justifies Helr’s existence. Generic HTTP pollers treat each request as independent: fire, parse, wait for the next interval. Real audit APIs require a loop within a single poll tick: fetch page one, extract the “next” pointer from the response, fetch page two, repeat until the API says there are no more pages. This is what Vector’s http_client cannot do, what Logstash’s http_poller has been asked to support for ten years, and what every custom script reimplements from scratch.

Helr supports four pagination strategies, selected per source via the strategy field in YAML:

• Link header follows Link: <url>; rel="next" in the response headers until the header is absent. Helr stores the last next_url in the state store so a restart can resume mid-pagination. Okta and GitHub both use this pattern.
• Cursor extracts a token from a JSON path in the response body and sends it back as a query parameter on GET requests or merges it into the request body on POST requests. This single strategy covers a surprising range of APIs because only the field names change: Google Workspace uses nextPageToken, Slack uses response_metadata.next_cursor, 1Password uses cursor in the POST body, Datadog uses a cursor in POST search responses, and Snyk uses a cursor in its REST v3 API. Each one is a different cursor_path/cursor_param pair in YAML, but the loop logic is identical.
• Page/Offset increments a page number (page=1,2,3,…) or an offset (offset=0,100,200,…) with a configurable limit parameter. This covers APIs with stable result ordering that support numeric pagination.
• Single page does no pagination at all, just one fetch per tick. This handles APIs that return all results in a single response or where the time window is narrow enough that pagination isn’t needed.

The dispatch is straightforward: PaginationConfig is a tagged enum. Each variant maps to a strategy-specific function. Adding a new strategy means adding a variant and a function without touching existing code.

Two related features fill gaps that pagination alone doesn’t cover. incremental_from stores the latest event timestamp from the previous tick and sends it as a query parameter (e.g. Slack’s oldest) on the first request of the next tick, so the API only returns events after that point. watermark_field and watermark_param do the same but derive the start-from value from the last event in each response, which is how Google Workspace’s startTime parameter works.

Streaming JSON Parsing

A typical audit API response is a JSON object with metadata (cursor, pagination tokens) wrapping an array of events. The naive parsing path downloads the full body, converts it to a string, parses the entire thing into a serde_json::Value tree, clones the events array, and iterates over the clones. For a 50 MB response, that’s roughly 250 MB of peak memory: about 5x the raw body size.

Helr addresses this with two streaming modes.

• Buffered streaming (the default) downloads the response body once but never builds the full value tree. A byte-level scanner locates the events array inside the JSON structure — either by following a configured dotted path (e.g. data.records) or by probing well-known keys like items, data, events, logs, entries. Once the array boundaries are found, Helr splits the response in two: the metadata shell (cursor tokens, pagination state) is parsed separately from the events. Events are then iterated one at a time using serde’s streaming deserializer, advancing a byte offset after each element. No intermediate collection of parsed events is ever allocated. For GraphQL-style responses where each array element wraps the actual event (e.g. edges[].node), the inner value is moved out in place rather than cloned.
• Full async streaming (feature-gated) goes further: it never buffers the full response at all. Bytes flow directly from the HTTP response stream into serde’s streaming deserializer. The main Rust challenge here is that serde’s from_reader expects synchronous I/O, but the HTTP response is an async byte stream. Helr bridges the two worlds by running the parser in a blocking task and feeding it bytes through a synchronous adapter from tokio-util. Parsed events are sent back to the async poll loop over a channel as they arrive. Size limits are enforced during download rather than after, and when session replay is enabled, a tee reader writes bytes to disk without additional buffering.

The result is a significant reduction in peak memory: the naive approach uses roughly 5x the response size, buffered streaming brings it down to about 2x, and full async streaming approaches 1x.

Extending with JavaScript Hooks

The declarative model covers the majority of audit APIs, but the research turned up several that require logic YAML cannot express: GraphQL APIs like Linear and New Relic (NerdGraph), APIs with HMAC request signing like Duo Security, cookie-based auth flows that involve a login POST before the real request, and pagination schemes that compute the next offset from the previous response body rather than following a cursor.

For these cases, Helr supports optional JavaScript hooks powered by Boa, a pure-Rust JS engine. Build with --features hooks to enable them. The released binaries always contain this feature.

Five hook points cover the full request lifecycle:

• getAuth(ctx) runs once per poll tick (not per page) and returns auth artifacts: headers, cookies, query parameters, or body fields. Results are cached across polls with a configurable TTL. This is where you’d implement a login flow that exchanges credentials for a session cookie.
• buildRequest(ctx) overrides the URL, headers, query parameters, or body for each request. The context includes environment variables, a snapshot of the state store, the source ID, and source-configured headers.
• parseResponse(ctx, response) extracts events from non-standard response shapes. It receives the status code, headers, and parsed body, and returns an array of { ts, source, event, meta } objects.
• getNextPage(ctx, response, request) derives the next page. It also receives the request that was sent (URL and body), so hooks can compute offsets. Return null to stop pagination.
• commitState(ctx, events) runs custom state commit logic after a successful poll.

Each hook runs in a sandbox with a configurable execution timeout and no filesystem access. When allow_network: true is set, Boa exposes a fetch() function so hooks can make HTTP requests, like calling a token endpoint or a login API. Source-configured headers (e.g. User-Agent) are passed to hooks via ctx.headers so fetch() calls can reuse them.

Here’s a condensed example of a getAuth hook that exchanges an API token for a session cookie, using fetch():

async function getAuth(ctx) {
  var res = await fetch("https://api.example.com/login", {
    method: "POST",
    headers: Object.assign({ "Content-Type": "application/json" }, ctx.headers),
    body: JSON.stringify({ code: ctx.env.API_TOKEN })
  });
  if (res.status >= 300) throw new Error("login failed: " + res.status);
  var cookie = (res.headers.get("set-cookie") || "").split(";")[0].trim();
  return { cookie: cookie };
}

And a getNextPage hook for skip-based GraphQL pagination:

function getNextPage(ctx, request, response) {
  var edges = response.body.data.auditLog.edges || [];
  if (edges.length < PAGE_SIZE) return null;
  var nextSkip = (request.body.variables.pageArgs.skip || 0) + PAGE_SIZE;
  return { body: buildQuery(nextSkip) };
}

These hooks combine naturally. A source can use getAuth for cookie login, buildRequest for GraphQL query construction, parseResponse for edges[].node extraction, and getNextPage for skip-based or cursor-based pagination, all in a single script file referenced by name in the YAML config.

Downstream and Wrapping Up

Helr emits NDJSON to stdout or file. Each line is one JSON object with ts, source, endpoint, event (the raw API payload), and meta (optional cursor, request ID). This format is designed to be consumed by any log agent:

helr run | alloy ...                          # pipe to Grafana Alloy
helr run --output /var/log/helr/events.ndjson  # Alloy/Vector/Fluent Bit tails the file
helr run | rsigma eval -r rules/              # pipe directly to RSigma for detection

Helr started from a simple idea: collecting audit logs from SaaS APIs shouldn’t require a custom script per vendor. It turned into a full-featured collector because doing it properly means handling auth, pagination, state, retries, rate limits, backpressure, streaming parsing, and all the edge cases those bring. Building it in Rust yielded a single static binary with low memory footprint and safe concurrency through Tokio, which matters for a process that runs 24/7 and handles sensitive data.

The project is open source under the MIT license. You can find it on GitHub and install it from crates.io:

cargo install helr

If you’re already piping audit logs through custom scripts, Helr can replace them with a YAML file. If you’re using RSigma to detect threats in JSON events, Helr can collect the events. They are complementary tools: Helr collects, RSigma detects.

Declarative Audit Log Collection from HTTP APIs was originally published in ITNEXT on Medium, where people are continuing the conversation by highlighting and responding to this story.