Writing a Fetcher

A converter reads a file a tool already produced. A fetcher goes and gets it — a live API client that retrieves security data from a remote tool and hands the bytes to that tool's converter.

Fetchers live beside their converters, in hdf-converters/fetchers/<tool>/, so the network-edge concerns (auth, pagination, rate limits) sit next to the format transform they feed. The fetcher inventory lists what is in tree and which languages each supports.

This page is the human walkthrough. If you are using agents for coding, the repository ships a build-fetcher skill that automates the same process; this page is what to read when you want to understand the conventions rather than delegate them.

When something is a fetcher

It belongs in fetchers/ when it calls a remote API and returns bytes a converter can consume, and the converter for that source format already exists — or lands in the same change.

It does not belong there when it is a pure file transform with no network calls, which is a converter; or when it is CLI orchestration such as command flags, output paths and validation gating, which stays in hdf-cli/cmd/hdf/cmd/.

The Go convention: two constructors

Every Go fetcher offers two ways to build a client, and the difference is who owns transport. The parameters differ by service — the HTTP-based fetchers take TLS options and build a stdlib client, while the AWS ones take a context and build an SDK client — but the pairing is constant.

// Default discovery. Reads TLS options and builds a stdlib http.Client with
// system-CA defaults. This is what hdf-cli uses: the CLI passes flags and
// environment through and never handles raw credentials in process.
fetcher, err := splunk.NewSplunkFetcher(params, shared.TLSOptions{})

// Client injection. The caller hands in a pre-configured client and the
// library never touches transport or TLS.
fetcher, err := splunk.NewSplunkFetcherWithClient(params, client)

The AWS fetchers take the same pair in SDK terms — NewAWSConfigFetcher(ctx, params) for discovery, NewAWSConfigFetcherWithClient(client) where client satisfies the SDK interface the fetcher declares.

Reach for the injection form whenever the application layer needs to own transport — corporate proxies, TLS interception, or a mocked client in tests.

Injection does not hand over credentials, and that catches people out. An HTTP fetcher resolves its own token inside Fetch, from environment variables and, where the tool has one, its CLI's config file: GITLAB_TOKEN / GLAB_TOKEN / the glab config for gitlab and gitlab-vulnerabilities, DEFECTDOJO_API_TOKEN, SONARQUBE_TOKEN, SPLUNK_TOKEN. Both constructors do this. The library never accepts a token as a parameter, never logs one and never lets one into an error message. Only the AWS fetchers differ, because there the injected SDK client carries the credentials itself.

That split is deliberate: Go is what hdf-cli drives, and resolving the token in the library keeps it out of the CLI's process and out of ps. The auth-agnostic TypeScript rule below is the other half of the same decision.

The shared helpers behind the default form live in fetchers/shared/go: TLSOptions, NewHTTPClient, ValidateAndBuildAPIURL and ReadLimitedBody.

The TypeScript convention: auth-agnostic

Go is required, because hdf-cli drives it; TypeScript is added when a JavaScript consumer needs it, and most fetchers in tree are Go-only.

TypeScript fetchers accept no credentials, file paths, environment lookups or TLS configuration. They take a pre-authenticated transport and use it for every call; the caller — heimdall2, saf-cli, or any downstream consumer — acquires credentials entirely on its own.

The shape varies by service, and both forms are in tree today:

• An SDK client — aws-securityhub takes a configured SecurityHubClient. The SDK discovers credentials through its own chain; the library never reads environment variables or ~/.aws/credentials.
• A bespoke REST API — defectdojo takes an authFetch callable shaped (path: string, init?: RequestInit) => Promise<Response> that injects whatever headers, cookies or tokens the service wants.

A service with its own JavaScript SDK would follow the first shape, passing that SDK's configured client.

That boundary is the security contract, not a style preference: a library that never receives credentials cannot log, persist or leak them.

The network edge is the security boundary

A converter's first obligation is a size guard on its input. A fetcher's equivalents are at the network edge, and they are requirements rather than suggestions — a fetcher reaches an attacker-influenced endpoint and reads an unbounded response.

• Validate the endpoint before using it. An unvalidated host or region string handed to an endpoint constructor is an SSRF vector. An HTTP fetcher builds its request URLs through the shared ValidateAndBuildAPIURL — see sonarqube or defectdojo. A fetcher that hands a string to an SDK validates it first in its own terms: awsconfig constrains the region with a DNS-label regex before the SDK ever sees it.
• Bound every response read with ReadLimitedBody — defectdojo and splunk are the examples. A fetcher that reads no raw bodies because its SDK owns the transport, like awsconfig, has nothing to bound.
• Bound every pagination loop by a maximum page count, and check ctx.Err() at the top of each iteration rather than only propagating the context. sonarqube is the fullest example.
• Never truncate a nested list silently. A page of findings often carries its own connections — a triage history, a list of overrides. Those paginate too, and a fixed first: on them returns whatever end the API orders from, which is usually the oldest while the consumer reads the newest as current. Ask for the page-info alongside the nodes, then either complete the list with a follow-up request or fail; gitlab-vulnerabilities does the former.
• Do not follow redirects with a credential attached. A 3xx re-sends the token to whatever host it names. Refuse redirects outright where the API never legitimately issues one, as gitlab-vulnerabilities does; where one is expected — the gitlab artifact fetcher follows GitLab's redirect to object storage — the credential header has to be stripped when the host changes.
• Apply a default deadline when the caller sets none.
• Never accept a secret as a CLI flag — environment, profile or token file only — and keep credentials out of logs and error messages.

These are checked in review, and the checks are listed in the repository's build-fetcher skill.

What the API returns is not always what it means

Two failure modes look like success and are worth designing against before writing the first request.

Request everything the converter maps. A narrow field selection silently strips data the converter would have translated, and nothing fails — the output is simply poorer than the source. Ask for the expansions, nested objects and union branches the converter has code for, and re-check the request whenever the converter grows a branch.

An empty result is not always an empty report. On a tiered product, or one that ingests asynchronously, the API answers a query about a feature you do not have with an empty list rather than an error, and a fetcher that trusts it emits a document that reads as a clean bill of health. Find the signal that ingestion itself leaves behind, judge the empty case against that, and refuse to write a document when the answer is "this was never populated" — naming which of the reasons it was. gitlab-vulnerabilities is the worked example: its CI-derived scanner fields read identically on every tier, so it uses the statistics row and scan statuses instead. The same trap applies to a filtered query: record the filter in what you hand the converter, or a selection that matched nothing is indistinguishable from a source with nothing to report.

Adding one

1. Pick a tool name in kebab-case and create fetchers/<tool>/{go,typescript}/.
2. Implement Go with the two constructors. Add the TypeScript twin, auth-agnostic, only when a JavaScript consumer needs it.
3. Write tests that validate request parameters, not just paths: the mock must assert headers, query parameters and request bodies, or it passes against a client that sends the wrong request to the right URL.
4. Wire the CLI command in hdf-cli/cmd/hdf/cmd/, using the default-discovery constructor, and register it with an AddCommand line in fetch.go — creating the file alone compiles but leaves the subcommand invisible. The file name does not always follow the tool name: the awsconfig fetcher is driven by fetch_aws_config.go.
5. Add the tool to the "Currently in tree" table in fetchers/README.md, which is where that inventory is maintained.