Destination secrets

Every destination needs a credential to authenticate to its backend — a Grafana Cloud access token, a Splunk HEC token, a basic-auth password, a bearer API key. LinkMesh keeps those out of the config it commits: a credential is stored once as a named, encrypted secret, referenced by name in the destination config, and the real value is substituted only at the moment config is delivered to a collector. Your GitOps repo, config history, and API responses hold the reference, never the value.

This is separate from the four control-plane credentials in Credentials & tokens — those authenticate collectors and users to LinkMesh; destination secrets authenticate collectors to their backends.

The secrets vault

A secret is a named, write-only value in an encrypted store:

• Named — grafana-cloud-token, splunk-hec, loki-password. One secret can be referenced by any number of destinations.
• Write-only — you set the value at creation (or on rotation); the API never returns it again. Reads yield only metadata: name, description, version, when it was last rotated, and how many configs reference it.
• Encrypted at rest — AES-256-GCM. The data key never lives in the database, so a database dump or backup yields only ciphertext.

Manage them under Settings → Secrets: create, rotate, and delete, gated by the secrets:read / secrets:write permissions.

Referencing a secret

In a destination’s (or source’s) configuration, a credential field holds a reference instead of a value:

api_token: ${secret:grafana-cloud-token}

In the UI, credential fields offer a picker: choose an existing secret or create a new one inline. The reference string — ${secret:<name>} — is what gets stored in the database, committed to the GitOps repo, returned by the API, and shown in a config diff. The plaintext value is none of those things.

An unknown reference is a hard error at delivery time, surfaced in the UI — a config that references a missing secret is never shipped with a blank credential.

Substituted only at delivery

The real value is resolved server-side, at the two delivery points only — when config is pushed to an OpAMP-managed collector, and when an Alloy collector polls for its config. The bytes the collector receives contain the real credential (it must, to authenticate to its backend); everything upstream of that moment — the working clone, the committed history, every API read — holds only the reference. Rotating a secret re-renders and re-delivers to every collector that references it, with no per-collector edit.

Inline credentials are redacted too

If a credential is entered directly into a config field instead of as a ${secret:} reference, LinkMesh still masks it on every read path (API, config preview, diffs) and in the config committed to git. References are the clean path; masking is the safety net so a hand-pasted key never leaks.

The data key

Vault encryption uses one operator-scoped data key (32 bytes, AES-256):

• Provide your own with LINKMESH_SECRETS_KEY (a 32-byte key, base64 or hex) or LINKMESH_SECRETS_KEY_FILE (a path to it). This is the recommended production posture — keep the key in a Kubernetes Secret or your secret manager, outside the data volume, so a stolen volume yields only ciphertext.
• Or let the server generate one. With no key configured, the server generates and persists a key on the data volume on first start, so the vault works out of the box. This protects credentials on the API, git, and diff read paths, but because the key sits next to the data it does not protect against theft of the volume itself — provide LINKMESH_SECRETS_KEY for that.

The data key is critical state

Losing the data key makes every existing secret unrecoverable — they must be re-entered. Treat it like the ServerSecret: back it up. Changing the key is safe as long as you do it as a rotation (keep the old key until the re-seal completes) — simply swapping the key without the old one leaves the vault unable to open what it sealed. Set secrets.autoGenerate: false if you’d rather the vault stay disabled until you provide a key.

Rotating the data key

To move to a new data key without downtime, run a dual-key rotation — the new key seals, the old key stays available to open existing values until they are re-sealed:

1. Set the new key as primary and keep the old one as previous. Put the new 32-byte key in LINKMESH_SECRETS_KEY (or secrets.key), and the old one in LINKMESH_SECRETS_KEY_PREVIOUS (comma-separated; or secrets.previousKeys as a list). Restart. The vault keeps working immediately — every stored secret still opens under the previous key.
2. The server re-seals automatically. On startup, with a previous key present, LinkMesh re-seals every vault secret and its own git-settings tokens under the new key. It is idempotent and, in a multi-replica deployment, a single replica performs the re-seal while the others keep serving. When it finishes it logs that the previous key can be removed.
3. Remove the previous key. Once that completion log appears, drop LINKMESH_SECRETS_KEY_PREVIOUS / secrets.previousKeys and restart. The old key is now retired.

A value that opens under neither the new nor the previous key is left untouched and logged — add the correct old key to the previous list, or re-enter that secret’s value. Nothing is ever destroyed by a rotation.

Rotation and upgrades

Rotating a secret — set a new value under Settings → Secrets. Every destination that references it re-renders and the affected collectors are re-delivered automatically.

After upgrading to a version with the vault, LinkMesh re-renders your committed configs so their tip holds references and masks, not plaintext. Git history, however, still contains any credential that was committed before the upgrade — history is not rewritten. The honest fix is to rotate any credential that ever sat in committed config, so the old value in history is dead.

See also

• Credentials & tokens — the four control-plane credentials (enrollment, service-account, own-metrics, UI session).
• Security & encryption — the trust model and what the server can and can’t see.