Set up alerting

LinkMesh watches your fleet’s self-telemetry and raises an alert when a collector’s throughput, error rate, or exporter queue crosses a threshold you set. Alerts fire on the server, deliver to a webhook, and land in an alert history you can acknowledge and track to resolution.

This page covers the whole loop: define a rule, send its notifications somewhere, and work the history.

How evaluation works

The server runs an alert evaluator on a fixed interval (about once a minute). On each pass it loads every enabled rule and checks it against the latest self-telemetry for each collector:

• When a collector’s metric satisfies the rule’s condition, the alert fires — a notification is created and delivered to your channels.
• When the metric recovers so the condition no longer holds, the alert resolves automatically.
• A short cooldown after an alert resolves keeps a metric hovering around the threshold from flapping — re-firing repeatedly within a couple of minutes.

Because evaluation reads the same self-telemetry the collectors already push, alerting needs no extra instrumentation — but it does need the collectors to be reporting. A collector that isn’t pushing self-telemetry has no metrics to evaluate. See Self-telemetry if the numbers you’d alert on aren’t arriving.

1. Create an alert rule

Open Alerts in the sidebar and click New Rule. A rule is a single threshold condition:

1. Name — what this rule watches, in words. It shows on every notification, so make it read well in a history row (Checkout throughput dropped, not rule 3).

2. Metric — the collector signal to watch:

   Metric	Unit	What it measures
   Throughput	records/s	Records a collector is processing — the workhorse metric. Alert below a floor to catch a source that went quiet.
   Error Rate	errors/s	Records a collector is failing to process or export. Alert above a ceiling.
   Queue Depth	count	How many records are buffered in an exporter’s send queue. A rising queue is the earliest sign a destination can’t keep up.
   Bytes/s	bytes/s	Measured egress. Only non-zero on collectors with detailed telemetry enabled (on the collector’s Telemetry settings); others report 0.

3. Operator and Value — the comparison: greater than, less than, greater-or-equal, or less-or-equal, against the number you set. “Throughput < 10” fires when a collector drops below 10 records/s.

4. Evaluation window — the period the metric is assessed over (1 minute up to 1 hour). A wider window smooths out momentary spikes; a narrow one reacts faster.

5. Severity — Critical or Warning. This is a label for triage and filtering in the history; it doesn’t change how the rule evaluates.

6. Enabled — leave on to evaluate immediately, or toggle off to stage a rule without arming it.

Save. The rule appears in the list with its condition and an enable toggle; the evaluator picks it up on its next pass.

Alert on the floor, not just the ceiling

The most useful pipeline alert is often a lower bound on throughput. A source that stops sending doesn’t raise errors — it goes silent, and silence is invisible until someone goes looking. A “throughput < N” rule turns that silence into a notification.

2. Send notifications to a webhook

A rule that fires with nowhere to go only shows in the UI. To get pushed notifications, add a channel under Notifications → Channels (or the Notification Channels page). Click New Channel and choose Webhook:

1. Name — how the channel reads in the list.

2. Webhook URL — where LinkMesh POSTs each alert. An incoming-webhook URL from Slack, or any HTTP endpoint you control.

3. HTTP method — POST (default) or PUT.

4. Headers — optional, one Key: Value per line, for endpoints that need auth (Authorization: Bearer …) or routing headers.

Every firing (and resolving) alert is delivered to all enabled channels. The payload is JSON with the rule name, collector, metric, severity, status, current value, threshold, and message — plus a Slack-compatible text field, so a Slack incoming webhook renders a readable message with no extra mapping.

Use the channel’s Test button to send a sample payload and confirm the endpoint accepts it — the row shows a pass or fail marker.

Webhooks can't reach private addresses by default

As a safety measure the server refuses to deliver webhooks to private/loopback addresses. If you’re posting to an endpoint inside your own network, enable outbound delivery to internal addresses in the server configuration — see Configuration.

3. Work the alert history

Alerts → History is the operational view. It opens with summary counts — Firing, Acknowledged, Resolved — over a table of every notification, with filters for severity, status, and a text search.

Each alert moves through a lifecycle:

• Firing — the condition is currently true. The row shows the collector, the metric, the message (e.g. records/s below threshold on edge-01), when it fired, and how long it’s been active.
• Acknowledged — you’ve seen it and are on it. Acknowledging doesn’t silence or resolve the alert; it marks that it’s being handled, so a second responder doesn’t duplicate the work. Acknowledge from the row action.
• Resolved — the metric recovered and the evaluator cleared the alert automatically. The row shows the total firing duration.

The bell in the top bar surfaces active alerts from anywhere in the UI, so you don’t have to sit on the history page to notice one fire.

See also

• Self-telemetry — the metrics alerting evaluates, and how to confirm a collector is reporting them.
• Reading throughput on the canvas — the live view of the same throughput, error, and queue signals you alert on.
• Monitor the server — watching the LinkMesh server’s own health, alongside your collector fleet.