Build your first pipeline

A pipeline is a reusable processing recipe — a linear chain of processors applied to telemetry on its way to a destination. This walkthrough builds one from scratch in the UI, verifies it with the dry-run panel, and attaches it to a route on a live collector.

By the end you’ll have a pipeline that strips emails, phone numbers, and credit-card patterns out of log bodies before they leave your network, attached to one route, verified against a real sample. Total time: about 8 minutes.

Before you start

You need a working LinkMesh server with at least one online collector and an active source — the Quickstart covers both. Any source that emits log records works for this walkthrough; the examples assume a log source on a collector named checkout-api.

1. Open the chain editor

Open Pipelines in the sidebar.

The Pipelines page lists every reusable processing recipe in your tenant. + New Pipeline opens the chain editor with an empty chain.

Click + New Pipeline. The chain editor opens with two anchors: Input (the schema the pipeline expects) and Output (the schema it produces). Between them is the chain — empty for a new pipeline.

The chain editor. Input and Output anchor the chain; each processor you add becomes a step in between, evaluated top to bottom.

2. Name and describe

In the header, set:

• Name: PII Redaction
• Description: Masks email addresses, phone numbers, and credit card patterns in event bodies before they leave the cluster.
• Signal types: logs
• Tags: compliance

The name is what routes reference, so pick something that reads well in a route list. The description is shown in the route attach picker and in the pipeline list — keep it one sentence, action-oriented.

3. Add a transform step

Click + Add Step between Input and Output. The processor picker opens.

The processor picker. Transform is the workhorse for any OTTL-based mutation — masking, attribute mutation, body rewrites; the picker also offers filter, sample, enrich, and route-on-attribute steps for v1.

Pick Transform (custom) and name the step Mask PII. In the rule builder, add three OTTL replace_pattern statements against body:

replace_pattern(body, "[a-zA-Z0-9._%+-]+@[a-zA-Z0-9.-]+\\.[a-zA-Z]{2,}", "<email>")
replace_pattern(body, "\\+?[0-9]{8,15}", "<phone>")
replace_pattern(body, "[0-9]{4}[ -]?[0-9]{4}[ -]?[0-9]{4}[ -]?[0-9]{4}", "<card>")

Each statement rewrites every match in the log body to a placeholder. The expression language is OTTL — the same one OpenTelemetry processors use, so the same rules you’d write in a hand-edited collector config work here.

Save the step. It appears in the chain between Input and Output.

Note

Pattern order matters: card numbers run last because the phone-number regex (\d{8,15}) would otherwise match the digit run inside a card number first and leave a <phone> instead of <card>. If you reorder, the dry-run panel in the next section will catch the regression immediately.

4. Dry-run against a sample

Click Dry-Run in the editor toolbar. The panel opens to the right of the chain with two columns — the input sample on the left, the output preview on the right.

Dry-run split-right layout. Paste or load a sample on the left; each chain step's output appears on the right with per-step deltas. Sample-data templates seed the OTLP envelope so you only have to edit the body.

Pick logs as the signal and OTLP JSON as the format, then paste this sample into the input pane:

{
  "resourceLogs": [{
    "resource": {
      "attributes": [
        {"key": "service.name", "value": {"stringValue": "checkout-api"}}
      ]
    },
    "scopeLogs": [{
      "logRecords": [{
        "severityNumber": 9,
        "severityText": "INFO",
        "body": {
          "stringValue": "POST /api/v2/cart/checkout user=jdoe@example.com card=4242 4242 4242 4242"
        }
      }]
    }]
  }]
}

Click Run. The output pane shows what comes out the other end — same envelope, redacted body:

{
  "resourceLogs": [{
    "resource": {
      "attributes": [
        {"key": "service.name", "value": {"stringValue": "checkout-api"}}
      ]
    },
    "scopeLogs": [{
      "logRecords": [{
        "severityNumber": 9,
        "severityText": "INFO",
        "body": {
          "stringValue": "POST /api/v2/cart/checkout user=<email> card=<card>"
        }
      }]
    }]
  }]
}

The email is now <email>, the card number is now <card>. That’s the contract — verify it on every change.

5. Inspect per-step diffs

Click the Diff tab in the dry-run panel. Each step shows what it removed, added, or rewrote — invaluable when a multi-step chain produces unexpected output and you need to localise the bug.

Per-step diff view. Red is removed, green is added, yellow is rewritten. Click a step header to expand its full input and output.

For this pipeline the Mask PII step shows the body field rewritten — the matched runs replaced with their placeholders, everything else untouched. If your output doesn’t match the expected JSON above, the diff is where you’ll see why.

6. Save and attach to a route

Click Save in the editor toolbar. The pipeline is now reusable across every collector — but it doesn’t run anywhere until a route references it.

Open the collector that owns the log source (checkout-api in the example) and switch to the Routing tab. Either edit an existing route or create a new one. In the route editor, set:

• Source: the log source
• Pipeline: PII Redaction (the one you just built)
• Destinations: whatever you’re sending to — Loki, OTLP, debug

Route editor with the log source on the left, the PII Redaction pipeline in the middle, and the destinations on the right.

Save. The collector picks up the new routing within a few seconds — the Last applied timestamp on the collector detail page is the authoritative confirmation.

7. Verify in the topology canvas

Open Topology from the sidebar. The canvas renders every collector, every pipeline, and every destination across the fleet as nodes, with active routes drawn as links between them.

The topology canvas visualises every collector and every connection across the fleet, with throughput annotations on each edge.

If the link from your log source through PII Redaction to your destination shows non-zero throughput, the pipeline is live and processing real records. Drop a synthetic log line containing an email + card number into the source; the matching record should arrive at the destination with both already masked.

What’s next

• Concepts: Pipeline — the full model: parameter overrides, schema contracts, and why pipelines are routes-agnostic
• Concepts: Route — how routes wire pipelines to sources and destinations on each collector
• Concepts: Source — the receiver side of the chain
• Drop noisy logs — Filter-processor patterns for stripping health-check probes and other access-log noise before they ever reach a redaction step
• Mask PII — broader PII redaction recipes (internal IDs, IBANs, custom regex catalogues) using the same transform processor

Tip

The dry-run panel re-runs against the same sample every time you change a step. Keep it open while you build — every save round-trips through the sample in under a second, so you see the effect of each change immediately.