Run behind a reverse proxy

Use this guide when LinkMesh is exposed via a public hostname with a Let’s Encrypt certificate, but Collectors connect through that same hostname over the internet. If your Collectors can reach the LinkMesh VM directly (VPC, VPN, bastion, or flat network), the direct VM install is simpler — skip this page.

LinkMesh speaks two protocols on two ports:

• HTTP API on :8080 — the web UI, REST API, and the OpAMP WebSocket endpoint at /v1/opamp that OpAMP-managed Collectors connect to. Terminate TLS at your proxy.
• gRPC on :50051 — the mTLS channel linkmesh-agent-managed Collectors use to connect. Pass through at the TCP layer. Do not terminate.

Two snags catch operators here, both addressed in the nginx config below:

• The default gRPC listener binds :50051 on every interface, so nginx can’t claim that port for its stream block. You’ll move LinkMesh’s internal gRPC listener to a separate port (we use :50061 below) and let nginx own the public :50051 socket. See step 3.
• The /v1/opamp endpoint upgrades to WebSocket. The nginx server block must explicitly forward the Upgrade/Connection headers and must not negotiate HTTP/2 on the same listener (HTTP/2 has no Connection: Upgrade semantics, so an http2 listener returns 400 Bad Request for every OpAMP handshake). The snippet below handles both.

Three steps

1. Install the package as in the direct VM install.

2. Add your public hostname to the gRPC server cert SANs. Cert subcommands need exclusive database access, so stop the service first; run as the linkmesh user so the service can read the new key:

   sudo systemctl stop linkmesh-server
   sudo runuser -u linkmesh -- linkmesh-server cert init --san dns:linkmesh.example.com

   Note

   Once grpc.externalUrl is set in config.yaml (step 3), cert init folds that hostname into the SANs automatically — the explicit --san here is belt-and-suspenders and lets you do step 2 before step 3.

3. Tell LinkMesh about its public URLs and which proxies to trust:

   sudo nano /etc/linkmesh/config.yaml
   sudo systemctl restart linkmesh-server

The relevant config.yaml snippet

externalUrl: "https://linkmesh.example.com"
http:
  trustedProxies: ["127.0.0.1/32", "10.0.0.0/8"]
grpc:
  port: 50061
  externalUrl: "linkmesh.example.com:50051"

• externalUrl is what LinkMesh prints in enrollment scripts and emails. Must match what users type in their browser.
• http.trustedProxies tells LinkMesh which X-Forwarded-* headers to honor. List your proxy’s loopback / VPC ranges.
• grpc.port is the internal port LinkMesh’s own gRPC listener binds to. Move it off the default 50051 (anything free works — we use 50061) so nginx’s stream block can take the public :50051 socket. Without this, nginx -t reloads cleanly but the service fails to start with bind() to 0.0.0.0:50051 failed (98: Address already in use) because LinkMesh’s gRPC server already owns that port on every interface.
• grpc.externalUrl is the address agents dial — host and public gRPC port. It must point at nginx’s stream listener (:50051, public), not the internal grpc.port and not the HTTPS API port. The host must match a SAN you added in step 2 (it’s used to verify the server cert) and is printed into bootstrapped agent configs.

Install nginx + issue a Let’s Encrypt certificate

On Debian/Ubuntu, install nginx (its built-in stream module handles the gRPC pass-through — no extra package needed) and certbot:

1. Install nginx and certbot:

   sudo apt-get update
   sudo apt-get install -y nginx certbot

2. Point DNS at the proxy and open the firewall. Add an A/AAAA record for linkmesh.example.com → the proxy’s public IP, and allow inbound 80, 443, and 50051. Port 80 is only needed for the ACME challenge and the HTTP→HTTPS redirect.

3. Issue the certificate with certbot’s standalone server. It binds port 80, so stop nginx for the ~10 seconds it takes:

   sudo systemctl stop nginx
   sudo certbot certonly --standalone -d linkmesh.example.com \
     --agree-tos -m you@example.com --non-interactive \
     --deploy-hook "systemctl reload nginx"
   sudo systemctl start nginx

   This writes fullchain.pem + privkey.pem to /etc/letsencrypt/live/linkmesh.example.com/ — the exact paths the nginx config below references. Certbot’s systemd timer auto-renews; the --deploy-hook reloads nginx so it picks up each renewed cert.

Two certs, two trust domains

This Let’s Encrypt cert secures the HTTP API edge only — nginx terminates it on 443. The gRPC channel uses LinkMesh’s own mTLS cert from cert init (step 2 above), passed straight through untouched. Don’t point nginx’s ssl_certificate at the gRPC cert or vice versa — they’re separate certificates for separate trust domains.

nginx config — HTTPS terminate for HTTP API + stream pass-through for gRPC

The HTTP API is plain HTTP between nginx and LinkMesh on :8080; nginx terminates TLS at the edge with your Let’s Encrypt cert. The gRPC channel on :50051 is passed through at the TCP layer — nginx does not terminate it. The OpAMP endpoint is a WebSocket upgrade carried over that same HTTPS listener and gets its own location block with the upgrade headers wired up.

/etc/nginx/sites-enabled/linkmesh.conf

# ── WebSocket upgrade map ────────────────────────────────────
# Used by the /v1/opamp location below. Lives at the http {} top level
# (outside any server block) — drop it in /etc/nginx/conf.d/ws-upgrade.conf
# if you prefer to keep this site file focused on the linkmesh server blocks.
map $http_upgrade $connection_upgrade {
    default upgrade;
    ''      close;
}

# ── HTTP API + OpAMP (port 443) — TLS terminated here ────────
server {
    # No `http2` — WebSocket Upgrade is HTTP/1.1 only. HTTP/2 has no
    # `Connection: Upgrade` semantics and returns 400 for every OpAMP
    # handshake. If you also serve a heavy web UI from this host and miss
    # h2 multiplexing, split OpAMP onto its own server_name on a separate
    # listener instead of re-enabling http2 here.
    listen 443 ssl;
    server_name linkmesh.example.com;

    ssl_certificate     /etc/letsencrypt/live/linkmesh.example.com/fullchain.pem;
    ssl_certificate_key /etc/letsencrypt/live/linkmesh.example.com/privkey.pem;

    # OpAMP WebSocket — preserves Upgrade headers, long idle timeouts so
    # the persistent OpAMP connection isn't reaped between heartbeats.
    location /v1/opamp {
        proxy_pass http://127.0.0.1:8080;
        proxy_http_version 1.1;
        proxy_set_header Upgrade    $http_upgrade;
        proxy_set_header Connection $connection_upgrade;
        proxy_set_header Host       $host;
        proxy_set_header X-Forwarded-Proto $scheme;
        proxy_set_header X-Forwarded-For   $proxy_add_x_forwarded_for;
        proxy_set_header X-Real-IP         $remote_addr;
        proxy_read_timeout 3600s;
        proxy_send_timeout 3600s;
    }

    # Everything else — REST API, web UI, enrollment endpoints.
    location / {
        proxy_pass http://127.0.0.1:8080;
        proxy_set_header Host              $host;
        proxy_set_header X-Forwarded-Proto $scheme;
        proxy_set_header X-Forwarded-Host  $host;
        proxy_set_header X-Forwarded-For   $proxy_add_x_forwarded_for;
        proxy_set_header X-Real-IP         $remote_addr;
        proxy_http_version 1.1;
        proxy_set_header Connection        "";
    }
}

# ── HTTP → HTTPS redirect ────────────────────────────────────
server {
    listen 80;
    server_name linkmesh.example.com;
    return 301 https://$host$request_uri;
}

The stream {} block for gRPC pass-through goes in a different file: /etc/nginx/nginx.conf at the top level. It can’t live in sites-enabled/ because that directory is included from inside the http {} context, and stream {} must sit alongside http {}, not inside it.

/etc/nginx/nginx.conf

# Append at the top level (NOT inside http {}).

# ── gRPC (port 50051) — TCP pass-through, NOT terminated ────
# Public :50051 → LinkMesh's internal :50061 (grpc.port in config.yaml).
stream {
    server {
        listen 50051;
        proxy_pass 127.0.0.1:50061;
    }
}

Then validate and reload:

1. Validate, then reload — nginx -t catches a missing stream {} block or a bad cert path before you take the proxy down:

   sudo nginx -t && sudo systemctl reload nginx

2. Confirm all three surfaces answer from a host that can reach the proxy:

   # HTTP API → 200/302
   curl -sI https://linkmesh.example.com/ | head -1
   
   # OpAMP WebSocket — must return 401 (auth required), NOT 400/301.
   # 400 means nginx is negotiating HTTP/2 or stripping Upgrade headers;
   # 301 means the request hit :80 instead of :443.
   curl -skI -o /dev/null -w '%{http_code}\n' \
     -H 'Upgrade: websocket' -H 'Connection: Upgrade' \
     -H 'Sec-WebSocket-Key: dGhlIHNhbXBsZSBub25jZQ==' \
     -H 'Sec-WebSocket-Version: 13' \
     https://linkmesh.example.com/v1/opamp
   
   # gRPC pass-through open
   nc -vz linkmesh.example.com 50051

OpAMP-managed Collector config — use wss://, not ws://

When you point an OpAMP-managed Collector at this proxy, the endpoint must be wss://linkmesh.example.com/v1/opamp (note the two ss). Plain ws:// hits nginx on :80, gets a 301 to HTTPS, and the OpAMP client gives up with websocket: bad handshake because it doesn’t follow redirects. The linkmesh-server install page emits the correct wss:// URL automatically; if you hand-edit the Collector config, double-check the scheme.

Why pass-through (not termination) for gRPC?

The gRPC channel uses mTLS: the agent presents a client certificate that is the collector’s identity. If nginx terminated TLS, the client cert would stop at nginx — LinkMesh would only see nginx’s loopback IP, with no way to know which collector is talking. By passing the raw TCP stream through to the LinkMesh server, the mTLS handshake terminates inside LinkMesh, the client cert is verified against the internal CA, and the agent’s identity is preserved end-to-end.

A full annotated example config ships with the package at /usr/share/doc/linkmesh-server/nginx-reverse-proxy.conf.example. It also documents the SNI-based alternative if you want HTTP API and gRPC to share port 443.