Latency profiling

:::danger Unavailable in Nexo 0.2.0

Nexo 0.2.0 publishes no supported profiling endpoint, flame-graph endpoint, profiling CLI, persistent histogram store, or request-trace header. The reference below preserves the former design and all examples for operational parity; do not enable, expose, or automate these interfaces unless a future selected release explicitly provides them.

:::

Understand exactly where time is spent inside the Nexo proxy pipeline, from the moment a MongoDB wire-protocol message arrives to the instant the response leaves.

Legacy UI feature labels:

• Per-Step Latency
• Flame Graphs
• OpenTelemetry

Why Profile?

Nexo sits between your application and MongoDB. Every request traverses a configurable pipeline of steps — logging, filtering, caching, metrics, WASM plugins, and more — before being forwarded upstream to the database. Each step adds latency. Profiling tells you exactly where that latency comes from, which pipeline steps are slow, and how much overhead the proxy adds compared to a direct connection.

Without profiling you are flying blind. A "slow query" might actually be a fast query wrapped in a slow pipeline. Profiling separates upstream database time from proxy overhead so you can optimise the right layer.

• Identify which pipeline steps contribute the most latency
• Compare proxy overhead against upstream database time
• Catch regressions early — a new plugin might double P99 latency
• Right-size your pipeline: remove steps that cost more than they save
• Feed data into capacity planning and SLA calculations

Built-in Profiling Endpoint

Nexo exposes a lightweight HTTP profiling endpoint that returns a JSON breakdown of per-step latency statistics. The endpoint runs on a separate port so it never interferes with MongoDB wire-protocol traffic.

Enabling the Endpoint

Add the following block to your nexo.json (or the equivalent YAML / TOML key):

{
  "profiling": {
    "enabled": true,
    "port": 9091
  }
}

Once enabled, GET /api/profile returns a full latency snapshot. The data resets when the proxy restarts unless you enable persistent storage (see the Profiling in Production section below).

Example Response

Below is a typical response from a proxy that has processed ~15 000 requests through a pipeline with logging, filtering, caching, and metrics steps:

GET http://localhost:9091/api/profile

HTTP/1.1 200 OK
Content-Type: application/json

{
  "total_requests": 15234,
  "avg_total_latency_us": 1250,
  "avg_upstream_latency_us": 980,
  "avg_pipeline_overhead_us": 270,
  "steps": [
    {
      "name": "logging",
      "phase": "request",
      "avg_latency_us": 12,
      "p50_latency_us": 8,
      "p95_latency_us": 25,
      "p99_latency_us": 45,
      "max_latency_us": 312,
      "invocations": 15234,
      "errors": 0
    },
    {
      "name": "filter",
      "phase": "request",
      "avg_latency_us": 35,
      "p50_latency_us": 20,
      "p95_latency_us": 85,
      "p99_latency_us": 150,
      "max_latency_us": 890,
      "invocations": 15234,
      "errors": 2
    },
    {
      "name": "cache",
      "phase": "request",
      "avg_latency_us": 65,
      "p50_latency_us": 40,
      "p95_latency_us": 120,
      "p99_latency_us": 250,
      "max_latency_us": 1520,
      "invocations": 15234,
      "errors": 5
    },
    {
      "name": "metrics",
      "phase": "request",
      "avg_latency_us": 18,
      "p50_latency_us": 12,
      "p95_latency_us": 35,
      "p99_latency_us": 60,
      "max_latency_us": 410,
      "invocations": 15234,
      "errors": 0
    },
    {
      "name": "metrics",
      "phase": "response",
      "avg_latency_us": 15,
      "p50_latency_us": 10,
      "p95_latency_us": 30,
      "p99_latency_us": 55,
      "max_latency_us": 390,
      "invocations": 15234,
      "errors": 0
    },
    {
      "name": "filter",
      "phase": "response",
      "avg_latency_us": 30,
      "p50_latency_us": 18,
      "p95_latency_us": 70,
      "p99_latency_us": 130,
      "max_latency_us": 720,
      "invocations": 15234,
      "errors": 1
    },
    {
      "name": "logging",
      "phase": "response",
      "avg_latency_us": 10,
      "p50_latency_us": 6,
      "p95_latency_us": 20,
      "p99_latency_us": 40,
      "max_latency_us": 280,
      "invocations": 15234,
      "errors": 0
    }
  ]
}

Response Field Reference

• total_requests — number of requests sampled since the last reset
• avg_total_latency_us — average end-to-end latency in microseconds
• avg_upstream_latency_us — average time waiting for MongoDB
• avg_pipeline_overhead_us — total minus upstream; this is the cost of the proxy
• steps[].name — the pipeline step identifier
• steps[].phase — request or response
• steps[].avg / p50 / p95 / p99 / max — latency distribution in μs
• steps[].invocations — how many times the step executed
• steps[].errors — non-fatal errors encountered in the step

Per-Step Latency Breakdown

The Nexo dashboard renders the profiling data as a sortable, filterable table. Each row represents a single pipeline step in a specific phase.

Dashboard Table Columns

• Step Name — identifier of the pipeline step
• Phase — request or response
• Avg (μs) — mean latency across all invocations
• P50 — median latency
• P95 — 95th percentile latency
• P99 — 99th percentile latency
• Max — worst-case observed latency
• Error Rate — percentage of invocations that produced a non-fatal error

Click any column header to sort. Sorting by P99 descending is the fastest way to find bottlenecks — high P99 means occasional slow outliers that hurt tail latency.

Table View

Historical Trend

The dashboard stores profiling snapshots every 30 seconds and displays a time-series chart of per-step latency. This lets you see how latency changes over time — correlate spikes with deployments, traffic surges, or configuration changes.

Flame Graph View

Flame graphs give you an instant visual understanding of where time is spent. Nexo generates interactive SVG flame graphs that you can view in the dashboard or download for offline analysis.

Accessing Flame Graphs

The profiling endpoint exposes a dedicated flame-graph route:

GET http://localhost:9091/api/profile/flamegraph

## Returns: image/svg+xml
## Query parameters:
## ?duration=30s    — sample window (default: 60s)
## ?min_width=0.5   — hide frames narrower than 0.5% (default: 0.1)
## ?title=MyProxy   — custom title for the SVG

How to Read the Flame Graph

• Width — proportional to time spent; wider bars mean more latency
• Nesting (vertical depth) — represents the call hierarchy; child frames are operations within a parent
• Top level — shows the full request lifecycle: request pipeline → upstream → response pipeline
• Second level — individual steps within each phase (logging, filter, cache, etc.)
• Third level — internal step operations such as BSON parse, config lookup, cache hit/miss, regex evaluation
• Interactive — click any bar to zoom in, hover to see exact timing, press Escape or click the root frame to zoom out

Detailed Flame Graph

The interactive flame graph expands request, upstream, and response work so you can drill into expensive steps, host calls, cache lookups, and plugin execution without relying on static ASCII snapshots.

Flame Graph Colour Coding

• Orange / Red — pipeline steps (request + response phases)
• Blue — upstream database communication
• Green — cache operations (hits and misses)
• Purple — WASM plugin execution
• Yellow — serialisation / deserialisation (BSON, JSON)
• Grey — idle / waiting time

Slow Step Detection

Nexo can automatically detect when a pipeline step exceeds its expected latency budget and emit alerts. This is useful for catching regressions early — a misconfigured cache, a poorly-written filter regex, or a WASM plugin that blocks on I/O.

Configuring Thresholds

Set a global threshold for all steps, then optionally override per step:

{
  "profiling": {
    "enabled": true,
    "port": 9091,
    "slow_step_threshold_us": 1000,
    "alert_on_slow_step": true,
    "per_step_thresholds": {
      "cache": 500,
      "filter": 100,
      "logging": 50,
      "wasm:auth": 2000,
      "metrics": 80
    }
  }
}

How Detection Works

• Every invocation is checked against the configured threshold
• If the step latency exceeds the threshold, a counter is incremented and a log line is emitted
• The log line includes the step name, phase, actual latency, and the threshold it exceeded
• A Prometheus counter is exported so you can build alerts in Grafana or PagerDuty

Log Output

2025-01-15T14:32:01.123Z  WARN  nexo::profiling: slow step detected
    step=cache phase=request latency_us=1520 threshold_us=500

2025-01-15T14:32:01.456Z  WARN  nexo::profiling: slow step detected
    step=filter phase=request latency_us=890 threshold_us=100

2025-01-15T14:32:05.789Z  WARN  nexo::profiling: slow step detected
    step=wasm:auth phase=request latency_us=2100 threshold_us=2000

Prometheus Metrics

Nexo exports a set of Prometheus metrics for slow step detection:

## HELP nexo_step_slow_total Number of times a step exceeded its latency threshold
## TYPE nexo_step_slow_total counter
nexo_step_slow_total{step="cache",phase="request"} 42
nexo_step_slow_total{step="filter",phase="request"} 17
nexo_step_slow_total{step="wasm:auth",phase="request"} 3

## HELP nexo_step_latency_us Per-step latency histogram
## TYPE nexo_step_latency_us histogram
nexo_step_latency_us_bucket{step="cache",phase="request",le="100"} 8500
nexo_step_latency_us_bucket{step="cache",phase="request",le="250"} 14200
nexo_step_latency_us_bucket{step="cache",phase="request",le="500"} 15100
nexo_step_latency_us_bucket{step="cache",phase="request",le="1000"} 15220
nexo_step_latency_us_bucket{step="cache",phase="request",le="+Inf"} 15234
nexo_step_latency_us_sum{step="cache",phase="request"} 990210
nexo_step_latency_us_count{step="cache",phase="request"} 15234

Grafana Alert Rule Example

## Alert when any step exceeds its threshold more than 10 times in 5 minutes
groups:
  - name: nexo-slow-steps
    rules:
      - alert: NexoSlowStep
        expr: rate(nexo_step_slow_total[5m]) > 10
        for: 2m
        labels:
          severity: warning
        annotations:
          summary: "Nexo step {{ $labels.step }} is consistently slow"
          description: >
            Step {{ $labels.step }} in {{ $labels.phase }} phase
            exceeded its latency threshold {{ $value | humanize }}
            times per second over the last 5 minutes.

Optimization Tips

Once you have profiling data, here are concrete ways to reduce pipeline latency:

1. Step Ordering Matters

Put cheap filtering steps first in the pipeline. If a filter step rejects a request in 35μs, every subsequent step is skipped entirely. Place expensive steps (WASM plugins, cache lookups) after filters to short-circuit early.

// nexo.json — optimal step order
{
  "pipeline": {
    "request": [
      "filter",      // cheap: rejects disallowed ops early (35μs)
      "logging",     // cheap: records the request (12μs)
      "metrics",     // cheap: increments counters (18μs)
      "cache",       // moderate: cache lookup (65μs)
      "wasm:auth"    // expensive: WASM host calls (120μs)
    ],
    "response": [
      "metrics",
      "filter",
      "logging"
    ]
  }
}

2. Disable Unused Steps

Even an idle step that does nothing adds approximately 5μs of overhead per invocation due to the function call, context setup, and BSON traversal. If you are not using a step, remove it from the pipeline entirely.

## Before: 7 steps × 15,000 req/s = 525,000 step invocations/s
## After removing 2 unused steps: 5 steps × 15,000 = 375,000 invocations/s
## Savings: ~10μs per request = 150ms total per second

3. WASM Plugin Performance

WASM plugins run in a sandboxed runtime. Each host function call (e.g., reading a config value, checking the database) crosses the WASM boundary and adds overhead. Minimise the number of host function calls per invocation:

• Batch multiple reads into a single host call where possible
• Cache configuration values inside the WASM module instead of re-reading each time
• Use memory.grow sparingly — pre-allocate buffers
• Profile the WASM plugin separately with nexo profile --step wasm:auth

## Profile a specific WASM step in detail
nexo profile --step wasm:auth --duration 30s

Step: wasm:auth
  Total invocations:  4,521
  Avg latency:        120μs
  Host calls/invoke:  3.2
  Avg host call time: 28μs
  WASM compute time:  30μs
  Boundary overhead:  6μs × 3.2 = 19μs

4. Connection Pooling

Upstream connection setup time can dominate latency for short queries. Nexo maintains a connection pool to MongoDB. Tune the pool size to avoid connection churn:

{
  "upstream": {
    "uri": "mongodb://mongo:27017",
    "pool_size": 50,
    "min_idle": 10,
    "max_idle_time_ms": 30000,
    "connect_timeout_ms": 5000
  }
}

5. Monitor P99, Not Just Average

Average latency can hide problems. A step with 10μs average but 5000μs P99 has a serious tail-latency issue. Always sort the profiling table by P99 to find the real culprits.

## Example: average looks fine, P99 reveals the problem
Step       Avg    P50    P95     P99     Max
─────────────────────────────────────────────
cache      65μs   40μs   120μs   250μs   1520μs   ← P99 is 4x the average
wasm:auth  120μs  80μs   250μs   480μs   2100μs   ← max is 17x the average
filter     35μs   20μs   85μs    150μs   890μs    ← spikes from complex regexes

6. Benchmark Locally

Use the built-in benchmark command to establish a local performance baseline before deploying configuration changes:

## Run a benchmark with 100 concurrent connections
nexo bench --connections 100 --duration 30s --target mongodb://localhost:27017

Benchmark Results:
  Connections:    100
  Duration:       30s
  Total Requests: 245,320
  Throughput:     8,177 req/s
  Avg Latency:    12.2ms
  P50 Latency:    9.8ms
  P95 Latency:    25.1ms
  P99 Latency:    48.3ms
  Max Latency:    312.5ms
  Pipeline Overhead:
    Avg: 280μs (2.3% of total)
    P99: 1.2ms (2.5% of total)

OpenTelemetry Integration

Nexo supports exporting distributed traces via the OpenTelemetry Protocol (OTLP). This lets you visualise individual request traces in tools like Jaeger, Zipkin, Datadog, or Honeycomb and correlate proxy latency with your application's end-to-end traces.

Enabling OTLP Export

{
  "telemetry": {
    "otlp_endpoint": "http://otel-collector:4317",
    "service_name": "nexo-proxy",
    "sample_rate": 0.1,
    "propagation": "tracecontext",
    "resource_attributes": {
      "deployment.environment": "production",
      "service.version": "1.4.2",
      "service.namespace": "data-platform"
    }
  }
}

Trace Structure

Every sampled request produces a trace with the following span hierarchy:

• Root span — nexo.request — covers the entire request lifecycle
• Child spans (request phase) — one per pipeline step, e.g. nexo.step.filter
• Child span (upstream) — nexo.upstream — time spent communicating with MongoDB
• Child spans (response phase) — one per response pipeline step

Span Attributes

Span: nexo.step.filter
  Attributes:
    nexo.step.name       = "filter"
    nexo.step.phase      = "request"
    nexo.step.duration_us = 35
    nexo.step.error      = false
    nexo.step.order      = 1
    db.system            = "mongodb"
    db.operation         = "find"
    db.mongodb.collection = "users"
    net.peer.name        = "mongo-primary.internal"
    net.peer.port        = 27017

Compatible Backends

• Jaeger — full support, recommended for self-hosted deployments
• Zipkin — supported via OTLP-to-Zipkin bridge in the OTel collector
• Datadog — native OTLP ingestion, maps Nexo spans to Datadog APM traces
• Honeycomb — excellent for high-cardinality exploration of step attributes
• Grafana Tempo — pairs well with the Prometheus metrics Nexo exports
• AWS X-Ray — supported via the OTel collector's X-Ray exporter

Example Trace

Trace ID: 4bf92f3577b34da6a3ce929d0e0e4736

nexo.request (1250μs)
  - nexo.step.logging [req] (12μs)
  - nexo.step.filter [req] (35μs)
    - regex.eval (22μs)
  - nexo.step.cache [req] (65μs)
    - cache.lookup (30μs)
    - bson.deserialize (25μs)
  - nexo.step.metrics [req] (18μs)
  - nexo.upstream (980μs)
    - mongodb.find (950μs)
    - connection.acquire (30μs)
  - nexo.step.metrics [resp] (15μs)
  - nexo.step.filter [resp] (30μs)
    - field.redact (20μs)
  - nexo.step.logging [resp] (10μs)

Trace Context Propagation

Nexo propagates the W3C traceparent header through the MongoDB wire protocol using a custom command wrapper. This means your application traces can link directly to the Nexo proxy spans:

Application span: app.api.getUser (45ms)
  mongodb.find (12ms)
    linked Nexo request span (1.25ms)
      - nexo.step.filter
      - nexo.upstream
      - nexo.step.logging

Profiling in Production

Nexo's profiling subsystem is designed for always-on production use. The overhead is minimal — typically less than 2% CPU — because statistics are maintained with lock-free atomic counters and pre-allocated histogram buckets.

Recommended Production Settings

{
  "profiling": {
    "enabled": true,
    "port": 9091,
    "slow_step_threshold_us": 1000,
    "alert_on_slow_step": true,
    "histogram_buckets": [10, 25, 50, 100, 250, 500, 1000, 2500, 5000, 10000],
    "snapshot_interval_s": 30,
    "snapshot_retention": "24h"
  },
  "telemetry": {
    "otlp_endpoint": "http://otel-collector:4317",
    "service_name": "nexo-proxy",
    "sample_rate": 0.01,
    "propagation": "tracecontext"
  }
}

Performance Impact

## Benchmark: profiling OFF vs ON (100 concurrent connections, 30s)
Throughput (rps): 8,320 -> 8,180 (-1.7%)
Avg Latency: 12.0ms -> 12.2ms (+1.7%)
P99 Latency: 47.1ms -> 48.3ms (+2.5%)
CPU Usage: 34.2% -> 34.9% (+0.7%)
Memory (RSS): 82 MB -> 88 MB (+7.3%)

Sampling for High-Traffic Deployments

For OTLP export on high-traffic proxies (> 10 000 req/s), reduce the sample rate to avoid overwhelming the collector:

• Development — sample_rate: 1.0 (trace every request)
• Staging — sample_rate: 0.1 (10% of requests)
• Production — sample_rate: 0.01 (1% of requests)
• High traffic (>100k rps) — sample_rate: 0.001 (0.1%)

Dashboard Auto-Refresh

The Nexo dashboard automatically refreshes profiling data every 5 seconds when the profiling tab is active. Snapshot data is persisted to disk every 30 seconds, so you retain up to 24 hours of historical data by default.

Exporting Data

For long-term analysis beyond the 24-hour retention window, export profiling data:

## Export profiling snapshots to JSON Lines format
nexo profile --export --format jsonl --output profile-data.jsonl

## Export to CSV for spreadsheet analysis
nexo profile --export --format csv --output profile-data.csv

## Stream to a remote endpoint
nexo profile --export --format jsonl | curl -X POST \
  -H "Content-Type: application/x-ndjson" \
  --data-binary @- \
  https://analytics.example.com/ingest/nexo

CLI Profiling

The nexo profile command lets you capture and analyse profiling data directly from the terminal — no dashboard required.

One-Shot Profile

## Profile for 60 seconds and write a JSON report
nexo profile --duration 60s --output report.json

Profiling for 60s... ████████████████████████████████████████ 100%

Wrote report to report.json
  Total requests:    92,410
  Avg total latency: 1,180μs
  Pipeline overhead: 260μs (22.0%)
  Slowest step:      wasm:auth (P99: 480μs)

Live Tail of Slow Steps

## Watch for any step invocation exceeding 500μs in real time
nexo profile --live --threshold 500us

[14:32:01.123] SLOW  cache       request   1520μs  (threshold: 500μs)
[14:32:01.456] SLOW  wasm:auth   request    890μs  (threshold: 500μs)
[14:32:05.789] SLOW  cache       request   1200μs  (threshold: 500μs)
[14:32:08.012] SLOW  wasm:auth   request   2100μs  (threshold: 500μs)
[14:32:12.345] SLOW  filter      request    720μs  (threshold: 500μs)
[14:32:15.678] SLOW  cache       request    980μs  (threshold: 500μs)
^C
Summary (14.6s):
  Total slow invocations: 6
  Most frequent: cache (3 times)
  Slowest: wasm:auth (2100μs)

Generate Flame Graph SVG

## Capture 30 seconds of data and render a flame graph
nexo profile --flamegraph --duration 30s --output flame.svg

Profiling for 30s... ████████████████████████████████████████ 100%
Generating flame graph...

Wrote flame.svg (248 KB)
  Open in browser: file:///path/to/flame.svg
  Or serve: python3 -m http.server 8080

Step-Level Detail

## Deep-dive into a specific step
nexo profile --step cache --duration 30s

Step: cache (request phase)
──────────────────────────────────
  Invocations:     45,230
  Hit Rate:        78.3%
  Miss Rate:       21.7%

  Latency Distribution:
    Avg:     65μs
    P50:     40μs
    P95:    120μs
    P99:    250μs
    Max:   1520μs

  Sub-operations:
    cache.lookup:       30μs avg  (45,230 calls)
    bson.deserialize:   25μs avg  (35,415 calls — hits only)
    bson.serialize:     18μs avg  ( 9,815 calls — misses only)
    upstream.store:      8μs avg  ( 9,815 calls — misses only)

  Latency Histogram:
    [  0μs -  25μs] ████████████████░░░░░░░░░░░░░░░░  35.2%
    [ 25μs -  50μs] ████████████████████░░░░░░░░░░░░  42.1%
    [ 50μs - 100μs] ████████░░░░░░░░░░░░░░░░░░░░░░░░  15.3%
    [100μs - 250μs] ███░░░░░░░░░░░░░░░░░░░░░░░░░░░░░   5.8%
    [250μs - 500μs] █░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░   1.2%
    [500μs -  1ms]  ░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░   0.3%
    [  1ms -  2ms]  ░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░   0.1%

Compare Configurations

## A/B test two pipeline configurations
nexo profile --duration 30s --output baseline.json
## ... change nexo.json ...
nexo profile --duration 30s --output experiment.json

nexo profile --compare baseline.json experiment.json

Pipeline Comparison:
- Throughput: 8,177 rps -> 9,320 rps (+14.0%)
- Avg Latency: 1,250μs -> 1,080μs (-13.6%)
- P99 Latency: 4,410μs -> 3,200μs (-27.4%)
- Pipeline Overhead: 270μs -> 180μs (-33.3%)
- Error Rate: 0.01% -> 0.01% (0.0%)

Per-Step Changes:
- cache: 65μs -> 45μs (-30.8%) — switched to LRU eviction
- wasm:auth: 120μs -> removed — moved upstream into the application service
- filter: 35μs -> 35μs (0.0%) — unchanged

Advanced Profiling Topics

Request-Level Trace Headers

You can request per-request profiling data by setting a custom header in your MongoDB driver configuration. Nexo will attach profiling data to the response metadata:

// In your MongoDB driver options, set appName to include the trace flag
const client = new MongoClient("mongodb://nexo-proxy:27017", {
  appName: "myapp?nexo_profile=true"
});

// The response metadata will include:
// {
//   "nexo_profile": {
//     "total_us": 1250,
//     "upstream_us": 980,
//     "steps": [
//       { "name": "filter", "phase": "request", "us": 35 },
//       { "name": "cache", "phase": "request", "us": 65 },
//       ...
//     ]
//   }
// }

Conditional Profiling

Profile only specific operations or collections to reduce noise:

{
  "profiling": {
    "enabled": true,
    "port": 9091,
    "filters": {
      "operations": ["find", "aggregate", "update"],
      "collections": ["users", "orders"],
      "min_upstream_latency_us": 1000
    }
  }
}

With conditional profiling, only requests matching the filter criteria are sampled. This is useful for focusing on slow queries or high-value collections without the noise of fast, simple operations.

Memory-Mapped Histogram Storage

For zero-allocation profiling in ultra-high-throughput deployments, Nexo can use memory-mapped files for histogram storage:

{
  "profiling": {
    "enabled": true,
    "storage": "mmap",
    "mmap_path": "/dev/shm/nexo-profile",
    "mmap_size_mb": 64
  }
}

This eliminates heap allocations for profiling data and allows external tools to read the histogram data directly from shared memory without going through the HTTP API.

Distributed Profiling

When running multiple Nexo instances behind a load balancer, aggregate profiling data across all instances:

## Aggregate profiles from multiple instances
nexo profile --aggregate \
  --instances nexo-1:9091,nexo-2:9091,nexo-3:9091 \
  --duration 60s \
  --output aggregate-report.json

Aggregating from 3 instances...

Instance Summary:
- nexo-1: requests 31,204, avg 1,180μs, p99 4,200μs, errors 12
- nexo-2: requests 30,876, avg 1,250μs, p99 4,410μs, errors 15
- nexo-3: requests 30,330, avg 1,210μs, p99 4,350μs, errors 11
- aggregate: requests 92,410, avg 1,213μs, p99 4,320μs, errors 38

Troubleshooting Profiling

Common Issues

Profiling endpoint returns 404

## Check that profiling is enabled in your config
cat nexo.json | jq '.profiling.enabled'
## Should output: true

## Check that the profiling port is not blocked
curl -v http://localhost:9091/api/profile
## If connection refused, verify the port in nexo.json

Latency numbers seem too high

• Check if you are running in debug mode — debug builds add significant overhead
• Verify that the proxy is not under memory pressure (check RSS vs available RAM)
• Look for GC pauses if using a WASM runtime with garbage collection
• Check for noisy neighbours on shared infrastructure

OTLP traces not appearing in Jaeger

## 1. Verify the collector is reachable
curl -v http://otel-collector:4317

## 2. Check Nexo logs for OTLP export errors
nexo logs --filter telemetry

## 3. Verify sample rate is not too low
cat nexo.json | jq '.telemetry.sample_rate'
## For debugging, temporarily set to 1.0

## 4. Check the collector's own logs
docker logs otel-collector 2>&1 | grep -i error

Flame graph is empty or too sparse

• Increase the sampling duration: --duration 120s
• Lower the minimum width: ?min_width=0.01
• Ensure there is active traffic during the profiling window
• Check that the proxy is processing requests (not just health checks)

Legacy footer navigation:

• ← Pipeline Architecture
• WASM Plugins →

Legacy footer label: Nexo Documentation — Latency Profiling