Version: 0.2.0 (Private Preview)

Canary deployments

:::danger Unavailable in Nexo 0.2.0

The pipeline-level canary controller, per-request traffic splitter, canary API, nexo canary commands, automatic promotion, and automatic rollback described below are not implemented in Nexo 0.2.0. Use the supported console deployment flow, verify the active graph, and perform an explicit configuration or release rollback. In 0.2.0, router-phase traffic-split behavior is not executed.

:::

Roll out pipeline changes gradually, monitor their impact in real time, and automatically roll back if anything goes wrong — all without provisioning additional infrastructure.

Legacy UI feature labels:

Pipeline-Level Canary
Auto-Rollback
Progressive Rollout

What Canary Means in Nexo

Traditional canary deployments work by sending a fraction of traffic to a new binary running on a small subset of servers. If the new binary misbehaves, traffic is shifted back to the old binary. This requires spinning up new infrastructure, deploying an updated application, and managing two separate pools of servers.

Nexo takes a fundamentally different approach. Because all request handling is defined by pipelines — ordered sequences of filter, transform, and routing steps — a canary deployment in Nexo means deploying a new pipeline configuration alongside the existing one. Both pipelines run on the exact same proxy instances, and the proxy's traffic splitter decides which pipeline handles each incoming request.

Key Differences from Traditional Canary

No extra infrastructure: Both baseline and candidate pipelines run on every proxy instance. There is no need for additional servers, containers, or load balancers.
Pipeline-level granularity: You are not deploying a new version of the proxy binary. You are deploying a new configuration — a different sequence of processing steps.
Per-request splitting: Traffic is split at the request level, not the connection level. This gives you statistically accurate traffic percentages even with long-lived connections.
Same upstream: Both pipelines ultimately talk to the same upstream MongoDB deployment, so you can compare apples to apples.
Instant rollback: Rolling back means routing 100% of traffic back to the baseline pipeline. No pods to scale down, no deploys to revert.

When to Use Canary

Canary deployments are ideal whenever you are making a non-trivial change to your pipeline and want to validate it against real production traffic:

Adding or modifying a filter step (e.g., a new rate limiter, auth check, or field redaction filter)
Installing a new marketplace plugin into the pipeline
Changing routing rules (e.g., directing certain queries to a secondary read replica)
Updating rate-limit thresholds or circuit-breaker settings
Testing a new transform step that rewrites queries before they reach MongoDB
Validating performance after changing the order of pipeline steps

💡 Tip: Canary is especially valuable when testing marketplace plugins from third-party authors. Even if a plugin passed automated review, you should canary it before routing all production traffic through it.

How It Works

When you start a canary, the proxy loads two pipeline configurations into memory: the baseline (the currently active pipeline) and the candidate (the new pipeline you want to test). A traffic splitter sits in front of both pipelines and makes a routing decision for every incoming request.

Traffic Splitting

For each incoming MongoDB wire-protocol request, the proxy generates a random number between 0 and 100. If the number is below the configured traffic percentage, the request is routed to the candidate pipeline; otherwise, it goes to the baseline. This is a per-request decision, not per-connection.

Per-request splitting is important because MongoDB connections are typically long-lived (pooled by the application driver). If you split at the connection level, a single busy connection could skew the traffic ratio significantly. By splitting per-request, each individual operation has an independent probability of landing on the candidate.

Architecture Diagram

Detailed Request Flow

Let's trace a single request through the canary architecture:

A MongoDB driver sends a wire-protocol message (e.g., OP_MSG containing a find command) to the Nexo proxy.
The proxy's connection handler accepts the message and passes it to the traffic splitter.
The splitter generates a random value. At 5% canary traffic, roughly 1 in 20 requests will be routed to the candidate pipeline.
The chosen pipeline (baseline or candidate) processes the request through its configured steps: auth → rate-limit → transform → route → upstream.
The response from MongoDB is sent back through the same pipeline (for response-phase filters) and returned to the client.
Metrics for the request (latency, status, error) are tagged with either pipeline=baseline or pipeline=candidate so they can be compared independently.

Metrics Collection

The proxy emits separate metric streams for baseline and candidate traffic. All standard proxy metrics are duplicated with a pipeline label:

## Baseline metrics
nexo_requests_total{pipeline="baseline", status="ok"} 12450
nexo_request_duration_seconds{pipeline="baseline", quantile="0.5"} 0.0012
nexo_request_duration_seconds{pipeline="baseline", quantile="0.95"} 0.0058
nexo_request_duration_seconds{pipeline="baseline", quantile="0.99"} 0.0123

## Candidate metrics
nexo_requests_total{pipeline="candidate", status="ok"} 4147
nexo_requests_total{pipeline="candidate", status="error"} 3
nexo_request_duration_seconds{pipeline="candidate", quantile="0.5"} 0.0013
nexo_request_duration_seconds{pipeline="candidate", quantile="0.95"} 0.0061
nexo_request_duration_seconds{pipeline="candidate", quantile="0.99"} 0.0130

These metrics feed into the rollback rule engine, auto-promote evaluator, and the dashboard comparison widget.

Progressive Rollout

Rather than jumping from 0% to 100%, a progressive rollout moves through a series of stages, each sending a larger percentage of traffic to the candidate. This gives you increasing confidence that the new pipeline is safe before committing fully.

Typical Progression

A common pattern is four stages: 5% → 25% → 50% → 100%. Each stage has a minimum duration and a minimum number of requests that must pass through the candidate before advancing to the next stage. This ensures that you don't promote prematurely based on an insufficient sample size.

Stage 1 — 5% traffic: Smoke test. Catch catastrophic failures (crashes, auth errors, connection leaks) with minimal blast radius.
Stage 2 — 25% traffic: Broader validation. Enough traffic to see statistically meaningful latency and error-rate differences.
Stage 3 — 50% traffic: Near-production load. Validates that the candidate can handle half of your real traffic without degradation.
Stage 4 — 100% traffic: Full promotion. The candidate becomes the new baseline. The old baseline is retained for quick manual rollback.

Stage Configuration

Stages are defined as an array in the canary configuration. Each stage specifies the traffic percentage, the minimum time to remain at that stage, and the minimum number of requests the candidate must handle before promotion is allowed.

{
  "canary": {
    "stages": [
      { "traffic_pct": 5, "min_duration": "10m", "min_requests": 1000 },
      { "traffic_pct": 25, "min_duration": "30m", "min_requests": 5000 },
      { "traffic_pct": 50, "min_duration": "1h", "min_requests": 10000 },
      { "traffic_pct": 100, "min_duration": "0", "min_requests": 0 }
    ],
    "auto_promote": true
  }
}

⚠️ Note: The final stage (100%) typically has min_duration: "0" and min_requests: 0 because once you reach 100%, the canary is effectively complete. The old baseline is retained in memory for a configurable grace period (default 15 minutes) in case you need to manually roll back.

Manual vs. Automatic Promotion

You can choose between manual and automatic promotion between stages:

Manual promotion ("auto_promote": false): The canary pauses at each stage and waits for an operator to explicitly promote it via the API or CLI. This is ideal for high-risk changes or the first time you deploy a new marketplace plugin.
Automatic promotion ("auto_promote": true): The canary automatically advances to the next stage when all of the auto-promote conditions are met (see "Auto-Promote Rules" below). This is ideal for routine configuration changes where you have high confidence in your rollback rules.

Custom Stage Schedules

You are not limited to four stages. You can define as many or as few as you like. For a very conservative rollout of a critical pipeline change, you might use:

{
  "canary": {
    "stages": [
      { "traffic_pct": 1, "min_duration": "30m", "min_requests": 500 },
      { "traffic_pct": 5, "min_duration": "1h", "min_requests": 3000 },
      { "traffic_pct": 10, "min_duration": "2h", "min_requests": 10000 },
      { "traffic_pct": 25, "min_duration": "2h", "min_requests": 25000 },
      { "traffic_pct": 50, "min_duration": "4h", "min_requests": 50000 },
      { "traffic_pct": 75, "min_duration": "2h", "min_requests": 50000 },
      { "traffic_pct": 100, "min_duration": "0", "min_requests": 0 }
    ],
    "auto_promote": false
  }
}

Conversely, for a low-risk change (e.g., adjusting a log-level filter), you might use just two stages:

{
  "canary": {
    "stages": [
      { "traffic_pct": 50, "min_duration": "5m", "min_requests": 500 },
      { "traffic_pct": 100, "min_duration": "0", "min_requests": 0 }
    ],
    "auto_promote": true
  }
}

Rollback Rules

Rollback rules define the conditions under which the canary is automatically aborted and all traffic is returned to the baseline pipeline. These rules run continuously in the background while a canary is active. They compare the candidate's metrics against the baseline's metrics in real time.

Available Thresholds

Error rate threshold (error_rate_delta_pct): Rollback if the candidate's error rate exceeds the baseline's error rate by more than this percentage. For example, if the baseline has a 0.02% error rate and the threshold is 5.0, rollback triggers if the candidate exceeds 5.02%.
Latency threshold (latency_p95_delta_ms): Rollback if the candidate's P95 latency exceeds the baseline's P95 latency by more than this many milliseconds. This catches performance regressions.
Sustained duration (sustained_duration): The condition must persist for this long before rollback triggers. This prevents flapping caused by momentary spikes. A single slow request should not cause an immediate rollback.
Check interval (check_interval): How frequently the rollback engine evaluates the metrics. Shorter intervals mean faster detection but slightly more overhead.

Rollback Configuration

{
  "rollback_rules": {
    "error_rate_delta_pct": 5.0,
    "latency_p95_delta_ms": 50,
    "sustained_duration": "30s",
    "check_interval": "10s"
  }
}

How Rollback Evaluation Works

Every check_interval seconds, the rollback engine performs the following:

Fetch the current error rate and P95 latency for both baseline and candidate from the metrics store.
Compute the delta: candidate_metric - baseline_metric.
If the delta exceeds the threshold, start (or continue) a sustained-duration timer.
If the delta drops below the threshold before the sustained duration elapses, reset the timer.
If the timer reaches sustained_duration, execute the rollback.

What Happens on Rollback

When rollback triggers, the following sequence occurs:

The traffic splitter immediately routes 100% of traffic to the baseline pipeline. Any in-flight requests on the candidate are allowed to complete.
The candidate pipeline configuration is removed from memory.
The canary status is updated to rolled_back.
An alert is sent via the configured notification channels (webhook, email, Slack, PagerDuty, etc.).
A detailed rollback report is generated containing the metrics that triggered the rollback, a timeline of the canary's stages, and the final metric comparison.

🚨 Important: Rollback is immediate. Once triggered, there is no "undo rollback" action. If you want to retry the candidate, you must start a new canary deployment.

Advanced Rollback: Custom Metrics

In addition to the built-in error rate and latency thresholds, you can define rollback rules based on custom metrics emitted by your pipeline plugins:

{
  "rollback_rules": {
    "error_rate_delta_pct": 5.0,
    "latency_p95_delta_ms": 50,
    "sustained_duration": "30s",
    "check_interval": "10s",
    "custom_metrics": [
      {
        "metric": "nexo_plugin_cache_miss_rate",
        "threshold_delta": 0.1,
        "comparison": "greater_than"
      },
      {
        "metric": "nexo_plugin_auth_rejects_total",
        "threshold_delta": 10,
        "comparison": "greater_than"
      }
    ]
  }
}

Auto-Promote Rules

When auto_promote is enabled, the canary engine evaluates promotion conditions at each check interval. If all conditions are met, the canary automatically advances to the next stage.

Promotion Conditions

All of the following conditions must be true for auto-promotion to occur:

Minimum duration: The canary has been at the current stage for at least min_duration.
Minimum requests: The candidate has processed at least min_requests since entering the current stage.
Error rate delta: The candidate's error rate does not exceed the baseline by more than max_error_rate_delta_pct.
Latency delta: The candidate's P95 latency does not exceed the baseline by more than max_latency_p95_delta_ms.

Auto-Promote Configuration

{
  "auto_promote_rules": {
    "min_duration": "10m",
    "min_requests": 1000,
    "max_error_rate_delta_pct": 1.0,
    "max_latency_p95_delta_ms": 20
  }
}

Notice that the auto-promote thresholds are typically tighter than the rollback thresholds. Rollback catches catastrophic regressions (5% error rate delta), while auto-promote requires near-parity (1% error rate delta) before advancing.

Promotion Timeline Example

Here's an example timeline for a canary with auto-promote enabled:

T+0m    Canary started at Stage 1 (5% traffic)
T+10m   min_duration met, min_requests met, deltas within limits → promote to Stage 2
T+10m   Stage 2 (25% traffic) begins
T+40m   min_duration met, min_requests met, deltas within limits → promote to Stage 3
T+40m   Stage 3 (50% traffic) begins
T+1h40m min_duration met, min_requests met, deltas within limits → promote to Stage 4
T+1h40m Stage 4 (100% traffic) — canary complete
T+1h40m Baseline is replaced with candidate. Old baseline retained for 15m grace period.

Combining Auto-Promote with Manual Gates

You can mix automatic and manual promotion by setting auto_promote to true but adding a manual_gate flag on specific stages:

{
  "canary": {
    "stages": [
      { "traffic_pct": 5, "min_duration": "10m", "min_requests": 1000 },
      { "traffic_pct": 25, "min_duration": "30m", "min_requests": 5000, "manual_gate": true },
      { "traffic_pct": 50, "min_duration": "1h", "min_requests": 10000 },
      { "traffic_pct": 100, "min_duration": "0", "min_requests": 0 }
    ],
    "auto_promote": true
  }
}

In this configuration, the canary auto-promotes from Stage 1 to Stage 2, then pauses and waits for manual approval before moving to Stage 3. Stages 3 to 4 are again automatic.

API Reference

All canary operations are available through the Nexo control-plane REST API. All endpoints require authentication via a bearer token.

Start a Canary

Creates a new canary deployment for a given pipeline.

POST /api/v1/canary
Content-Type: application/json
Authorization: Bearer <token>

{
  "pipeline_id": "pipe-abc123",
  "candidate_config": {
    "steps": [
      { "type": "rate_limit", "config": { "max_rps": 500 } },
      { "type": "auth", "config": { "provider": "oidc" } },
      { "type": "transform", "plugin_id": "plg-query-rewrite-v2" },
      { "type": "route", "config": { "upstream": "mongodb://primary:27017" } }
    ]
  },
  "initial_traffic_pct": 5,
  "rollback_rules": {
    "error_rate_delta_pct": 5.0,
    "latency_p95_delta_ms": 50,
    "sustained_duration": "30s",
    "check_interval": "10s"
  },
  "stages": [
    { "traffic_pct": 5, "min_duration": "10m", "min_requests": 1000 },
    { "traffic_pct": 25, "min_duration": "30m", "min_requests": 5000 },
    { "traffic_pct": 50, "min_duration": "1h", "min_requests": 10000 },
    { "traffic_pct": 100, "min_duration": "0", "min_requests": 0 }
  ]
}

Response:

HTTP/1.1 201 Created

{
  "canary_id": "cnry-7f3a9b2e",
  "pipeline_id": "pipe-abc123",
  "status": "active",
  "current_stage": 1,
  "traffic_pct": 5,
  "created_at": "2025-01-15T10:30:00Z",
  "baseline_version": "v2.0.3",
  "candidate_version": "v2.1.0",
  "metrics_url": "/api/v1/canary/cnry-7f3a9b2e/metrics"
}

Get Canary Status

Retrieves the current status and metrics for an active canary deployment.

GET /api/v1/canary/{canary_id}
Authorization: Bearer <token>

Response:

HTTP/1.1 200 OK

{
  "canary_id": "cnry-7f3a9b2e",
  "pipeline_id": "pipe-abc123",
  "status": "active",
  "current_stage": 2,
  "traffic_pct": 25,
  "started_at": "2025-01-15T10:30:00Z",
  "stage_entered_at": "2025-01-15T10:40:00Z",
  "elapsed": "45m",
  "metrics": {
    "baseline": {
      "requests": 12450,
      "error_rate_pct": 0.02,
      "latency_p50_ms": 1.2,
      "latency_p95_ms": 5.8,
      "latency_p99_ms": 12.3
    },
    "candidate": {
      "requests": 4150,
      "error_rate_pct": 0.03,
      "latency_p50_ms": 1.3,
      "latency_p95_ms": 6.1,
      "latency_p99_ms": 13.0
    }
  },
  "rollback_status": {
    "error_rate_sustained_for": "0s",
    "latency_sustained_for": "0s",
    "healthy": true
  }
}

Promote to Next Stage

Manually promotes the canary to the next stage. Only valid when auto-promote is disabled or the canary is paused at a manual gate.

POST /api/v1/canary/{canary_id}/promote
Authorization: Bearer <token>

Response:

HTTP/1.1 200 OK

{
  "canary_id": "cnry-7f3a9b2e",
  "previous_stage": 2,
  "current_stage": 3,
  "traffic_pct": 50,
  "promoted_at": "2025-01-15T11:15:00Z",
  "message": "Promoted from stage 2 (25%) to stage 3 (50%)"
}

Manual Rollback

Immediately rolls back the canary to the baseline, regardless of current metrics.

POST /api/v1/canary/{canary_id}/rollback
Authorization: Bearer <token>

{
  "reason": "Noticed unusual query patterns in candidate logs"
}

Response:

HTTP/1.1 200 OK

{
  "canary_id": "cnry-7f3a9b2e",
  "status": "rolled_back",
  "rolled_back_at": "2025-01-15T11:20:00Z",
  "reason": "Noticed unusual query patterns in candidate logs",
  "triggered_by": "manual",
  "final_metrics": {
    "baseline": { "requests": 18200, "error_rate_pct": 0.02, "latency_p95_ms": 5.9 },
    "candidate": { "requests": 6050, "error_rate_pct": 0.04, "latency_p95_ms": 6.3 }
  }
}

Update Traffic Percentage

Manually adjusts the traffic percentage without changing the stage. Useful for fine-grained control during manual canary management.

PATCH /api/v1/canary/{canary_id}
Authorization: Bearer <token>
Content-Type: application/json

{
  "traffic_pct": 25
}

Response:

HTTP/1.1 200 OK

{
  "canary_id": "cnry-7f3a9b2e",
  "traffic_pct": 25,
  "updated_at": "2025-01-15T11:25:00Z",
  "message": "Traffic percentage updated to 25%"
}

List Active Canaries

Returns all currently active canary deployments.

GET /api/v1/canary?status=active
Authorization: Bearer <token>

Response:

HTTP/1.1 200 OK

{
  "canaries": [
    {
      "canary_id": "cnry-7f3a9b2e",
      "pipeline_id": "pipe-abc123",
      "status": "active",
      "current_stage": 2,
      "traffic_pct": 25,
      "started_at": "2025-01-15T10:30:00Z"
    }
  ],
  "total": 1
}

The Nexo dashboard includes a real-time canary comparison widget that shows baseline and candidate metrics side by side. The widget updates every 5 seconds and provides instant visibility into the health of your canary deployment.

Metrics Displayed

Request rate: Total requests processed by each pipeline
Error rate: Percentage of requests that resulted in an error
P50 latency: Median response time
P95 latency: 95th percentile response time
P99 latency: 99th percentile response time

The dashboard shows a side-by-side comparison view with action buttons at the bottom:

Real-time updates: Metrics refresh every 5 seconds via WebSocket subscription.
Color-coded deltas: If a candidate metric is worse than the baseline, it is highlighted in red. If it is better, it is green. Neutral differences remain white.
Rollback timer: When a rollback threshold is being approached, the widget shows a warning indicator with the sustained duration counter.
Historical chart: Click any metric to expand a time-series chart comparing baseline and candidate over the lifetime of the canary.
Action buttons: Promote, rollback, and edit rules directly from the widget without navigating to a separate page.

You can embed the canary comparison widget in your own dashboards via an iframe or the Nexo React component:

import { CanaryWidget } from '@nexo/dashboard-components';

function MyDashboard() {
  return (
    <CanaryWidget
      canaryId="cnry-7f3a9b2e"
      refreshInterval={5000}
      showActions={true}
      theme="dark"
    />
  );
}

CLI Commands

The Nexo CLI provides a complete set of commands for managing canary deployments from the terminal.

Start a Canary

## Start a canary with auto-promote enabled
nexo canary start --pipeline pipe-abc123 \
  --candidate-config new-config.json \
  --traffic 5 \
  --auto-promote

## Start with custom stages from a file
nexo canary start --pipeline pipe-abc123 \
  --candidate-config new-config.json \
  --stages stages.json \
  --rollback-rules rollback.json

## Start a manual canary (no auto-promote)
nexo canary start --pipeline pipe-abc123 \
  --candidate-config new-config.json \
  --traffic 5

Check Status

## Get status of a specific canary
nexo canary status pipe-abc123

## Get detailed status with metrics
nexo canary status pipe-abc123 --verbose

## Watch status in real-time (refreshes every 5s)
nexo canary status pipe-abc123 --watch

Example output:

Canary cnry-7f3a9b2e (pipe-abc123)
  Status:     active
  Stage:      2/4 (25% traffic)
  Duration:   45m
  Baseline:   12,450 requests | 0.02% errors | P95 5.8ms
  Candidate:  4,150 requests  | 0.03% errors | P95 6.1ms
  Health:     ✓ All metrics within thresholds
  Next:       Auto-promote in ~15m (pending min_requests)

Promote and Rollback

## Manually promote to next stage
nexo canary promote pipe-abc123

## Force promote (skip min_duration / min_requests checks)
nexo canary promote pipe-abc123 --force

## Rollback with a reason
nexo canary rollback pipe-abc123 --reason "elevated error rates in logs"

## Rollback all active canaries (emergency)
nexo canary rollback --all --reason "global incident"

List Active Canaries

## List all active canaries
nexo canary list

## List with JSON output for scripting
nexo canary list --output json

## List including completed/rolled-back canaries
nexo canary list --all

Example output:

ID                PIPELINE       STAGE  TRAFFIC  STATUS   STARTED
cnry-7f3a9b2e     pipe-abc123    2/4    25%      active   45m ago
cnry-1a2b3c4d     pipe-xyz789    3/3    50%      active   2h ago

Inspect Canary History

## View the history of a completed canary
nexo canary history cnry-7f3a9b2e

## Export metrics as CSV
nexo canary metrics cnry-7f3a9b2e --format csv --output metrics.csv

Best Practices

Follow these guidelines to get the most out of Nexo's canary deployment system:

1. Start Small

Always begin your canary at 5% traffic or lower. This minimizes the blast radius if the candidate has a severe issue. Even at 5%, you'll process enough requests within a few minutes to get statistically meaningful results on most production workloads.

2. Set Conservative Rollback Thresholds

When in doubt, use tighter thresholds. It is far better to roll back a healthy candidate (false positive) than to let a broken candidate reach 100% traffic (false negative). You can always re-deploy with looser thresholds once you understand the candidate's behavior.

// Conservative rollback rules (recommended starting point)
{
  "rollback_rules": {
    "error_rate_delta_pct": 2.0,
    "latency_p95_delta_ms": 20,
    "sustained_duration": "20s",
    "check_interval": "5s"
  }
}

3. Monitor for Sufficient Duration

Don't rush through stages. Allow at least 10 minutes per stage to collect enough data for a meaningful comparison. Some issues only manifest under sustained load — a candidate might look fine for the first few minutes but degrade after connections pool up or caches warm.

4. One Canary at a Time

Do not run canary deployments on multiple pipelines simultaneously. If you canary Pipeline A and Pipeline B at the same time, and metrics degrade, you won't know which candidate caused the issue. Run canaries sequentially and isolate variables.

⚠️ Warning: Nexo will prevent you from starting a second canary on the same pipeline, but it cannot prevent you from starting canaries on different pipelines that share the same upstream. Use operational discipline to avoid concurrent canaries on related pipelines.

5. Test on Staging First

Before canarying on production, run the same candidate configuration through your staging environment. Staging may not have production-level traffic, but it will catch configuration errors, plugin compatibility issues, and other obvious problems.

6. Use Auto-Promote for Routine Changes

For well-understood, low-risk changes (e.g., bumping a rate limit, updating a plugin to a patch version), use auto-promote to reduce operator toil. For high-risk changes (e.g., adding a brand-new pipeline step, installing an untested marketplace plugin), use manual promotion so a human reviews metrics at each stage.

7. Keep Rollback Detection Fast

Configure sustained_duration to be under 30 seconds and check_interval to be 5–10 seconds. The faster you detect a problem, the fewer requests are affected. With a 5% traffic split and 20-second detection, you limit exposure to a small number of requests.

8. Document Candidate Changes

Always document what changed in the candidate configuration. When you start a canary, include a description:

nexo canary start --pipeline pipe-abc123 \
  --candidate-config new-config.json \
  --traffic 5 \
  --auto-promote \
  --description "Added query-rewrite plugin v2.1.0 to transform step. \
    Expected to reduce P95 latency by ~10% for aggregation queries."

This description appears in the canary history and rollback reports, making post-mortems significantly easier.

Notifications and Alerting

Nexo can send notifications at key canary lifecycle events. Configure notification channels in your project settings or pass them inline when starting a canary.

Supported Events

canary.started: A new canary deployment has begun
canary.promoted: Canary advanced to the next stage
canary.rolled_back: Canary was rolled back (auto or manual)
canary.completed: Canary reached 100% and candidate is now the baseline
canary.threshold_warning: A rollback threshold is being approached

Notification Configuration

{
  "notifications": {
    "channels": [
      {
        "type": "webhook",
        "url": "https://hooks.slack.com/services/T00/B00/xxx",
        "events": ["canary.rolled_back", "canary.completed"]
      },
      {
        "type": "email",
        "addresses": ["oncall@example.com"],
        "events": ["canary.rolled_back"]
      },
      {
        "type": "pagerduty",
        "integration_key": "abc123",
        "events": ["canary.rolled_back"],
        "severity": "warning"
      }
    ]
  }
}

Troubleshooting

Canary Starts but No Traffic Reaches Candidate

Verify that the candidate pipeline configuration is valid and the proxy successfully loaded both pipelines. Check the proxy logs for pipeline loading errors:

nexo logs --filter "canary" --level error --tail 100

Rollback Triggers Immediately

If rollback triggers within the first few seconds, it's usually because the candidate pipeline has a configuration error causing 100% error rate. Check:

Plugin IDs in the candidate config are valid and installed
Upstream connection strings are correct
Auth provider settings match your environment
Rate-limit settings are not set to zero

Metrics Show No Difference

If baseline and candidate metrics are identical, the change may not affect the metrics you're monitoring. This is actually a good sign — it means the candidate is performing identically to the baseline. Promote with confidence.

Auto-Promote Not Advancing

If the canary stays at a stage longer than expected with auto-promote enabled, check which condition is not met:

nexo canary status pipe-abc123 --verbose

## Output includes:
## Auto-promote conditions:
## ✓ min_duration: 10m (elapsed: 12m)
## ✗ min_requests: 1000 (current: 450)    ← this is blocking
## ✓ max_error_rate_delta_pct: 1.0 (current: 0.01)
## ✓ max_latency_p95_delta_ms: 20 (current: 0.3)

Explore related topics to deepen your understanding of Nexo's deployment and pipeline management capabilities:

Pipeline Configuration — Learn how to define and manage pipeline steps, filters, and routing rules.
Plugin Development — Build custom pipeline plugins with the Nexo SDK and publish to the marketplace.
Monitoring & Metrics — Configure Prometheus metrics, Grafana dashboards, and alerting for your proxy.
CLI Reference — Complete reference for the Nexo command-line interface and all available commands.
Rollback Strategies — Deep dive into rollback mechanisms, custom metrics, and incident-response workflows.
API Reference — Full REST API documentation for Nexo Cloud, including authentication.

Legacy footer label: Last updated January 2025

Legacy footer navigation:

What Canary Means in Nexo​

Key Differences from Traditional Canary​

When to Use Canary​

How It Works​

Traffic Splitting​

Architecture Diagram​

Detailed Request Flow​

Metrics Collection​

Progressive Rollout​

Typical Progression​

Stage Configuration​

Manual vs. Automatic Promotion​

Custom Stage Schedules​

Rollback Rules​

Available Thresholds​

Rollback Configuration​

How Rollback Evaluation Works​

What Happens on Rollback​

Advanced Rollback: Custom Metrics​

Auto-Promote Rules​

Promotion Conditions​

Auto-Promote Configuration​

Promotion Timeline Example​

Combining Auto-Promote with Manual Gates​

API Reference​

Start a Canary​

Get Canary Status​

Promote to Next Stage​

Manual Rollback​

Update Traffic Percentage​

List Active Canaries​

Dashboard Widget​

Metrics Displayed​

Widget Layout​

Widget Features​

Embedding the Widget​

CLI Commands​

Start a Canary​

Check Status​

Promote and Rollback​

List Active Canaries​

Inspect Canary History​

Best Practices​

1. Start Small​

2. Set Conservative Rollback Thresholds​

3. Monitor for Sufficient Duration​

4. One Canary at a Time​

5. Test on Staging First​

6. Use Auto-Promote for Routine Changes​

7. Keep Rollback Detection Fast​

8. Document Candidate Changes​

Notifications and Alerting​

Supported Events​

Notification Configuration​

Troubleshooting​

Canary Starts but No Traffic Reaches Candidate​

Rollback Triggers Immediately​

Metrics Show No Difference​

Auto-Promote Not Advancing​

Related Documentation​

What Canary Means in Nexo

Key Differences from Traditional Canary

When to Use Canary

How It Works

Traffic Splitting

Architecture Diagram

Detailed Request Flow

Metrics Collection

Progressive Rollout

Typical Progression

Stage Configuration

Manual vs. Automatic Promotion

Custom Stage Schedules

Rollback Rules

Available Thresholds

Rollback Configuration

How Rollback Evaluation Works

What Happens on Rollback

Advanced Rollback: Custom Metrics

Auto-Promote Rules

Promotion Conditions

Auto-Promote Configuration

Promotion Timeline Example

Combining Auto-Promote with Manual Gates

API Reference

Start a Canary

Get Canary Status

Promote to Next Stage

Manual Rollback

Update Traffic Percentage

List Active Canaries

Dashboard Widget

Metrics Displayed

Widget Layout

Widget Features

Embedding the Widget

CLI Commands

Start a Canary

Check Status

Promote and Rollback

List Active Canaries

Inspect Canary History

Best Practices

1. Start Small

2. Set Conservative Rollback Thresholds

3. Monitor for Sufficient Duration

4. One Canary at a Time

5. Test on Staging First

6. Use Auto-Promote for Routine Changes

7. Keep Rollback Detection Fast

8. Document Candidate Changes

Notifications and Alerting

Supported Events

Notification Configuration

Troubleshooting

Canary Starts but No Traffic Reaches Candidate

Rollback Triggers Immediately

Metrics Show No Difference

Auto-Promote Not Advancing

Related Documentation