Incident runbooks

These runbooks provide a safe first response. Follow your organization’s incident, change, evidence-preservation, and escalation procedures. Do not copy secrets or customer data into tickets or chat.

First five minutes

1. Record start time, affected environments, applications, and operations.
2. Identify the active Nexo release and pipeline revision.
3. Compare direct MongoDB health with the Nexo path only when an approved diagnostic path exists.
4. Preserve sanitized events, conditions, logs, and recent configuration history.
5. Stop unrelated deployments and avoid untracked manual patches.

Runbook matrix

Symptom	Check first	Safe response
Proxy unavailable	Pod state, rollout, image pull, probes, node pressure, Service endpoints	Roll back a bad release or configuration; restore node and registry access; verify readiness before traffic.
High error rate	Error type, recent policy change, backend health, active graph revision	Revert the offending revision, restore backend health, or re-establish management synchronization.
High latency	Per-component latency, upstream latency, CPU throttling, memory pressure, pool utilization	Disable or revert the slow optional component, restore upstream capacity, or scale tested replicas.
MongoDB backend failure	DNS, Service/endpoints, in-pod connectivity, certificates, connection resets	Restore or fail over MongoDB through the approved database procedure; fix discovery or pool capacity.
Suspected tenant violation	Identity source, tenant policy, cache-key scope, routing revision, audit evidence	Contain traffic, preserve evidence, roll back policy/routing changes, and invalidate unsafe cached state.
Unexpected rate limiting	Limited identity and rule, total versus limited traffic, burst pattern	Correct the identity/rule, tune an approved quota, or contain abusive traffic.
Cache degradation	Hit/miss/eviction trend, TTL, capacity, key shape, tenant boundary	Roll back key changes, correct isolation, resize tested capacity, or adjust TTL.
Control Plane unreachable	Control-plane health, network path, DNS, certificate and credential validity	Restore management connectivity; confirm the still-active graph before resuming changes.
License expiry or invalidity	Expiry, environment scope, validation path, approved license source	Renew or correct the approved license, update through the supported workflow, and verify runtime status.
Pod crash loop	Previous logs, events, exit reason, OOM, Secret/config references, dependencies	Fix the invalid input or capacity issue, roll back, then restart through the controller.

Proxy unavailable

CrashLoopBackOff, ImagePullBackOff, failed readiness, or empty Service endpoints can all present as connection failure. Determine whether the incident began with a rollout, node event, registry failure, or invalid configuration before restarting pods.

High errors or latency

Separate policy rejection from runtime failure and MongoDB failure. Compare error codes, component-level telemetry, upstream latency, and resource throttling. A fast rollback is safer than live-editing several policies during an incident.

Tenant isolation incident

Treat possible cross-tenant access as a security incident. Contain affected traffic, preserve audit and revision evidence, and inspect authenticated identity, routing, cache keys, and response transformation. In 0.2.0, router-phase Tenant Isolation is not executed and must not be claimed as a data-path boundary.

Control-plane interruption

Do not assume that inability to publish a new revision means the existing data plane is down. Confirm the last active revision and traffic health separately. After connectivity returns, reconcile desired and active state before making another change.

Diagnostic bundle

nexoctl diagnostics and nexoctl support-bundle generate can collect Kubernetes state when the operator has appropriate cluster access. Diagnostics output is unredacted and can include raw pod logs; sanitize it before sharing. Support-bundle redaction is best-effort and every archive still requires inspection.

Preserved legacy operations reference

:::danger Unavailable in Nexo 0.2.0

This frozen section preserves every legacy workflow, command, flag, example, table, troubleshooting item, limitation, and related link for parity. It does not make those interfaces available in 0.2.0. The 0.2.0 Private Preview has no public Docker/Compose or VM/binary distribution, no public chart repository, no supported end-to-end nexoctl login/status/upgrade workflow, no profiling API or profiling CLI, no canary control API, no automatic promotion/rollback engine, and no executable router or connection phase. Every shell command below is historical, not a copy/paste procedure. Never pipe an unverified download into a shell, consume a mutable latest artifact, or place connect tokens, license JWTs, passwords, or API keys in process arguments. Require a pinned artifact plus an approved signature/checksum and the licensed secret-delivery workflow. Use placeholders only and follow the supported 0.2.0 guidance above this section.

:::

Operational runbooks for diagnosing and resolving common Nexo issues. Each runbook follows a consistent structure: detection → diagnosis → resolution → validation.

Legacy UI severity labels:

• P0 — Critical Security
• P1 — Service Impacting
• P2 — Degraded
• P3 — Low Impact | ID | Title | Category | Severity | | --- | --- | --- | --- | | RB-001 | Proxy Unavailable | Availability | P1 | | RB-002 | High Latency | Performance | P1 | | RB-003 | Tenant Access Denied | Tenant Isolation | P1 | | RB-004 | Rate Limiting Misconfigured | Rate Limiting | P2 | | RB-005 | Cache Not Working | Cache | P2 | | RB-006 | Query Rewrite Failures | Rewrite | P2 | | RB-007 | Traffic Routing Incorrect | Routing | P1 | | RB-008 | Control Plane Disconnected | Control Plane | P1 | | RB-009 | MongoDB Backend Unreachable | Backend | P1 | | RB-010 | TLS Handshake Failures | Security | P1 | | RB-011 | License Expired | License | P2 | | RB-012 | Shadow Traffic Failures | Shadow | P3 | | RB-013 | PII Masking Errors | Security | P2 | | RB-014 | Audit Logging Failures | Audit | P2 | | RB-015 | Connection Pool Exhaustion | Performance | P1 | | RB-016 | SLO Violations | Performance | P2 | | RB-017 | Config Deployment Failed | Deployment | P2 | | RB-018 | Pipeline Ordering Conflict | Configuration | P2 | | RB-019 | Cross-Tenant Data Leak | Security | P0 | | RB-020 | Version Incompatibility | Upgrade | P3 |

Legacy UI labels: P1 · RB-001 · Availability

Proxy Unavailable

Symptoms

• MongoDB clients receive connection refused or timeout errors
• Application logs show "connection reset by peer"
• Kubernetes reports pod CrashLoopBackOff or not ready

Possible Causes

• Proxy pod OOMKilled — memory limit exceeded
• Liveness probe failure — ingress process hung
• TLS certificate expired — cannot establish connections
• Backend MongoDB unreachable — health check fails
• Invalid pipeline configuration — proxy fails to start

Detection

Alert: NexoProxyDown or NexoProxyNotReady
Metric: up{job="nexo-proxy"} == 0

Diagnosis Steps

1. kubectl get pods -l nexo.io/component=proxy -n nexo
2. kubectl describe pod <pod-name> -n nexo | grep -A5 "State:"
3. kubectl logs <pod-name> -n nexo --tail=100 --previous
4. kubectl top pod -l nexo.io/component=proxy -n nexo
5. curl http://<proxy-service>:9090/health

Resolution

• OOMKilled: Increase memory limits in deployment profile or NexoProxy CR
• Probe failure: Check ingress logs for deadlock/panic, restart pod
• TLS expired: Renew certificate, update Secret, pod will auto-reload
• Backend unreachable: Verify MongoDB connectivity from proxy namespace
• Invalid config: Check NexoPipeline status conditions for validation errors

Validation

• kubectl get pods -l nexo.io/component=proxy — all pods Running/Ready
• curl http://<proxy-service>:9090/health returns 200
• mongosh --host <proxy-service> --port 27018 — connection succeeds
• Check metrics: rate(nexo_requests_total[1m]) > 0

Prevention

• Set resource requests/limits based on load testing
• Enable PodDisruptionBudget (shipped in Helm chart)
• Monitor certificate expiry with cert-manager alerts
• Use deployment readiness score before deploying config changes Legacy UI labels: P1 · RB-002 · Performance

High Latency

Symptoms

• Application response times degraded
• P95/P99 latency exceeds SLO thresholds
• Users report slow queries

Possible Causes

• Backend MongoDB overloaded — slow query execution
• Connection pool exhaustion — requests queued
• Pipeline step bottleneck — expensive rewrite or policy evaluation
• Network latency — cross-zone or cross-region traffic
• Cache miss storm — cold cache after restart or eviction

Detection

Alert: NexoHighLatency
Metric: histogram_quantile(0.95, rate(nexo_operation_duration_seconds_bucket[5m])) > 0.5

Diagnosis Steps

1. Check per-step latency: histogram_quantile(0.95, rate(nexo_operation_duration_seconds_bucket{step!=""}[5m])) by (step)
2. Check connection pool: nexo_connections_active vs max configured
3. Check backend latency: nexo_backend_duration_seconds
4. Check cache hit rate: rate(nexo_cache_hits_total[5m]) / (rate(nexo_cache_hits_total[5m]) + rate(nexo_cache_misses_total[5m]))
5. kubectl top pod -l nexo.io/component=proxy — check CPU throttling

Resolution

• Backend overloaded: Scale MongoDB, add read replicas, enable caching
• Pool exhaustion: Increase max_server_connections, add replicas
• Step bottleneck: Check rewrite regex complexity, simplify policies
• Network: Move proxy co-located with MongoDB, use topology-aware routing
• Cache miss: Pre-warm cache, increase TTL, check invalidation rules

Validation

• histogram_quantile(0.95, ...) returns below SLO threshold
• nexo_connections_active < 80% of max
• Application latency returns to baseline

Prevention

• Set connection pool limits based on capacity planning
• Enable cache for read-heavy workloads
• Use index advisor to identify missing indexes
• Monitor with SLO violation alerts Legacy UI labels: P1 · RB-003 · Tenant Isolation

Tenant Access Denied

Symptoms

• Application receives "unable to determine tenant identity" error
• Specific tenants cannot access their data
• nexo_tenant_isolation_violations_total increasing

Possible Causes

• Missing tenant identification header/claim — request lacks required context
• Tenant not in allowlist — new tenant not registered
• Tenant identification strategy mismatch — wrong header name or JWT path
• Rate limit per tenant exceeded — tenant-specific throttling

Detection

Alert: NexoTenantViolation
Metric: rate(nexo_tenant_isolation_violations_total[5m]) > 0

Diagnosis Steps

1. Check violation metrics by tenant: nexo_tenant_isolation_violations_total by (tenant, reason)
2. Check proxy logs: kubectl logs -l nexo.io/component=proxy | grep "tenant"
3. Verify identification strategy in pipeline config
4. Check if tenant header is being forwarded by upstream load balancer
5. Test with explicit header: mongosh --host proxy --eval "db.runCommand({ping:1})" --apiStrict

Resolution

• Missing header: Ensure upstream (ingress/LB) forwards tenant identification header
• Not in allowlist: Add tenant to allowlist in tenant isolation config
• Strategy mismatch: Update identificationStrategy to match actual header/JWT structure
• Rate limited: Increase per-tenant rate limit or contact tenant about excessive usage

Validation

• nexo_tenant_isolation_violations_total stops increasing
• Affected tenant can successfully connect and query
• Verify in logs: tenant field populated correctly in structured logs

Prevention

• Set unidentifiedRequestBehavior to "log" in staging before "reject" in production
• Use allowlist + default behavior together for defense-in-depth
• Monitor tenant violation metrics continuously Legacy UI labels: P2 · RB-004 · Rate Limiting

Rate Limiting Misconfigured

Symptoms

• Legitimate requests being throttled unexpectedly
• nexo_ratelimit_rejected_total spiking
• Application receiving 429-equivalent MongoDB errors

Possible Causes

• Rate limit too low for actual traffic volume
• Burst bucket too small — spiky traffic exhausts tokens instantly
• Wrong bucket key — all tenants sharing single bucket
• Cooldown period active — previous burst triggered extended cooldown
• Circuit breaker open — error threshold exceeded

Detection

Alert: NexoRateLimitExcessive
Metric: rate(nexo_ratelimit_rejected_total[5m]) > 100

Diagnosis Steps

1. Check rejection reasons: rate(nexo_ratelimit_rejected_total[5m]) by (reason)
2. Check current bucket state in logs: grep "rate-limit" with decision field
3. Compare configured limit vs actual traffic: rate(nexo_requests_total[5m])
4. Check if circuit breaker is open: grep "circuit" in proxy logs
5. Verify bucket key config — is it per-tenant, per-connection, or global?

Resolution

• Too low: Increase requests_per_second based on actual traffic baseline
• Burst too small: Increase burst_size to 2-3x per_second value
• Wrong key: Configure per-tenant or per-source bucketing
• Cooldown active: Wait for cooldown to expire, or disable burst_cooldown
• Circuit open: Fix upstream errors causing circuit to trip, then reset

Validation

• rate(nexo_ratelimit_rejected_total[5m]) returns to baseline
• Legitimate requests flowing without throttling
• Check rate limiter decision logs show "allowed" for normal traffic

Prevention

• Set rate limits based on load testing, not guesses
• Use adaptive rate limiting for variable workloads
• Configure webhook alerts for rate limit threshold warnings
• Start with logging-only mode before enforcing Legacy UI labels: P2 · RB-005 · Cache

Cache Not Working

Symptoms

• Cache hit rate near 0% despite repeated queries
• Backend MongoDB load not reduced after enabling cache
• nexo_cache_misses_total increasing rapidly

Possible Causes

• Cache rules not matching query patterns — wrong collection/command filter
• TTL too short — entries expire before reuse
• Cache size too small — eviction before queries repeat
• Invalidation rules too aggressive — writes clearing valid cache entries
• Cache key includes non-deterministic fields (timestamps, random IDs)

Detection

Alert: NexoCacheMissRateHigh
Metric: miss rate > 90% for 10 minutes

Diagnosis Steps

1. Check cache metrics: nexo_cache_hits_total, nexo_cache_misses_total, nexo_cache_evictions_total
2. Check cache memory: nexo_cache_memory_bytes vs configured max
3. Check cache rules in pipeline config — which collections/commands are cacheable
4. Look at query patterns — are they using parameterized queries or unique per-request?
5. Check invalidation events: are writes clearing the cache too often?

Resolution

• Rules not matching: Update cache rules to include target collections/commands
• TTL too short: Increase DefaultTTL based on data freshness requirements
• Cache too small: Increase MaxMemoryMB or MaxEntries
• Aggressive invalidation: Narrow invalidateOn rules to specific write operations
• Non-deterministic keys: Exclude timestamp/random fields from cache key generation

Validation

• rate(nexo_cache_hits_total[5m]) / total > 50% (or target hit rate)
• Backend query rate decreased
• Application latency improved for cached query patterns

Prevention

• Start with read-heavy API preset which configures sensible defaults
• Monitor cache hit rate and eviction rate continuously
• Use negative caching for frequently-queried-but-empty results
• Size cache based on working set, not total data size Legacy UI labels: P2 · RB-006 · Rewrite

Query Rewrite Failures

Symptoms

• nexo_rewrite_errors_total increasing
• Queries returning unexpected results
• Application errors after enabling rewrite rules

Possible Causes

• Rewrite rule regex syntax error — rule fails to compile at runtime
• Rewrite produces invalid BSON — malformed output document
• Rule ordering conflict — multiple rules transforming same field
• Missing field in request — rule assumes field exists

Detection

Alert: NexoRewriteErrors
Metric: rate(nexo_rewrite_errors_total[5m]) > 0

Diagnosis Steps

1. Check rewrite error metrics by rule: nexo_rewrite_errors_total by (rule)
2. Check audit log for rewrite decisions: kubectl logs -l nexo.io/component=proxy | grep "rewrite"
3. Review the specific rule that is failing in pipeline config
4. Test the rewrite rule with a known query in staging
5. Check for conflicting rules modifying the same document path

Resolution

• Syntax error: Fix regex/rule pattern, redeploy pipeline config
• Invalid BSON: Validate rewrite output structure matches expected schema
• Ordering conflict: Reorder rules or merge conflicting transforms
• Missing field: Add conditional check in rule, or use default values

Validation

• nexo_rewrite_errors_total stops increasing
• Audit log shows successful rewrites: "applied" without errors
• Application receives correct query results
• Run test queries to verify rewrite produces expected output

Prevention

• Test rewrite rules in staging/shadow mode before production
• Use dry-run mode to preview rewrite effects
• Keep rules simple — prefer multiple simple rules over one complex rule
• Version control pipeline configs for easy rollback Legacy UI labels: P1 · RB-007 · Routing

Traffic Routing Incorrect

Symptoms

• Traffic going to wrong backend cluster
• Specific region/tenant receiving data meant for another
• Canary deployment receiving production traffic beyond configured weight

Possible Causes

• Routing rule priority conflict — higher priority rule catches unintended traffic
• Header routing header not forwarded — upstream strips custom headers
• Sticky session expired — client reconnected to different target
• Weighted routing math error — percentages don't sum to 100
• Gradual rollout misconfigured — canary percentage higher than intended

Detection

Metric: nexo_trafficsplit_routed_total by (target) — unexpected distribution

Diagnosis Steps

1. Check routing distribution: nexo_trafficsplit_routed_total by (target, rule)
2. Check proxy logs for routing decisions: grep "traffic-split" with target field
3. Verify routing rules in pipeline config match intended behavior
4. Check if routing headers are present in requests reaching the proxy
5. Verify canary/gradual rollout configuration percentages

Resolution

• Priority conflict: Reorder rules, make catch-all rules lowest priority
• Header missing: Configure upstream LB to forward required headers
• Sticky expired: Increase sticky session duration or use connection-based routing
• Weight error: Correct percentages to sum to 100%
• Canary misconfigured: Reduce canary weight, validate with metrics

Validation

• nexo_trafficsplit_routed_total by (target) shows correct distribution
• Specific test requests route to expected targets
• No cross-tenant or cross-region data leakage

Prevention

• Use canary health monitoring with auto-rollback
• Test routing rules with shadow traffic before production
• Monitor routing distribution dashboards continuously
• Set explicit priorities on all rules — never rely on implicit ordering Legacy UI labels: P1 · RB-008 · Control Plane

Control Plane Disconnected

Symptoms

• Dashboard shows environment as "disconnected"
• Configuration changes not propagating to edge
• Agent logs show gRPC connection errors

Possible Causes

• Network policy blocking agent → control plane communication
• Control plane service down or unhealthy
• TLS certificate mismatch between agent and control plane
• Connect token expired or revoked
• DNS resolution failure for control plane hostname

Detection

Alert: NexoControlPlaneDown
Dashboard: Environment status shows "disconnected"

Diagnosis Steps

1. Check agent logs: kubectl logs -l nexo.io/component=agent -n nexo --tail=100
2. Check control plane health: curl http://controlplane-service:8080/health
3. Check network policies: kubectl get networkpolicy -n nexo
4. Verify DNS: kubectl exec -it <agent-pod> -- nslookup <controlplane-host>
5. Check connect-token metadata only: kubectl get secret nexo-connect-token -n nexo -o jsonpath='{.metadata.name}{"\t"}{.metadata.creationTimestamp}{"\n"}'

Resolution

• Network policy: Add egress rule allowing agent → control plane port
• CP down: Check control plane pod status, restart if needed
• TLS mismatch: Regenerate certificates, update secrets
• Token expired: Mint a short-lived, rotatable credential through the licensed dashboard and approved secret-delivery workflow. Inject it without printing it or placing it in shell arguments, then revoke the previous credential.
• DNS failure: Check CoreDNS logs, verify service DNS is resolvable

Validation

• Agent logs show successful heartbeat
• Dashboard shows environment as "connected"
• Push a config change and verify it propagates within 60s

Prevention

• Monitor heartbeat intervals — alert if gap > 2 minutes
• Prefer short-lived, rotatable connect credentials; monitor expiry and rehearse rotation and revocation.
• Keep network policies tested as part of deployment validation Legacy UI labels: P1 · RB-009 · Backend

MongoDB Backend Unreachable

Symptoms

• All proxy requests failing with connection errors
• nexo_errors_total{type="connection"} spiking
• Application receiving "no reachable servers" errors

Possible Causes

• MongoDB instance down or unrestarting
• Network partition between proxy and MongoDB
• MongoDB authentication failure (credentials rotated)
• MongoDB connection limit reached (maxIncomingConnections)
• DNS resolution failure for MongoDB hostname

Detection

Alert: NexoBackendUnreachable
Metric: nexo_errors_total{type="connection"} spike

Diagnosis Steps

1. Test direct connectivity: kubectl exec -it <proxy-pod> -- mongosh <backend-uri> --eval "db.runCommand({ping:1})"
2. Check connection pool metrics: nexo_connections_active, nexo_connection_errors_total
3. Check MongoDB status: rs.status() on the replica set
4. Check DNS: nslookup <mongodb-host> from proxy pod
5. Check credentials: verify Secret content matches MongoDB user

Resolution

• MongoDB down: Restart MongoDB, check for disk/memory issues
• Network partition: Check security groups, network policies, firewall rules
• Auth failure: Update credentials in Secret, proxy will reconnect
• Connection limit: Increase MongoDB maxIncomingConnections, or reduce proxy pool size
• DNS: Fix DNS resolution, or use IP-based target address

Validation

• Direct connectivity test succeeds from proxy pod
• nexo_connections_active shows healthy pool
• Application requests succeeding through proxy
• nexo_errors_total{type="connection"} stops increasing

Prevention

• Use Connection Pipeline pooling to manage connections efficiently
• Monitor backend connection metrics with alerts
• Configure multiple replica set members for failover
• Test credential rotation procedures regularly Legacy UI labels: P1 · RB-010 · Security

TLS Handshake Failures

Symptoms

• Clients receiving "TLS handshake failure" or "certificate verify failed"
• Proxy logs show "x509: certificate has expired"
• Connections dropping during TLS negotiation

Possible Causes

• Certificate expired — auto-renewal failed or not configured
• CA mismatch — client does not trust proxy certificate CA
• SNI mismatch — certificate doesn't match requested hostname
• TLS version incompatibility — client requires newer/older TLS
• Intermediate certificate missing — incomplete chain

Detection

Proxy logs with "tls" or "x509" errors
Connection failure metrics

Diagnosis Steps

1. Check certificate expiry: kubectl get secret nexo-tls -n nexo -o jsonpath="{.data.tls\.crt}" | base64 -d | openssl x509 -noout -dates
2. Check proxy logs for TLS errors: kubectl logs -l nexo.io/component=proxy | grep -i "tls\|x509\|certificate"
3. Verify cert chain: openssl s_client -connect <proxy>:27018 -servername <hostname>
4. Check cert-manager status (if used): kubectl get certificate -n nexo
5. Verify CA bundle in client configuration matches proxy cert issuer

Resolution

• Expired: Renew certificate manually or fix cert-manager issuer
• CA mismatch: Distribute correct CA certificate to clients
• SNI mismatch: Update certificate SANs to include correct hostnames
• TLS version: Configure proxy TLS minVersion to match client capabilities
• Missing intermediate: Include full certificate chain in TLS Secret

Validation

• openssl s_client -connect <proxy>:27018 shows "Verify return code: 0 (ok)"
• Client connections succeed without TLS errors
• Certificate expiry is >30 days in the future

Prevention

• Use cert-manager with auto-renewal
• Alert on certificate expiry < 14 days
• Include certificate validation in deployment readiness checks
• Document CA distribution procedure for client teams Legacy UI labels: P2 · RB-011 · License

License Expired

Symptoms

• Admin alert: "license expired"
• Proxy operating in degraded mode
• Enterprise features disabled

Possible Causes

• License key not renewed before expiry
• License secret not updated in cluster
• License validation endpoint unreachable

Detection

Admin Alerts page shows license warning
Proxy logs mention "license"

Diagnosis Steps

1. Check admin alerts: dashboard → Admin → Alerts
2. Check license secret: kubectl get secret nexo-license -n nexo
3. Check proxy logs: kubectl logs -l nexo.io/component=proxy | grep "license"
4. Verify license expiry date in Nexo dashboard settings

Resolution

• Renew license through Nexo portal or contact sales
• Update the referenced license Secret through the licensed console or an approved external-secret/CSI delivery workflow. Do not print the JWT, place it in Helm values or process arguments, or persist it in shell history. Rotate or revoke the previous credential after validation.
• Restart proxy pods to pick up new license: kubectl rollout restart deployment -l nexo.io/component=proxy -n nexo

Validation

• Admin alerts cleared
• Proxy logs show "license valid"
• Enterprise features re-enabled

Prevention

• Set calendar reminder 30 days before license expiry
• Monitor license admin alerts
• Prefer short-lived or explicitly expiring, rotatable license credentials where the licensed workflow supports them. Legacy UI labels: P3 · RB-012 · Shadow

Shadow Traffic Failures

Symptoms

• Shadow target not receiving traffic
• Shadow diff results missing
• Shadow target errors not appearing in metrics

Possible Causes

• Shadow target unreachable — wrong address or network blocked
• Sample rate set too low — barely any traffic mirrored
• Max concurrent shadow requests exhausted — shadow dropped
• Timeout too short — shadow target slower than primary

Detection

Shadow metrics showing zero activity
No diff results generated

Diagnosis Steps

1. Check shadow metrics: nexo_shadow_requests_total, nexo_shadow_errors_total
2. Verify shadow target connectivity from proxy pod
3. Check shadow config: sample_rate, max_concurrent, timeout_ms
4. Check proxy logs for shadow-related warnings

Resolution

• Target unreachable: Fix network path, verify shadow target address
• Sample rate: Increase sample_rate (e.g., 0.1 → 1.0 for 100%)
• Concurrency exhausted: Increase max_concurrent or reduce shadow target latency
• Timeout: Increase timeout_ms for slower shadow targets

Validation

• nexo_shadow_requests_total increasing
• Diff results appearing (if diff_enabled)
• Shadow target logs showing received traffic

Prevention

• Start with low sample rate and increase gradually
• Monitor shadow error rate separately from primary
• Set reasonable timeouts based on shadow target SLA Legacy UI labels: P2 · RB-013 · Security

PII Masking Errors

Symptoms

• Sensitive data appearing in logs or responses
• PII masking step throwing errors
• Compliance audit failing

Possible Causes

• Masking rules not covering all sensitive fields
• Rule pattern mismatch — field path changed in application
• Masking step not in pipeline — was removed or never added
• Masking step positioned incorrectly — after logging step

Detection

Audit review
Compliance scanning
pii-masking error metrics

Diagnosis Steps

1. Verify PII masking step exists in pipeline config
2. Check step ordering: PII masking must be before logging and audit steps
3. Review masking rules against actual document schemas
4. Check pii-masking logs for pattern match failures

Resolution

• Missing rules: Add field paths for all PII fields in masking config
• Pattern mismatch: Update field paths to match current schema
• Step missing: Add builtin:pii-masking to pipeline
• Wrong order: Move PII masking before logging and audit steps

Validation

• Query with PII fields — verify masked in logs
• No PII visible in audit events
• Compliance scan passes

Prevention

• Pipeline validation (readiness score) checks PII masking before logging
• Regular compliance scans of log output
• Automate PII field discovery from application schemas Legacy UI labels: P2 · RB-014 · Audit

Audit Logging Failures

Symptoms

• Audit events missing or incomplete
• nexo_audit_events_total not increasing
• Compliance audit gaps

Possible Causes

• Audit output misconfigured — wrong file path or output target
• Disk full — audit log file cannot be written
• Filter too restrictive — audit only capturing subset of events
• Audit step missing from pipeline

Detection

Metric: nexo_audit_events_total stalled
Disk space alerts

Diagnosis Steps

1. Check audit metrics: nexo_audit_events_total, nexo_audit_bytes_total
2. Check audit step config: output type, file path, filters
3. Check disk space in pod: kubectl exec <pod> -- df -h
4. Verify audit step is in pipeline config and in correct position

Resolution

• Misconfigured output: Fix output path or switch to stdout/external
• Disk full: Increase PVC size, enable log rotation, clean old files
• Filter too restrictive: Broaden filter to capture required events
• Step missing: Add builtin:audit to pipeline config

Validation

• nexo_audit_events_total increasing with traffic
• Audit output contains expected event types
• No gaps in audit timeline

Prevention

• Monitor audit metrics with alerts
• Configure log rotation and retention
• Use deployment readiness check for audit compliance Legacy UI labels: P1 · RB-015 · Performance

Connection Pool Exhaustion

Symptoms

• New connections timing out
• Application receiving "connection pool timeout" errors
• nexo_connections_active at or near maximum

Possible Causes

• Traffic spike exceeding pool capacity
• Slow queries holding connections too long
• Connection leak — connections not returned to pool
• Backend MongoDB slow — queries blocking pool
• Pool misconfigured — max too low for workload

Detection

Alert: NexoConnectionPoolExhausted
Metric: nexo_connections_active / max > 0.9

Diagnosis Steps

1. Check connection metrics: nexo_connections_active, nexo_connections_total, nexo_connection_wait_duration
2. Check for slow queries: nexo_logging_slow_queries_total
3. Check backend latency: is MongoDB responding slowly?
4. Check pool config: max_server_connections, max_client_connections
5. Monitor connection age: are connections being reused or created fresh?

Resolution

• Traffic spike: Scale proxy replicas horizontally
• Slow queries: Enable query guard, optimize queries, add indexes
• Connection leak: Restart proxy pod, check for connection pool bug
• Backend slow: Scale MongoDB, add read replicas
• Pool too small: increase pooling.maxServerConnections in the Connection Pipeline config

Validation

• nexo_connections_active < 80% of max
• Connection wait times normal
• Application connections succeeding without timeout

Prevention

• Size pool based on load testing results
• Enable Connection Pipeline pooling with queue overflow handling
• Monitor pool utilization and alert at 80%
• Use idle timeout to release unused connections Legacy UI labels: P2 · RB-016 · Performance

SLO Violations

Symptoms

• nexo_slo_violations_total increasing
• Latency or error rate exceeding defined SLOs

Possible Causes

• Upstream load causing latency to exceed SLO threshold
• Error rate spike from backend or configuration issue
• SLO threshold too tight for actual workload characteristics

Detection

Alert: NexoSLOViolation
Metric: rate(nexo_slo_violations_total[5m]) > 0

Diagnosis Steps

1. Check which SLO is violated: nexo_slo_violations_total by (slo_name, type)
2. Check if it's latency or error-based violation
3. Correlate with other alerts — is there a root cause elsewhere?
4. Check if SLO thresholds match actual service requirements

Resolution

• Latency-based: See RB-002 High Latency runbook
• Error-based: Investigate error source (backend, config, etc.)
• SLO too tight: Adjust thresholds to realistic levels

Validation

• nexo_slo_violations_total stops increasing
• Actual latency/error rate within SLO bounds

Prevention

• Set SLOs based on baseline measurements, not aspirational targets
• Review SLOs quarterly against actual performance Legacy UI labels: P2 · RB-017 · Deployment

Config Deployment Failed

Symptoms

• Pipeline config change not taking effect
• NexoPipeline CRD shows error conditions
• Agent reports deployment failure

Possible Causes

• Invalid pipeline configuration — validation error
• Agent disconnected — cannot receive updated config
• CRD schema validation failure — invalid field types
• Operator not reconciling — operator pod unhealthy

Detection

Dashboard deploy status shows failure
kubectl get nexopipeline -o yaml shows error condition

Diagnosis Steps

1. Check NexoPipeline status: kubectl get nexopipeline -n nexo -o yaml | yq .status
2. Check operator logs: kubectl logs -l nexo.io/component=operator -n nexo --tail=50
3. Check agent logs: kubectl logs -l nexo.io/component=agent -n nexo --tail=50
4. Verify config validity in dashboard deploy preview

Resolution

• Invalid config: Fix validation errors shown in CRD status conditions
• Agent disconnected: See RB-008 Control Plane Disconnected
• Schema error: Update CRD fields to match expected schema
• Operator unhealthy: Restart operator pod, check for resource issues

Validation

• NexoPipeline status shows "Ready" condition
• status.configHash updated to new value
• Proxy logs show "configuration reloaded"
• New behavior observable in traffic

Prevention

• Always use deployment readiness score before deploying
• Test config changes in staging first
• Use pipeline validation to catch errors before deploy Legacy UI labels: P2 · RB-018 · Configuration

Pipeline Ordering Conflict

Symptoms

• Unexpected behavior from pipeline steps
• Cache serving unfiltered results
• Tenant isolation not enforcing correctly

Possible Causes

• Cache step placed before policy enforcement — cached results bypass security
• Router consumes untrusted identity or runs without completed request-phase access decisions; multiple mutually exclusive routers may also be configured
• Rewrite step after cache — cached queries don't get rewritten
• Body-capable logging or audit placed before PII masking — protected response fields can be recorded in cleartext

Detection

Deployment readiness score warnings
Unexpected functional behavior

Diagnosis Steps

1. Review pipeline step order in NexoPipeline CRD or dashboard
2. Check deployment readiness score for ordering warnings
3. Trace a request through the pipeline — which steps execute in what order?
4. Compare with recommended ordering in documentation

Resolution

Apply explicit phase invariants rather than treating request, router, and response components as one freely reorderable list:

1. Connection/request identity: Authenticate the client and establish a trusted tenant identity before request processing.
2. Request phase: Apply filter and access-control decisions before cache lookup or routing, then apply identity-aware rate limits and other protections.
3. Request transformation/cache: Rewrite the query before computing cache keys. Perform cache lookup only after authorization and scope every key by authenticated tenant.
4. Router phase: Configure only one router when the release permits a single router. Tenant Isolation and Traffic Split are alternative router choices, not an ordered pair. The selected router must consume the trusted identity and completed request-phase policy state.
5. Response phase: Apply PII masking before any logging or audit sink that can capture bodies or protected fields.
6. Observation: Emit only masked body records; metadata-only metrics or logs may observe earlier phases when they cannot capture sensitive payloads.

In 0.2.0 the router phase is not executed, so neither Tenant Isolation nor Traffic Split can be used as a data-path security boundary.

:::danger Unsafe legacy ordering anti-pattern

The former sequence placed Logging/Metrics first, then Rate Limiting, Tenant Isolation, Filter/Access Control, PII Masking, Query Rewrite, Cache, Traffic Split/Routing, and Audit. Do not deploy that order: it observes unauthenticated traffic, authorizes too late for safe cache/routing decisions, and can expose unmasked response data.

:::

Validation

• Deployment readiness score shows no ordering warnings
• Functional tests pass with correct ordering
• Authenticated tenant identity and request access controls are enforced before cache lookup; any selected router consumes that trusted state

Prevention

• Use deployment readiness checks (catches ordering issues)
• Follow recommended pipeline ordering template
• Use presets which include correct ordering by default Legacy UI labels: P0 · RB-019 · Security

Cross-Tenant Data Leak

Symptoms

• Tenant A receiving data belonging to Tenant B
• nexo_tenant_isolation_violations_total spiking
• Security audit finding cross-tenant access

Possible Causes

• Tenant isolation step missing from pipeline
• Tenant identification strategy returning wrong tenant
• Cache serving results across tenants — cache key not tenant-scoped
• Routing sending tenant traffic to wrong backend
• Cross-tenant blocking disabled — enforcement not active

Detection

CRITICAL SECURITY INCIDENT
Alert: NexoTenantViolation (should page immediately)

Diagnosis Steps

1. IMMEDIATE: Verify tenant isolation step is active and enforcing
2. Check cache configuration: is cache key scoped by tenant?
3. Check tenant identification: is tenant correctly extracted from every request?
4. Check routing rules: are tenant-specific routes correctly configured?
5. Review recent config changes: did someone modify tenant config?
6. Check audit log for cross-tenant access patterns

Resolution

• IMMEDIATE: Enable crossTenantBlocking: strict if not active
• IMMEDIATE: Flush cache if cache is suspected source: restart proxy pods
• Fix identification: Correct header/JWT/field extraction configuration
• Fix cache: Add tenant to cache key scope (tenant field in cache config)
• Fix routing: Ensure per-tenant routing rules are correct
• Notify affected tenants per incident response procedure

Validation

• Verify each tenant only sees their own data
• nexo_tenant_isolation_violations_total at zero
• Run cross-tenant test suite to confirm isolation
• Security audit passes

Prevention

• ALWAYS enable crossTenantBlocking: strict in production
• ALWAYS scope cache keys by tenant
• Use deployment readiness score — blocks deploy without tenant isolation
• Regular cross-tenant penetration testing
• Monitor tenant violation alerts at P0 severity Legacy UI labels: P3 · RB-020 · Upgrade

Version Incompatibility

Symptoms

• Proxy features not working after upgrade
• CRD validation errors after operator upgrade
• Agent cannot communicate with new control plane version

Possible Causes

• Operator upgraded without CRD update — schema mismatch
• Agent version behind control plane — API incompatibility
• MongoDB driver version mismatch — wire protocol differences

Detection

Post-upgrade functional failures
CRD status errors

Diagnosis Steps

1. Check component versions: kubectl get pods -l nexo.io -o jsonpath="{.items[*].spec.containers[*].image}"
2. Check CRD version: kubectl get crd nexoproxies.nexo.io -o yaml | head -20
3. Check control plane version: curl http://cp-service:8080/api/version
4. Compare versions against compatibility matrix

Resolution

• CRD mismatch: Apply latest CRD manifests before operator upgrade
• Agent behind: Upgrade agent to match control plane version
• Wire protocol: Use version-compat step to patch protocol differences

Validation

• All components running same major.minor version
• CRD status shows no validation errors
• End-to-end traffic flowing correctly

Prevention

• Follow upgrade order: CRDs → Operator → Agent → Proxy
• Use version-compat step during rolling upgrades
• Test upgrades in staging before production
• Read release notes for breaking changes