Incident runbooks

These runbooks provide a safe first response. Follow your organization’s incident, change, evidence-preservation, and escalation procedures. Do not copy secrets or customer data into tickets or chat.

First five minutes

1. Record start time, affected environments, applications, and operations.
2. Identify the active Nexo release and pipeline revision.
3. Compare direct MongoDB health with the Nexo path only when an approved diagnostic path exists.
4. Preserve sanitized events, conditions, logs, and recent configuration history.
5. Stop unrelated deployments and avoid untracked manual patches.

Runbook matrix

Symptom	Check first	Safe response
Proxy unavailable	Pod state, rollout, image pull, probes, node pressure, Service endpoints	Roll back a bad release or configuration; restore node and registry access; verify readiness before traffic.
High error rate	Error type, recent policy change, backend health, active graph revision	Revert the offending revision, restore backend health, or re-establish management synchronization.
High latency	Per-component latency, upstream latency, CPU throttling, memory pressure, pool utilization	Disable or revert the slow optional component, restore upstream capacity, or scale tested replicas.
MongoDB backend failure	DNS, Service/endpoints, in-pod connectivity, certificates, connection resets	Restore or fail over MongoDB through the approved database procedure; fix discovery or pool capacity.
Suspected tenant violation	Identity source, tenant policy, cache-key scope, routing revision, audit evidence	Contain traffic, preserve evidence, roll back policy/routing changes, and invalidate unsafe cached state.
Unexpected rate limiting	Limited identity and rule, total versus limited traffic, burst pattern	Correct the identity/rule, tune an approved quota, or contain abusive traffic.
Cache degradation	Hit/miss/eviction trend, TTL, capacity, key shape, tenant boundary	Roll back key changes, correct isolation, resize tested capacity, or adjust TTL.
Control Plane unreachable	Control-plane health, network path, DNS, certificate and credential validity	Restore management connectivity; confirm the still-active graph before resuming changes.
License expiry or invalidity	Expiry, environment scope, validation path, approved license source	Renew or correct the approved license, update through the supported workflow, and verify runtime status.
Pod crash loop	Previous logs, events, exit reason, OOM, Secret/config references, dependencies	Fix the invalid input or capacity issue, roll back, then restart through the controller.

Proxy unavailable

CrashLoopBackOff, ImagePullBackOff, failed readiness, or empty Service endpoints can all present as connection failure. Determine whether the incident began with a rollout, node event, registry failure, or invalid configuration before restarting pods.

Diagnosis steps:

kubectl get pods \
  --selector nexo.io/component=proxy \
  --namespace nexo-system

kubectl describe pod <pod-name> \
  --namespace nexo-system

kubectl logs <pod-name> \
  --namespace nexo-system \
  --tail=200 \
  --previous

Check NexoPipeline status conditions for validation errors that prevent startup:

kubectl get nexopipeline \
  --namespace nexo-system \
  --output yaml

Resolution patterns:

• OOMKilled: Increase memory limits through the NexoProxy CR or NexoDeploymentProfile.
• ImagePullBackOff: Restore registry access and verify the pinned image digest in the active release.
• Invalid configuration: Correct the field reported in NexoPipeline status conditions and publish a corrected revision; do not live-edit during an incident.
• Probe failure: Collect logs for the previous container, roll back the offending release, then restart through the controller — do not manually patch the Deployment.

High errors or latency

Separate policy rejection from runtime failure and MongoDB failure. Compare error codes, component-level telemetry, upstream latency, and resource throttling. A fast rollback is safer than live-editing several policies during an incident.

Diagnosis steps:

# Inspect Operator and Manager logs for recent reconciliation events
kubectl logs \
  --selector nexo.io/component=operator \
  --namespace nexo-system \
  --tail=100

kubectl logs \
  --selector nexo.io/component=manager \
  --namespace nexo-system \
  --tail=100

# Check resource pressure
kubectl top pod \
  --selector nexo.io/component=proxy \
  --namespace nexo-system

TLS handshake failures

TLS errors can originate on either the client-to-proxy leg or the proxy-to-MongoDB leg. Determine which leg fails before applying a fix.

Diagnosis steps:

# Check certificate expiry dates
kubectl get secret nexo-tls \
  --namespace nexo-system \
  --output jsonpath='{.data.tls\.crt}' \
  | base64 --decode \
  | openssl x509 -noout -dates

# Inspect proxy logs for TLS or x509 errors
kubectl logs \
  --selector nexo.io/component=proxy \
  --namespace nexo-system \
  --tail=200 \
  | grep -i "tls\|x509\|certificate"

Resolution patterns:

• Expired certificate: Renew and update the TLS Secret. The Proxy picks up the new certificate without a manual restart when cert-manager or a CSI provider is used. Verify the updated Secret before considering the incident resolved.
• CA mismatch: Distribute the correct CA certificate to clients through your approved CA-distribution path.
• SNI mismatch: Update the certificate Subject Alternative Names to include the hostnames clients use. Re-verify with openssl s_client.
• Missing intermediate: Include the full certificate chain in the TLS Secret, not just the leaf certificate.

MongoDB backend failure

Diagnosis steps:

# Prompt for the password; never put a credential-bearing URI in argv.
kubectl exec -it \
  --namespace nexo-system \
  <proxy-pod-name> \
  -- mongosh --host <mongodb-hostname> --port <mongodb-port> \
     --username <mongodb-username> --password \
     --eval "db.runCommand({ping:1})"

# Check DNS resolution from within the namespace
kubectl exec \
  --namespace nexo-system \
  <proxy-pod-name> \
  -- nslookup <mongodb-hostname>

Restore or fail over MongoDB through your approved database procedure. Fix Kubernetes Service endpoints, DNS records, or credential Secrets as needed. After restoring connectivity, verify the connection pool recovers before routing production traffic.

Connection pool exhaustion

Slow queries that hold connections, a traffic spike, or a pool sized below the workload all cause pool exhaustion. Do not restart pods without first understanding the cause.

Diagnosis steps:

# Check pool configuration in the active ConnectionPipeline CR
kubectl get nexoconnectionpipeline \
  --namespace nexo-system \
  --output yaml

# Review recent Proxy logs for wait or exhaustion signals
kubectl logs \
  --selector nexo.io/component=proxy \
  --namespace nexo-system \
  --tail=200 \
  | grep -i "pool\|connection"

Resolution patterns:

• Traffic spike: Scale proxy replicas through the NexoProxy CR.
• Slow backend: Identify and optimize slow queries; enable the Query Guard component to protect the pool.
• Pool limit too low: Increase maxServerConnections in the NexoConnectionPipeline configuration and publish a new pipeline revision.

Tenant isolation incident

Treat possible cross-tenant access as a security incident. Contain affected traffic, preserve audit and revision evidence, and inspect authenticated identity, routing, cache keys, and response transformation. In 0.2.0, router-phase Tenant Isolation is not executed and must not be claimed as a data-path boundary. Use database-native authorization as the primary isolation control for this release.

Immediate containment:

1. Preserve the current NexoPipeline revision identifier and evidence from your independently governed database, platform, and network audit sources before making any changes. The current Nexo Audit contract is not available as compliance evidence.
2. Roll back any recent pipeline or routing revision that coincides with the incident window.
3. Invalidate potentially cross-contaminated cache state by restarting affected Proxy pods through the controller — do not manually patch Deployments.
4. Follow your organization's security-incident and data-breach notification procedures.

Pipeline ordering

Incorrect component ordering is a security risk, not just a correctness issue.

:::danger Current release boundaries

The sequence below is a conceptual target for a future bundle with verified contracts. In the current release, NexoConnectionPipeline identity fields, Access Control, PII Masking, and router-phase isolation are not supported enforcement controls. Use MongoDB-native authentication and authorization, independently verified application or database masking, and an external audit source.

:::

For a future release that explicitly publishes and tests each contract, apply these phase invariants:

1. Authentication / identity: Authenticate the client and establish a trusted tenant identity before any request-phase component executes.
2. Request phase: Apply a verified filter/access-control contract before cache lookup or routing. Until that contract ships, enforce permissions in MongoDB.
3. Request transformation / cache: Rewrite the query before computing cache keys. Perform cache lookup only after authorization and scope every key by authenticated tenant.
4. Router phase: Configure at most one router. Tenant Isolation and Traffic Split are alternative router choices, not an ordered pair. The selected router must consume the trusted identity and completed request-phase policy state. In 0.2.0, the router phase does not execute.
5. Response phase: Apply independently verified masking before any logging or audit sink that can capture bodies or protected fields.
6. Observation: Emit only masked body records; metadata-only metrics or logs may observe earlier phases when they cannot capture sensitive payloads.

:::danger Unsafe legacy ordering anti-pattern

The former sequence placed Logging/Metrics first, then Rate Limiting, Tenant Isolation, Filter/Access Control, PII Masking, Query Rewrite, Cache, Traffic Split/Routing, and Audit. Do not deploy that order: it observes unauthenticated traffic, authorizes too late for safe cache/routing decisions, and can expose unmasked response data.

:::

Check the current pipeline ordering:

kubectl get nexopipeline \
  --namespace nexo-system \
  --output yaml

Control-plane interruption

Do not assume that inability to publish a new revision means the existing data plane is down. Confirm the last active revision and traffic health separately. After connectivity returns, reconcile desired and active state before making another change.

Diagnosis steps:

# Check Manager connectivity and last heartbeat events
kubectl logs \
  --selector nexo.io/component=manager \
  --namespace nexo-system \
  --tail=100

# Inspect Manager networking — egress to the Control Plane
kubectl get networkpolicy \
  --namespace nexo-system

# Check connect-token Secret metadata (do not print token value)
kubectl get secret nexo-connect-token \
  --namespace nexo-system \
  --output jsonpath='{.metadata.name}{"\t"}{.metadata.creationTimestamp}{"\n"}'

If the connect token must be rotated, mint a replacement through the licensed console's protected rotation path, inject it through the approved Secret delivery workflow without printing the value or placing it in shell arguments, then revoke the previous credential after the new heartbeat is confirmed.

License expiry or invalidity

Diagnosis steps:

# Inspect the license Secret metadata (never print the JWT value)
kubectl get secret nexo-license \
  --namespace nexo-system

# Inspect proxy logs for license signals
kubectl logs \
  --selector nexo.io/component=proxy \
  --namespace nexo-system \
  --tail=100 \
  | grep -i "license"

Renew through the licensed console and update the referenced Secret through an approved external-secret or CSI delivery workflow. Do not print the JWT, embed it in Helm values or process arguments, or persist it in shell history. After updating the Secret, trigger a rolling restart through the controller:

kubectl rollout restart deployment \
  --selector nexo.io/component=proxy \
  --namespace nexo-system

Revoke or invalidate the previous credential after confirming the new license is active.

Version incompatibility after upgrade

Apply CRDs before upgrading the Operator, and upgrade the Operator before upgrading the Manager. Check component version alignment:

kubectl get pods \
  --selector nexo.io \
  --namespace nexo-system \
  --output jsonpath="{range .items[*]}{.metadata.name}{'\t'}{.spec.containers[0].image}{'\n'}{end}"

kubectl get crd nexoproxies.nexo.io \
  --output jsonpath='{.metadata.resourceVersion}{"\n"}'

Consult the release notes and the Compatibility reference for the supported upgrade order and any field schema changes before applying new CRD manifests.

Diagnostic bundle

nexoctl diagnostics and nexoctl support-bundle generate can collect Kubernetes state when the operator has appropriate cluster access. Diagnostics output is unredacted and can include raw pod logs; sanitize it before sharing. Support-bundle redaction is best-effort and every archive still requires inspection.

nexoctl diagnostics \
  --namespace nexo-system \
  --output ./nexo-diagnostics.json

nexoctl support-bundle generate \
  --namespace nexo-system \
  --output ./nexo-support.tar.gz

Related pages

• Production checklist
• nexoctl command reference
• Troubleshooting overview
• Admission and drift behavior
• Support