CrashLoopBackOff is a Kubernetes backoff state: the container repeatedly starts and exits, and the platform delays each restart. It does not reveal the root cause. Common triggers after a release include a missing secret, bad command, unavailable dependency, startup probe failure or resource termination. Diagnose the container’s first exit and recent rollout changes before adjusting restarts.
Symptoms: identify the exact failure
Before changing a setting, reproduce the issue once and write down the affected product, operating-system or service version, error text, time, and what still works. One useful observation narrows the fault domain far more than several indiscriminate resets.
- Signal 1: New pods restart while older replica pods still serve traffic
- Signal 2: Exit code 137 appears shortly after startup
- Signal 3: App process exits before serving on its configured port
Why this happens
Different failure points can produce nearly identical messages. A fix is more reliable when it targets the failing layer rather than the nearest visible symptom.
- A new environment or secret value prevents application initialization
- Resource limits or startup probes terminate a slow process
- Entrypoint or dependency changes cause an immediate nonzero exit
Before making changes
Protect data and preserve a rollback path. Confirm the active Kubernetes context and namespace before executing commands. Save deployment manifests and sensitive incident evidence securely; do not print secrets or modify live resource limits without authorization. Preserve healthy replicas until rollback is validated.
Recommended approach: Perform the low-risk checks first. Record the original value of every setting you change, test the exact failing action after each step, and stop when the result is repeatable. Avoid applying several unrelated remedies at once.
Step-by-Step Troubleshooting
The steps below are stored in PRBLEM’s editable Beginner and Advanced troubleshooting fields. Select a method appropriate to your experience; the Beginner sequence focuses on reversible checks, while Advanced actions are intended for administrators or users comfortable collecting diagnostic evidence.
Step-by-Step Troubleshooting
Follow the reversible checks in order; use Advanced only if authorized and preserve the original settings.
Find the failing workload and revision
Record namespace, deployment and affected pod, image digest and rollout revision. Confirm that errors began after the targeted release and not an unrelated node event.
Inspect pod status and events
Use kubectl describe pod with the correct context and namespace. Note OOMKilled, Error, FailedMount, image pull or probe failure messages.
Read previous container logs
Query logs for the previous terminated container, not only the current short-lived instance. Capture sanitized first exception and the exact time of exit.
Compare startup configuration
Review recent changes to commands, environment, ConfigMaps, Secrets references and mounted paths without exposing secret values.
Differentiate probes from application exits
Check whether liveness and startup probe timing matches normal initialization. Only tune thresholds when evidence shows a healthy app being restarted prematurely.
Assess memory and CPU evidence
Correlate exit codes, node pressure and memory limits with workload metrics. Avoid raising quotas across the cluster as a speculative fix.
Roll back or redeploy the narrow change
If the new build is conclusively bad, perform the documented rollback, watch readiness and confirm healthy traffic before continuing rollout.
Compare rollout and runtime fingerprints
Check image digests, dependency versions, PodSpec diff and deployment history so a mutable tag cannot obscure which binary is failing.
Reproduce in staging with safe diagnostics
Use a representative replica and sanitized logs to test corrected startup flags, dependencies or resource requests without exposing production secrets.
Add deployment failure gates
Include smoke checks, startup-probe budgets and rollout monitoring to catch bad containers before all previous replicas are removed.
Interpret the results
Do not treat one successful retry as proof of a permanent fix. Compare the outcome with the following decision guide and note exactly which check changed the result.
| What you observe | Likely direction | Next safe action |
|---|---|---|
| New pods restart while older replica pods still serve traffic | A new environment or secret value prevents application initialization | Find the failing workload and revision: Record namespace, deployment and affected pod, image digest and rollout revision. Confirm that errors began after the targeted release and not an unrelated. |
| Exit code 137 appears shortly after startup | Resource limits or startup probes terminate a slow process | Read previous container logs: Query logs for the previous terminated container, not only the current short-lived instance. Capture sanitized first exception and the exact time of exit. |
| App process exits before serving on its configured port | Entrypoint or dependency changes cause an immediate nonzero exit | Differentiate probes from application exits: Check whether liveness and startup probe timing matches normal initialization. Only tune thresholds when evidence shows a healthy app being restarted prema. |
How to verify the fix
Pass condition: Replacement pods remain Ready across multiple restart observation windows, restart counts stop increasing and application health and user-facing requests return to baseline.
Repeat the original operation at least twice, including after a normal restart or reconnect where appropriate. Confirm that unrelated features still work and that security, backups and access controls remain enabled. Record the tested version and time so future incidents can be compared accurately.
Common mistakes to avoid
- Avoid: Deleting all pods without inspecting why they exited
- Avoid: Using a higher liveness threshold to hide a genuine fatal exception
- Avoid: Running kubectl against the wrong cluster or exposing secret payloads in logs
If the issue persists
If healthy replicas also become unavailable, follow the incident playbook and engage the cluster operator. Supply sanitized pod events, exit reasons, revision diff and exact deployment window.
Include with a support request: the reproducible steps, exact error or status, affected and known-good versions, what changed, and the checks already completed. Remove passwords, tokens, payment data, private messages and personal identifiers from screenshots and logs before sharing.
Solution Checklist
Official reference
For product-specific menus and current behavior, consult GitHub Actions documentation. Features and menu labels can vary by version; verify them against your environment before making lasting changes.