🚀 How to Debug a Pod with a Completed Status
🚀
When working with Kubernetes, it’s common to encounter pods that have a status of Completed. This status often means the pod has run its intended task and exited normally, but sometimes you might want to debug and ensure everything worked as expected.
In this blog post, we’ll walk through the key steps to debug a pod with a Completed status using a real-life example! Let’s get started! 💡
🔎 1. Check the Pod Logs
The first step in debugging a completed pod is to check the logs. This will show you what the pod did before it reached the Completed status.
Run this command to fetch the logs for the pod:
kubectl logs
In our example, the pod name is minio-post-job-thkcq and it’s in the minio namespace:
kubectl logs minio-post-job-thkcq -n minio
👉 Tip: The logs will provide insight into whether the pod completed its job successfully or if there were any errors during execution.
rifaterdemsahin@Rifats-MacBook-Pro code % kubectl logs minio-post-job-thkcq -n minio
Connecting to MinIO server: http://minio:9000
mc: <ERROR> Unable to initialize new alias from the provided credentials. Get "http://minio:9000/probe-bucket-sign-bqiyu0cfb5do/?location=": dial tcp 10.106.103.35:9000: i/o timeout.
"Failed attempts: 1"
Addedmyminiosuccessfully.
Creating user 'console'
Added userconsolesuccessfully.
Adding policy 'consoleAdmin' for 'console'
PolicyconsoleAdminis set on userconsolerifaterdemsahin@Rifats-MacBook-Pro code %
🔍 2. Describe the Pod
Next, use the describe command to get detailed information about the pod’s lifecycle and events. This command is super helpful in figuring out the reasons behind the pod’s state.
Run this command:
kubectl describe pod
For example:
kubectl describe pod minio-post-job-thkcq -n minio
This will give you a breakdown of the pod’s phases, conditions, and resource usage. It’s a handy tool to understand the overall behavior of the pod.
🔄 3. Check for Pod Restarts or Failures
Sometimes, a pod may complete successfully, while other times it could have been restarted or failed before completion. You can check the RESTARTS column in your kubectl get pods output to see if the pod was restarted:
kubectl get pods -n minio
A restart count greater than 0 could indicate underlying issues.
📋 4. Inspect the Pod’s Exit Code
One of the most useful pieces of information is the pod’s exit code. You can find this by inspecting the describe output under the State section of the container.
An exit code of 0 generally means the pod completed successfully. If you see a non-zero exit code, it might indicate that something went wrong, and you’ll need to investigate further.
📝 Example Output with Screenshot
Here’s a screenshot of a real debugging session for reference:
As you can see, the minio-post-job-thkcq pod is in a Completed state with no restarts, indicating that it likely finished its task successfully.
🔗 Connect with Me!
Feel free to reach out to me for more Kubernetes tips or just to connect. I love collaborating with fellow tech enthusiasts! 😄
-
💼 LinkedIn: Rifat Erdem Sahin
-
🐦 Twitter: Rifat Erdem Sahin
-
🎥 YouTube: Rifat Erdem Sahin
-
💻 GitHub: Rifat Erdem Sahin
By following these steps, you should now have a better idea of why your pod is in a Completed state and whether it encountered any issues during its lifecycle.
Happy debugging! 🛠️
Error
It looks like the log output shows an error when the minio-post-job pod tried to connect to the MinIO server but encountered an I/O timeout:
mc:
This indicates that the pod was unable to communicate with the MinIO server at http://minio:9000 due to a network or service issue. Here are some steps you can take to further debug the problem:
🛠️ 1. Check if the MinIO Service is Running
First, check if the MinIO service is properly exposed and running by using this command:
kubectl get svc -n minio
Ensure that the service is up and running and that it is bound to the correct IP and port (9000).
🔍 2. Verify Pod Connectivity to MinIO
Try checking the network connectivity from within a running pod in the same namespace (minio) to see if you can reach the MinIO service:
kubectl exec -it
If this fails, there may be a networking issue preventing the pods from communicating with each other.
🔄 3. Check Network Policies
If network policies are in place in your Kubernetes cluster, make sure they aren’t restricting communication between the MinIO server and the pods.
🧪 4. Test the MinIO Endpoint Manually
You can also test the MinIO endpoint manually from within any running pod using a basic curl or wget request to see if the service is reachable:
curl http://minio:9000/probe-bucket-sign-bqiyu0cfb5do/?location=
If this times out, you likely have a networking or service exposure issue that needs further troubleshooting.
By following these steps, you should be able to determine the root cause of the timeout and fix the communication issue with the MinIO service.
Error 2
From the output of the kubectl describe pod minio-post-job-thkcq -n minio, it looks like the pod minio-post-job-thkcq has successfully completed its job with an exit code of 0, which means it completed without any errors. Here's a breakdown of key details:
-
Pod Status:
-
State: Terminated (Reason: Completed, Exit Code: 0): This indicates the pod ran successfully and exited without any issues. -
Started and Finished Times: The pod started and finished in around 23 seconds, which is typical for short-running jobs like this one.
-
Events: The pod pulled the necessary image, created the container, and started it without any errors.
Summary:
-
The job completed successfully (
Exit Code 0). -
The error you encountered in the logs (
i/o timeout) seems to be related to the pod's attempts to connect to the MinIO service. Despite the error, the job was able to proceed with adding the user and policies.
Next Steps:
-
Verify MinIO Service:
Since the pod completed its task, the issue is likely with the connection to the MinIO service (i/o timeout). As mentioned earlier, you should: -
Verify the MinIO service is running properly by checking the service status:
bash kubectl get svc -n minio -
Check MinIO Server Logs:
Investigate the logs of the MinIO server itself to see if there are any issues that prevented the connection:
kubectl logs
- Check Network Connectivity:
If the service is up, there may be a network connectivity issue between the pods and the MinIO service. You can troubleshoot this by trying to ping orcurlthe MinIO service from within another pod in the same namespace.
By addressing the connection issue with the MinIO server, you should be able to resolve the timeout problem seen in the logs.
Imported from rifaterdemsahin.com · 2025