How to Use Must-Gather for Troubleshooting OpenShift Clusters

If you're an OpenShift administrator or a developer working with Kubernetes clusters, you've likely encountered scenarios where you need to debug cluster issues. OpenShift provides a powerful diagnostic tool called must-gather, which helps collect logs, configurations, and other important data from your cluster. In this blog post, we'll walk through what must-gather is, how it works, and how you can use it step-by-step to troubleshoot and resolve issues in your OpenShift environment.

What is Must-Gather?

Must-gather is a command-line tool provided by OpenShift that collects a variety of diagnostic data from your cluster. It gathers logs, resource information, and configuration files from different parts of the cluster, including nodes, pods, services, and operators. This data is invaluable for identifying and resolving issues in a Kubernetes-based environment.

Why Use Must-Gather?

When something goes wrong in an OpenShift cluster, the first step is often to collect data to understand the issue. Must-gather simplifies this process by providing a comprehensive snapshot of your cluster's state, which you can use for:

• Troubleshooting: Quickly gather relevant logs and configurations to diagnose problems.

• Support: Provide detailed information to Red Hat or other support teams.

• Compliance and Auditing: Collect configurations and state information for auditing purposes.

Step-by-Step Guide to Using Must-Gather

Let's go through the steps to effectively use must-gather in an OpenShift environment.

Step 1: Install the OpenShift CLI (oc)

To use must-gather, you first need the OpenShift CLI (oc). You can download it from the OpenShift web console or the Red Hat Customer Portal. Follow the instructions specific to your operating system to install it.

Step 2: Authenticate with Your OpenShift Cluster

Before running must-gather, ensure you're authenticated with your OpenShift cluster. Use the following command to log in:

oc login --server= --token=

Replace <your-server-url> with the API server URL of your OpenShift cluster and <your-token> with your access token.

Step 3: Run the Must-Gather Command

Once authenticated, you can run the must-gather command:

oc adm must-gather

This command initiates the data collection process. By default, it collects logs and configurations from the entire cluster.

Step 4: Specify a Custom Must-Gather Image (Optional)

You can customize the must-gather operation by using a specific image. For example, if you're troubleshooting networking issues, you might use a networking-focused must-gather image:

oc adm must-gather --image=quay.io/openshift/origin-must-gather:latest

This command will pull the specified image and use it to collect data tailored to the image's purpose.

Step 5: WAIT for Data Collection to Complete

Must-gather runs as a pod in your cluster and collects data from various nodes and components. This process might take some time, depending on the size and complexity of your cluster. You can monitor the progress by checking the status of the must-gather pod:

oc get pods -n openshift-must-gather

Once the pod completes its task, the collected data will be downloaded to your local machine in a timestamped directory.

Step 6: Analyze the Collected Data

Navigate to the directory where the must-gather output was saved. You'll see a folder structure that resembles your cluster components, such as nodes, pods, services, and namespaces:

cd must-gather.local.

Inside this directory, you'll find logs, YAML files, and configuration data. Here's a quick overview of what you might see:

• Logs: Detailed logs for various components and operators (current.log, previous.log, etc.).

• Configuration Files: YAML files that show the current configuration of resources (keepalived, render-config-keepalived, etc.).

• Cluster State Information: Details about the current state of the cluster and its components.

Step 7: Identify Issues and Take Action

Using the collected data, you can start troubleshooting your cluster. For instance:

• Log Files: Check log files for error messages, warnings, or other indicators of what might be going wrong.

• Configuration Mismatches: Compare current configurations with your desired state or documentation to identify discrepancies.

• Resource Health: Review the state of cluster resources to detect failed pods, misconfigured services, or network issues.

Step 8: Clean Up After Must-Gather

Once you've gathered and analyzed the necessary information, clean up the must-gather pod and any other resources it created:

oc delete project openshift-must-gather

This command will remove the project and all related resources, helping to keep your cluster tidy.

Tips for Effective Troubleshooting with Must-Gather

• Use Specific Must-Gather Images: Different images are available for specific areas like networking, storage, etc. Using a tailored image can help focus your data collection and analysis.

• Automate Repetitive Tasks: Consider scripting the must-gather process for repeated use, especially in environments where you frequently need to collect data.

• Keep a Record of Past Must-Gathers: Store previous must-gather outputs for historical comparisons and auditing purposes.

Conclusion

Must-gather is an essential tool for any OpenShift administrator or developer. By following these steps, you can quickly collect the necessary data to troubleshoot and resolve issues in your cluster. Remember to regularly update your skills and knowledge about OpenShift and Kubernetes, as these tools continuously evolve with new features and improvements.

With this guide, you're now equipped to handle common cluster issues effectively using must-gather. Happy troubleshooting!

🔗 Connect with me:

• 💼 LinkedIn: https://www.linkedin.com/in/rifaterdemsahin/

• 🐦 Twitter: https://x.com/rifaterdemsahin

• 🎥 YouTube: https://www.youtube.com/@RifatErdemSahin

• 💻 GitHub: https://github.com/rifaterdemsahin

Imported from rifaterdemsahin.com · 2025