Setting Up Prometheus and Thanos Together with TLS Certificate Authentication 🔒

Setting up Prometheus and Thanos for a secure and scalable monitoring system can be a game-changer for your infrastructure. In this post, I'll walk you through configuring these tools with TLS certificates to ensure secure communication 🔑.

Let’s dive in! 🚀

Step 1: Install Prometheus 📊

First, get Prometheus up and running on your Linux server.# Download Prometheus wget https://github.com/prometheus/prometheus/releases/download/v2.44.0/prometheus-2.44.0.linux-amd64.tar.gz # Extract Prometheus tar -xzvf prometheus-2.44.0.linux-amd64.tar.gz cd prometheus-2.44.0.linux-amd64/ # Start Prometheus ./prometheus --config.file=prometheus.yml

Once done, check that Prometheus is accessible at http://localhost:9090 🔍.

Step 2: Install Thanos 🛠️

Now, let's set up Thanos, which extends Prometheus for scalable long-term storage and query capabilities.# Download Thanos wget https://github.com/thanos-io/thanos/releases/download/v0.33.0/thanos-0.33.0.linux-amd64.tar.gz # Extract Thanos tar -xzvf thanos-0.33.0.linux-amd64.tar.gz cd thanos-0.33.0.linux-amd64/ # Ensure Thanos is accessible ./thanos --version

Thanos is now installed and ready for configuration! ⚙️

Step 3: Generate Certificates (TLS) 🔐

To secure communications between Prometheus and Thanos, we'll use TLS certificates. Here’s how you can generate them.

a. Generate CA Certificate

# Generate the private key for the CA openssl genrsa -out ca.key 4096 # Create the CA certificate openssl req -x509 -new -nodes -key ca.key -sha256 -days 365 -out ca.crt -subj "/CN=ThanosCA"

b. Generate Certificates for Prometheus and Thanos

Prometheus Certificate:

# Generate Prometheus private key openssl genrsa -out prometheus.key 2048 # Create a CSR for Prometheus openssl req -new -key prometheus.key -out prometheus.csr -subj "/CN=prometheus" # Sign Prometheus certificate with the CA openssl x509 -req -in prometheus.csr -CA ca.crt -CAkey ca.key -CAcreateserial -out prometheus.crt -days 365 -sha256

Thanos Certificate:

# Generate Thanos private key openssl genrsa -out thanos.key 2048 # Create a CSR for Thanos openssl req -new -key thanos.key -out thanos.csr -subj "/CN=thanos" # Sign Thanos certificate with the CA openssl x509 -req -in thanos.csr -CA ca.crt -CAkey ca.key -CAcreateserial -out thanos.crt -days 365 -sha256

With both certificates ready, we can now secure communication between Prometheus and Thanos. 🔗

Step 4: Configure Prometheus for TLS 🔧

To make Prometheus communicate over TLS, edit your prometheus.yml configuration.global: scrape_interval: 15s scrape_configs: - job_name: 'thanos' scheme: https static_configs: - targets: ['localhost:9091'] tls_config: cert_file: /path/to/prometheus.crt key_file: /path/to/prometheus.key ca_file: /path/to/ca.crt

Save the file and restart Prometheus:./prometheus --config.file=prometheus.yml

Step 5: Configure Thanos Sidecar 🏗️

Now, we need to configure the Thanos Sidecar to communicate with Prometheus securely../thanos sidecar \ --tsdb.path /path/to/prometheus/data \ --prometheus.url http://localhost:9090 \ --grpc-address 0.0.0.0:10901 \ --http-address 0.0.0.0:10902 \ --cert-file=/path/to/thanos.crt \ --key-file=/path/to/thanos.key \ --ca-file=/path/to/ca.crt

This sets up secure communication between Prometheus and Thanos 🔄.

Step 6: Run Thanos Querier 🌍

To query data across your Prometheus and Thanos instances, run the Thanos Querier../thanos query \ --http-address 0.0.0.0:10904 \ --grpc-address 0.0.0.0:10903 \ --store=localhost:10901 \ --cert-file=/path/to/thanos.crt \ --key-file=/path/to/thanos.key \ --ca-file=/path/to/ca.crt

Now you can access the Thanos Querier at https://localhost:10904 and query both Prometheus and Thanos! 🌟

Step 7: Deploying on Kubernetes 🐳

Deploying Prometheus and Thanos in Kubernetes is straightforward. Here’s an example YAML file to get Prometheus deployed with TLS.apiVersion: v1 kind: Pod metadata: name: prometheus spec: containers: - name: prometheus image: prom/prometheus:v2.44.0 args: - '--config.file=/etc/prometheus/prometheus.yml' - '--web.listen-address=:9090' volumeMounts: - name: config-volume mountPath: /etc/prometheus/ volumes: - name: config-volume configMap: name: prometheus-config --- apiVersion: v1 kind: ConfigMap metadata: name: prometheus-config data: prometheus.yml: | global: scrape_interval: 15s scrape_configs: - job_name: 'thanos' scheme: https static_configs: - targets: ['localhost:9091'] tls_config: cert_file: /etc/prometheus/prometheus.crt key_file: /etc/prometheus/prometheus.key ca_file: /etc/prometheus/ca.crt

This example showcases how you can define Prometheus configuration using Kubernetes ConfigMaps for TLS integration. 🧑‍💻

Step 8: Test Your Setup 🧪

Prometheus should be running on https://localhost:9090.
Thanos Querier should be accessible via https://localhost:10904.

Both tools should now be communicating securely using TLS certificates. 🔒

Wrapping Up 🎉

Setting up Prometheus and Thanos together with TLS certificates ensures a secure, scalable monitoring system for your infrastructure. Whether you're deploying them on Linux or Kubernetes, these steps provide you with the foundation to start monitoring your systems securely.

Happy Monitoring! 🎈