🚀 Thanos Endpoint Deployment with RemoteWrite: A Proof of Concept 💡

In this blog post, I’m sharing the steps and insights gained while deploying a Thanos endpoint with RemoteWrite enabled. This proof-of-concept is aimed at achieving a scalable, high-availability solution for your monitoring stack. Let's dive into how I made this happen and how you can replicate it.

🎯 Objective

Deploy Thanos endpoint and ensure it connects seamlessly with Prometheus via RemoteWrite for a highly available, efficient metric storage and query solution.

Start the Engines

🛠️ Tools and Technologies

• Thanos: A highly available Prometheus setup

• Prometheus: Collects and stores time-series data

• RemoteWrite: Prometheus feature for writing metrics to external storage

• Docker/Kubernetes: Container orchestration for deployment

🚀 Step-by-Step Process

• 🛑 Pause and Plan
  Before jumping into deployment, I took a moment to sketch out the architecture—ensuring I understood how Prometheus, Thanos sidecar, and the store API would interact with each other. This is critical to avoid misconfigurations later.

• 📦 Deploying Prometheus
  You’ll need a running instance of Prometheus. For my deployment, I used Kubernetes. Make sure to include the remote_write configuration in your prometheus.yml. Here’s a snippet:

remote_write:
- url: "http://thanos-receive:19291/api/v1/receive"

• 🧩 Installing Thanos
  Next, I deployed Thanos Receiver, Query, and Sidecar in my Kubernetes cluster using Helm charts. This creates a centralized place for all Prometheus metrics.

• 📡 Setting Up RemoteWrite
  Now for the key part! I configured Prometheus to send metrics to the Thanos Receiver through RemoteWrite. This ensures that metrics are sent in near real-time for storage and querying.

• 🖥️ Thanos Query Setup
  The Thanos Query component lets you query data across all Prometheus instances. I configured it to connect with Thanos Store and Prometheus so I could get a single-pane-of-glass view of all my metrics.

• 🔧 Testing the Setup
  After deployment, I tested the entire setup by generating synthetic traffic. The goal was to check if metrics were successfully written and queried from Thanos.

💡 Key Insights

• Scaling with RemoteWrite: RemoteWrite was efficient in transferring metrics to the Thanos Receiver, proving to be a highly scalable solution for large datasets.

• Pausing for Planning: Taking that extra time to plan architecture upfront saved me a lot of debugging time later. I recommend sketching out your architecture, even if it's on a napkin! 📝

📸 Screenshot of Setup

Here’s a snapshot of the Thanos query interface with real-time data flowing in from Prometheus.

🚀 Future Enhancements

• Adding AlertManager: I'll be integrating AlertManager for better alerting capabilities next.

• Scaling Up: Testing the setup with more Prometheus instances and expanding Thanos storage for long-term retention.

To deploy Thanos with RemoteWrite on an OpenShift cluster, follow these commands and YAML files. The setup includes deploying Prometheus, Thanos components (Receiver, Sidecar, Query, and Store), and configuring RemoteWrite.

1. Deploy Prometheus Operator

Prometheus will collect metrics and send them to Thanos using RemoteWrite. First, install the Prometheus Operator:

oc apply -f https://raw.githubusercontent.com/prometheus-operator/prometheus-operator/main/bundle.yaml

2. Create Prometheus Instance

Create a namespace for Prometheus:

oc create namespace monitoring

Now, create a Prometheus Custom Resource (CR) instance:

apiVersion: monitoring.coreos.com/v1
kind: Prometheus
metadata:
name: prometheus-instance
namespace: monitoring
spec:
replicas: 1
serviceAccountName: prometheus
serviceMonitorSelector:
matchLabels:
team: frontend
remoteWrite:
- url: "http://thanos-receive.monitoring:19291/api/v1/receive"
resources:
requests:
memory: 400Mi

Apply the configuration:

oc apply -f prometheus-instance.yaml

3. Thanos Sidecar Configuration

Next, deploy the Thanos Sidecar. This sidecar will be attached to Prometheus and send the data to the Thanos Receiver.

apiVersion: apps/v1
kind: Deployment
metadata:
name: thanos-sidecar
namespace: monitoring
spec:
replicas: 1
selector:
matchLabels:
app: thanos
template:
metadata:
labels:
app: thanos
spec:
containers:
- name: thanos-sidecar
image: thanosio/thanos:v0.31.0
args:
- sidecar
- --tsdb.path=/prometheus
- --prometheus.url=http://localhost:9090
- --objstore.config-file=/etc/thanos/config.yaml
ports:
- containerPort: 10902
volumeMounts:
- name: prometheus-data
mountPath: /prometheus
- name: thanos-config
mountPath: /etc/thanos
volumes:
- name: prometheus-data
emptyDir: {}
- name: thanos-config
configMap:
name: thanos-config

Create the Thanos configuration:

apiVersion: v1
kind: ConfigMap
metadata:
name: thanos-config
namespace: monitoring
data:
config.yaml: |
type: S3
config:
bucket: "your-s3-bucket-name"
endpoint: "s3.amazonaws.com"
access_key: "your-access-key"
secret_key: "your-secret-key"
insecure: false

Apply the configuration:

oc apply -f thanos-sidecar.yaml
oc apply -f thanos-config.yaml

4. Thanos Receiver Deployment

Thanos Receiver will collect metrics via RemoteWrite from Prometheus.

apiVersion: apps/v1
kind: Deployment
metadata:
name: thanos-receiver
namespace: monitoring
spec:
replicas: 1
selector:
matchLabels:
app: thanos-receiver
template:
metadata:
labels:
app: thanos-receiver
spec:
containers:
- name: thanos-receiver
image: thanosio/thanos:v0.31.0
args:
- receive
- --receive.local-endpoint=0.0.0.0:19291
- --tsdb.path=/var/thanos/data
ports:
- containerPort: 19291
volumeMounts:
- name: thanos-data
mountPath: /var/thanos/data
volumes:
- name: thanos-data
emptyDir: {}

Apply the deployment:

oc apply -f thanos-receiver.yaml

5. Thanos Query Deployment

Thanos Query allows querying metrics from all Thanos instances.

apiVersion: apps/v1
kind: Deployment
metadata:
name: thanos-query
namespace: monitoring
spec:
replicas: 1
selector:
matchLabels:
app: thanos-query
template:
metadata:
labels:
app: thanos-query
spec:
containers:
- name: thanos-query
image: thanosio/thanos:v0.31.0
args:
- query
- --http-address=0.0.0.0:9090
- --store=thanos-receiver.monitoring:10902
ports:
- containerPort: 9090

Apply the configuration:

oc apply -f thanos-query.yaml

6. Expose Thanos Query Service

You can expose Thanos Query as a service to allow access:

apiVersion: v1
kind: Service
metadata:
name: thanos-query
namespace: monitoring
spec:
ports:
- name: http
port: 9090
targetPort: 9090
selector:
app: thanos-query
type: LoadBalancer

Apply the service:

oc apply -f thanos-query-service.yaml

7. Thanos Store Gateway (Optional)

If you plan to store metrics long-term in object storage (S3, GCS), deploy the Thanos Store component to query old data.

apiVersion: apps/v1
kind: Deployment
metadata:
name: thanos-store
namespace: monitoring
spec:
replicas: 1
selector:
matchLabels:
app: thanos-store
template:
metadata:
labels:
app: thanos-store
spec:
containers:
- name: thanos-store
image: thanosio/thanos:v0.31.0
args:
- store
- --data-dir=/var/thanos/store
- --objstore.config-file=/etc/thanos/config.yaml
ports:
- containerPort: 10901
volumeMounts:
- name: thanos-data
mountPath: /var/thanos/store
- name: thanos-config
mountPath: /etc/thanos
volumes:
- name: thanos-data
emptyDir: {}
- name: thanos-config
configMap:
name: thanos-config

Apply the deployment:

oc apply -f thanos-store.yaml

To trigger these OpenShift commands on macOS, you'll need to have several prerequisites in place to ensure smooth execution. Here's a practical guide to set this up:

Prerequisites

• Install OpenShift CLI (OC Tool):

• Download and install OpenShift CLI from here.

• After downloading, place the oc binary in /usr/local/bin/ for easy access.

chmod +x oc
sudo mv oc /usr/local/bin/

• Install Kubernetes CLI (Kubectl):

• You might also want kubectl to manage Kubernetes resources alongside OpenShift.

brew install kubectl

• Ensure Access to OpenShift Cluster:

• Make sure you have access to an OpenShift cluster and are authenticated. You can use the following command to authenticate:
  bash oc login --server=<openshift_cluster_url> --token=<token>

• Prometheus Operator and Thanos Setup:

• Install the necessary Prometheus and Thanos components using the commands provided.

Execution

• Apply Prometheus Operator:
  This will install the Prometheus operator which will manage Prometheus instances in the cluster.

oc apply -f https://raw.githubusercontent.com/prometheus-operator/prometheus-operator/main/bundle.yaml

• Create Monitoring Namespace:
  Create the monitoring namespace where the Prometheus and Thanos instances will run.

oc create namespace monitoring

• Apply Prometheus and Thanos Configuration:
  You'll need the YAML files (prometheus-instance.yaml, thanos-sidecar.yaml, etc.) prepared. Apply each configuration file in sequence to set up Prometheus, Thanos sidecar, and other components.

oc apply -f prometheus-instance.yaml
oc apply -f thanos-sidecar.yaml
oc apply -f thanos-config.yaml
oc apply -f thanos-receiver.yaml
oc apply -f thanos-query.yaml
oc apply -f thanos-query-service.yaml
oc apply -f thanos-store.yaml

Practical Tips:

• YAML File Availability: Ensure the .yaml files you reference are accessible either on your local machine or via a URL.

• Monitoring the Setup: After applying the configurations, you can monitor the status of Prometheus and Thanos by running:

oc get pods -n monitoring

This process will deploy Prometheus with Thanos integration on your OpenShift cluster on macOS.

Conclusion

With these steps and YAML files, you have a working OpenShift deployment of Thanos with Prometheus using RemoteWrite. You can now query, store, and manage metrics from multiple Prometheus instances in a highly scalable way.

Let me know if you'd like more specific details or troubleshooting tips! 😊

🔗 Connect with me:

• 💼 LinkedIn

• 🐦 Twitter

• 🎥 YouTube

• 💻 GitHub

Have you tried deploying Thanos or using RemoteWrite? Drop a comment below, and let’s discuss your experience! 🎉

Imported from rifaterdemsahin.com · 2025