Proof of concept (PoC) that integrates Grafana Tempo, Thanos, Prometheus, and Grafana in a CRC (CodeReady Containers)

To create a proof of concept (PoC) that integrates Grafana Tempo, Thanos, Prometheus, and Grafana in a CRC (CodeReady Containers) OpenShift environment, we will guide you through deploying these components. This PoC will help you understand distributed tracing (using Tempo) and long-term metrics storage and scaling (using Thanos) while visualizing everything in Grafana.

Here’s how we’ll structure the project:

Deploy Prometheus for metrics collection.
Deploy Tempo for distributed tracing.
Deploy Thanos to extend Prometheus for long-term storage and metrics aggregation.
Deploy Grafana to visualize metrics and traces.
Configure OpenTelemetry in a sample application for tracing.

Step 1: Deploy Prometheus in CRC OpenShift

First, let’s deploy Prometheus to gather metrics from the environment.

Prometheus Deployment YAML:

apiVersion: apps/v1
kind: Deployment
metadata:
name: prometheus
namespace: monitoring
spec:
replicas: 1
selector:
matchLabels:
app: prometheus
template:
metadata:
labels:
app: prometheus
spec:
containers:
- name: prometheus
image: prom/prometheus:latest
ports:
- containerPort: 9090
volumeMounts:
- name: prometheus-config
mountPath: /etc/prometheus/prometheus.yml
subPath: prometheus.yml
volumes:
- name: prometheus-config
configMap:
name: prometheus-config

Prometheus ConfigMap:

Create the following ConfigMap to define the scrape configurations:

apiVersion: v1
kind: ConfigMap
metadata:
name: prometheus-config
namespace: monitoring
data:
prometheus.yml: |
global:
scrape_interval: 15s

scrape_configs:
                  - job_name: 'prometheus'
                    static_configs:
                      - targets: ['localhost:9090']

Apply these configurations:

oc apply -f prometheus-deployment.yaml
oc apply -f prometheus-configmap.yaml

This sets up Prometheus to collect metrics from various sources.

Step 2: Deploy Grafana Tempo for Tracing

Now, let’s deploy Grafana Tempo to collect and store traces.

Tempo Deployment YAML:

apiVersion: apps/v1
kind: Deployment
metadata:
name: tempo
namespace: monitoring
spec:
replicas: 1
selector:
matchLabels:
app: tempo
template:
metadata:
labels:
app: tempo
spec:
containers:
- name: tempo
image: grafana/tempo:latest
ports:
- containerPort: 3200

Apply the Tempo configuration:

oc apply -f tempo-deployment.yaml

This will deploy Tempo in your CRC OpenShift environment, allowing it to collect traces. You can later configure your application to send traces to this Tempo instance.

Step 3: Deploy Thanos for Metrics Aggregation and Long-Term Storage

Deploy Thanos to extend Prometheus with long-term metrics storage and global querying.

Thanos Deployment YAML:

apiVersion: apps/v1
kind: Deployment
metadata:
name: thanos
namespace: monitoring
spec:
replicas: 1
selector:
matchLabels:
app: thanos
template:
metadata:
labels:
app: thanos
spec:
containers:
- name: thanos
image: quay.io/thanos/thanos:v0.22.0
args:
- "sidecar"
- "--grpc-address=0.0.0.0:10901"
- "--http-address=0.0.0.0:10902"
- "--prometheus.url=http://prometheus:9090"
ports:
- containerPort: 10901
- containerPort: 10902

This deploys Thanos as a sidecar to Prometheus. Thanos will provide long-term metrics storage and scaling.

Apply the Thanos deployment:

oc apply -f thanos-deployment.yaml

Step 4: Deploy Grafana for Visualization

We’ll now deploy Grafana to visualize the data (metrics and traces) collected by Prometheus and Tempo.

Grafana Deployment YAML:

apiVersion: apps/v1
kind: Deployment
metadata:
name: grafana
namespace: monitoring
spec:
replicas: 1
selector:
matchLabels:
app: grafana
template:
metadata:
labels:
app: grafana
spec:
containers:
- name: grafana
image: grafana/grafana:latest
ports:
- containerPort: 3000

Expose Grafana service:

oc expose svc/grafana

Once Grafana is deployed, access it through the route created by OpenShift. You will need to set up Prometheus and Tempo as data sources in Grafana.

Add Prometheus and Tempo as Data Sources:

Go to Grafana (accessible via the exposed route).
Navigate to Configuration > Data Sources.
Add Prometheus with the following URL:
http://prometheus:9090
Add Tempo with the following URL:
http://tempo:3200

Step 5: Configure Sample Application with OpenTelemetry

To generate traces, we need a sample application that sends tracing data to Tempo. You can instrument any microservice using OpenTelemetry to send traces.

Example: OpenTelemetry Tracing Configuration (Python)

Here’s a simple Python application that sends traces to Tempo:

from opentelemetry import trace
from opentelemetry.exporter.otlp.proto.grpc.trace_exporter import OTLPSpanExporter
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import SimpleSpanProcessor
from opentelemetry.sdk.resources import Resource

Setup OpenTelemetry

trace.set_tracer_provider(TracerProvider(resource=Resource.create({"service.name": "my-service"})))
tracer = trace.get_tracer(name)

Export spans to Grafana Tempo

otlp_exporter = OTLPSpanExporter(endpoint="tempo.monitoring.svc.cluster.local:4317", insecure=True)
span_processor = SimpleSpanProcessor(otlp_exporter)
trace.get_tracer_provider().add_span_processor(span_processor)

Create a span for tracing

with tracer.start_as_current_span("example-request"):
print("This is a trace example")

This application will generate traces and send them to Tempo. Replace tempo.monitoring.svc.cluster.local:4317 with the actual Tempo endpoint in your OpenShift cluster.

Step 6: Monitor and Visualize

Visualize Metrics: In Grafana, create a new dashboard that uses Prometheus as the data source. Visualize metrics like CPU usage, memory consumption, and request rates.
Visualize Traces: Use Tempo as the data source in Grafana to visualize traces for specific requests. You can link these traces with metrics to get a complete view of your system’s health.

Conclusion

This PoC in CRC OpenShift sets up a complete observability stack with Grafana Tempo for tracing, Thanos for scaling Prometheus and providing long-term metrics storage, and Grafana to visualize metrics and traces. By following these steps, you’ll gain insight into how distributed systems perform, helping you troubleshoot issues faster and understand how your application is behaving across different services.

Let me know if you'd like further assistance with configurations or understanding any of the components!

Imported from rifaterdemsahin.com · 2024