When an application is running, knowing that it is up does not necessarily mean knowing what it is doing. For example, imagine an API request that takes several seconds to complete. We would want to answer questions such as: Is the application receiving more requests than usual? What happened during the request? How long does the request take? Did the application encounter an error? Which part of the request should we investigate? Instead of looking at behavior through a single source, we should collect different types of signal and make them available in one observability stack. What We Need to Observe To understand the application from different perspectives, we need three types of telemetry: metrics, logs, and traces. Each signal provides a different view of the application, and each becomes useful at a different stage of an investigation. Metrics Metrics helps us to understand the overall behavior of the application. How many requests are being processed? Are request rates increasing? Are errors becoming more frequent? Is the application behaving differently from normal? This gives us the high level view we need to detect if something might be wrong. Logs Logs give us details about individual events inside the application, such as a request starting, an operation completing, or an error occurring. They become especially useful when we need to understand the context behind an abnormal metric. Traces A trace allows us to follow a request through its execution and understand its timing. This becomes particularly useful when investigating slow requests or determining which part of an operation deserves further attention. Centralize Collection Instead of making the application communicate independently with several observability backends, we can place a telemetry collection layer between the application. This provide a consistent way to receive telemetry from the application and route it to the systems responsible for storing and analyzing each signal. Visualization Jumping between different interfaces makes investigation slower, especially when we need to correlate information from multiple signals. Therefore we need a visualization layer where metrics, logs, and traces can be explored together. Architecture Now that we know what we need to collect, the next question is how these signals should move through the system. The architecture looks like this: The application is the source of the telemetry. When the application generates metrics, logs, and traces, these signals are sent to the OpenTelemetry Collector using OTLP. The Collector then routes each signal to the appropriate backend. Metrics are exposed for Prometheus to collect, logs are sent to Loki, and traces are forwarded to Jaeger. Grafana sits on top of these systems and gives us a single place to explore the data. This creates a clear separation between the application and the systems that store and analyze its telemetry. Application to Collector Consider a request entering the API: GET /api/orders The application processes the request and generates telemetry while doing so. Instead of sending that telemetry directly to multiple backends, the application sends it to the Collector The application therefore only needs to know where the telemetry collection layer is located. This becomes especially useful as the observability stack grows. Additional processing or exporters can be introduced at the Collector without requiring the application to establish a new connection to every backend. Collector to Backends Once telemetry reaches the Collector, the signals follow different paths depending on their type. Each backend therefore has a specific responsibility. Prometheus handles the metrics data, Loki handles application logs, and Jaeger handles traces. From Backends to Grafana Once the telemetry reaches its respective backends, we need a way to explore and make sense of the data. Grafana provides this layer by connecting to Prometheus, Loki, and Jaeger, giving us a single interface to query and visualize the telemetry generated by our application. The Application Before sending any telemetry, we need an application that can generate it. For this project, we use a small Go API as the application being observed. The API exposes several endpoints that represent different application behaviors: GET /health GET /api/orders GET /api/transaction GET /api/users Each endpoint gives us a different scenario to observe. /health represents a simple successful request /api/orders returns application data /api/transaction simulates a slow operation /api/users intentionally returns an error This gives us enough variety to see how metrics, logs, and traces behave when we test the observability stack later. The code snippets in this article focus on the important parts of the implementation. For the complete configuration you can find the source code in the repository below. Github Repository Initializing Opentelemetry The telemetry setup is separated from the HTTP handlers in telemetry.go. The application first defines its service identity: res, err := resource.New( ctx, resource.WithAttributes( semconv.ServiceName("observability-api"), ), ) This gives the telemetry a consistent service name: observability-api. More importantly, each telemetry provider is configured to send its data to the OpenTelemetry Collector: otlptracegrpc.WithEndpoint("otel-collector:4317") The same Collector endpoint is used for metrics and logs. This means the application does not need to know where Prometheus, Loki, or Jaeger are running. It only needs to know the address of the Collector. The trace, metric, and log providers are then registered globally in the application. Instrumenting HTTP Requests The next step is making the HTTP server itself observable. mux.HandleFunc("/health", health) mux.HandleFunc("/api/orders", orders) mux.HandleFunc("/api/transaction", transaction) mux.HandleFunc("/api/users", users) Instead of passing this multiplexer directly to the HTTP server, we wrap it with OpenTelemetry's HTTP instrumentation: handler := otelhttp.NewHandler( mux, "HTTP Server", ) The resulting handler is then used by the HTTP server: server := &http.Server{ Addr: ":8080", Handler: handler, } This instrumentation sits in front of all four endpoints: As requests pass through this layer, OpenTelemetry can capture HTTP request telemetry and associate it with the request context. Logging Inside Each Endpoint For application specific events, we use a small helper called emitLog. The helper receives the request context, severity, message, and additional attributes: func emitLog( ctx context.Context, severity log.Severity, message string, attrs ...attribute.KeyValue, ) { ... logger.Emit(ctx, record) } The important part is the context: logger.Emit(ctx, record) Each handler passes r.Context() into emitLog, so the log is emitted using the context of the current HTTP request. The endpoint therefore does not need to know how the log will eventually reach Loki. It only creates the log event through the OpenTelemetry logging API. /health The simplest example is the health endpoint. When we call: GET /health the handler generates an informational log: emitLog( r.Context(), log.SeverityInfo, "health check", attribute.String("endpoint", "/health"), ) The application therefore produces telemetry as part of processing the request rather than requiring a separate logging pipeline inside the handler. /api/orders The orders endpoint follows the same pattern, but adds more context to the log. After creating the orders, the handler emits: emitLog( r.Context(), log.SeverityInfo, "orders fetched successfully", attribute.String("endpoint", "/api/orders"), attribute.Int("order_count", len(orders)), ) Now the telemetry contains not only the event itself, but also structured attributes such as the endpoint and number of orders. For example: message: orders fetched successfully endpoint: /api/orders order_count: 3 These attributes become useful later when we query the logs with Loki. /api/transaction The transaction endpoint gives us a more interesting example because it intentionally takes three seconds to complete. First, the handler records the start of the operation: start := time.Now() emitLog( r.Context(), log.SeverityInfo, "transaction started", attribute.String("endpoint", "/api/transaction"), ) After the simulated operation: time.Sleep(3 * time.Second) the handler calculates the duration and emits another log: duration := time.Since(start) emitLog( r.Context(), log.SeverityInfo, "transaction completed", attribute.String("endpoint", "/api/transaction"), attribute.Int64("duration_ms", duration.Milliseconds()), ) /api/users Finally, we have an endpoint that intentionally returns an error. emitLog( r.Context(), log.SeverityError, "failed to fetch users", attribute.String("endpoint", "/api/users"), attribute.Int("status_code", http.StatusInternalServerError), ) The request therefore produces an error-level log before returning HTTP 500. This gives us a different scenario from /api/transaction: instead of investigating latency, we can later use the observability stack to investigate an application error. Building the Observability Pipeline With the application ready, we can now configure how that telemetry is processed and delivered to each backend. Configuring the OpenTelemetry Collector The Collector accepts OTLP through gRPC: receivers: otlp: protocols: grpc: endpoint: 0.0.0.0:4317 The Collector then uses the batch processor: processors: batch: Rather than defining one large pipeline for every signal, the configuration separates metrics, logs, and traces: service: pipelines: metrics: receivers: - otlp processors: - batch exporters: - prometheus traces: receivers: - otlp processors: - batch exporters: - otlp/jaeger logs: receivers: - otlp processors: - batch exporters: - otlphttp/loki Collector to Prometheus Metrics use the Prometheus exporter: exporters: prometheus: endpoint: "0.0.0.0:8889" The Collector exposes the metrics endpoint on port 8889. This is slightly different from the way telemetry enters the Collector. The application sends telemetry to the Collector, but Prometheus does not receive metrics through a push connection. Instead, Prometheus periodically scrapes the endpoint exposed by the Collector. global: scrape_interval: 5s scrape_configs: - job_name: "otel-collector" static_configs: - targets: - "otel-collector:8889" Prometheus therefore scrapes the Collector every five seconds. Collector to Loki Logs use the otlphttp/loki exporter: otlphttp/loki: endpoint: http://loki:3100/otlp The Collector sends the processed logs to Loki through its OTLP HTTP endpoint. This works particularly well with the structured information we added to the application logs. For example, the /api/orders endpoint produces: message: orders fetched successfully endpoint: /api/orders order_count: 3 Instead of treating the log as only a text message, these additional attributes provide context that can be queried later. Loki is configured to support structured metadata: limits_config: allow_structured_metadata: true volume_enabled: true The project also uses filesystem storage for Loki, backed by the loki-data Docker volume. Collector to Jaeger Traces use a separate OTLP exporter: otlp/jaeger: endpoint: jaeger:4317 tls: insecure: true The traces pipeline connects this exporter to the OTLP receiver and batch processor. Jaeger is configured to receive OTLP through gRPC: receivers: otlp: protocols: grpc: endpoint: 0.0.0.0:4317 For this project Jaeger uses an in memory backend: extensions: jaeger_query: storage: traces: memory jaeger_storage: backends: memory: memory: max_traces: 100000 The trace exporter writes into that memory storage: exporters: jaeger_storage_exporter: trace_storage: memory Running the Pipeline All of these components are defined as separate Docker Compose services. The Collector mounts our configuration file into the container and exposes its OTLP and Prometheus endpoints: otel-collector: image: otel/opentelemetry-collector-contrib:latest command: - "--config=/etc/otelcol-contrib/otel-collector.yml" volumes: - ./otel/otel-collector.yml:/etc/otelcol-contrib/otel-collector.yml:ro ports: - "4317:4317" - "8889:8889" The services are connected through the research-observability Docker network, allowing the Collector to reach services such as loki and jaeger directly by their Docker service names. Bringing Everything Together with Grafana grafana: image: grafana/grafana:latest container_name: grafana environment: - GF_SECURITY_ADMIN_USER=admin - GF_SECURITY_ADMIN_PASSWORD=admin - GF_AUTH_ANONYMOUS_ENABLED=false volumes: - grafana-data:/var/lib/grafana ports: - "3001:3000" The container exposes Grafana on port 3001 of the host, while Grafana itself listens on port 3000 inside the container. Because Grafana, Prometheus, Loki, and Jaeger are connected to the same research-observability Docker network, Grafana can communicate with the other services using their Docker service names. This means we can configure the data sources using addresses such as: http://prometheus:9090 http://loki:3100 http://jaeger:16686 Adding Prometheus as a Data Source From the Grafana dashboard, open: Connections → Data sources Then select Add data source and choose Prometheus. For the server URL, we use the Docker service name instead of localhost: http://prometheus:9090 After entering the URL, select Save & test. If the connection is successful, Grafana confirms that it can communicate with Prometheus. Adding Loki We repeat the same process for Loki. Go to: Connections → Data sources → Add data source Then select Loki. For the URL, use: http://loki:3100 After entering the URL, select Save & test. Once Grafana confirms the connection, Loki is available as a data source for exploring application logs. Adding Jaeger Connections → Data sources → Add data source select Jaeger. For the URL, use Jaeger's query endpoint: http://jaeger:16686 This is different from the OTLP endpoint used by the Collector. The Collector sends traces to Jaeger through its OTLP gRPC endpoint, while Grafana connects to Jaeger's query interface to retrieve traces. After entering the URL, select Save & test. At this point, Grafana has access to all three observability backends. Verifying the Data Sources Before building any dashboard, we can verify that Grafana can actually query the data. Open: Explore From the data source selector, we can choose between: Prometheus Loki Jaeger For example, selecting Prometheus allows us to run a metrics query directly from Grafana. This gives us a simple way to verify each connection before creating dashboards. Investigation Now that Grafana is connected to our observability backends, we can use the stack to investigate actual application problems. Instead of simply checking whether telemetry is being collected, we want to answer a more practical question: What happens when something goes wrong? For this example, we will investigate /api/transaction A Slow Transaction As mention before our /api/transaction endpoint intentionally waits for three seconds before returning a response: time.Sleep(3 * time.Second) From the users perspective, the problem is simple: the API feels slow. However, knowing that a request is slow does not immediately tell us why. We start by sending a request to: GET /api/transaction After generating several requests, we can open our Grafana dashboard and look at the request duration. The metric tells us that the request is taking significantly longer than a normal API request. But metrics alone do not provide much context about what happened inside the request. This is where we move to the logs. Looking at the Logs In Grafana, we can open the Loki data source and search for logs generated by the transaction endpoint. The application produces two important events: transaction started transaction completed The completion log also contains the duration: message: transaction completed endpoint: /api/transaction duration_ms: 3000 Now we know that the request really did spend approximately three seconds processing the transaction. The logs give us more context than the metric alone, but we still do not have a detailed view of the request execution. Following the Trace The same request also generates a trace that is exported through the OpenTelemetry Collector and stored in Jaeger. From Grafana, we can open the corresponding trace and inspect its timeline. The trace gives us a view of how long the request took and where that time was spent. From Symptom to Cause This scenario demonstrate the main reason we built the observability stack. Each signal gives us a different level of information. Metrics help us detect that something unusual is happening. Logs give us additional application context. Traces allow us to follow the request and investigate its execution. The important part is not simply having all three signals available. The real value comes from being able to use them together when investigating a problem. Conclusion Building this observability stack gave us a practical way to understand what is happening inside an application instead of only knowing whether it is running. We started with a simple Go API and added OpenTelemetry to generate metrics, logs, and traces. The OpenTelemetry Collector then provided a central layer for processing and routing those signals to Prometheus, Loki, and Jaeger. Grafana brought everything together into a single interface. The /api/transaction example showed why having multiple signals matters. Metrics helped us identify that the request was slow. Logs provided additional context about the transaction, while the trace allowed us to inspect the request execution and its timing. The code snippets in this article focus on the important parts of the implementation. For the complete configuration you can find the source code in the repository below. GitHub Repository: https://github.com/muhammadyulasfipahrizal/E2E-observability-stack.git