How-to use OpenTelemetry with Agent metrics - Kubernetes

Introduction

The Terraform Cloud Agent has the option to export metrics to OpenTelemetry Collector as documented here

This article shows an example of using the OpenTelemetry fileexporter (see here for more details) to write the telemetry data to a file in a container

Expected Outcome

At the end you have a working example of using the Terraform Cloud Agent metrics and the possibilities. 

Prerequisites

• Terraform Cloud/Terraform Enterprise

Procedure

Below is an example configuration to use 

• Create the OpenTelemetry configuration as a configmap
  
  

apiVersion: v1
data:
  config.yaml: "extensions:\n  memory_ballast:\n    size_mib: 128\n\nreceivers:\n
    \ otlp/tfe_agent:\n    protocols:\n      grpc:\n        endpoint: 0.0.0.0:4317\n\nprocessors:\n
    \ batch:\n  memory_limiter:\n    check_interval: 1s\n    limit_mib: 1024\n    spike_limit_percentage:
    20\n\nexporters:\n  file/no_rotation:\n    path: /exported_data/telemetry_output.json\n
    \ logging:\n  file:\n    path: /dev/null\n\nservice:\n  extensions: [memory_ballast]\n
    \ pipelines: \n    traces: \n      receivers: [otlp/tfe_agent]\n      exporters:
    [file]\n    metrics/tfe_agent:\n      receivers: [otlp/tfe_agent]\n      processors:
    [memory_limiter, batch]\n      exporters: [file/no_rotation]\n  telemetry:"
kind: ConfigMap
metadata:
  name: otel-config
  namespace: terraform-enterprise-agents

• Create a pod with the OpenTelemetry container and a Busybox container to view the log that gets created

apiVersion: v1
kind: Pod
metadata:
  name: otel-collector
  namespace: terraform-enterprise-agents
  labels:
    app: otel-collector
spec:
  containers:
    - name: busybox
      image: busybox
      command: [ "sleep","infinity" ]
      volumeMounts:
        - name: data-volume
          mountPath: /exported_data
    - name: collector
      image: otel/opentelemetry-collector-contrib:0.73.0
      args: ["--config", "/etc/otel/config.yaml"]
      ports:
        - containerPort: 4317
      volumeMounts:
        - name: config-volume
          mountPath: /etc/otel/config.yaml
          subPath: config.yaml
        - name: data-volume
          mountPath: /exported_data
  volumes:
    - name: config-volume
      configMap:
        name: otel-config
    - name: data-volume
      emptyDir: {}

• Create a service for the OpenTelemetry pod
  
  

apiVersion: v1
kind: Service
metadata:
  name: otel-collector
  namespace: terraform-enterprise-agents
spec:
  selector:
    app: otel-collector
  ports:
    - name: otlp-grpc
      port: 4317
      targetPort: 4317

• Make sure that when Terraform Enterprise is starting a pod it uses the OpenTelemetry to send the logs. Add the following to your overrides.yaml used for the Terraform Enterprise helm installation

agentWorkerPodTemplate:
  spec:
    containers:
    - env:
      - name: TFC_AGENT_OTLP_ADDRESS
        value: otel-collector.terraform-enterprise-agents.svc.cluster.local:4317

• After doing a run you should see the metrics in the log output of OpenTelemetry. Verify this by using the busybox container

kubectl -n terraform-enterprise-agents exec -it otel-collector -c busybox -- sh

cat /exported_data/telemetry_output.json

• The file should contain something like below

{"key":"agent_name","value":{"stringValue":"agent-tele"}},{"key":"agent_pool_id","value":{"stringValue":"apool-RjP1Tazzy1HU1gGE"}}

Additional Information

• Terraform Cloud Agent metrics can be found here

• OpenTelemetry Collector can be found here
• OpenTelemetry fileexporter details can be found here