Serverless & Sidecars

On ECS Fargate, Cloud Run and Azure Container Apps there is no server to install on. The agent runs as one more container next to your app, configured from environment variables.

The sidecar is the same image as everywhere else, ghcr.io/zharp-io/agent, started with ZHARP_MODE=sidecar. It needs no config file and no shell: it builds its configuration from the environment when it starts. Containers of one task, service or app share a network, so your app sends to localhost and nothing is exposed outside it.

What the sidecar does

• Traces, metrics and logs over OTLP on localhost:4317 (gRPC) and localhost:4318 (HTTP), from your app's OpenTelemetry SDK. Set OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:4318 in the app.
• Container logs from ECS FireLens on localhost:24224 (Fluent Forward). Each line keeps the container's name and id and whether it came from stdout or stderr.
• StatsD / DogStatsD metrics on udp://localhost:8125, tags included (|#region:eu), for apps that already emit them.
• Log files on a shared volume, for Cloud Run and Container Apps, when ZHARP_LOGS_PATH is set.
• Platform attributes on everything it sends — see what's collected.
• A health check on port 13133. The image has no shell or curl, so container health checks run /zharp-collector health, which exits 0 when the agent is up.
• A clean stop. On SIGTERM it stops accepting data, sends everything it is holding, and exits. Data waits at most 2 seconds to be batched, so a task or instance that is stopped loses nothing it had already received.

It collects nothing about a host — no host metrics, no log files outside ZHARP_LOGS_PATH, no container discovery — because on these platforms there is no host to see. It uses about 25 MB of memory at rest and caps itself at 80% of its container's memory limit, so 128–256 MB is plenty.

Environment variables

Service, environment and version

What your app says about itself wins. For traces, metrics and OTLP logs, the service.name, deployment.environment.name and service.version the SDK sends are kept, and the sidecar's ZHARP_* values only fill in what is missing. An SDK left at its default unknown_service:… name counts as missing. For FireLens logs, a record's own service, env and version fields win, then ZHARP_SERVICE, then the container's name. With nothing set at all, the service is the platform's name for the workload: the ECS task family, the Cloud Run service or the container app.

Try it locally before deploying:

docker run --rm -e ZHARP_MODE=sidecar -e ZHARP_API_KEY=YOUR_API_KEY \
  -e ZHARP_SERVICE=checkout -e ZHARP_ENV=dev \
  ghcr.io/zharp-io/agent:latest print-config     # the config it would run with

# run it, then point an app at localhost:4318 from the same network namespace
docker run -d --name zharp-agent -e ZHARP_MODE=sidecar -e ZHARP_API_KEY=YOUR_API_KEY ghcr.io/zharp-io/agent:latest
docker run --rm --network container:zharp-agent your-app

AWS ECS Fargate

The task runs three containers: your app, AWS's FireLens log router, and the agent. The app's stdout and stderr go through FireLens, which forwards them to the agent; its traces and metrics go to the agent directly.

1. 1
   Store the API key

   aws secretsmanager create-secret --name zharp/api-key --secret-string 'YOUR_API_KEY'
   # the task execution role needs secretsmanager:GetSecretValue on it

2. 2
   Add the two containers to your task definition

   The agent is essential: false, so a problem with it never stops your app. The app waits for the agent to be healthy, so nothing it sends at startup is lost; when the task stops, ECS stops the app first and the agent last, so the app's last lines still get out.

   deploy/ecs/fargate-sidecar-task.json

   {
     "family": "checkout",
     "networkMode": "awsvpc",
     "requiresCompatibilities": ["FARGATE"],
     "cpu": "512",
     "memory": "1024",
     "runtimePlatform": { "operatingSystemFamily": "LINUX", "cpuArchitecture": "X86_64" },
     "executionRoleArn": "arn:aws:iam::ACCOUNT_ID:role/ecsTaskExecutionRole",
     "taskRoleArn": "arn:aws:iam::ACCOUNT_ID:role/checkout-task-role",
     "containerDefinitions": [
       {
         "name": "app",
         "image": "ACCOUNT_ID.dkr.ecr.REGION.amazonaws.com/checkout:2.4.1",
         "essential": true,
         "portMappings": [{ "containerPort": 8080, "protocol": "tcp" }],
         "environment": [
           { "name": "OTEL_EXPORTER_OTLP_ENDPOINT", "value": "http://localhost:4318" },
           { "name": "OTEL_EXPORTER_OTLP_PROTOCOL", "value": "http/protobuf" },
           { "name": "OTEL_SERVICE_NAME", "value": "checkout" },
           { "name": "OTEL_RESOURCE_ATTRIBUTES", "value": "deployment.environment.name=production,service.version=2.4.1" }
         ],
         "logConfiguration": {
           "logDriver": "awsfirelens",
           "options": {
             "Name": "forward",
             "Host": "127.0.0.1",
             "Port": "24224"
           }
         },
         "dependsOn": [
           { "containerName": "zharp-agent", "condition": "HEALTHY" },
           { "containerName": "log_router", "condition": "START" }
         ]
       },
       {
         "name": "log_router",
         "image": "public.ecr.aws/aws-observability/aws-for-fluent-bit:stable",
         "essential": true,
         "firelensConfiguration": {
           "type": "fluentbit",
           "options": { "enable-ecs-log-metadata": "true" }
         },
         "memoryReservation": 50,
         "logConfiguration": {
           "logDriver": "awslogs",
           "options": {
             "awslogs-group": "/ecs/checkout/firelens",
             "awslogs-region": "REGION",
             "awslogs-stream-prefix": "log_router",
             "awslogs-create-group": "true"
           }
         },
         "dependsOn": [{ "containerName": "zharp-agent", "condition": "START" }]
       },
       {
         "name": "zharp-agent",
         "image": "ghcr.io/zharp-io/agent:latest",
         "essential": false,
         "environment": [
           { "name": "ZHARP_MODE", "value": "sidecar" },
           { "name": "ZHARP_SERVICE", "value": "checkout" },
           { "name": "ZHARP_ENV", "value": "production" },
           { "name": "ZHARP_VERSION", "value": "2.4.1" }
         ],
         "secrets": [
           {
             "name": "ZHARP_API_KEY",
             "valueFrom": "arn:aws:secretsmanager:REGION:ACCOUNT_ID:secret:zharp/api-key"
           }
         ],
         "healthCheck": {
           "command": ["CMD", "/zharp-collector", "health"],
           "interval": 10,
           "timeout": 5,
           "retries": 3,
           "startPeriod": 10
         },
         "stopTimeout": 20,
         "cpu": 128,
         "memory": 256,
         "memoryReservation": 128,
         "logConfiguration": {
           "logDriver": "awslogs",
           "options": {
             "awslogs-group": "/ecs/checkout/zharp-agent",
             "awslogs-region": "REGION",
             "awslogs-stream-prefix": "zharp",
             "awslogs-create-group": "true"
           }
         }
       }
     ]
   }

3. 3
   Register it and roll the service

   aws ecs register-task-definition --cli-input-json file://fargate-sidecar-task.json
   aws ecs update-service --cluster YOUR_CLUSTER --service checkout --task-definition checkout

Each log line carries the container's name, so one task with several containers shows them apart. To give containers of one task different services, add the fields in a FireLens config file of your own (FireLens tags a container's records <container>-firelens-<task id>):

# zharp.conf, baked into your own aws-for-fluent-bit image, then in the log_router container:
#   "firelensConfiguration": { "type": "fluentbit", "options": {
#     "enable-ecs-log-metadata": "true",
#     "config-file-type": "file", "config-file-value": "/fluent-bit/etc/zharp.conf" } }
[FILTER]
    Name   modify
    Match  worker-firelens*
    Add    service billing-worker
    Add    env production
    Add    version 9.9.0

Google Cloud Run

Cloud Run services can run sidecars. The agent starts first (the app depends on it), and the app sends OTLP to localhost:4318.

1. 1
   Make the image and the key available

   # Cloud Run pulls from Artifact Registry: proxy ghcr.io once
   gcloud artifacts repositories create ghcr --repository-format=docker \
     --location=REGION --mode=remote-repository --remote-docker-repo=https://ghcr.io
   
   # the API key, readable by the service's account
   printf '%s' 'YOUR_API_KEY' | gcloud secrets create zharp-api-key --data-file=-
   gcloud secrets add-iam-policy-binding zharp-api-key \
     --member=serviceAccount:SERVICE_ACCOUNT --role=roles/secretmanager.secretAccessor
   
   gcloud run services replace service.yaml --region=REGION

2. 2
   Deploy the service

   deploy/cloudrun/service.yaml

   apiVersion: serving.knative.dev/v1
   kind: Service
   metadata:
     name: checkout
   spec:
     template:
       metadata:
         annotations:
           # The agent starts (and passes its startup probe) before the app.
           run.googleapis.com/container-dependencies: '{"app":["zharp-agent"]}'
           # Instance-based billing: with request-based billing the CPU is throttled between
           # requests and the agent can't export until the next request arrives.
           run.googleapis.com/cpu-throttling: "false"
       spec:
         serviceAccountName: SERVICE_ACCOUNT
         timeoutSeconds: 300
         containers:
           - name: app
             image: REGION-docker.pkg.dev/PROJECT_ID/apps/checkout:2.4.1
             ports:
               - containerPort: 8080
             env:
               - name: OTEL_EXPORTER_OTLP_ENDPOINT
                 value: http://localhost:4318
               - name: OTEL_EXPORTER_OTLP_PROTOCOL
                 value: http/protobuf
               - name: OTEL_SERVICE_NAME
                 value: checkout
               - name: OTEL_RESOURCE_ATTRIBUTES
                 value: deployment.environment.name=production,service.version=2.4.1
               # Optional: where the app writes a copy of its log for the agent to tail.
               - name: LOG_FILE
                 value: /var/log/app/app.log
             resources:
               limits:
                 cpu: "1"
                 memory: 512Mi
             volumeMounts:
               - name: app-logs
                 mountPath: /var/log/app
           - name: zharp-agent
             image: REGION-docker.pkg.dev/PROJECT_ID/ghcr/zharp-io/agent:latest
             env:
               - name: ZHARP_MODE
                 value: sidecar
               - name: ZHARP_API_KEY
                 valueFrom:
                   secretKeyRef:
                     name: zharp-api-key
                     key: latest
               - name: ZHARP_SERVICE
                 value: checkout
               - name: ZHARP_ENV
                 value: production
               - name: ZHARP_VERSION
                 value: 2.4.1
               - name: ZHARP_LOGS_PATH
                 value: /var/log/app/*.log
             resources:
               limits:
                 cpu: 500m
                 memory: 256Mi
             startupProbe:
               httpGet:
                 path: /
                 port: 13133
               periodSeconds: 2
               failureThreshold: 15
               timeoutSeconds: 1
             livenessProbe:
               httpGet:
                 path: /
                 port: 13133
               periodSeconds: 30
               failureThreshold: 3
               timeoutSeconds: 3
             volumeMounts:
               - name: app-logs
                 mountPath: /var/log/app
         volumes:
           - name: app-logs
             emptyDir:
               medium: Memory
               sizeLimit: 64Mi

Azure Container Apps

Container Apps runs every container of a revision together. The agent is one of them; the app sends OTLP to localhost:4318.

1. 1
   Fill in the YAML

   Replace the placeholders and the API key. To keep the key in Key Vault, swap value for keyVaultUrl and identity: system.

   deploy/azure-container-apps/app.yaml

   location: REGION
   properties:
     managedEnvironmentId: /subscriptions/SUBSCRIPTION_ID/resourceGroups/RESOURCE_GROUP/providers/Microsoft.App/managedEnvironments/ENVIRONMENT
     configuration:
       activeRevisionsMode: Single
       ingress:
         external: true
         targetPort: 8080
         transport: auto
       secrets:
         - name: zharp-api-key
           value: your_api_key
     template:
       # SIGTERM to SIGKILL; the agent flushes what it holds when a replica stops.
       terminationGracePeriodSeconds: 30
       containers:
         - name: app
           image: REGISTRY.azurecr.io/checkout:2.4.1
           env:
             - name: OTEL_EXPORTER_OTLP_ENDPOINT
               value: http://localhost:4318
             - name: OTEL_EXPORTER_OTLP_PROTOCOL
               value: http/protobuf
             - name: OTEL_SERVICE_NAME
               value: checkout
             - name: OTEL_RESOURCE_ATTRIBUTES
               value: deployment.environment.name=production,service.version=2.4.1
             - name: LOG_FILE
               value: /var/log/app/app.log
           resources:
             cpu: 0.5
             memory: 1Gi
           volumeMounts:
             - volumeName: app-logs
               mountPath: /var/log/app
         - name: zharp-agent
           image: ghcr.io/zharp-io/agent:latest
           env:
             - name: ZHARP_MODE
               value: sidecar
             - name: ZHARP_API_KEY
               secretRef: zharp-api-key
             - name: ZHARP_SERVICE
               value: checkout
             - name: ZHARP_ENV
               value: production
             - name: ZHARP_VERSION
               value: 2.4.1
             - name: ZHARP_LOGS_PATH
               value: /var/log/app/*.log
           resources:
             cpu: 0.25
             memory: 0.5Gi
           probes:
             - type: Startup
               httpGet:
                 path: /
                 port: 13133
               periodSeconds: 2
               failureThreshold: 15
             - type: Liveness
               httpGet:
                 path: /
                 port: 13133
               periodSeconds: 30
               failureThreshold: 3
           volumeMounts:
             - volumeName: app-logs
               mountPath: /var/log/app
       scale:
         minReplicas: 1
         maxReplicas: 10
       volumes:
         - name: app-logs
           storageType: EmptyDir

2. 2
   Create or update the app

   az containerapp create --name checkout --resource-group RESOURCE_GROUP --yaml app.yaml
   # or, for an app that already exists
   az containerapp update --name checkout --resource-group RESOURCE_GROUP --yaml app.yaml

Logs work as on Cloud Run: stdout goes to Log Analytics, so send logs over OTLP, or write them to the shared EmptyDir volume for the agent to tail. Container Apps can't order containers, so an app that sends telemetry in its first second may find the agent not yet listening; OpenTelemetry SDK exporters retry, so this usually costs nothing.

What's collected on each platform

Limits

• No host or container metrics: Fargate, Cloud Run and Container Apps don't expose a host. CPU and memory of the task or instance stay in CloudWatch, Cloud Monitoring and Azure Monitor.
• stdout and stderr are only collected on ECS (through FireLens). On Cloud Run and Container Apps, use OTLP logs or a shared file.
• StatsD counters, gauges and sets are sent. Timers, histograms and distributions are summarised by the agent but not shown in Zharp yet.
• StatsD values are added up over 10 seconds; when a container stops, the last unsent window is lost. OTLP data is not.
• Each running task, instance or replica is listed under Zharp Agent hosts as a Serverless sidecar while it runs, and drops off the list once it has been offline for 15 minutes — scaling up and down doesn't leave a trail of offline hosts.