# OpenTelemetry Send your sandboxes' metrics and events to Datadog, Grafana Cloud, Honeycomb or any OTLP collector. agental sends what happens to your sandboxes to one OTLP/HTTP collector that you choose. Nothing is installed in the sandboxes. | What | Where | When | | --- | --- | --- | | Sandbox and account metrics | `/v1/metrics` | Every 60 seconds | | Sandbox events, as log records | `/v1/logs` | As each happens | ## Set the collector `PUT /v1/telemetry` ```shell curl -X PUT https://agental.io/v1/telemetry \ -H "Authorization: Bearer $AGENTAL_API_KEY" \ -H "Content-Type: application/json" \ -d '{"endpoint":"https://otlp.datadoghq.com","headers":{"dd-api-key":""}}' ``` - `endpoint` is the collector's base address. `/v1/metrics` and `/v1/logs` are added to it, and one you paste with either on the end is taken off. - It has to be `https://`, on port 443 or 4318, on the public internet. Redirects are not followed. - `headers` are sent with every export: what your collector checks, such as its API key. At most 20. Their values are kept and never shown again, not even to you. - An account has one collector. Setting it again replaces it. - Setting it sends nothing. The first metrics go at the next whole minute, and nothing from before it was set is sent. ## Test it `POST /v1/telemetry/test` ```shell curl -X POST https://agental.io/v1/telemetry/test \ -H "Authorization: Bearer $AGENTAL_API_KEY" ``` Sends one empty metrics export with your headers, and answers with what the collector said: `ok`, its HTTP `status`, and a `message` when something is wrong. A wrong key or address shows here, not after a minute of waiting. In the CLI, `agental telemetry set -H name=value` sets it and tests it. ## Vendor settings | Vendor | Endpoint | Headers | | --- | --- | --- | | Datadog | `https://otlp.datadoghq.com`, or the OTLP host of your Datadog site | `dd-api-key=` | | Grafana Cloud | The OTLP endpoint on your stack's OpenTelemetry card, such as `https://otlp-gateway-prod-eu-west-2.grafana.net/otlp` | `Authorization=Basic ` | | Honeycomb | `https://api.honeycomb.io`, or `https://api.eu1.honeycomb.io` | `x-honeycomb-team=` | | Your own OpenTelemetry Collector | Its OTLP/HTTP receiver behind TLS, on port 443 or 4318 | Whatever it checks | ## Metrics - All are gauges. - Every point of a sandbox has `agental.sandbox.id` and `agental.sandbox.name`, so each sandbox is its own series. - A sandbox sends points only while it runs. Each export holds the minutes since the last one the collector took. | Metric | Unit | What it is | | --- | --- | --- | | `agental.sandbox.cpu.utilization` | `1` | Share of the sandbox's CPU time that was busy, 0 to 1. | | `agental.sandbox.memory.utilization` | `1` | Share of its memory in use, 0 to 1. | | `agental.sandbox.memory.usage` | `By` | Memory in use. | | `agental.sandbox.filesystem.utilization` | `1` | Share of its root disk in use, 0 to 1. | | `agental.sandbox.filesystem.usage` | `By` | Root disk space in use. | | `agental.sandbox.network.io` | `By/s` | Bytes a second over its network link, with `network.io.direction` `receive` or `transmit`. | | `agental.sandbox.disk.io` | `By/s` | Bytes a second read from and written to its disk, with `disk.io.direction` `read` or `write`. | | `agental.account.sandboxes.active` | `{sandbox}` | Sandboxes running now. | | `agental.account.sandboxes.held` | `{sandbox}` | Sandboxes the account holds, running or not. | | `agental.account.sandboxes.limit` | `{sandbox}` | The most sandboxes the account may hold at once, which `held` counts against. Not sent where there is no limit. | There is no figure for time stalled on disk. The host does not measure it for a virtual machine, and the CPU's own `iowait` is not the same thing, so it is not sent in its place. | Resource attribute | Value | | --- | --- | | `service.name` | `agental` | | `service.instance.id`, `agental.sandbox.id` | The sandbox's id | | `agental.sandbox.name` | Its name | | `agental.sandbox.size` | Its size | | `agental.sandbox.memory.size`, `agental.sandbox.disk.size` | Its memory and its root disk, in bytes | | `agental.account.id` | Your account's id | ## Events Each [sandbox event](https://agental.io/docs/webhooks) is one log record. Its name is in the record's `eventName` and in its `event.name` attribute, and its body is the webhook's body, word for word. | Attribute | Value | | --- | --- | | `agental.event.id` | The same id as the webhook's. Use it to remove duplicates. | | `agental.sandbox.id`, `agental.sandbox.name` | The sandbox | | `agental.sandbox.status`, `agental.sandbox.previous_status` | Where it stands now, and where it stood | | `agental.sandbox.reason` | Why it stopped, on `sandbox.stopped` | | `agental.sandbox.error` | What went wrong, on `sandbox.error` | `sandbox.error` is sent with the severity `ERROR`, and every other event with `INFO`. ## Delivery and errors | Collector answers | What agental does | | --- | --- | | `2xx` | Done. A partial success shows its message in `lastError`. | | `429`, `502`, `503`, `504`, or no answer in 10 seconds | Sends the same again later, after `Retry-After` when the collector sends one. | | Any other status | Lets that export go and shows the error in `lastError`. Fix the key or the address. The next minute is sent as usual. | - Metrics are sent in time order. - After an outage, the metrics of the last 10 minutes that the collector missed are sent, and no older. They are read again from where they are kept, not held for you meanwhile. - Events the collector missed are sent again, from the last 24 hours at most. ## Read and remove `GET /v1/telemetry` ```shell curl -X GET https://agental.io/v1/telemetry \ -H "Authorization: Bearer $AGENTAL_API_KEY" ``` The answer's `telemetry` is null when none is set. Otherwise it has the `endpoint`, the names of its `headers`, `lastExportAt`, when the collector last took the metrics, and `lastError`, the last thing that went wrong with the metrics or the events. `DELETE /v1/telemetry` ```shell curl -X DELETE https://agental.io/v1/telemetry \ -H "Authorization: Bearer $AGENTAL_API_KEY" ``` Nothing more is sent to the collector, and its headers are forgotten. ## In the console Settings has an OpenTelemetry export section: set the collector, send a test, and see the last export and the last error. ## Errors | Answer | Meaning | | --- | --- | | `400 invalid_request` | The endpoint or a header cannot be used. `message` says which and why. | | `404 not_found` | A test or a removal, with no collector set. | --- Every page of these docs as Markdown: [llms.txt](https://agental.io/llms.txt)