Provider packages¶

A provider package holds everything Tapyr knows about an API: the Provider (base URL, auth, pagination, limits, errors), job templates, a simulator, scenarios, a contract test, a dashboard and examples. Packages are declarative: YAML and CEL, no Go, except for the rare auth flows written as plugins. The layout is described in 03-configuration.md.

Built-in packages¶

tapyr provider list lists them with the packages of providerPaths, which override built-in packages of the same name; --output json adds their catalog metadata, templates and limits (provider catalog output).

Looking for an API to package? Provider ideas lists candidates in other domains, with their rate-limit patterns and the engine features some of them still need.

Writing a package¶

1. Read the vendor documentation and write provider.yaml. Every limit cites its source in its source (vendor-docs with the link, measured with the date, or assumed), and in its description and the README for humans. The metadata describes the package in the catalog: displayName, categories from providers/categories.yaml, maintainers (GitHub handles), tier (official or community) and validation (simulator, or real-platform with a date and the platform version), see 03-configuration.md. Prefer endpoints that return many objects per request (organization-wide lists, large pages): the budget engine spends quota deliberately, the templates should too.
2. Write the job templates (templates/*.yaml) with estimate.pagesPerRun. Metric names follow <prefix>_<object>_<measure>_<unit> in base units, descriptive attributes go to _info metrics, and no high-cardinality label (client MAC addresses, IPs) is used.
3. Build the simulator (sim/sim.yaml): a synthetic inventory with template fixtures (see 08-simulator.md), or static fixtures from a recording. Ship the default, quota-tight and bad-day scenarios.
4. Write the contract test (tests/contract.yaml, tests/contract.env) and generate tests/expected.prom with tapyr provider lint --update <dir>. Review the samples: they are the contract of the package.
5. Add a Grafana dashboard (dashboards/*.json, with a datasource variable, an API quota row when the API has a quota, and a link to Tapyr / Overview, following the dashboard conventions), examples (examples/*.yaml) and the README. Every package README follows the same skeleton, as in example-sim:
6. an introduction (the API, what is collected) with the Validation status (real platform or simulator, versions, dates);
7. Requirements, when there are (firmware versions, permissions, network access);
8. Coverage: a table Template | Endpoint | Default interval | Requests | Metrics (requests per run), then the labels, joins and conversions;
9. Auth setup: how to get the credentials and give them to Tapyr;
10. Limits: a table Limit | Value | Source, then the quota maths with the default intervals;
11. Simulator: the command, with the port the simulator gets under tapyr sim --all --listen-base 127.0.0.1:8081 (8081 plus its portOffset), the credentials it accepts and a Scenario | Effect table;
12. Dashboard, Known caveats, and End-to-end test when there is one.
13. tapyr provider lint --catalog <dir> must pass.

Dashboard conventions¶

The dashboards of the packages and Tapyr / Overview read the same way, whatever the API:

• Colors carry a meaning. Grafana's named colors (green, orange, red, blue, text), so that the dashboards follow the theme of each Grafana:
• a count of problems (devices down, failed runs) is green at 0, then red from 1 when it needs action, or orange when it is a warning (alerting devices, a backup uplink, a device not seen for an hour);
• a count or a measure with no good or bad value (devices, clients, uptime) is text, never the default thresholds of Grafana (red from 80);
• the share of a quota left is red under 10%, orange under 25%, green above;
• quota charts draw the limit in dashed red, Tapyr in blue and other clients in orange;
• status codes: 2xx green, 429 orange, other errors red.
• States are state timelines. A 0/1 metric (an interface up, an uplink active) is a state timeline with value mappings (Up/Down), not a line between 0 and 1.
• Thresholds come from a source. Mobile signal (RSRP, RSRQ, SINR, RSSI) uses the categories of the Teltonika recommendations as dashed lines; resource use (memory, flash) is orange from 80% and red from 90%. A measure without a sound threshold (temperatures, load) has none.
• Rankings are bars. "The 10 busiest access points" is a bar gauge of the current values, not ten lines that change members over time.
• Empty is an answer. A panel that can have nothing to show (no device down, no exhaustion projected) says so with noValue (None, 0).
• Queries survive push outputs. With remote_write or otlp, a series receives a sample only when its job runs, and queries look back 5 minutes for the last one (outputs). Gauges are read with last_over_time(metric[window]), the window two to three times the default interval of the job that produces them (15m for a 5-minute job, 2h for an hourly one), and the _info metrics of joins with max by (<id>, <labels>) (last_over_time(..._info[window])). Rates of counters need no window of their own.

CEL mapping recipes¶

Patterns taken from the built-in templates. The CEL environment and its functions are described in 06-mapping-and-outputs.md.

Enveloped lists. extract.items points to the list inside the envelope (teltonika-rms, aruba-central-classic):

extract:
  items: response.body.data          # {"success": true, "data": [...], "meta": {...}}

Paginate such a list in the simulator with response.itemsField (see 08-simulator.md).

Flattening nested lists. One item per appliance and uplink (meraki-appliance-uplink-statuses):

items: >-
  response.body.map(a, default(a.uplinks, []).map(u, {'serial': a.serial, 'interface': u.interface, 'status': default(u.status, '')})).flatten()

Absent or null fields. default(value, fallback) covers a missing or null field; has() tests a key before reading below it; skipIf drops a sample rather than inventing a value (meraki-dashboard, aruba-central-classic):

labels:
  name: "default(item.name, '')"
  network_id: "has(item.network) ? default(item.network.id, '') : ''"
metrics:
  - name: aruba_ap_clients
    value: toFloat(item.client_count)
    skipIf: "!has(item.client_count) || item.client_count == null"

Timestamps. parseTimestamp reads RFC 3339 by default, or a Go layout; a timestamp metric is a number of seconds (rms-devices):

value: "double(int(parseTimestamp(item.last_connection_at, '2006-01-02 15:04:05')))"

As the timestamp of a sample, it carries the source time of the value (example-sim-devices): timestamp: parseTimestamp(item.updatedAt).

_info metrics. Descriptive attributes go to an _info metric (value 1) that shares an identifier label with the measures, so that queries join them (meraki-devices):

info:
  - name: meraki_device_info
    labels:
      serial: item.serial
      name: "default(item.name, '')"
      model: "default(item.model, '')"

(meraki_device_up == 0) * on (serial) group_left (name, model) meraki_device_info

Recording real responses¶

tapyr record runs jobs once against the real API, through the budget engine, and writes their responses anonymized:

tapyr record --config ./config --job devices --out ./recording [--anonymize ./anonymize.yaml]

• Serials, MAC and IP addresses, identifiers, names, emails, phone numbers, coordinates and addresses are replaced by consistent pseudonyms that keep their format (a MAC stays a MAC, an identifier keeps its length and its digits); the same value gets the same pseudonym in every page and in the request variables, so the recording stays consistent.
• anonymize.yaml adds rules and exceptions:

replace:
  - { path: "data[*].company_name", as: name }   # serial, digits, mac, ipv4, ipv6, email, hostname, name, id, coordinate, address, redact
keep:
  - "data[*].model"

• report.txt lists every replaced path and rule, never a value. Review the recording before committing anything: free-text fields can still hold identifying data.
• Only the quota headers of the provider and Retry-After are recorded; request credentials never are.
• Run it with the configuration and TAPYR_STATE_KEY of tapyr run: it reads the token pair an auth plugin stored (aruba-central-classic), read-only, and never renews a rotating refresh token, which would leave tapyr run with a spent one.

The teltonika-rms fixtures follow the shape of such a recording.

Linting and contract tests¶

tapyr provider lint providers/teltonika-rms        # one package
tapyr provider lint providers                      # every package of a directory
tapyr provider lint --update providers/my-api      # accept the new samples
tapyr provider lint --catalog providers/my-api     # also require the catalog metadata

The lint loads the package, compiles every template, checks the dashboards (valid JSON, a datasource variable, only metrics of the package or tapyr_ self-metrics), the examples, the required scenarios, then runs the contract test: the jobs of tests/contract.yaml run once against the simulator, on a fake clock starting at 2026-01-01T00:00:00Z, and their samples must equal tests/expected.prom. Every template must be exercised. With --catalog, the catalog metadata of the package and the source of every limit are required too, with existing categories and dates. make lint and the CI lint the built-in packages with --catalog.

End-to-end tests¶

The packages tested against real platforms have e2e tests in test/e2e/ (build tag e2e), skipped unless their credentials are set:

TAPYR_E2E_MERAKI_API_KEY=... make e2e
TAPYR_E2E_RMS_TOKEN=... make e2e
TAPYR_E2E_RUTOS_URL=https://192.0.2.10 TAPYR_E2E_RUTOS_USERNAME=... TAPYR_E2E_RUTOS_PASSWORD=... make e2e

They run tapyr run for one or two minutes and spend a few requests.

Auth plugins¶

When an auth flow cannot be declared (Aruba Central rotates its refresh tokens), it is written in Go under internal/auth/plugins/<name> and registered in internal/auth/plugins. A plugin validates its auth.plugin.config (secret references as {secret: name}), creates an authenticator per account with a sender of auth requests through the budget engine and a sealed store for its state, and provides its simulator side. See 05-collection.md.