Environments
What is inside a Tier3 Labs environment.
Six parts, shipped together as containerized tasks. Everything the agent can touch is logged. Everything the grader needs is outside the agent’s reach.
- Product
- An unmodified, self-hosted SaaS product running in containers with its own databases and the services around it: mail relay, webhook receivers, an identity provider where the tickets call for one. Each tenant in the product is one fictional customer.
- World
- Fictional companies, named people with roles, per-tenant data, dashboards, rules, integrations and permissions. Generated from a fixed seed, so every episode starts from a byte-identical state and a reset takes seconds.
- Tasks
- Each task is a support ticket written the way customers actually write them: imperfect, sometimes blaming the wrong thing. It comes with a fault script, a scope of what may change, a list of checks where every requirement in the ticket maps to one check, a reply rubric and a reference solution.
- Tools
- A small JSON tool set, identical for every model: the product’s API through a logging proxy, logs, read-only SQL, documentation search over a pinned snapshot, functional probes such as health checks and test notifications, escalate and reply. No shell, no filesystem, no network.
- Grader
- Runs in its own container after the episode is frozen. It queries the product and databases directly, fires test notifications and reads the sinks, logs in as customer users, and diffs a snapshot of the whole world against what the task allowed to change.
- Runners
- A harness for any API model and a trajectory format you can train on. The same prompt and tool schemas for every model, so baselines are comparable.
Task families
The failure types a support engineer actually sees.
Our first environments model multi-tenant infrastructure SaaS: monitoring, identity, and the integrations between them. Families are added per customer around the products and workflows you want.
- Data sources and integrations: rotated credentials, wrong hosts, TLS and permission mismatches.
- Dashboards and configuration: missing data, wrong tenant, broken variables, restricted folders.
- Alerting and notifications: paused rules, misrouted policies, broken contact points, mute timings.
- Users, teams and permissions: locked-out viewers, disabled accounts, users in the wrong organization.
- Authentication and SSO: identity-provider changes that break login, role mapping, redirect errors.
- Security judgment and policy: requests that should be refused, escalated or only partly fulfilled.
- Diagnosis: tickets with a false premise, two independent faults behind one symptom, the wrong fix already applied.
Tiers
A gradient from clear to compound.
Tier 1
One misdescribed setting, a clear symptom. The agent has to find the right object and make one careful change.
Tier 2
The ticket blames the wrong thing. Diagnosis needs logs, the database or a probe, and the fix takes two or three changes.
Tier 3
Multiple causes, red herrings, cross-system faults, or a request that should be refused. Where frontier models still fail.
Scoring
One number, and how it is earned.
State is binary: every check in the task passes or the episode scores zero. Checks query the product, not the transcript.
Penalties multiply the state score down for writes outside the ticket’s scope, destructive deletes, and collateral changes found by diffing the world before and after.
Reply is judged against a short rubric by a model given only the verified facts, carries a small weight, and is gated to zero unless the state passed. A good explanation cannot rescue a wrong fix.
A task counts as passed when every check passes and the penalty factor stays above a floor. Pass rates are reported per task as pass@1 and pass^k so that consistency, not just a lucky run, is what you see.
state = 1 if every check passes else 0 penalty = product of (1 - p_i) over penalties reply = rubric fraction, only if state == 1 score = state * 0.85 * penalty + 0.15 * reply
Want a set built around your tickets?
Tell us the products, workflows and failure types you care about. We will scope the families, tiers and volume with you before anything is built.
Talk to us