Approach
An environment is only as good as its grader.
These are the rules we build to. They are the reason a score from a Tier3 Labs environment means what it says.
- Grade the world, not the transcript
- A support agent’s message can claim a fix that never happened. Our graders read the product’s state and the server-side action log, never the agent’s words. The written reply is scored last, with a small weight, and only when the state is already right.
- Assume the agent will cheat
- Reward hacking is the audited risk in RL environments. Before any model runs, scripted gaming agents try the obvious exploits: claim it is fixed, escalate everything, delete and recreate, open every permission, address the grader directly. A model told to cheat runs against every release. Each attempt, whether it worked, and the fix are logged.
- Measure difficulty, do not assert it
- Labs train on tasks in a measured band: hard enough that a model fails often, solvable enough that it sometimes succeeds. We measure pass rates per task against real checkpoints, retire what saturates, add compound tasks where the ceiling is too low, and ship the numbers.
- Every task must be real and fair
- A do-nothing agent must score zero on every task. The reference solution, written through the agent’s own tools, must pass every task. A validator confirms that each fault actually breaks what the ticket says is broken. Failures from model runs are read one by one, and graders that punish a harmless, reversible action are corrected.
- No real customer data
- Worlds are fictional: companies, people, data and configuration are generated from a fixed seed, and names are checked against the public record so they do not collide with real organizations. Failure types are learned from how products break in public, without copying anyone’s tickets.
- Open-source substrates, used properly
- Environments are built on unmodified open-source products, pinned to a version, with their licenses and notices retained. We do not ship enterprise builds or modify the product.
- Reproducible by construction
- Deterministic seeds, byte-identical resets in seconds, pinned product versions and documentation snapshots, serialized episodes and a complete action log. Rerun any episode and get the same world.
What this is not
- Not a dataset of support transcripts.
- Not a chat simulator that scores tone.
- Not a benchmark with an answer key to memorize.
The agent changes a live system, and the system is what gets checked.
Questions about our process?
Research, evaluation and legal teams are welcome to ask for our build and grading documentation. Contact us.