agentenv environment · MuJoCo 3.17 · Genesis 1.4.5 · Apache-2.0

GenesisWorkbenchEnv: fix an object’s physics from one complaint

A sim-ready object comes out of a VLM-to-CAD step with a physics bug. A contributor says what feels wrong, and that sentence is the whole task prompt. An LLM agent works out the cause and edits the object through MCP tools while Genesis simulates and renders it. When the agent stops, a probe suite it never sees runs the edited object in Genesis and compares its behaviour with a reference.

On this page: contributor sessions · a knife block, validated · any MJCF, validated (scanned objects, ReplicaCAD furniture, kitchen scenes)

The agent getsA complaint and an object

One MJCF file with one physics bug from the CAD step, and a contributor’s sentence about how it misbehaves. Nothing says which parameter is wrong.

It can useSeven MCP tools

inspect, render (returns PNG frames), set_inertial, set_material, set_joint, run_probe and diff, all backed by a live Genesis scene.

It is graded byA hidden probe suite, in MuJoCo

MuJoCo, the engine the objects ship for, runs the edited object through the task’s probes and checks it against reference behaviour, for a score from 0 to 1 and a pass or fail. Genesis runs the same checks as a second opinion. An approved object is exported as MJCF and OBJ that load in plain MuJoCo.

The mug, oven door and drawer as delivered beside the agent's fix, simulated and rendered in Genesis — Each object as delivered (red) beside the agent's fix (teal), simulated and rendered by Genesis from the MJCF recorded in the episodes: the mug on a tilt table, the oven door unlatched at 5°, the drawer pulled with 10 N.

A knife block, validated in MuJoCo and exported as MJCF and OBJ

The workbench grades in MuJoCo, the engine these objects ship for, and Genesis runs the same checks as a second opinion. This knife block is a set: a beech block and six free knives that sit in its slots. Its probes ask what a robot would. Do the knives stay seated? Do they drop into their slots? Do they pull out without lifting the block? How far can the table tilt? As delivered, the chef's and bread knives were each given to MuJoCo as one collision mesh. MuJoCo collides a mesh as its convex hull, so those knives are as wide as their handles, and they can't enter a 4 mm slot. The fix gives them a convex hull per part. One check asks whether anything goes into anything else: after every probe nothing may rest more than 0.5 mm inside another body or the floor, or ever go more than 3 mm in. A reviewer caught a chef's knife blade showing through the block's front wall in the lean replay. MuJoCo had pushed it there. Its default contacts are soft, so on impact the blade sank further into the wall than it is thick, and the wall then pushed it out sideways. The block now collides as whole walls rather than strips a few mm wide, contacts are stiffer, and the check fails any object where that happens. Pick a state and a probe to replay what MuJoCo simulated.

State

Probe

simulated in MuJoCo

How far it can tilt before it tips

The static margin: the angle at which the set's centre of mass, with every knife in, passes the edge of its rubber feet. It's least to the back, where the block and the handles lean.

This probe, knife by knife

Download the MJCF + OBJ (zip)

The zip has scene.xml, which attaches each object from its own MJCF file, plus the OBJ meshes and report.json. Approving an object in the workbench writes the same export, and loads it back in MuJoCo before it counts.

Any MJCF, validated

scripts/validate_mjcf.py checks a MuJoCo asset the way the workbench checks its own. Give it an MJCF file, a folder with a scene.xml, an export zip, or a bare OBJ. It reads the model: where mass and inertia come from, density over the collision volume, centre of mass, friction and contact softness, how much a mesh's convex hull fills in, thin and floating collision pieces, joints. Then it simulates the asset on a floor at its own timestep and settings: does it settle, does anything go into anything else, does it end the same at half the step, how far can it tilt, how does it drop, and do its joints hold and reach their range. Below are twelve scanned kitchen objects from the Google Scanned Objects set as converted for MuJoCo, one of them given as its OBJ alone, and four of the workbench's own exports. Then come five pieces of ReplicaCAD's articulated furniture, converted from URDF with scripts/convert_urdf.py, and kitchen scenes built from them with scripts/compose_kitchen.py. Each scene has the counter and the fridge fixed in place, with the approved objects set out on the counter top, one layout per seed. For a scene, the checks ask three things. Does everything come to rest? Does anything go into anything else? Does every drawer and door open all the way with the objects in place?

Replay

simulated in MuJoCo

The report as Markdown

Every scanned object here fails the same way. It has no joint, so MuJoCo welds it to the world. It has no <inertial>, so MuJoCo gives it mass at water's density, and the visual mesh counts too, so half or more of that mass is a second copy of the object. Its friction and contact softness are MuJoCo's defaults. The workbench's exports set all of these. ReplicaCAD's furniture has placeholder inertia (a unit inertia on every link) and door centres of mass at their hinges. Some of its URDFs are authored for a Y-up world, with a root rotation of 1.565 rad where π/2 was meant, and one has a malformed vector. Its counter top is drawn about 5 mm above its collision box.

Scanned objects: Google Scanned Objects, CC BY 4.0. Furniture: ReplicaCAD, CC BY 4.0.

With a contributor in the loop

In the contributor-* tasks a contributor plays with the object on a page the env serves. They lift, lean, drop, fill or shake it, or open its doors, and Genesis simulates each action. They chat with the agent, which reads what they did with last_interaction, and they approve or reject the object. Below, each session is replayed on one clock. The 3D view shows the object as it was at the selected moment, and it plays back the motion Genesis recorded for each action, from the same meshes Genesis simulated. The red cylinder is a 330 ml can, for scale. Drag to orbit, and pick any event in the timeline or the transcript.

speed as delivered

contributor actioncontributor messageagent replyagent editagent probedecision

Screen recordings of the contributor page, and every session

contributor-knife-block Two knives that collide as one convex hull each and can't enter their slots. Insert shows it; the agent gives them a hull per part; graded in MuJoCo.

contributor-cad-mug A VLM-to-CAD mug built at 0.7× scale. The agent asks for the real size, scales it, and keeps the mass right.

contributor-cad-frother A loose power-cord ring the CAD left floating 25 mm from the frother. Shake finds it; the agent removes it.

contributor-rcad-cupboard ReplicaCAD doors that jam on each other. Open shows which; the agent turns off self-collision and says what that costs.

Task	Contributor	Outcome	Grade	Agent replies	The agent's edits	Sim asset
`contributor-cad-bowl`	scripted	approved	1.00	2	`set_scale(factor=1.99)`; `set_shell(part=body, thickness=0.006)`; `set_shell(part=body)`; `set_material(part=body, material=ceramic)`; `set_inertial(link=body, mass=2)`	v1
`contributor-cad-bowl`	scripted	approved	1.00	2	`set_scale(factor=1.99)`; `set_shell(part=body, thickness=0.008)`; `set_shell(part=body, thickness=0.0029)`	v2
`contributor-cad-coffeemaker`	scripted	approved	1.00	2	`set_shell(part=housing, thickness=0.003)`; `set_shell(part=steel, thickness=0.001)`; `set_shell(part=steel_2, thickness=0.001)`; `set_shell(part=housing, thickness=0.002)`; `set_shell(part=steel, thickness=0.0009)`; `set_shell(part=steel_2, thickness=0.0009)`	v1
`contributor-cad-coffeemaker`	scripted	approved	1.00	1	`set_shell(part=housing, thickness=0.003)`; `set_shell(part=steel, thickness=0.0015)`; `set_shell(part=steel_2, thickness=0.0015)`; `set_shell(part=steel, thickness=0.001)`; `set_shell(part=steel_2, thickness=0.001)`; `set_shell(part=housing, thickness=0.0025)`	v2
`contributor-cad-frother`	browser (recorded)	approved	1.00	1	`remove_part(part=cord, restore=False)`	v1
`contributor-cad-frother`	browser (recorded)	approved	1.00	1	`remove_part(part=cord, restore=False)`	v2
`contributor-cad-mug`	scripted	approved	1.00	3	`set_scale(factor=1.4)`; `set_shell(part=body, thickness=0.004)`; `set_shell(part=body)`; `set_inertial(link=body, mass=0.3)`; `set_scale(factor=1.516)`; `set_inertial(link=body, mass=0.3)`	v2
`contributor-cad-mug`	scripted	approved	1.00	3	`set_scale(factor=1.4)`; `set_shell(part=body, thickness=0.005)`; `set_shell(part=body)`; `set_inertial(link=body, mass=0.3)`; `set_scale(factor=1.516)`; `set_inertial(link=body, mass=0.3)`	v5
`contributor-cad-mug`	browser (recorded)	approved	1.00	3	`set_scale(factor=1.38)`; `set_inertial(link=body, mass=0.3)`; `set_scale(factor=1.516)`; `set_inertial(link=body, mass=0.3)`	v3
`contributor-cad-mug`	browser (recorded)	approved	1.00	3	`set_scale(factor=1.4)`; `set_shell(part=body, thickness=0.004)`; `set_shell(part=body)`; `set_inertial(link=body, mass=0.3)`; `set_scale(factor=1.516)`; `set_inertial(link=body, mass=0.3)`	v4
`contributor-knife-block`	browser (recorded)	approved	1.00	1	`set_collision(body=chef_20cm, mode=parts)`; `set_collision(body=bread_20cm, mode=parts)`	v1
`contributor-rcad-cupboard`	scripted	approved	1.00	1	`set_self_collision(enabled=False)`	v1
`contributor-rcad-cupboard`	browser (recorded)	approved	1.00	1	`set_self_collision(enabled=False)`	v2
`contributor-rcad-cupboard`	browser (recorded)	approved	1.00	1	`set_self_collision(enabled=False)`	v3
`contributor-scan-mug`	scripted	approved	1.00	1	`set_material(part=body, material=ceramic)`; `set_shell(part=body, thickness=0.004)`	v1
`contributor-scan-mug`	scripted	approved	1.00	1	`set_material(part=body, material=ceramic)`; `set_shell(part=body, thickness=0.004)`; `set_shell(part=body, thickness=0.005)`	v2

Agent: Claude Code with claude-sonnet-5-5. A script plays the contributor, through the page in Chrome or over its API. It knows each object's real size and weight from the scan the CAD was built from, and it never sees the hidden checks. Without a contributor, the mug and bowl scale tasks had scored 0.50 (Sonnet) and 0.75 (Opus), because the real size isn't in the asset. Every session is in the dataset's sessions.jsonl.

How an episode runs

The agent loops through steps 2 to 4 as many times as it wants: it calls a tool, Genesis answers with numbers or rendered frames, and the agent edits again. The probe suite runs once, after the agent’s last turn.

The agent’s own run_probe tool lets it test its fix, but the grading probe runs with settings and tolerances it never sees. A pass needs the edited object to behave like the reference, not just to look edited.

Episodes

One tab per task. Each shows the complaint, every tool call the agent made with the frames Genesis rendered, the hidden probe’s curves before and after the edit, the checks, and the MJCF diff.

Loading episodes…

Try the lean test

The mug-tip task, in your browser. mug_017 came out of VLM-to-CAD with its centre of mass at the base, so it rights itself like a weighted toy. Lean it past 50° and let go, then load the agent’s fix and try again. Drag the view to orbit.

COM height 0.3 cm tips past 85.7° real mug tips past 45.4°

Lean0°

Lean the mug, then let go.

The demo uses the rigid-body tipping rule, not Genesis: a body released at a lean falls once its centre of mass passes the edge it pivots on, at atan(radius ÷ COM height). With a 4 cm radius that is 85.7° before the fix and 45.4° after.

Run it yourself

The plugin installs into an agentenv-framework environment and ships a bundle named genesis-workbench with eleven tasks: three on objects built from primitives and eight on real assets (scanned objects, VLM-to-CAD output and ReplicaCAD furniture). Genesis runs on its CPU backend, so no GPU is needed.

# install agentenv-framework, then add the plugin
agent-env plugin add 'agentenv-genesis @ git+https://github.com/earakely-scale/agentenv-genesis-plugin'

# run all eleven tasks
agent-env run genesis-workbench

Claude Code needs a model endpoint that serves Anthropic’s Messages API, set in [model] of .agentenv/config.toml. Every run is recorded in agentenv’s local store.

mug-tipThe inertial frame of mug_017 sits at its base. Probe: a tilt table, final tilt for each lean angle.
oven-doorA 1.2 kg door with 4.0 N·m of hinge friction never opens. Probe: hinge angle over time after release at 5°.
drawer-slideThe drawer has no joint at all. Probe: drawer travel over time under a 10 N pull.