IO42 wires LLM agents to robot fleets through MCP, with ROS2 as the execution substrate. Agents call bounded semantic tools — never actuators — and every call is stamped so a decision can be replayed against the sensor data that produced it.
Every command travels this path before reaching an actuator. Telemetry and reasoning traces travel back up it, tagged with a single correlation ID.
Single-robot and fleet-coordinator agents. Neither holds a direct robot endpoint — every request starts here and ends at the gateway.
The only address an agent ever calls. Discovers live robots, locks actuators per task, routes validated calls to the right domain server.
URDF merged with a platform capability manifest, then versioned. New robot or new capability produces a new schema, not rewritten code.
Generated per robot from its capability descriptor. A fixed-base arm simply has no mobility domain to call.
YOLO for speed, CLIP for open vocabulary, VLM for scene reasoning — normalized to one schema and projected into robot frame before an agent sees them.
The only layer with actuator access. Validates every goal against deterministic sensors before Nav2 or MoveIt2 moves anything.
Every MCP call and every sensor frame, joined by one episode_id — the join that makes agent behaviour debuggable and trainable.
Mines the trace store for repeat patterns and exports them as behaviour trees. Mature behaviours graduate off the agent path entirely.
All generated from the same registry. A robot exposes only the domains its manifest declares.
| Domain | Example tools | ROS2 backing |
|---|---|---|
| Perception | describe_scene · detect_objects · locate_object | Perception stack + tf |
| Mobility | navigate_to · follow_path · get_pose · dock | Nav2 |
| Manipulation | pick · place · move_to_pose · grasp | MoveIt2 |
| Diagnostics | get_health · get_battery · run_self_test | Diagnostics aggregator |
A single task, traced end to end — including the part where perception is unsure and the agent asks again.
Coordinator agent generates an episode_id and calls gateway discovery.
detect_objects("wrench") returns a CLIP hit below the confidence floor.
Agent calls describe_scene() for a VLM read before committing to a grasp.
Perception returns a pose_3d already resolved into base_link.
Agent calls pick(object_id) on the manipulation domain.
Safety layer checks reach and force limits; MoveIt2 plans and executes.
Calls, confidences, trajectory and force-torque land under one episode_id — later mined into a standing rule for low-light picks.
Five rules that decide most arguments before they start.
Agents call move_to_pose. Never raw torque.
Deterministic sensors are truth. No exceptions.
episode_id threads MCP calls, ROS goals and rosbag windows.
Simple surface for agents, full resolution underneath for mining.
Old traces stay replayable as capabilities evolve.
Four decisions still open. Each changes the shape of the control plane.