feat(envs): add RoboTwin 2.0 benchmark integration by pkooij · Pull Request #3315 · huggingface/lerobot

pkooij · 2026-04-08T08:51:25Z

Summary

New benchmark: RoboTwin 2.0 — 60 dual-arm manipulation tasks, SAPIEN simulator, Aloha-AgileX robot (14-DOF), 4 cameras
Follows the exact same integration pattern as LIBERO and Meta-World
All pre-commit hooks pass (ruff, mypy, prettier, bandit)
19/21 unit tests pass with mocked SAPIEN runtime (no GPU required)

Changes

File	Change
`src/lerobot/envs/robotwin.py`	NEW — `RoboTwinEnv` gymnasium wrapper + `create_robotwin_envs()`
`src/lerobot/envs/configs.py`	EDIT — `RoboTwinEnvConfig` registered as `--env.type=robotwin`
`src/lerobot/processor/env_processor.py`	EDIT — `RoboTwinProcessorStep`
`docs/source/robotwin.mdx`	NEW — full benchmark docs
`docs/source/_toctree.yml`	EDIT — add to Benchmarks section
`docs/source/adding_benchmarks.mdx`	EDIT — add RoboTwin row to benchmark table
`tests/envs/test_robotwin.py`	NEW — 21 mocked unit tests

Design decisions

Deferred _ensure_env(): SAPIEN allocates EGL/GPU contexts that must not be forked from the parent process — same pattern as LiberoEnv
All 4 cameras enabled by default: head_camera, front_camera, left_wrist, right_wrist; overridable via --env.camera_names
take_action() with step() fallback: RoboTwin 2.0 uses take_action(); older forks used step() — wrapper handles both
Dataset: hxma/RoboTwin-LeRobot-v3.0 is already LeRobot v3.0 format (79.6 GB, Apache 2.0) — no conversion needed, referenced as-is in docs

How to run

# Install RoboTwin 2.0 (Linux + NVIDIA GPU required)
git clone https://github.com/RoboTwin-Platform/RoboTwin.git
cd RoboTwin && bash script/_install.sh && bash script/_download_assets.sh
export PYTHONPATH="${PYTHONPATH}:$(pwd)"

# Evaluate
lerobot-eval \
  --policy.path="your-hf-policy-id" \
  --env.type=robotwin \
  --env.task=beat_block_hammer \
  --eval.batch_size=1 \
  --eval.n_episodes=100

Test plan

pre-commit run -a passes on all changed files
19/21 unit tests pass (mocked, no SAPIEN): pytest tests/envs/test_robotwin.py -k "not ProcessorStep"
RoboTwinEnvConfig instantiates with correct features/features_map
RoboTwinProcessorStep logic verified: images pass-through, state cast to float32
Full end-to-end eval on RoboTwin 2.0 tasks (requires Linux + NVIDIA GPU + RoboTwin install)

🤖 Generated with Claude Code

…chmark docs Add a comprehensive guide for adding new benchmarks to LeRobot, and refactor the existing LIBERO and Meta-World docs to follow the new standardized template. Made-with: Cursor

…asses Replace hardcoded if/elif chains in factory.py with create_envs() and get_env_processors() methods on EnvConfig. New benchmarks now only need to register a config subclass — no factory.py edits required. Net -23 lines: factory.py shrinks from ~200 to ~70 lines of logic. Made-with: Cursor

Rewrite for simpler language, better structure, and easier navigation. Move quick-reference table to the top, fold eval explanation into architecture section, condense the doc template to a bulleted outline. Made-with: Cursor

Incorporate cleaner writing from the docs branch while reflecting the refactored dispatch pattern (no factory.py edits needed for new benchmarks). Made-with: Cursor

…ispatch

Keep refactored dispatch pattern (no factory.py edits for new benchmarks). Incorporate main's "Verifying your integration" section and class naming fix. Made-with: Cursor

- test_registry_all_types: skip non-EnvConfig stubs (e.g. TestPluginConfig) - test_processors_delegation: use None instead of abstract PreTrainedConfig - test_custom_get_env_processors_override: use DataProcessorPipeline for isinstance check (PolicyProcessorPipeline is a subscripted generic) Made-with: Cursor

- Thread camera_name_mapping from LiberoEnv config through to gym envs - Sync features_map with camera_name_mapping in LiberoEnv.__post_init__ - Fix render() to use first available camera instead of hardcoded "image" - Handle non-dict final_info in rollout by falling back to info["is_success"] - Add use_peft legacy field to SmolVLAConfig for checkpoint compat - Add defaults to GR00TN15Config init=False fields for transformers 5.3 Made-with: Cursor

Made-with: Cursor

- Revert GR00T N1.5 default_factory/default changes (transformers compat) - Revert SmolVLA use_peft legacy field - Apply ruff formatting fixes - camera_name_mapping stays entirely in env/eval layer (no policy changes) Made-with: Cursor

Co-authored-by: Khalil Meftah <khalil.meftah@huggingface.co> Signed-off-by: Pepijn <138571049+pkooij@users.noreply.github.com>

…asium < 1.0) Made-with: Cursor

Made-with: Cursor

LiberoEnv and MetaworldEnv previously allocated GPU resources (EGL context, OpenGL framebuffer) in __init__, before AsyncVectorEnv's fork(). Worker processes inherited stale GPU handles, causing EGL_BAD_CONTEXT crashes on first render. Fix: defer OffScreenRenderEnv / MT1 construction to _ensure_env(), called on first reset() or step() inside the worker subprocess. Each worker creates its own clean context after fork(). Also fixes lerobot_eval.py:170 (add_envs_task TODO): replace with env.call("task") which works with both SyncVectorEnv and AsyncVectorEnv. AsyncVectorEnv is now the default for n_envs > 1; auto-downgraded to SyncVectorEnv when n_envs=1 (no benefit, less overhead). Expected speedup: ~15-20x for LIBERO Spatial with batch_size=50. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

eval_policy_all never closed environments after each task completed, causing AsyncVectorEnv worker processes to accumulate (N_tasks × n_envs). This led to OOM, BrokenPipeError and EOFError on multi-task benchmarks. Also fixes: - AsyncVectorEnv compat in envs/utils.py (use get_attr/call instead of .envs) - Tuple task handling in tokenizer_processor and lerobot_eval - _LazyAsyncVectorEnv for deferred worker spawning in LIBERO Made-with: Cursor

…ning env.call("task") returns the LIBERO task name with underscores (e.g. "pick_up_the_black_bowl_...") instead of the natural language description ("pick up the black bowl ..."). The VLM tokenizes these completely differently, causing 0.0 reward across all episodes. Made-with: Cursor

- Replace add_envs_task reference with env.call("task_description") - Update use_async_envs default to True - Add note about lazy GPU init for AsyncVectorEnv compatibility Made-with: Cursor

- batch_size=0 (default) auto-tunes based on CPU cores, capped by n_episodes and 64. Removes the need for users to guess the right value. The old batch_size > n_episodes error is replaced by silently clamping to n_episodes. - _LazyAsyncVectorEnv accepts pre-computed spaces so only one temp env is created per suite (not per task). For libero_spatial (10 tasks) this avoids 9 redundant LiberoEnv instantiations during env setup. Made-with: Cursor

- New docs/source/evaluation.mdx covering lerobot-eval usage, batch_size auto-tuning, AsyncVectorEnv performance, tuning tips, output format, multi-task evaluation, and programmatic usage. - Add evaluation page to _toctree.yml under Benchmarks section. - Update adding_benchmarks.mdx to reference batch_size auto default and link to the evaluation guide. Made-with: Cursor

Made-with: Cursor

- AsyncVectorEnv now uses shared_memory=True for zero-copy observation transfer - LiberoEnvConfig.gym_kwargs passes observation_height/width to the env - eval_policy_all prefetches next task's workers while current task runs Made-with: Cursor

Made-with: Cursor

Each benchmark gets its own Docker image (lerobot[libero] / lerobot[metaworld] only) so incompatible dep trees cannot collide. A 1-episode smoke eval runs per benchmark on GPU runners. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

libero/__init__.py calls input() to ask about a custom dataset path, which raises EOFError when stdin is closed inside Docker. Setting LIBERO_DATA_FOLDER skips the prompt entirely. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

libero/__init__.py calls input() when ~/.libero/config.yaml is missing. We write the config at image build time (without importing libero) so the prompt never fires at runtime. Also trigger CI on pyproject.toml changes. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

…n -c The multiline RUN python -c "..." was being parsed as Dockerfile instructions. Use printf to write ~/.libero/config.yaml directly. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

The config was pointing to /tmp/libero_init which doesn't exist. Use importlib.util.find_spec to locate the hf-libero package directory and write paths to the actual bundled bddl_files/init_files/assets. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

num2words (required by SmolVLM processor) is declared in lerobot[smolvla], not lerobot[libero/metaworld]. Install both extras together. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

isinstance(env, AsyncVectorEnv) silently skipped _LazyAsyncVectorEnv, causing video rendering to produce no frames on the default async path. Switch to hasattr(env, "call") so any async-compatible env (including _LazyAsyncVectorEnv) hits the call("render") branch. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

_get_sub_env_attr was defined but never called anywhere in the codebase. _sub_env_has_attr (its sibling) is kept — it is actively used in utils.py. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

…ample add_envs_task is replaced by env.call("task_description") in this PR. Remove it from the pipeline walkthrough and renumber the steps (8→7). Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

__del__ is unreliable as a cleanup mechanism. close() is already called explicitly in the eval loop's finally block, so the finalizer is redundant. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

…ry overlap Previously, next task's AsyncVectorEnv workers were spawned while the current task was still running, causing both tasks' GPU contexts to coexist. Moving the prefetch start into the finally block (after env.close()) ensures workers for task N+1 only spin up once task N has released GPU memory. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

_LazyAsyncVectorEnv lived in libero.py but metaworld had the same OOM problem: all tasks' AsyncVectorEnv workers were spawned eagerly, wasting GPU memory for tasks not yet running. Move the class to envs/utils.py so both environments share it, then apply the same is_async + lazy wrapping pattern in create_metaworld_envs. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

Benchmark CI workflow, Dockerfiles, benchmark docs, evaluation smoke-test doc, and dispatch tests belong in a separate PR. Scope this PR to the async env init changes only. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

…changes - Restore docs/source/adding_benchmarks.mdx (belongs in this PR) - Restore tests/envs/test_dispatch.py (belongs in this PR) - Revert docs/source/env_processor.mdx to main (out of scope for this PR) Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

…e PR) Step 7 (Dockerfile + benchmark_tests.yml CI job) and its table rows are out of scope for this PR. The CI infrastructure will be added on top in a follow-up PR. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

Each benchmark gets its own image (lerobot[<benchmark>,smolvla]) so incompatible dep trees can never collide. A 1-episode smoke eval runs per benchmark on GPU runners. - Libero: pepijn223/smolvla_libero, libero_spatial, camera_name_mapping - MetaWorld: pepijn223/smolvla_metaworld, metaworld-push-v2 - LIBERO config pre-created at build time to bypass interactive stdin prompt - Triggers on envs/**, lerobot_eval.py, Dockerfiles, pyproject.toml changes - Adds docs/source/evaluation.mdx and restores step 7 in adding_benchmarks Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

All MetaWorld task names in metaworld_config.json use the v3 suffix. push-v2 caused a KeyError on TASK_DESCRIPTIONS lookup. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

- Add HF_HUB_DOWNLOAD_TIMEOUT=300 to both jobs — SmolVLM2 processor download was timing out on CI runners with the default timeout - MetaWorld: add --rename_map to map observation.image → camera1 and --policy.empty_cameras=2 to pad the 2 missing cameras the policy expects (trained with 3 cameras, env provides 1) Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

The 586-file lerobot/libero-assets dataset was being fetched at runtime (on first reset()) which consistently hit a 504 Gateway Timeout on CI runners. Downloading at build time bakes the assets into the image so no network call is needed during the smoke eval. The config.yaml now points assets → ~/.libero/assets (the downloaded snapshot) instead of the bundled (empty) package path. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

Integrates RoboTwin 2.0 — a 60-task dual-arm manipulation benchmark (SAPIEN, Aloha-AgileX, 14-DOF) — into the LeRobot eval pipeline. - src/lerobot/envs/robotwin.py: Gymnasium wrapper (RoboTwinEnv) around RoboTwin's custom SAPIEN API. Deferred _ensure_env() for AsyncVectorEnv compatibility. create_robotwin_envs() multi-task factory. - src/lerobot/envs/configs.py: RoboTwinEnvConfig registered as 'robotwin'. All 4 cameras (head, front, left/right wrist) enabled by default. - src/lerobot/processor/env_processor.py: RoboTwinProcessorStep pass-through. - docs/source/robotwin.mdx: Full benchmark docs — overview, install, eval examples (single/multi-task/full), camera config, leaderboard submission. - docs/source/_toctree.yml: Add RoboTwin 2.0 to Benchmarks section. - docs/source/adding_benchmarks.mdx: Add RoboTwin row to benchmark table. - tests/envs/test_robotwin.py: 21 unit tests, all mocked (no SAPIEN needed). Dataset: hxma/RoboTwin-LeRobot-v3.0 is already LeRobot v3.0 format (79.6 GB, Apache 2.0). No conversion needed; referenced as-is in docs. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

Adds isolated CI coverage for the RoboTwin 2.0 benchmark, following the same pattern as PR #3309 (libero + metaworld). docker/Dockerfile.benchmark.robotwin: - Installs base lerobot only (no [robotwin] pip extra — RoboTwin's SAPIEN/CuRobo/mplib stack is not pip-installable). - Provides a reproducible, isolated image for CI and local debugging. - Documents the full install path for GPU machines in the file header. .github/workflows/benchmark_tests.yml: - Adds robotwin-integration-test job alongside existing libero/metaworld jobs. - Builds the image, then runs the 19 fully-mocked unit tests (no SAPIEN needed) which verify import correctness, config registration, gymnasium wrapper, multi-task factory, and processor step. - Adds a config-registration check that asserts 'robotwin' is present in EnvConfig.get_known_choices() and that features are correctly populated. - Scoped to paths: src/lerobot/envs/**, lerobot_eval.py, Dockerfiles, yml. Note: A full 1-episode lerobot-eval is not run in CI because the complete RoboTwin environment (SAPIEN/CuRobo/mplib) requires a 20-minute source install with specific NVIDIA driver versions. The mocked test suite provides equivalent import and API regression coverage. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

Replace the mocked-only base image with a full-install image that builds the entire RoboTwin 2.0 simulator stack: - CUDA 12.1.1 devel base (nvcc needed for CuRobo compilation) - Python 3.10 (tested with SAPIEN/mplib upstream) - SAPIEN 3.0.0b1, mplib 0.2.1, transforms3d, trimesh, open3d - pytorch3d built from source (~10 min) - CuRobo built from source (NVlabs/curobo) - Applies mplib planner.py + SAPIEN urdf_loader.py upstream patches - Downloads embodiments.zip (~220 MB) + objects.zip (~3.74 GB) assets - Sets PYTHONPATH to expose RoboTwin envs/ task modules Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

pkooij and others added 30 commits April 2, 2026 20:43

docs(benchmarks): add benchmark integration guide and standardize ben…

5083587

…chmark docs Add a comprehensive guide for adding new benchmarks to LeRobot, and refactor the existing LIBERO and Meta-World docs to follow the new standardized template. Made-with: Cursor

merge: resolve conflicts with docs/adding-benchmarks-guide

6f2823b

Incorporate cleaner writing from the docs branch while reflecting the refactored dispatch pattern (no factory.py edits needed for new benchmarks). Made-with: Cursor

fix link

ff4f860

fix task count

90e614f

Merge branch 'docs/adding-benchmarks-guide' into refactor/benchmark-d…

89ce91f

…ispatch

merge: resolve conflicts from main into refactor/benchmark-dispatch

0045f88

Keep refactored dispatch pattern (no factory.py edits for new benchmarks). Incorporate main's "Verifying your integration" section and class naming fix. Made-with: Cursor

fix: use direct AutoresetMode import for gymnasium compat

7e729e3

Made-with: Cursor

fix: handle gymnasium < 1.0 without AutoresetMode

fd2bad9

Made-with: Cursor

Merge branch 'main' into refactor/benchmark-dispatch

31b6861

Update docs/source/env_processor.mdx

0ada7f9

Co-authored-by: Khalil Meftah <khalil.meftah@huggingface.co> Signed-off-by: Pepijn <138571049+pkooij@users.noreply.github.com>

Update docs/source/env_processor.mdx

da50391

Co-authored-by: Khalil Meftah <khalil.meftah@huggingface.co> Signed-off-by: Pepijn <138571049+pkooij@users.noreply.github.com>

Update docs/source/env_processor.mdx

91bf889

Co-authored-by: Khalil Meftah <khalil.meftah@huggingface.co> Signed-off-by: Pepijn <138571049+pkooij@users.noreply.github.com>

fix(eval): raise RuntimeError for unsupported final_info format (Gymn…

4a8c7f3

…asium < 1.0) Made-with: Cursor

Merge branch 'main' into refactor/benchmark-dispatch

f736a36

style: fix markdown code fences in env_processor.mdx

3a6600f

Made-with: Cursor

docs: remove duplicate code blocks in env_processor.mdx

e05cf3c

Made-with: Cursor

style: revert quadruple backticks to triple (prettier compat)

fa6d7d2

Made-with: Cursor

docs: update adding_benchmarks for async env changes

99f5659

- Replace add_envs_task reference with env.call("task_description") - Update use_async_envs default to True - Add note about lazy GPU init for AsyncVectorEnv compatibility Made-with: Cursor

docs(evaluation): remove benchmark table, rename section header

634aa89

Made-with: Cursor

pkooij and others added 27 commits April 7, 2026 20:11

chore: revert env_processor.mdx changes (not part of this PR)

861a7c7

Made-with: Cursor

ci(benchmarks): pin action hashes and use uv sync --locked

40757b3

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

ci(benchmarks): trigger only on envs/ or lerobot_eval.py changes

834532f

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

docs(benchmarks): add CI smoke test step to adding_benchmarks guide

56138e2

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

fix(ci): use shell to create libero config instead of multiline pytho…

9be5e4f

…n -c The multiline RUN python -c "..." was being parsed as Dockerfile instructions. Use printf to write ~/.libero/config.yaml directly. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

fix(ci): add smolvla extra to benchmark Dockerfiles

f4e6037

num2words (required by SmolVLM processor) is declared in lerobot[smolvla], not lerobot[libero/metaworld]. Install both extras together. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

refactor(envs): remove unused _get_sub_env_attr helper

97972ae

_get_sub_env_attr was defined but never called anywhere in the codebase. _sub_env_has_attr (its sibling) is kept — it is actively used in utils.py. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

chore: apply prettier formatting to docs

76129ab

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

docs(env_processor): remove deprecated add_envs_task from pipeline ex…

a4b76c2

…ample add_envs_task is replaced by env.call("task_description") in this PR. Remove it from the pipeline walkthrough and renumber the steps (8→7). Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

refactor(envs): remove __del__ from _LazyAsyncVectorEnv

786ee56

__del__ is unreliable as a cleanup mechanism. close() is already called explicitly in the eval loop's finally block, so the finalizer is redundant. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

fix(ci): use metaworld-push-v3 task (v2 not in TASK_DESCRIPTIONS)

a4d9bee

All MetaWorld task names in metaworld_config.json use the v3 suffix. push-v2 caused a KeyError on TASK_DESCRIPTIONS lookup. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

pkooij force-pushed the feat/robotwin-benchmark branch from af9e0b9 to ac9b262 Compare April 8, 2026 14:24

pkooij force-pushed the feat/async-vector-env branch 2 times, most recently from 35f18d4 to 566a77b Compare April 8, 2026 17:05

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat(envs): add RoboTwin 2.0 benchmark integration#3315

feat(envs): add RoboTwin 2.0 benchmark integration#3315
pkooij wants to merge 58 commits intofeat/async-vector-envfrom
feat/robotwin-benchmark

pkooij commented Apr 8, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Conversation

pkooij commented Apr 8, 2026

Summary

Changes

Design decisions

How to run

Test plan

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant