feat(ci): benchmark smoke tests with isolated Docker images (LIBERO + MetaWorld)#3319

Open

pkooij wants to merge 61 commits intomainfrom

feat/benchmark-ci

Commits on Apr 8, 2026

docs(benchmarks): add benchmark integration guide and standardize benchmark docs
pkooij
committed
refactor(envs): move dispatch logic from factory into EnvConfig subclasses
pkooij
committed
docs(benchmarks): clean up adding-benchmarks guide for clarity
pkooij
committed
fix link
pkooij
committed
fix task count
pkooij
committed
fix: enable SmolVLA eval on LIBERO with custom camera mappings
pkooij
committed
fix: use direct AutoresetMode import for gymnasium compat
pkooij
committed
fix: handle gymnasium < 1.0 without AutoresetMode
pkooij
committed
refactor: revert policy changes, keep env-only camera mapping fixes
pkooij
committed
Update docs/source/env_processor.mdx

pkooij
and
s1lent4gnt
committed
feat(envs): lazy env init + AsyncVectorEnv as default for n_envs > 1

pkooij
and
claude
committed
fix: close envs between tasks to prevent worker process accumulation
pkooij
committed
fix(eval): use task_description instead of task for language conditioning
pkooij
committed
docs: update adding_benchmarks for async env changes
pkooij
committed
feat(eval): batch_size=auto + faster env loading
pkooij
committed
docs: add evaluation guide and update benchmarks doc
pkooij
committed
docs(evaluation): remove benchmark table, rename section header
pkooij
committed
perf(eval): shared memory, observation passthrough, task prefetch
pkooij
committed
style: ruff format
pkooij
committed
chore: revert env_processor.mdx changes (not part of this PR)
pkooij
committed
ci(benchmarks): add isolated integration tests for libero and metaworld

pkooij
and
claude
committed
ci(benchmarks): pin action hashes and use uv sync --locked

pkooij
and
claude
committed
ci(benchmarks): trigger only on envs/ or lerobot_eval.py changes

pkooij
and
claude
committed
fix(ci): set LIBERO_DATA_FOLDER to bypass interactive stdin prompt

pkooij
and
claude
committed
docs(benchmarks): add CI smoke test step to adding_benchmarks guide

pkooij
and
claude
committed
fix(ci): pre-create libero config in Dockerfile to bypass stdin prompt

pkooij
and
claude
committed
fix(ci): use shell to create libero config instead of multiline python -c

pkooij
and
claude
committed
fix(ci): point libero config to bundled package init_files

pkooij
and
claude
committed
fix(ci): add smolvla extra to benchmark Dockerfiles

pkooij
and
claude
committed
fix(eval): render_frame covers _LazyAsyncVectorEnv

pkooij
and
claude
committed
refactor(envs): remove unused _get_sub_env_attr helper

pkooij
and
claude
committed
chore: apply prettier formatting to docs

pkooij
and
claude
committed
docs(env_processor): remove deprecated add_envs_task from pipeline example

pkooij
and
claude
committed
refactor(envs): remove __del__ from _LazyAsyncVectorEnv

pkooij
and
claude
committed
fix(eval): prefetch next task's workers after close to avoid GPU memory overlap

pkooij
and
claude
committed
refactor(envs): move _LazyAsyncVectorEnv to utils and apply to metaworld

pkooij
and
claude
committed
chore: remove out-of-scope benchmark/CI/docs files from PR

pkooij
and
claude
committed
chore: restore adding_benchmarks + test_dispatch, drop env_processor changes

pkooij
and
claude
committed
docs(adding_benchmarks): remove CI smoke test step (coming in separate PR)

pkooij
and
claude
committed
refactor(envs): remove unused add_envs_task

pkooij
and
claude
committed
style: fix prettier formatting in env_processor.mdx

pkooij
and
claude
committed
fix(ci): use root container chmod to fix PermissionError on artifact dirs

pkooij
and
claude
committed
fix(ci): re-chmod artifacts after eval to fix unreadable files

pkooij
and
claude
committed
feat(ci): add monthly schedule trigger for benchmark tests

pkooij
and
claude
committed
fix(ci): change benchmark schedule from monthly to weekly (every Monday)

pkooij
and
claude
committed
fix(ci): use docker cp instead of bind mounts for artifacts

pkooij
and
claude
committed
fix(ci): write eval output to /tmp inside container

pkooij
and
claude
committed

Commits on Apr 9, 2026