Submitted · ICRA 2027
Same Scene, Different Story
A real-world benchmark of perception under manipulation.
* Equal contribution
- real scenes
- 100
- phases and view
- 3 + ego
- single objects
- 70
- models / model–task evaluations
- 49 / 65
Perception under manipulation
Official dataset showcase · silent, click to play. This web copy is resized and compressed from the public RPX video.
One scene, changing conditions
The same scene is recorded before, during, and after manipulation, with an additional egocentric view.
Illustrative captures from scene 049. These four views are not frame-synchronized. They show released observations, not model predictions.
A shared benchmark across six tasks
Accuracy and robustness are different questions
How does perception change when hands enter a scene and objects move? RPX measures quality within each phase and robustness across phase changes. Across the reported 65 model–task evaluations, none meets the authors’ thresholds for both accuracy and robustness.
From model adapters to comparable evaluations
Task-specific adapters connect a model or API to the evaluation protocol. The toolkit handles data selection, scoring, and phase-level diagnostics. Coverage of the six benchmark tasks does not imply that every task ships with pretrained model sweeps.
| Task | Input | Evaluation |
|---|---|---|
| Image depth | RGB image | Metric-depth accuracy |
| Video depth | RGB sequence | Depth accuracy and temporal consistency |
| Object tracking | Video and initialization | Tracking and identity consistency |
| Relative camera pose | Image pairs | Relative-pose errors and threshold AUC |
| General VQA | Image and question | Visual grounding and localization |
| In-context VQA | Image, question, and reference | Reference-conditioned localization |
Released data include RGB, metric depth, instance masks, calibrated fisheye stereo, and camera pose. See the dataset and toolkit documentation for the exact capture and task contracts.