ISSN 3083-4471 API Registry JSON Sitemap
TIRAS Transactions on Intelligent Robotics and Autonomous Systems An open access journal of Kaggleyes Academy
TIRAS > Vol. 1, No. 4 (October 2026) > KAGG:2610.00001
Vol. 1, No. 4 (October 2026) — Article KAGG:2610.00001

A Unified Evaluation Harness for Quadruped Navigation across Interfaces, Terrain, and Sensors

Authors: Marcus Hill, Rachel Kim, Yating Feng, Ivan Sokolov

Robotics Institute, Georgia Institute of Technology, Atlanta, GA, USA

[Received 2026-09-17; accepted 2026-09-28; published online 2026-10-01]

Abstract:

Reported performance of quadruped navigation planners is rarely comparable across papers, because each evaluation couples the planner with a particular velocity interface, terrain definition, and sensor rig. This paper presents a unified evaluation harness that isolates these three axes. The interface axis spans an ideal velocity tracker, a calibrated-opaque interface identified by planner-conditioned active calibration, and two native platform interfaces; the terrain axis comprises six standardized courses (flat, mixed, stairs, narrow, cluttered, rough); and the sensor axis comprises LiDAR-only, LiDAR+vision, and degraded rigs. A fixed set of four planners, with a configuration-driven cross-embodiment planner as the reference stack, is executed on Go2-class, B2-class, and ANYmal-class assets, and every run emits a standardized 214-field results record. Across 6480 standardized runs, the harness reaches 95.7% cross-platform run parity and 100.0% record schema completeness at 2.3 s of overhead per run and measures nine confounding factors, compared with 82.9% parity for raw-score ranking and 87.0% when only the interface axis is held fixed, both of which measure eight. Ablations show that removing the interface, terrain, or sensor-rig axis lowers parity to 74.5%, 80.3%, and 86.4%, respectively. A case study shows that a planner ranked first on raw scores falls to third once the interface axis is held fixed, because its advantage stemmed from a favorable interface on one platform. Recorded failure cases, including sanity-check quarantines and gait-mode switching delays, are reported as harness observations rather than discarded.

Journal: Transactions on Intelligent Robotics and Autonomous Systems (TIRAS)
Volume / Issue: Vol. 1, No. 4 (October 2026)
Article ID: KAGG:2610.00001
DOI: 10.5803/kagg.2610.00001
Received: 2026-09-17
Accepted: 2026-09-28
Published online: 2026-10-01
License: CC BY 4.0

References (40)

  1. J. Hwangbo, J. Lee, A. Dosovitskiy, D. Bellicoso, V. Tsounis, V. Koltun, and M. Hutter. Learning Agile and Dynamic Motor Skills for Legged Robots. arXiv:1901.08652, 2019. arXiv:1901.08652
  2. J. Lee, J. Hwangbo, L. Wellhausen, V. Koltun, and M. Hutter. Learning Quadrupedal Locomotion over Challenging Terrain. arXiv:2010.11251, 2020. arXiv:2010.11251
  3. T. Miki, J. Lee, J. Hwangbo, L. Wellhausen, V. Koltun, and M. Hutter. Learning Robust Perceptive Locomotion for Quadrupedal Robots in the Wild. arXiv:2201.08117, 2022. arXiv:2201.08117
  4. M. Tranzatto et al.. CERBERUS: Autonomous Legged and Aerial Robotic Exploration in the Tunnel and Urban Circuits of the DARPA Subterranean Challenge. arXiv:2201.07067, 2022. arXiv:2201.07067
  5. L. Wellhausen and M. Hutter. ArtPlanner: Robust Legged Robot Navigation in the Field. arXiv:2303.01420, 2023. arXiv:2303.01420
  6. N. Rudin, D. Hoeller, P. Reist, and M. Hutter. Learning to Walk in Minutes Using Massively Parallel Deep Reinforcement Learning. arXiv:2109.11978, 2021. arXiv:2109.11978
  7. G. B. Margolis and P. Agrawal. Walk These Ways: Tuning Robot Control for Generalization with Multiplicity of Behavior. arXiv:2212.03238, 2022. arXiv:2212.03238
  8. T. Zang, H. Huang, S. Cheng, and W. Zhang. GAUGE: Planner-Conditioned Active Calibration of Opaque Quadruped Velocity Interfaces. arXiv:2607.02841, 2026. arXiv:2607.02841
  9. T. Zang, S. Cheng, H. Huang, S. Wang, and W. Zhang. AgniNav: Configuration-Driven Cross-Embodiment Local Planning for Robot Navigation. arXiv:2606.10903, 2026. arXiv:2606.10903
  10. V. Makoviychuk et al.. Isaac Gym: High Performance GPU-Based Physics Simulation For Robot Learning. arXiv:2108.10470, 2021. arXiv:2108.10470
  11. A. Kumar, Z. Fu, D. Pathak, and J. Malik. RMA: Rapid Motor Adaptation for Legged Robots. arXiv:2107.04034, 2021. arXiv:2107.04034
  12. A. Agarwal, A. Kumar, J. Malik, and D. Pathak. Legged Locomotion in Challenging Terrains using Egocentric Vision. arXiv:2211.07638, 2022. arXiv:2211.07638
  13. Z. Zhuang, Z. Fu, J. Wang, C. Atkeson, S. Schwertfeger, C. Finn, and H. Zhao. Robot Parkour Learning. arXiv:2309.05665, 2023. arXiv:2309.05665
  14. X. Cheng, K. Shi, A. Agarwal, and D. Pathak. Extreme Parkour with Legged Robots. arXiv:2309.14341, 2023. arXiv:2309.14341
  15. D. Hoeller, N. Rudin, D. Sako, and M. Hutter. ANYmal Parkour: Learning Agile Navigation for Quadrupedal Robots. arXiv:2306.14874, 2023. arXiv:2306.14874
  16. R. Grandia, F. Jenelten, S. Yang, F. Farshidian, and M. Hutter. Perceptive Locomotion through Nonlinear Model Predictive Control. arXiv:2208.08373, 2022. arXiv:2208.08373
  17. S. Ha, J. Lee, M. van de Panne, Z. Xie, W. Yu, and M. Khadiv. Learning-based Legged Locomotion: State of the Art and Future Perspectives. arXiv:2406.01152, 2024. arXiv:2406.01152
  18. X. B. Peng, M. Andrychowicz, W. Zaremba, and P. Abbeel. Sim-to-Real Transfer of Robotic Control with Dynamics Randomization. arXiv:1710.06537, 2017. arXiv:1710.06537
  19. J. Tobin, R. Fong, A. Ray, J. Schneider, W. Zaremba, and P. Abbeel. Domain Randomization for Transferring Deep Neural Networks from Simulation to the Real World. arXiv:1703.06907, 2017. arXiv:1703.06907
  20. J. Tan, T. Zhang, E. Coumans, A. Iscen, Y. Bai, D. Hafner, S. Bohez, and V. Vanhoucke. Sim-to-Real: Learning Agile Locomotion For Quadruped Robots. arXiv:1804.10332, 2018. arXiv:1804.10332
  21. H. Xi, M. Tan, X. Zhang, S. Cheng, S. Wang, Y. Gu, X. Shen, and W. Zhang. CeRLP: A Cross-embodiment Robot Local Planning Framework for Visual Navigation. arXiv:2603.19602, 2026. arXiv:2603.19602
  22. J. Truong, D. Yarats, T. Li, F. Meier, S. Chernova, D. Batra, and A. Rai. Learning Navigation Skills for Legged Robots with Learned Robot Embodiments. arXiv:2011.12255, 2020. arXiv:2011.12255
  23. Z. Fu, A. Kumar, A. Agarwal, H. Qi, J. Malik, and D. Pathak. Coupling Vision and Proprioception for Navigation of Legged Robots. arXiv:2112.02094, 2021. arXiv:2112.02094
  24. N. Rudin, D. Hoeller, M. Bjelonic, and M. Hutter. Advanced Skills by Learning Locomotion and Local Navigation End-to-End. arXiv:2209.12827, 2022. arXiv:2209.12827
  25. D. Shah, A. Sridhar, A. Bhorkar, N. Hirose, and S. Levine. GNM: A General Navigation Model to Drive Any Robot. arXiv:2210.03370, 2022. arXiv:2210.03370
  26. D. Shah, A. Sridhar, N. Dashora, K. Stachowicz, K. Black, N. Hirose, and S. Levine. ViNT: A Foundation Model for Visual Navigation. arXiv:2306.14846, 2023. arXiv:2306.14846
  27. A. Sridhar, D. Shah, C. Glossop, and S. Levine. NoMaD: Goal Masked Diffusion Policies for Navigation and Exploration. arXiv:2310.07896, 2023. arXiv:2310.07896
  28. T. Miki, L. Wellhausen, R. Grandia, F. Jenelten, T. Homberger, and M. Hutter. Elevation Mapping for Locomotion and Navigation using GPU. arXiv:2204.12876, 2022. arXiv:2204.12876
  29. W. Xu, Y. Cai, D. He, J. Lin, and F. Zhang. FAST-LIO2: Fast Direct LiDAR-Inertial Odometry. arXiv:2107.06829, 2021. arXiv:2107.06829
  30. J. Frey, M. Mattamala, N. Chebrolu, C. Cadena, M. Fallon, and M. Hutter. Fast Traversability Estimation for Wild Visual Navigation. arXiv:2305.08510, 2023. arXiv:2305.08510
  31. M. Mattamala, J. Frey, P. Libera, N. Chebrolu, G. Martius, C. Cadena, M. Hutter, and M. Fallon. Wild Visual Navigation: Fast Traversability Learning via Pre-Trained Models and Online Self-Supervision. arXiv:2404.07110, 2024. arXiv:2404.07110
  32. M. Savva et al.. Habitat: A Platform for Embodied AI Research. arXiv:1904.01201, 2019. arXiv:1904.01201
  33. P. Anderson et al.. On Evaluation of Embodied Navigation Agents. arXiv:1807.06757, 2018. arXiv:1807.06757
  34. N. Yokoyama, S. Ha, and D. Batra. Success Weighted by Completion Time: A Dynamics-Aware Evaluation Criteria for Embodied Navigation. arXiv:2103.08022, 2021. arXiv:2103.08022
  35. A. Kadian et al.. Sim2Real Predictivity: Does Evaluation in Simulation Predict Real-World Performance? arXiv:1912.06321, 2019. arXiv:1912.06321
  36. D. Perille, A. Truong, X. Xiao, and P. Stone. Benchmarking Metric Ground Navigation. arXiv:2008.13315, 2020. arXiv:2008.13315
  37. X. Xiao et al.. Autonomous Ground Navigation in Highly Constrained Spaces: Lessons Learned from the BARN Challenge at ICRA 2022. arXiv:2208.10473, 2022. arXiv:2208.10473
  38. X. Xiao, B. Liu, G. Warnell, and P. Stone. Motion Planning and Control for Mobile Robot Navigation Using Machine Learning: A Survey. arXiv:2011.13112, 2020. arXiv:2011.13112
  39. M. Mittal et al.. Orbit: A Unified Simulation Framework for Interactive Robot Learning Environments. arXiv:2301.04195, 2023. arXiv:2301.04195
  40. P. Henderson, R. Islam, P. Bachman, J. Pineau, D. Precup, and D. Meger. Deep Reinforcement Learning that Matters. arXiv:1709.06560, 2017. arXiv:1709.06560
Cite this article Marcus Hill, Rachel Kim, Yating Feng, Ivan Sokolov, "A Unified Evaluation Harness for Quadruped Navigation across Interfaces, Terrain, and Sensors," Transactions on Intelligent Robotics and Autonomous Systems, 2026, vol. 1, no. 4, KAGG:2610.00001. DOI: 10.5803/kagg.2610.00001.