ISSN 3083-4471 API Registry JSON Sitemap
TIRAS Transactions on Intelligent Robotics and Autonomous Systems An open access journal of Kaggleyes Academy
TIRAS > Vol. 1, No. 1 (July 2026) > KAGG:2607.00004
Vol. 1, No. 1 (July 2026) — Article KAGG:2607.00004

A Benchmark Protocol for Evaluating Opaque Velocity Interfaces in Legged Robots

Authors: Grace Lin, Wei Xu, Arjun Mehta, Ethan Cole

Autonomous Robotics Center, Carnegie Mellon University, Pittsburgh, PA, USA

[Received 2026-07-05; accepted 2026-07-16; published online 2026-07-19]

Abstract:

Velocity commands are the de facto interface between navigation software and the locomotion controllers of legged robots, yet these controllers behave as opaque, stateful, and platform-specific systems whose command-to-motion response is rarely documented and never characterized in a common way. This paper proposes a benchmark protocol for evaluating such opaque velocity interfaces. The protocol consists of a fixed suite of 42 probe command sequences drawn from five families (step, sweep, spiral, saturation, and disturbance rejection), five standardized metrics (command fidelity, phase lag, steady-state bias, saturation envelope, and recovery time), and a reporting template that records firmware, gait mode, payload, and terrain alongside the results. GAUGE-style active calibration serves as the reference characterization method. The protocol is demonstrated on two ANYmal B units with identical firmware and different service histories and on one Unitree Go2, on a lab floor and a stair course, with three repetitions per sequence and platform. The characterization produced by the full suite attains a command fidelity error of 8.4 percent RMS and a 95th-percentile phase lag of 62 ms, compared with 9.0 percent and 66 ms for the reference-unit characterization. A probe-budget sweep shows that fidelity error falls from 17.8 to 8.4 percent as the suite grows from 6 to 42 sequences, with diminishing returns beyond 27. The recorded anomalies, including an unmodeled 0.4 s gait-mode transient and payload-dependent early saturation, illustrate behavior that single-unit characterizations miss. The probe suite and exact probe parameters are released with the paper.

Journal: Transactions on Intelligent Robotics and Autonomous Systems (TIRAS)
Volume / Issue: Vol. 1, No. 1 (July 2026)
Article ID: KAGG:2607.00004
DOI: 10.5803/kagg.2607.00004
Received: 2026-07-05
Accepted: 2026-07-16
Published online: 2026-07-19
License: CC BY 4.0

References (42)

  1. J. Di Carlo, P. M. Wensing, B. Katz, G. Bledt, and S. Kim. Dynamic Locomotion in the MIT Cheetah 3 Through Convex Model-Predictive Control. IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), 2018.
  2. D. Kim, J. Di Carlo, B. Katz, G. Bledt, and S. Kim. Highly Dynamic Quadruped Locomotion via Whole-Body Impulse Control and Model Predictive Control. arXiv:1909.06586, 2019. arXiv:1909.06586
  3. C. D. Bellicoso, F. Jenelten, C. Gehring, and M. Hutter. Dynamic Locomotion Through Online Nonlinear Motion Optimization for Quadrupedal Robots. IEEE Robotics and Automation Letters, 3(3), 2018.
  4. J. Hwangbo, J. Lee, A. Dosovitskiy, D. Bellicoso, V. Tsounis, V. Koltun, and M. Hutter. Learning Agile and Dynamic Motor Skills for Legged Robots. arXiv:1901.08652, 2019. arXiv:1901.08652
  5. J. Lee, J. Hwangbo, L. Wellhausen, V. Koltun, and M. Hutter. Learning Quadrupedal Locomotion over Challenging Terrain. arXiv:2010.11251, 2020. arXiv:2010.11251
  6. T. Miki, J. Lee, J. Hwangbo, L. Wellhausen, V. Koltun, and M. Hutter. Learning Robust Perceptive Locomotion for Quadrupedal Robots in the Wild. arXiv:2201.08117, 2022. arXiv:2201.08117
  7. N. Rudin, D. Hoeller, P. Reist, and M. Hutter. Learning to Walk in Minutes Using Massively Parallel Deep Reinforcement Learning. arXiv:2109.11978, 2021. arXiv:2109.11978
  8. P. Fankhauser, M. Bloesch, and M. Hutter. Probabilistic Terrain Mapping for Mobile Robots with Uncertain Localization. IEEE Robotics and Automation Letters, 3(4), 2018.
  9. L. Wellhausen and M. Hutter. ArtPlanner: Robust Legged Robot Navigation in the Field. arXiv:2303.01420, 2023. arXiv:2303.01420
  10. D. Shah, A. Sridhar, A. Bhorkar, N. Hirose, and S. Levine. GNM: A General Navigation Model to Drive Any Robot. arXiv:2210.03370, 2022. arXiv:2210.03370
  11. D. Shah, A. Sridhar, N. Dashora, K. Stachowicz, K. Black, N. Hirose, and S. Levine. ViNT: A Foundation Model for Visual Navigation. arXiv:2306.14846, 2023. arXiv:2306.14846
  12. G. Williams, N. Wagener, B. Goldfain, P. Drews, J. M. Rehg, B. Boots, and E. A. Theodorou. Information Theoretic MPC for Model-Based Reinforcement Learning. IEEE International Conference on Robotics and Automation (ICRA), 2017.
  13. T. Zang, H. Huang, S. Cheng, and W. Zhang. GAUGE: Planner-Conditioned Active Calibration of Opaque Quadruped Velocity Interfaces. arXiv:2607.02841, 2026. arXiv:2607.02841
  14. M. Hutter, C. Gehring, D. Jud, A. Lauber, C. D. Bellicoso, V. Tsounis, J. Hwangbo, K. Bodie, P. Fankhauser, M. Bloesch, R. Diethelm, S. Bachmann, A. Melzer, and M. Hoepflinger. ANYmal – A Highly Mobile and Dynamic Quadrupedal Robot. IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), 2016.
  15. J. Tan, T. Zhang, E. Coumans, A. Iscen, Y. Bai, D. Hafner, S. Bohez, and V. Vanhoucke. Sim-to-Real: Learning Agile Locomotion For Quadruped Robots. arXiv:1804.10332, 2018. arXiv:1804.10332
  16. A. Kumar, Z. Fu, D. Pathak, and J. Malik. RMA: Rapid Motor Adaptation for Legged Robots. arXiv:2107.04034, 2021. arXiv:2107.04034
  17. G. B. Margolis, G. Yang, K. Paigwar, T. Chen, and P. Agrawal. Rapid Locomotion via Reinforcement Learning. arXiv:2205.02824, 2022. arXiv:2205.02824
  18. L. Smith, I. Kostrikov, and S. Levine. A Walk in the Park: Learning to Walk in 20 Minutes With Model-Free Reinforcement Learning. arXiv:2208.07860, 2022. arXiv:2208.07860
  19. X. B. Peng, M. Andrychowicz, W. Zaremba, and P. Abbeel. Sim-to-Real Transfer of Robotic Control with Dynamics Randomization. arXiv:1710.06537, 2017. arXiv:1710.06537
  20. G. Ji, J. Mun, H. Kim, and J. Hwangbo. Concurrent Training of a Control Policy and a State Estimator for Dynamic and Robust Legged Locomotion. arXiv:2202.05481, 2022. arXiv:2202.05481
  21. I. M. A. Nahrendra, B. Yu, and H. Myung. DreamWaQ: Learning Robust Quadrupedal Locomotion With Implicit Terrain Imagination via Deep Reinforcement Learning. arXiv:2301.10602, 2023. arXiv:2301.10602
  22. G. B. Margolis and P. Agrawal. Walk These Ways: Tuning Robot Control for Generalization with Multiplicity of Behavior. arXiv:2212.03238, 2022. arXiv:2212.03238
  23. S. Gangapurwala, M. Geisert, R. Orsolino, M. Fallon, and I. Havoutis. RLOC: Terrain-Aware Legged Locomotion using Reinforcement Learning and Optimal Control. arXiv:2012.03094, 2020. arXiv:2012.03094
  24. Z. Fu, X. Cheng, and D. Pathak. Deep Whole-Body Control: Learning a Unified Policy for Manipulation and Locomotion. arXiv:2210.10044, 2022. arXiv:2210.10044
  25. R. Yang, M. Zhang, N. Hansen, H. Xu, and X. Wang. Learning Vision-Guided Quadrupedal Locomotion End-to-End with Cross-Modal Transformers. arXiv:2107.03996, 2021. arXiv:2107.03996
  26. A. Agarwal, A. Kumar, J. Malik, and D. Pathak. Legged Locomotion in Challenging Terrains using Egocentric Vision. arXiv:2211.07638, 2022. arXiv:2211.07638
  27. X. Cheng, K. Shi, A. Agarwal, and D. Pathak. Extreme Parkour with Legged Robots. arXiv:2309.14341, 2023. arXiv:2309.14341
  28. Z. Zhuang, Z. Fu, J. Wang, C. Atkeson, S. Schwertfeger, C. Finn, and H. Zhao. Robot Parkour Learning. arXiv:2309.05665, 2023. arXiv:2309.05665
  29. D. Hoeller, N. Rudin, D. Sako, and M. Hutter. ANYmal Parkour: Learning Agile Navigation for Quadrupedal Robots. arXiv:2306.14874, 2023. arXiv:2306.14874
  30. M. Frey, M. Mattamala, N. Chebrolu, C. Cadena, M. Fallon, and M. Hutter. Fast Traversability Estimation for Wild Visual Navigation. arXiv:2305.08510, 2023. arXiv:2305.08510
  31. N. Rudin, D. Hoeller, M. Bjelonic, and M. Hutter. Advanced Skills by Learning Locomotion and Local Navigation End-to-End. arXiv:2209.12827, 2022. arXiv:2209.12827
  32. H. Xi, M. Tan, X. Zhang, S. Cheng, S. Wang, Y. Gu, X. Shen, and W. Zhang. CeRLP: A Cross-embodiment Robot Local Planning Framework for Visual Navigation. arXiv:2603.19602, 2026. arXiv:2603.19602
  33. T. Zang, S. Cheng, H. Huang, S. Wang, and W. Zhang. AgniNav: Configuration-Driven Cross-Embodiment Local Planning for Robot Navigation. arXiv:2606.10903, 2026. arXiv:2606.10903
  34. L. Ljung. System Identification: Theory for the User, 2nd ed.. Prentice Hall, 1999.
  35. R. Pintelon and J. Schoukens. System Identification: A Frequency Domain Approach, 2nd ed.. Wiley-IEEE Press, 2012.
  36. M. Bloesch, M. Hutter, M. A. Hoepflinger, S. Leutenegger, C. Gehring, C. D. Remy, and R. Siegwart. State Estimation for Legged Robots: Consistent Fusion of Leg Kinematics and IMU. Robotics: Science and Systems (RSS), 2012.
  37. R. Hartley, M. Ghaffari, R. M. Eustice, and J. W. Grizzle. Contact-Aided Invariant Extended Kalman Filtering for Legged Robot State Estimation. arXiv:1805.10410, 2018. arXiv:1805.10410
  38. D. Wisth, M. Camurri, and M. Fallon. VILENS: Visual, Inertial, Lidar, and Leg Odometry for All-Terrain Legged Robots. arXiv:2107.07243, 2021. arXiv:2107.07243
  39. B. Calli, A. Walsman, A. Singh, S. Srinivasa, P. Abbeel, and A. M. Dollar. Benchmarking in Manipulation Research: The YCB Object and Model Set and Benchmarking Protocols. arXiv:1502.03143, 2015. arXiv:1502.03143
  40. P. Anderson et al.. On Evaluation of Embodied Navigation Agents. arXiv:1807.06757, 2018. arXiv:1807.06757
  41. N. Yokoyama, S. Ha, and D. Batra. Success Weighted by Completion Time: A Dynamics-Aware Evaluation Criteria for Embodied Navigation. arXiv:2103.08022, 2021. arXiv:2103.08022
  42. A. Kadian et al.. Sim2Real Predictivity: Does Evaluation in Simulation Predict Real-World Performance? arXiv:1912.06321, 2019. arXiv:1912.06321
Cite this article Grace Lin, Wei Xu, Arjun Mehta, Ethan Cole, "A Benchmark Protocol for Evaluating Opaque Velocity Interfaces in Legged Robots," Transactions on Intelligent Robotics and Autonomous Systems, 2026, vol. 1, no. 1, KAGG:2607.00004. DOI: 10.5803/kagg.2607.00004.