Forum Discussion on Open-Source Robotic Manipulation and Benchmarking

ROS-Industrial Consortium Americas 2023 Annual Meeting

This workshop was held during the ROS-Industrial Consortium Americas 2023 Annual Meeting held on May 25, 2023 in Detroit, Michigan.

Workshop Overview

To support and improve the development and deployment of robotic manipulation systems, the open-source ecosystem should facilitate development and dissemination of open-source assets (i.e., hardware, software, datasets), benchmarking practices, and sharing of results. Such a distributed, community-driven venue would enable researchers and developers to share and learn about open-source resources, find tools to utilize them, collaborate on developing systematic robot experimentation methodologies, and disseminate their findings. During this workshop, we will facilitate forum discussions around two topics: (1) identification of current gaps that limit the effectiveness of the ecosystem (e.g., hardware access, simulation fidelity, lack of relevant assets), and (2) proposing solutions to the identified issues and improve the state of the ecosystem (e.g., establishing advisory boards, integrating benchmark promotion within ROS, developing streamlined infrastructure). The primary goal is to receive feedback from workshop attendees to drive the development and implementation of new activities for an improved open-source ecosystem.

Key Takeaways

Based on the discussions had at the workshop, a set of key takeaways have been summarized below:

How to Enhance the Open-Source Ecosystem?

Current Gaps

Based on an online survey of over 100 respondents to provide feedback on the current state of open-source assets and benchmarking resources for robotic manipulation, those that were manipulation researchers (57%) rated the following statements as Never, Rarely, Sometimes, Often, or Frequently (results below are ordered from highest to lowest frequency):

Barriers (highest to lowest frequency)

Activity (highest to lowest frequency)

Future Solutions

Modular Benchmarking Software Pipelines

Distributed Physical Benchmarking Facilities

Online Community Resources

Working Groups and Advocacy


Funded by the National Science Foundation, Pathways to Enable Open-Source Ecosystems (POSE), Award TI-2229577