Collaborative Open-source Manipulation Performance Assessment for Robotics Enhancment (COMPARE)

Funded by the National Science Foundation, Pathways to Enable Open-Source Ecosystems (POSE), Award TI-2229577

Survey on Open-Source and Benchmarking for Robotics

We conducted a survey for researchers to provide feedback on the current state of open-source assets and benchmarking resources for robotics, and future activities for improvement. The results as of September 2023 from over 100 online respondents and ~100 workshop participants are summarized below.

Current Limitations

The barriers most frequently faced are: (1) a lack of relevant comparable benchmarks, (2) limitations of simulation capabilities, and (3) issues when integrating open-source products. While the proposed OSE does not seek to explicitly improve (2), increasing benchmarking throughout the community will lead to higher demand for better simulation capabilities. The activities which respondents least frequently performed are: (1) contributing to open-source, and (2) benchmarking to compare to others in the field. The lack of clear instructions in published benchmarks was highlighted in open response. During workshop discussions (see links to individual workshop content below) and deep dive meetings, several bottlenecks to conducting quality robot manipulation performance evaluation were identified, summarized as follows:

Recommendations for Improvement

The survey included several potential activities and mechanisms for respondents to rate in terms of benefit. Those rated as highest benefit were: organized repositories of (1) robot manipulation benchmarking results and (2) open-source products for robot manipulation. The remaining items were rated just below those, but all of similar benefit: (3) distributed robot benchmarking centers, (4) dedicated conference tracks and journals for benchmarking, (5) remote access to robot hardware, and (6) review panels to ensure open-source contributions meet established standards. Open response comments further highlighted the need for (6) as well as the benefit of having truly modular software to enable improved performance benchmarking. Our community discussions also led to several recommendations of OSE activities in order to build incentives for the community to conduct quality robot manipulation performance evaluation, summarized as follows:

COMPARE Workshops

Charter for Strategic Development and Ecosystem Functions

To support the development, benchmarking, and deployment of robot systems, an open-source ecosystem (OSE) is under development through the National Science Foundation (NSF) Pathways to Enable Open-Source Ecosystems (POSE) program.

The OSE will facilitate the development and dissemination of the open-source assets for robotic manipulation, i.e., robot hardware, software, benchmarking practices. It will create a community-driven platform that researchers and developers can share and learn about these open-source resources, find tools to easily utilize them, collaborate on developing systematic robot experimentation methodologies, and disseminate their findings effectively. As such the OSE will address the issues facing the advancement of robot manipulation due to the lack of systematic development and benchmarking methodologies, as well as the unique challenges related to physical assets (both equipment and benchmarking tools), preventing the robotic manipulation community from effectively utilizing such resources compared to other research domains like computer vision.

The OSE will support a wide variety of open-source assets (OSAs) available or to be developed in the future for robot manipulation, including physical object sets and hardware designs, digital datasets and simulations, instructional task protocols and metrics, and functional algorithms and robotic behaviors. 

Participants in the OSE will include all stakeholders of these OSAs, including developers (those who develop and maintain the OSA), contributors (those who iterate on an existing OSA), and users (those who implement OSAs in their research and development practices). 

OSE structure, governance, functions, and activities will be facilitated by a distributed set of OSE participants in various roles (e.g., leadership and participatory positions), diverse in terms of terms of backgrounds, research domains, and developmental maturity, spanning industry, academia, and government entities. 

Development of the OSE will be conducted transparently, with continuous input solicited from the community and outputs shared from the initial core development team, through a series of workshops, meetings, surveys, and asynchronous communication via online platforms.

It is intended that the Open Source Security Foundation (OpenSSF) best practices for their Free/Libre and Open Source Software (FLOSS) projects be followed for all OSAs utilized within the OSE, although additional consideration will be needed for non-code-based OSAs such as those that are physical and instructional.

During Phase I of the project (2022 - 2023), the scope of the OSE shall be investigated and discovered, towards a proposal for Phase II (2024 - 2026). During Phase II, the OSE will be formally developed and established through the execution of multiple activities and functions, including, but not limited to, the following:

Principal Investigators

Holly Yanco

University of Massachusetts Lowell

Adam Norton

University of Massachusetts Lowell

Berk Calli

Worcester Polytechnic University

Aaron Dollar

Yale University


If you are interested in contributing to the development of the COMPARE open-source ecosystem or have any questions or comments, please e-mail Adam Norton at