Collaborative Open-source Manipulation and Perception Assets for Robotics Ecosystem (COMPARE)

Funded by the National Science Foundation, Pathways to Enable Open-Source Ecosystems (POSE), Award TI-2229577

Survey on Open-Source and Benchmarking for Robotics

We are inviting you to take a survey to provide feedback on the current state of open-source assets and benchmarking resources for robotics, and future activities for improvement. Participation in this study is completely voluntary. You will not be compensated for filling out the survey. Completing the survey should take no more than 15 minutes. To be eligible for this study you must be ≥18 years of age and able to read and write in English. 

If you are interested in participating, you can access the survey by clicking here

COMPARE Workshops

Charter for Strategic Development and Ecosystem Functions

To support the development, benchmarking, and deployment of robot systems, an open-source ecosystem (OSE) is under development through the National Science Foundation (NSF) Pathways to Enable Open-Source Ecosystems (POSE) program.

The OSE will facilitate the development and dissemination of the open-source assets for robotic manipulation, i.e., robot hardware, software, benchmarking practices. It will create a community-driven platform that researchers and developers can share and learn about these open-source resources, find tools to easily utilize them, collaborate on developing systematic robot experimentation methodologies, and disseminate their findings effectively. As such the OSE will address the issues facing the advancement of robot manipulation due to the lack of systematic development and benchmarking methodologies, as well as the unique challenges related to physical assets (both equipment and benchmarking tools), preventing the robotic manipulation community from effectively utilizing such resources compared to other research domains like computer vision.

The OSE will support a wide variety of open-source assets (OSAs) available or to be developed in the future for robot manipulation, including physical object sets and hardware designs, digital datasets and simulations, instructional task protocols and metrics, and functional algorithms and robotic behaviors. 

Participants in the OSE will include all stakeholders of these OSAs, including developers (those who develop and maintain the OSA), contributors (those who iterate on an existing OSA), and users (those who implement OSAs in their research and development practices). 

OSE structure, governance, functions, and activities will be facilitated by a distributed set of OSE participants in various roles (e.g., leadership and participatory positions), diverse in terms of terms of backgrounds, research domains, and developmental maturity, spanning industry, academia, and government entities. 

Development of the OSE will be conducted transparently, with continuous input solicited from the community and outputs shared from the initial core development team, through a series of workshops, meetings, surveys, and asynchronous communication via online platforms.

It is intended that the Open Source Security Foundation (OpenSSF) best practices for their Free/Libre and Open Source Software (FLOSS) projects be followed for all OSAs utilized within the OSE, although additional consideration will be needed for non-code-based OSAs such as those that are physical and instructional.

During Phase I of the project (2022 - 2023), the scope of the OSE shall be investigated and discovered, towards a proposal for Phase II (2024 - 2026). During Phase II, the OSE will be formally developed and established through the execution of multiple activities and functions, including, but not limited to, the following:

Principal Investigators

Holly Yanco

University of Massachusetts Lowell

Adam Norton

University of Massachusetts Lowell

Berk Calli

Worcester Polytechnic University

Aaron Dollar

Yale University


If you are interested in contributing to the development of the COMPARE open-source ecosystem or have any questions or comments, please e-mail Adam Norton at