Key Takeaways

Based on the discussions had at the workshop, a set of key takeaways have been summarized and organized into topics below:

Current landscape: Issues

• Ensure the inclusion of simulation assets in ecosystem definition

• Ontologies of tasks are needed such that performance thresholds in each can be set

• For publications and funding opportunities, novel solutions are favored over replication

  • ACM Emerging Interest Group (EIG) on Reproducibility and Replicability is working to integrate these practices into ACM journals and conferences https://reproducibility.acm.org/ 

• Developing open-source is thankless and payless; but it can lead to more citations

• Different robot platforms cause lack of replication of open-source between labs

• Need to start promoting generalization of methods over specific, narrow solutions

• Lack of diversity of tasks with benchmarks, yet there is a lot of research in these other application areas

  • Paper length restrictions can inhibit inclusion of sufficient task description detail

  • Supplementary material/data sometimes allowed in paper submissions, which could be the right method to include this info

• No existing review/qualification process for generating and sharing datasets

Current landscape: Existing efforts to leverage

• Datasheets for datasets: https://dl.acm.org/doi/10.1145/3458723 Can this be adopted for use in robotics to document open-source and benchmarking for replication purposes? 

• HRI conference is starting to incorporate mechanisms similar to those identified through this project, such as standard reporting of study parameters

  • Advocates found their way onto the organizing committee and helped to make these changes from inside

• HRI conference now has short contributions for code, datasets, and replication studies where documentation is reviewed in addition to the paper itself

  • This practice would be useful in multiple areas of robotics

• Robotics competitions are good ways to get the community focused around a common task

• Software Bill of Materials (SBOM) practices could be adopted for robotics for software + hardware + the relationships between the two 

  • Recording the packages used, the computing environment it runs in, the hardware of the robot it runs on, etc.

Future solutions: Modular benchmarking software pipelines

• Will labs that develop RL solutions, end-to-end solutions, etc., be able to use the pipeline?

  • We will have to develop it to allow for two distinct type of usage profiles: step-by-step (individual components) or end-to-end (complete systems)

• Benchmarking protocols and metrics should apply equally to solutions that use the pipeline and those that do not

Future solutions: Distributed physical benchmarking facilities

• Start with a centralized effort conducted at a single facility (e.g., UMass Lowell NERVE Center) to work out the logistics and infrastructure needed for others to participate, then slowly bring on volunteer-based organizations to provide labs/hardware/time

• Incentives to participate:

  • Develop some form of “badging” for participating labs in this network

  • Agreement on shared authorship of papers for research conducted in a participating lab

• When planning benchmarking exercises, consider broad vs. narrow challenge spaces

• Similar community-driven effort: Dialogue System Technology Challenge (DSTC) https://dstc11.dstc.community/

• If a company that provides resources sponsors a competition or workshop, the reward can be in terms of resources rather than cash (e.g., Amazon sponsored a workshop that awarded some amount of MTurk credits as a prize)

Future solutions: Online community resources

• Some existing leaderboards could be used or modeled after, such as the AI leaderboards on https://leaderboard.allenai.org/ 

• LLM model cards include characteristics of a model in a standard format; the proposed ROS Enhancement Proposal (REP) for characterization of benchmarking data in open-source assets could be developed similarly

Future solutions: Working groups and advocacy

• Participants must be diverse and decentralized, not heavily influenced by a single entity

• Do not set requirements for leveraging open-source and requiring benchmarking comparisons until open-source assets are well documented enough for proper replication and benchmarking methods have been established

• Let’s limit the tasks/domains we work in to ensure that we can require/build criteria AND have the best practices to require

Discussion Topics

The forum will be split into two distinct categories: (1) the current landscape of robotic manipulation open-source and benchmarking practices, and (2) proposed new activities to improve robotic manipulation open-source and benchmarking practices. A note taker will record meeting minutes for all topics discussed and salient takeaways will be distilled. Topics for discussion include the following:

• Limitations to robotics research and development caused by access to robotic hardware, robot simulation capabilities, lack of relevant comparable benchmarks, and lack of relevant open-source assets.

• Current ability of researchers to benchmark their robotic manipulation research to others in the field using existing protocols, or utilize open-source assets in their research and contribute to their development.

• Possible activities to improve the current state of open-source and benchmarking for robotic manipulation, including organized repositories of open-source assets and benchmarking results, establishing advisory committees, developing physical distributed benchmarking facilities, and dedicated conference tracks.

Participation

The workshop will be hybrid, with a focus on in-person participation given the time focus on forum discussions, but a virtual option for remote attendees will be available. Questions will be collected from the online chat and asked by one of the workshop organizers to the speaker. A Slack workspace is being established for pre, during, and post-workshop discussions and coordination, to serve as an open communication platform for the open-source ecosystem. 

Contact

Please contact Adam Norton with any questions or comments via e-mail: adam_norton@uml.edu