Abstract

Advancement in robot manipulation is limited by a lack of systematic development and benchmarking methodologies, causing inefficiencies and even stagnation. There are several assets available in the robotics literature (e.g., YCB, NIST-ATB), yet an active and effective mechanism to disseminate and use them is lacking, which significantly reduces their impact and utility. This workshop will take a step towards removing the roadblocks to the development and assessment of robot manipulation hardware and software by reviewing, discussing, and laying the groundwork for an open-source ecosystem. The workshop aims to determine the needs and wants of robot manipulation researchers regarding open-source asset development, utilization, and dissemination. As such, the workshop will play a crucial role for identifying the preconditions and requirements to develop an open-source ecosystem that provides physical, digital, instructional, and functional assets for performance benchmarking and comparison. Discussions will include ways for maintaining ecosystem activity over time and identifying methods and principles to achieve a sustainable open-source effort. Accordingly, the invited speakers of this workshop includes experts that led well-established, successful open-source efforts (e.g., Robotarium, ROS-Industrial) along with experimentation experts and developers of newer open-source assets (e.g., NIST-MOAD, Household Cloth Object Set). The overarching goal is to learn from the successful examples and open communication channels between new and experienced researchers.

Overview

Presentations and guided discussion will take place across three categories:

1. Open-source Hardware and Infrastructure: hardware designs that incorporate commercial off-the-shelf (COTS) materials and rapid-prototyping (e.g., Yale OpenHand), test apparatuses (e.g., ACRV Picking Benchmark), and remotely accessible infrastructure for conducting robot manipulation experiments (e.g., Robotarium).

2. Open-source Software and Functionality: algorithms and behaviors that drive robot manipulation operations (e.g., MoveIt motion planner) and calibration (e.g., DREAM camera calibration), typically accessible via the Robot Operating System (ROS).

3. Open-source Benchmarking Protocols and Datasets: task protocols and metrics (e.g., RGMC service and manufacturing tasks) and datasets generated from physical testing (e.g., Cornell Grasp Dataset) or simulations (e.g., Jacquard Dataset).

After each category sessions' presentations and Q+A for each presenter is completed, a guided discussion will be facilitated amongst the workshop participants. The following topics, among others, will be put forth to motivate these discussions:

• Availability: What open-source assets are available in this category? What types of assets are there too many or too few of? How is the availability of these assets promoted, or how should it be?

• Composition: What formats or structures of open-source assets in this category are used? What characteristics are they missing and which are unnecessary? 

• Applicability: Are the open-source assets reviewed in this category applicable to your research? What uses cases would they be applicable to? Are there particular domains or applications that would benefit greatly from open-source assets?

• Benefits: What are the benefits of having this open-source asset available? How do you use it for your own work? Are there missing features that would provide greater benefit to you or others?

• Implementation: What are the barriers to using open-source assets in this category? Are there existing instructions and documentation that assist in implementation, or are these features lacking? What level of support is desired to ease implementation?

Speakers

Contributions

Short papers are sought to be presented that discuss issues faced, successes achieved, and/or analyses of the current landscape of robotic manipulation when developing or utilizing open-source assets and conducting benchmarking. Submissions may be in the form of position papers, proposals for new efforts, or reporting of new results, 2-4 pages, with the expectation that authors of accepted papers will provide a presentation at the workshop and participate in topic discussions. 

Submissions of papers should use the ICRA 2023 format, 2-4 pages in length (excluding references), anonymization not required. Contributed papers should fit into one or more of the following topics of the workshop: open-source hardware and infrastructure, open-source software and functionality, open-source benchmarking protocols and datasets, the availability of open-source assets, their composition, applicability or lack there of, benefits of open-source, and barriers to implementation, among others.

All submissions will be reviewed and authors of accepted papers will be asked to give a 10 minute talk at the workshop. At least one author of each accepted submission must register for the workshop and attend in-person; remote presentation will not be allowed.

• February 22, 2023: Call for submissions open

• March 17, 2023, 23:59 Anywhere on Earth (AoE): Early submission deadline for short papers to ensure decision by ICRA 2023 early registration deadline (April 1)

• March 24, 2023: Notification of acceptance of early workshop submissions

• April 21, 2023, 23:59 AoE: Submission deadline for short papers

• April 28, 2023: Notification of acceptance for workshop submissions

Submissions should be e-mailed to adam_norton@uml.edu with the text “[ICRA 2023 Workshop Submission]” in the subject line. Authors of accepted submissions will be offered the option of having their papers uploaded to a workshop-specific archive on arXiv.org. Inclusion in this archive will not be mandatory since it may create problems for authors who wish to submit follow-on work to venues with strict prior publication rules.

Participation

The workshop will be hybrid, with a focus on in-person participation, but a virtual option for remote attendees to watch presentations and participate in discussion will be available. Join the COMPARE project Slack workspace, channel #icra-2023-workshop, to participate in discussions pre, during, and post-workshop: https://join.slack.com/t/compare-ecosystem/shared_invite/zt-1nfgdwq4z-_8_PsXVhJ6H1FAZuQizjTA 

Schedule (tentative)

Invited talks: 30 minutes each (20 presentation + 10 questions)

Submitted talks: 15 minutes each (10 presentation + 5 questions)

Discussion: 30 minutes each

All times given are in Greenwich Mean Time (GMT)

Introduction

• 9:00 Introduction of workshop participants

• 9:10 Current State of Open-source Robot Manipulation Landscape and User Experiences, to be provided by one of the workshop organizers

Open-source Hardware and Infrastructure

• 9:30 Robotarium, Sean Wilson

• 10:00 REMOTE: Remote Experimentation of Manipulation for Online Test and Evaluation, Cindy Grimm

• 10:30 The TriFinger Platform and the Real Robot Challenge for Learning Agile Manipulation, Felix Widmaier

• 11:00 Submitted talks 

• 11:30 Discussion

Lunch break

• 12:00 - 1:00

Open-source Software and Functionality

• 1:00 ROS-Industrial, Lily Baye-Wallace

• 1:30 What has Benchmarking ever done for us (in Robotic Manipulation)?, Jürgen “Juxi” Leitner

• 2:00 Submitted talks

• 2:30 Discussion

• 3:00 Break

Open-source Benchmarking Protocols and Datasets

• 3:30 NIST Manufacturing Object Assemblies Dataset, Megan Zimmerman

• 4:00 Challenges in Comparing Cloth Manipulation, Irene Garcia-Camacho

• 4:30 Submitted talks

• 5:00 Discussion

Next Steps

• 5:30 Next Steps for Establishing an Open-source Ecosystem for Robot Manipulation, to be provided by one of the workshop organizers

• 6:00 Workshop end

IEEE RAS Technical Committees

This workshop is endorsed by:

• The IEEE RAS Technical Committee on Robotic Hands, Grasping & Manipulation (TC RHGM): https://www.ieee-ras.org/robotic-hands-grasping-and-manipulation

• The IEEE RAS Technical Committee for Performance Evaluation & Benchmarking of Robotic and Automation Systems (TC PEBRAS): https://www.ieee-ras.org/performance-evaluation

Organizers

• Adam Norton, University of Massachusetts Lowell

• Holly Yanco, University of Massachusetts Lowell 

• Berk Calli, Worcester Polytechnic Institute

• Aaron Dollar, Yale University

Contact

Please contact Adam Norton with any questions or comments via e-mail: adam_norton@uml.edu 

Funded by the National Science Foundation, Pathways to Enable Open-Source Ecosystems (POSE), Award TI-2229577