RSS 2022 Workshop

Envisioning an Infrastructure for Multi-Robot and Collaborative Autonomy Testing and Evaluation


Recent years have seen significant progress made in standardizing robot software, thanks to ROS and its associated simulation tools such as Gazebo. This has enabled researchers to build on each other’s work without having to reinvent the wheel. Nevertheless, there is a critical lack of infrastructure when it comes to multi-robot systems and collaborative autonomy. The key question that we would like to discuss at this workshop is: what should the testing and validation infrastructure for multi-robot and collaborative autonomy look like.

Researchers who are inventing and enhancing new collaboration algorithms sometimes resort to testing their innovations on a small set of simplified scenarios that they have used in their previous work in order to reduce the time and cost of testing. Such testing may be merely a demonstration that the collaboration algorithm works; it may not yield comparisons with existing algorithms that generate insights into the performance and cost tradeoffs; and it may not consider a broad enough set of missions and scenarios needed for test and evaluation, validation and verification (TEVV). The multi-robot systems community needs an infrastructure that supports system development and testing from concept to realization in multiple domains. The goal of this workshop is to bring together researchers and practitioners to understand the infrastructure needs of collaborative autonomy testing.

The University of Maryland recently received a planning grant from the CISE Community Research Infrastructure (CCRI) program to envision what an Infrastructure for Collaborative Autonomy Testing (ICAT) would look like. Through this workshop, we seek to foster a discussion with the broader community, understand the needs of the stakeholders, and identify effective resources that might be incorporated into ICAT. Through this workshop, a larger portion of the CISE MRS research community will be able to participate in the design of ICAT.

Invited Speakers

Prof. Sebastian Elbaum

University of Virginia

Dr. Simon Jones

University of Bristol

Dr. Wolfgang Hönig

TU Berlin

Dr. Nina Mahmoudian

Purdue University

Dr. Morgan Quigley

Open Robotics

Dr. Michael Grey

Open Robotics

Call for Contributions

We invite participants to submit their research in RSS format (up to 4 pages including figures, excluding references). We encourage the submission of early ideas, late-breaking results, position papers, or open research questions that are likely to generate interesting discussions. Accepted papers will be presented in a poster session and selected papers as spotlight talks. All submitted contributions will go through a single blind review process.

Discussion Topics

The specific goals of the workshop will be to discuss ideas around following topics:

  • The need for infrastructure for multi-robot systems (MRS) research and collaborative autonomy test and evaluation;
  • Hardware and software system architecture for Infrastructure for Collaborative Autonomy Testing (ICAT) and use cases for the infrastructure;
  • Technologies that can be used as part of the ICAT; their strengths and opportunities for improvement.


Please email submissions (in PDF format) to:

RSS Submission Template:

For more information, see

Important Dates

  • Submission deadline: May 30, 2022 Extended June 4, 2022 (Anywhere on Earth).
  • Notification of acceptance (rolling basis): June 10, 2022.
  • Camera Ready Submission: June 22, 2022.
  • Workshop day: July 1, 2022.


In-person Venue: 1024 Mudd (Morningside Campus, Columbia University)

Remote Participation Link: Zoom link

Time Event
9:00 Opening Remarks
9:15 Invited Talk - Dr. Nina Mahmoudian (Purdue University)
10:00 Coffee Break
10:30 Invited Talk - Dr. Morgan Quigley and Dr. Michael Grey (Open Robotics)
11:15 Contributed presentations
12:00 Lunch Break
13:30 Invited Talk - Prof. Sebastian Elbaum (University of Virginia)
14:15 Invited Talk - Dr. Wolfgang Hönig (TU Berlin)
15:00 Coffee Break
15:30 Invited Talk - Dr. Simon Jones (University of Bristol)
16:15 Panel Discussion
17:00 Closing Remarks

Detailed Program

Invited Talk 1 (July 1, 9:15)

Low-cost maritime fleet planning and operation

Nina Mahmoudian

Abstract: This talk will present a holistic approach to low-cost fleet planning and operation for a variety of applications such as physical oceanography, Arctic observation, coastal surveillance, and target detection/classification that depend on the persistent operation of unmanned systems in maritime environments. We accomplish this through addressing the fleet planning problem of diverse fleets along with designing, simulating and validating custom and off the shelf Unmanned Surface Vehicles (USVs), Autonomous Underwater Vehicles (AUVs), and Underwater Gliders (UGs). In this talk, I will describe recent progress on mission planning considering realistic constraints on performance and degradation, design and operation of a custom small underwater glider ROUGHIE, development, optimization, testing, and validation of a collapsible underwater docking station for a range of AUVS, development and testing of a custom low cost USV, and validation of mission planning maneuvers using off the shelf AUVs. Simultaneous validation of multiple vehicles is achieved through surrogate models such as a fleet of low-cost, compact aerial platforms. This approach enables a lower cost and faster transition of novel persistent fleet technology to the field compared to directly transitioning from simulation to multiple high-cost assets.

Bio: Dr. Nina Mahmoudian is an associate professor of Mechanical Engineering at Purdue University. Previously, she was the Lou and Herbert Wacker Associate Professor in Autonomous Mobile Systems with the Department of Mechanical Engineering at Michigan Tech. She received her Ph.D. in Aerospace Engineering from Virginia Tech. Dr. Mahmoudian is a recipient of the 2015 NSF-CAREER and 2015 ONR-YIP awards. She is serving as editor of ICRA and associate editor of RA-L.

Invited Talk 2 (July 1, 10:30)

On simulating multi-fleet operations with Gazebo and Open-RMF

Morgan Quigley and Michael Grey

Abstract: In recent years, many applications of mobile robots have become commercially viable. Large buildings in a variety of industries now operate fleets of robots for cleaning, delivery, security patrols, and so on. Some buildings are now starting to operate multiple fleets of robots from multiple vendors, and discovering the challenges inherent in multi-vendor operations. In particular, fleets are often unaware of the other fleets in the facility, and although intra-fleet deconfliction is a common feature, inter-fleet deconfliction is often achieved operationally only through spatial or temporal isolation. Such isolation techniques can work up to a point, but often fail to scale beyond two or three fleets. In this talk, we will present Open-RMF, an open-source multi-fleet deconfliction and operations framework. We will describe our Gazebo-based development and testing process. We will also discuss the challenges of "emulating" the behavior of proprietary fleets, and share some thoughts on how simulation testing could be improved in the future, potentially by using vendor-provided fleet management software "in the loop" to reduce surprises in real-world testing.

Dr. Quigley is one of the original developers of the Robot Operating System (ROS), an open-source software framework that encourages collaborative software development and reuse. Morgan joined the founding team of Open Robotics after completing a PhD in Computer Science in the AI lab at Stanford University in 2012. His research interests include open-source collaboration at all levels of firmware and software, distributed embedded systems, hardware/software co-design, and heterogeneous multi-robot systems.
Dr. Grey has been working with Open Robotics for five years, after completing a PhD in Robotics at Georgia Tech. Grey has been based in Singapore for the last four years as a primary designer and author of Open-RMF, an open-source multi-fleet robot deconfliction and operations platform. His current research focuses are multi-agent motion planning, optimal resource allocation, and negotiation across distributed systems.

Contributed Presentations
11:15-11:22 Apurva Badithela, Josefine Graebener, Richard M. Murray. Minimally Constrained Testing for Autonomy with Temporal Logic Specifications.
11:22-11:29 Micah Corah and Sebastian Scherer. On Performance Impacts of Coordination via Submodular Maximization for Multi-Robot Perception Planning and the Dynamics of Target Coverage and Cinematography.
11:29-11:36 Mickey Li, Robert Clarke, Arthur Richards. Starling: Containerisation Architecture for Scalable Local Development, Deployment and Testing of Multi-UAV Systems.
11:36-11:43 Vasu Sharma, Prasoon Goyal, Kaixiang Lin, Govind Thattai, Qiaozi Gao, Gaurav S. Sukhatme. CHMARL: A Multimodal Benchmark for Cooperative, Heterogeneous Multi-Agent Reinforcement Learning.
11:43-11:50 Georgios Tzoumas, Lucio Salinas, Lenka Pitonakova, Alex Horlock, Jonathon Waters and Sabine Hauert. Developing digital twins of swarms of UAVs for wildfire monitoring.
11:50-12:00 Q&A session
Invited Talk 3 (July 1, 13:30)

Emerging Testing Techniques to Validate Robot Systems

Sebastian Elbaum

Abstract: The soaring number of autonomous systems are bringing to prominence the insidiousness and potential impact of their faults. In turn, these faults highlight the limitations of existing analysis techniques, which are being challenged by robots’ immense and rich input space, the integration of cyber and physical semantics, the multiple sources of uncertainty, and the incorporation of machine learning components. In this talk I will provide an overview of some mobile robotic platforms and frameworks that we have targeted to ground our research efforts in addressing these challenges, the range of properties that those systems are meant to satisfy from robustness to ethical concerns, and the body of techniques we have developed to address those challenges. I will argue that such techniques must be an integral part of any testing infrastructure aiming at validating robots.

Bio: Dr. Elbaum is Anita Jones Professor of Computer Science at the University of Virginia, and one of the founders of the Laboratory for Engineering Safe Software (LESS Lab). His research aims to build dependable systems through domain-specific analysis techniques. His teaching focuses on instilling cost-effective software development principles. He is the recipient of an NSF Career Award, an IBM Innovation Award, a Google Faculty Research Award, an FSE Test of Time Award 2018, several ACM SigSoft Distinguished Paper Awards (FSE2006, ICSE2008, ICSE2012, ISSTA2013, ICSE2016), several best paper and distinguished artifacts awards (ESEM2011, IROS2016, ISSTA2017, ISSTA2020), mostly for empirically studying program analysis and software testing challenges, and developing automated techniques for addressing them. His current development and analysis work is focused on autonomous systems. He served as Program Co-Chair of the 2015 International Conference on Software Engineering, Program Chair for the 2007 International Symposium on Software Testing and Analysis, Program Co-Chair for the 2008 Empirical Software Engineering Symposium, Co-Editor for the Information and Software Technology Journal, and as Associate Editor of the ACM Transactions on Software Engineering and Methodologies Journal. He was the Steering Committee Chair for ICSE from 2015-2019. He was recognized as a Distinguished Scientist by the Association for Computing Machinery for his contributions to computing, and as a Fellow by the IEEE for his contributions to testing evolving systems. He received his Ph.D. from the University of Idaho, and a Systems Engineering degree from Universidad Catolica de Cordoba, Argentina. He spent a significant part of his academic career at the University of Nebraska, where he co-founded two internationally recognized labs, the E2 Software Engineering Lab and the Nimbus Robotics Lab. He has spent his sabbaticals as a research scientist or research fellow at Google (Mountain View, USA), CNR (Pisa, Italy), and UCL (London, UK).

Invited Talk 4 (July 1, 14:15)

Crazyswarm(2): A Testbed for Aerial Robot Teams

Wolfgang Hönig

Abstract: I will present key concepts and use-cases for the Crazyswarm, a ROS-based test-bed for operating large aerial multi-robot teams that use the commercially available Bitcraze Crazyflie as autopilot. A particular focus will be on how we solved challenges around localization, communication, and simulation, which are often an issue for any multi-robot system. I will also provide an overview of the current state of Crazyswarm2, our re-designed, ROS2-based version of the testbed. Finally, I will present examples and my future vision of how Crazyswarm(2) has been and will be useful for testing and validation of autonomous multi-robot teams.

Bio: Wolfgang Hönig is an independent junior research group leader at TU Berlin, Germany, heading the Intelligent Multi-Robot Coordination Lab. Previously, he was a postdoctoral scholar at the California Institute of Technology, USA. He received the diploma in Computer Science from TU Dresden, Germany in 2012, and the M.S. and Ph.D. degrees from the University of Southern California (USC), USA in 2016 and 2019, respectively. His research focuses on enabling large teams of physical robots to collaboratively solve real-world tasks, using tools from informed search, optimization, and machine learning.

Invited Talk 1 (July 1, 15:30)

DOTS: An Open Testbed for Industrial Swarm Robotic Solutions

Simon Jones

Abstract: We present DOTS, a new open access testbed for industrial swarm robotics experimentation. It consists of 20+ fast agile robots with high sensing and computational performance, and real-world payload capability. They are housed in an arena equipped with private 5G, motion capture, multiple cameras, and are openly accessible via an online portal. Our aim is to reduce barriers to entry by providing a complete platform-agnostic pipeline to develop, simulate, and deploy experimental applications to the swarm. As a proof-of-concept, we show how the testbed can be used to design new intralogistics solutions using out-of-the-box swarms.

Bio: Simon is a Senior Researcher and Doctoral Research Prize Fellow in the Hauert Lab and Bristol Robotics Laboratory where he works on distributed situational awareness for robot swarms. He's been Co-I and lead researcher on several grants related to building a state-of-the-art swarm testbed for intralogistics in collaboration with industry. He is also researching the automatic design of controllers for robot swarms that are human readable using artificial evolution of behaviour trees, including onboard custom built swarms with GPUs. Prior to academia, Simon was principal engineer at ARM.


Jeffrey Herrmann

University of Maryland

Craig Lawrence

Applied Research Laboratory
for Intelligence and Security

Dinesh Manocha

University of Maryland

Derek Paley

University of Maryland

Adam Porter

University of Maryland

Pratap Tokekar

University of Maryland

Nare Karapetyan

University of Maryland