Departmental Papers (ESE)


Streaming Media


This paper presents a reactive planning system that enriches the topological representation of an environment with a tightly integrated semantic representation, achieved by incorporating and exploiting advances in deep perceptual learning and probabilistic semantic reasoning. Our architecture combines object detection with semantic SLAM, affording robust, reactive logical as well as geometric planning in unexplored environments. Moreover, by incorporating a human mesh estimation algorithm, our system is capable of reacting and responding in real time to semantically labeled human motions and gestures. New formal results allow tracking of suitably non-adversarial moving targets, while maintaining the same collision avoidance guarantees. We suggest the empirical utility of the proposed control architecture with a numerical study including comparisons with a state-of-the-art dynamic replanning algorithm, and physical implementation on both a wheeled and legged platform in different settings with both geometric and semantic goals.

For more information: Kod*lab

Sponsor Acknowledgements

This work was supported by AFRL grant FA865015D1845 (subcontract 669737-1), AFOSR grant FA9550-19-1-0265 (Assured Autonomy in Contested Environments), and ONR grant #N00014-16-1-2817, a Vannevar Bush Fellowship held by the last author, sponsored by the Basic Research Office of the Assistant Secretary of Defense for Research and Engineering. The authors thank Diedra Krieger for assistance with video recording.

Document Type

Journal Article

Subject Area

GRASP, Kodlab

Date of this Version


Publication Source

IEEE Robotics and Automation Letters (RA-L)





Start Page


Last Page




Copyright/Permission Statement

This material is posted here with permission of the IEEE. Such permission of the IEEE does not in any way imply IEEE endorsement of any of the University of Pennsylvania's products or services. Internal or personal use of this material is permitted. However, permission to reprint/republish this material for advertising or promotional purposes or for creating new collective works for resale or redistribution must be obtained from the IEEE by writing to By choosing to view this document, you agree to all provisions of the copyright laws protecting it.


Reactive and Sensor-Based Planning, Motion and Path Planning, Semantic Scene Understanding

Bib Tex

@article{Vasilopoulos_RAL_2020, author = {V. Vasilopoulos and G. Pavlakos and S. L. Bowman and J. D. Caporale and K. Daniilidis and G. J. Pappas and D. E. Koditschek}, title = {{Reactive Semantic Planning in Unexplored Semantic Environments Using Deep Perceptual Feedback}}, journal = {{IEEE Robotics and Automation Letters (RA-L)}}, volume = {5}, number = {3}, pages = {4455-4462}, DOI = {10.1109/LRA.2020.3001496}}



Date Posted: 12 June 2020

This document has been peer reviewed.