Jeannette Bohg
Assistant Professor of Computer Science
Bio
Jeannette Bohg is an Assistant Professor of Computer Science at Stanford University. She was a group leader at the Autonomous Motion Department (AMD) of the MPI for Intelligent Systems until September 2017. Before joining AMD in January 2012, Jeannette Bohg was a PhD student at the Division of Robotics, Perception and Learning (RPL) at KTH in Stockholm. In her thesis, she proposed novel methods towards multi-modal scene understanding for robotic grasping. She also studied at Chalmers in Gothenburg and at the Technical University in Dresden where she received her Master in Art and Technology and her Diploma in Computer Science, respectively. Her research focuses on perception and learning for autonomous robotic manipulation and grasping. She is specifically interesting in developing methods that are goal-directed, real-time and multi-modal such that they can provide meaningful feedback for execution and learning. Jeannette Bohg has received several awards, most notably the 2019 IEEE International Conference on Robotics and Automation (ICRA) Best Paper Award, the 2019 IEEE Robotics and Automation Society Early Career Award and the 2017 IEEE Robotics and Automation Letters (RA-L) Best Paper Award.
Academic Appointments
-
Assistant Professor, Computer Science
-
Member, Bio-X
-
Faculty Affiliate, Institute for Human-Centered Artificial Intelligence (HAI)
2024-25 Courses
- Principles of Robot Autonomy I
AA 274A, CS 237A, EE 260A, ME 274A (Aut) - Principles of Robot Autonomy II
AA 174B, AA 274B, CS 237B, EE 260B, ME 274B (Win) - Topics in Advanced Robotic Manipulation
CS 326 (Aut) -
Independent Studies (12)
- Advanced Reading and Research
CS 499 (Aut, Win, Spr) - Advanced Reading and Research
CS 499P (Aut, Win, Spr) - Curricular Practical Training
CS 390A (Aut, Win, Spr) - Curricular Practical Training
CS 390B (Aut, Win, Spr) - Experimental Investigation of Engineering Problems
ME 392 (Aut, Win, Spr) - Independent Project
CS 399 (Aut, Win, Spr) - Independent Work
CS 199 (Aut, Win, Spr) - Master's Research
CME 291 (Aut, Win, Spr) - Part-time Curricular Practical Training
CS 390D (Aut, Win, Spr) - Senior Project
CS 191 (Aut, Win, Spr) - Supervised Undergraduate Research
CS 195 (Aut, Win, Spr) - Writing Intensive Senior Research Project
CS 191W (Aut, Win, Spr)
- Advanced Reading and Research
-
Prior Year Courses
2023-24 Courses
- Computer Vision: From 3D Perception to 3D Reconstruction and Beyond
CS 231A (Spr) - Principles of Robot Autonomy II
AA 174B, AA 274B, CS 237B (Win) - Topics in Advanced Robotic Manipulation
CS 326 (Aut)
2022-23 Courses
- Computer Vision: From 3D Reconstruction to Recognition
CS 231A (Win) - Principles of Robot Autonomy I
AA 174A, AA 274A, CS 237A, EE 160A, EE 260A (Aut) - Principles of Robot Autonomy II
AA 174B, AA 274B, CS 237B, EE 260B (Win)
2021-22 Courses
- Computer Vision: From 3D Reconstruction to Recognition
CS 231A (Win) - Principles of Robot Autonomy II
AA 174B, AA 274B, CS 237B, EE 260B (Win) - Robotics and Autonomous Systems Seminar
AA 289, CS 529 (Win, Spr) - Topics in Advanced Robotic Manipulation
CS 326 (Aut)
- Computer Vision: From 3D Perception to 3D Reconstruction and Beyond
Stanford Advisees
-
Doctoral Dissertation Reader (AC)
Dane Brouwer, William Chong, Won Kyung Do, Shivani Guptasarma, Colton Stearns, Gadi Sznaier Camps, Bohan Wu -
Doctoral Dissertation Advisor (AC)
Claire Chen, Marion Lepert, Krishnan Srinivasan -
Master's Program Advisor
Anshika Agarwal, Sochima Ezema, Megan Ja, Sahil Jain, Andrew Lee, Yousef Liang, Cameron Mohne, Hannah Norman, Erik Rozi, Khanh Tran, Xiaoyue Wang, Polycarpos Yiorkadjis, Abe Yosef -
Doctoral Dissertation Co-Advisor (AC)
Tyler Lum, Matt Strong, Priya Sundaresan -
Doctoral (Program)
Chris Agia, Claire Chen, Krishnan Srinivasan, Jingyun Yang -
Postdoctoral Research Mentor
Francis Engelmann
All Publications
-
ShaSTA: Modeling Shape and Spatio-Temporal Affinities for 3D Multi-Object Tracking
IEEE ROBOTICS AND AUTOMATION LETTERS
2024; 9 (5): 4273-4280
View details for DOI 10.1109/LRA.2023.3323124
View details for Web of Science ID 001192358500009
-
Design and Control of Roller Grasper V3 for In-Hand Manipulation
IEEE TRANSACTIONS ON ROBOTICS
2024; 40: 4222-4234
View details for DOI 10.1109/TRO.2024.3454388
View details for Web of Science ID 001317680300001
-
Text2Motion: from natural language instructions to feasible plans
AUTONOMOUS ROBOTS
2023; 47 (8): 1345-1365
View details for DOI 10.1007/s10514-023-10131-7
View details for Web of Science ID 001281797200002
-
TidyBot: personalized robot assistance with large language models
AUTONOMOUS ROBOTS
2023
View details for DOI 10.1007/s10514-023-10139-z
View details for Web of Science ID 001101530300001
-
Deep Learning Approaches to Grasp Synthesis: A Review
IEEE TRANSACTIONS ON ROBOTICS
2023
View details for DOI 10.1109/TRO.2023.3280597
View details for Web of Science ID 001019463600001
-
The OBJECTFOLDER BENCHMARK: Multisensory Learning with <i>Neural</i> and <i>Real</i> Objects
IEEE COMPUTER SOC. 2023: 17276-17286
View details for DOI 10.1109/CVPR52729.2023.01657
View details for Web of Science ID 001062531301056
-
KITE: Keypoint-Conditioned Policies for Semantic Manipulation
JMLR-JOURNAL MACHINE LEARNING RESEARCH. 2023
View details for Web of Science ID 001221201501003
-
In-Hand Manipulation of Unknown Objects with Tactile Sensing for Insertion
IEEE. 2023: 8765-8771
View details for DOI 10.1109/IROS55552.2023.10341456
View details for Web of Science ID 001136907802128
-
CARTO: Category and Joint Agnostic Reconstruction of ARTiculated Objects
IEEE COMPUTER SOC. 2023: 21201-21210
View details for DOI 10.1109/CVPR52729.2023.02031
View details for Web of Science ID 001062531305052
-
STAP: Sequencing Task-Agnostic Policies
IEEE. 2023: 7951-7958
View details for DOI 10.1109/ICRA48891.2023.10160220
View details for Web of Science ID 001048371101041
-
Visuomotor Control in Multi-Object Scenes Using Object-Aware Representations
IEEE. 2023: 9515-9522
View details for DOI 10.1109/ICRA48891.2023.10160888
View details for Web of Science ID 001048371102030
-
TidyBot: Personalized Robot Assistance with Large Language Models
IEEE. 2023: 3546-3553
View details for DOI 10.1109/IROS55552.2023.10341577
View details for Web of Science ID 001133658802091
-
Active Task Randomization: Learning Robust Skills via Unsupervised Generation of Diverse and Feasible Tasks
IEEE. 2023: 1924-1931
View details for DOI 10.1109/IROS55552.2023.10341727
View details for Web of Science ID 001133658801066
-
Learning Tool Morphology for Contact-Rich Manipulation Tasks with Differentiable Simulation
IEEE. 2023: 1859-1865
View details for DOI 10.1109/ICRA48891.2023.10161453
View details for Web of Science ID 001036713001096
-
A Bayesian Treatment of Real-to-Sim for Deformable Object Manipulation
IEEE ROBOTICS AND AUTOMATION LETTERS
2022; 7 (3): 5819-5826
View details for DOI 10.1109/LRA.2022.3157377
View details for Web of Science ID 000778903600002
-
Predicting Hand-Object Interaction for Improved Haptic Feedback in Mixed Reality
IEEE ROBOTICS AND AUTOMATION LETTERS
2022; 7 (2): 3851-3857
View details for DOI 10.1109/LRA.2022.3148458
View details for Web of Science ID 000756831900029
-
Vision-Only Robot Navigation in a Neural Radiance World
IEEE ROBOTICS AND AUTOMATION LETTERS
2022; 7 (2): 4606-4613
View details for DOI 10.1109/LRA.2022.3150497
View details for Web of Science ID 000766269000016
-
Symbolic State Estimation with Predicates for Contact-Rich Manipulation Tasks
IEEE. 2022: 1702-1709
View details for DOI 10.1109/ICRA46639.2022.9811675
View details for Web of Science ID 000941265701027
-
OBJECTFOLDER 2.0: A Multisensory Object Dataset for Sim2Real Transfer
IEEE COMPUTER SOC. 2022: 10588-10598
View details for DOI 10.1109/CVPR52688.2022.01034
View details for Web of Science ID 000870759103065
-
DiffCloud: Real-to-Sim from Point Clouds with Differentiable Simulation and Rendering of Deformable Objects
IEEE. 2022: 10828-10835
View details for DOI 10.1109/IROS47612.2022.9981101
View details for Web of Science ID 000909405302132
-
Whisker-Inspired Tactile Sensing for Contact Localization on Robot Manipulators
IEEE. 2022: 7817-7824
View details for DOI 10.1109/IROS47612.2022.9982122
View details for Web of Science ID 000909405300071
-
Category-Independent Articulated Object Tracking with Factor Graphs
IEEE. 2022: 3800-3807
View details for DOI 10.1109/IROS47612.2022.9982029
View details for Web of Science ID 000908368202123
-
Grounding Predicates through Actions open(drawer) open(drawer)
IEEE. 2022: 3498-3504
View details for DOI 10.1109/ICRA46639.2022.9812016
View details for Web of Science ID 000941265702038
-
Dynamic multi-robot task allocation under uncertainty and temporal constraints
AUTONOMOUS ROBOTS
2021
View details for DOI 10.1007/s10514-021-10022-9
View details for Web of Science ID 000714998700002
-
Concept2Robot: Learning manipulation concepts from instructions and human demonstrations
INTERNATIONAL JOURNAL OF ROBOTICS RESEARCH
2021
View details for DOI 10.1177/02783649211046285
View details for Web of Science ID 000709456500001
-
Learning latent actions to control assistive robots
AUTONOMOUS ROBOTS
2021: 1-33
Abstract
Assistive robot arms enable people with disabilities to conduct everyday tasks on their own. These arms are dexterous and high-dimensional; however, the interfaces people must use to control their robots are low-dimensional. Consider teleoperating a 7-DoF robot arm with a 2-DoF joystick. The robot is helping you eat dinner, and currently you want to cut a piece of tofu. Today's robots assume a pre-defined mapping between joystick inputs and robot actions: in one mode the joystick controls the robot's motion in the x-y plane, in another mode the joystick controls the robot's z-yaw motion, and so on. But this mapping misses out on the task you are trying to perform! Ideally, one joystick axis should control how the robot stabs the tofu, and the other axis should control different cutting motions. Our insight is that we can achieve intuitive, user-friendly control of assistive robots by embedding the robot's high-dimensional actions into low-dimensional and human-controllable latent actions. We divide this process into three parts. First, we explore models for learning latent actions from offline task demonstrations, and formalize the properties that latent actions should satisfy. Next, we combine learned latent actions with autonomous robot assistance to help the user reach and maintain their high-level goals. Finally, we learn a personalized alignment model between joystick inputs and latent actions. We evaluate our resulting approach in four user studies where non-disabled participants reach marshmallows, cook apple pie, cut tofu, and assemble dessert. We then test our approach with two disabled adults who leverage assistive devices on a daily basis.
View details for DOI 10.1007/s10514-021-10005-w
View details for Web of Science ID 000681168800001
View details for PubMedID 34366568
View details for PubMedCentralID PMC8335729
-
How to train your differentiable filter
AUTONOMOUS ROBOTS
2021
View details for DOI 10.1007/s10514-021-09990-9
View details for Web of Science ID 000659371400001
-
Detect, Reject, Correct: Crossmodal Compensation of Corrupted Sensors
IEEE. 2021: 909-916
View details for DOI 10.1109/ICRA48506.2021.9561847
View details for Web of Science ID 000765738801022
-
Differentiable Factor Graph Optimization for Learning Smoothers
IEEE. 2021: 1339-1345
View details for DOI 10.1109/IROS51168.2021.9636300
View details for Web of Science ID 000755125501017
-
TrajectoTree: Trajectory Optimization Meets Tree Search for Planning Multi-contact Dexterous Manipulation
2021 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)
IEEE. 2021: 8262-8268
View details for DOI 10.1109/IROS51168.2021.9636346
-
Probabilistic 3D Multi-Modal, Multi-Object Tracking for Autonomous Driving
IEEE. 2021: 14227-14233
View details for DOI 10.1109/ICRA48506.2021.9561754
View details for Web of Science ID 000771405405099
-
Interpreting Contact Interactions to Overcome Failure in Robot Assembly Tasks
IEEE. 2021: 3410-3417
View details for DOI 10.1109/ICRA48506.2021.9560825
View details for Web of Science ID 000765738802107
-
OmniHang: Learning to Hang Arbitrary Objects using Contact Point Correspondences and Neural Collision Estimation
IEEE. 2021: 5921-5927
View details for DOI 10.1109/ICRA48506.2021.9560971
View details for Web of Science ID 000765738804066
-
Combining learned and analytical models for predicting action effects from sensory data
INTERNATIONAL JOURNAL OF ROBOTICS RESEARCH
2020
View details for DOI 10.1177/0278364920954896
View details for Web of Science ID 000569523800001
-
Making Sense of Vision and Touch: Learning Multimodal Representations for Contact-Rich Tasks
IEEE TRANSACTIONS ON ROBOTICS
2020; 36 (3): 582–96
View details for DOI 10.1109/TRO.2019.2959445
View details for Web of Science ID 000543027200001
-
Learning Task-Oriented Grasping From Human Activity Datasets
IEEE ROBOTICS AND AUTOMATION LETTERS
2020; 5 (2): 3352–59
View details for DOI 10.1109/LRA.2020.2975706
View details for Web of Science ID 000520954200016
-
Self-Supervised Learning of State Estimation for Manipulating Deformable Linear Objects
IEEE ROBOTICS AND AUTOMATION LETTERS
2020; 5 (2): 2372–79
View details for DOI 10.1109/LRA.2020.2969931
View details for Web of Science ID 000526572000036
-
Object-Centric Task and Motion Planning in Dynamic Environments
IEEE ROBOTICS AND AUTOMATION LETTERS
2020; 5 (2): 844–51
View details for DOI 10.1109/LRA.2020.2965875
View details for Web of Science ID 000510751900008
-
UniGrasp: Learning a Unified Model to Grasp With Multifingered Robotic Hands
IEEE ROBOTICS AND AUTOMATION LETTERS
2020; 5 (2): 2286–93
View details for DOI 10.1109/LRA.2020.2969946
View details for Web of Science ID 000526572000025
-
Concept2Robot: Learning Manipulation Concepts from Instructions and Human Demonstrations
MIT PRESS. 2020
View details for Web of Science ID 000570976900082
-
Accurate Vision-based Manipulation through Contact Reasoning
IEEE. 2020: 6738-6744
View details for Web of Science ID 000712319504073
-
Learning Hierarchical Control for Robust In-Hand Manipulation
IEEE. 2020: 8855-8862
View details for Web of Science ID 000712319505116
-
Learning to Scaffold the Development of Robotic Manipulation Skills
IEEE. 2020: 5671-5677
View details for Web of Science ID 000712319503133
-
Learning User-Preferred Mappings for Intuitive Robot Control
IEEE. 2020: 10960-10967
View details for DOI 10.1109/IROS45743.2020.9340909
View details for Web of Science ID 000724145800103
-
Learning Topological Motion Primitives for Knot Planning
IEEE. 2020: 9457-9464
View details for DOI 10.1109/IROS45743.2020.9341330
View details for Web of Science ID 000724145803058
-
Multimodal Sensor Fusion with Differentiable Filters
IEEE. 2020: 10444-10451
View details for DOI 10.1109/IROS45743.2020.9341579
View details for Web of Science ID 000724145800046
-
Dynamic Multi-Robot Task Allocation under Uncertainty and Temporal Constraints
MIT PRESS. 2020
View details for Web of Science ID 000570976900068
-
Predicting grasp success in the real world - A study of quality metrics and human assessment
ROBOTICS AND AUTONOMOUS SYSTEMS
2019; 121
View details for DOI 10.1016/j.robot.2019.103274
View details for Web of Science ID 000491214500017
-
Variable Impedance Control in End-Effector Space: An Action Space for Reinforcement Learning in Contact-Rich Tasks
IEEE. 2019: 1010–17
View details for Web of Science ID 000544658400108
-
MeteorNet: Deep Learning on Dynamic 3D Point Cloud Sequences
IEEE. 2019: 9245–54
View details for DOI 10.1109/ICCV.2019.00934
View details for Web of Science ID 000548549204037
-
Leveraging Contact Forces for Learning to Grasp
IEEE. 2019: 3615–21
View details for Web of Science ID 000494942302094
-
Making Sense of Vision and Touch: Self-Supervised Learning of Multimodal Representations for Contact-Rich Tasks
IEEE. 2019: 8943–50
View details for Web of Science ID 000494942306083
-
Learning to Estimate Pose and Shape of Hand-Held Objects from RGB Images
IEEE. 2019: 3980–87
View details for Web of Science ID 000544658403042
-
Motion-Based Object Segmentation Based on Dense RGB-D Scene Flow
IEEE ROBOTICS AND AUTOMATION LETTERS
2018; 3 (4): 3797–3804
View details for DOI 10.1109/LRA.2018.2856525
View details for Web of Science ID 000441444700016
-
Interactive Perception: Leveraging Action in Perception and Perception in Action
IEEE TRANSACTIONS ON ROBOTICS
2017; 33 (6): 1273–91
View details for DOI 10.1109/TRO.2017.2721939
View details for Web of Science ID 000417841500001
-
Reports on the 2017 AAAI Spring Symposium Series
AI MAGAZINE
2017; 38 (4): 99–106
View details for DOI 10.1609/aimag.v38i3.2754
View details for Web of Science ID 000419468800015
-
Probabilistic Articulated Real-Time Tracking for Robot Manipulation
IEEE ROBOTICS AND AUTOMATION LETTERS
2017; 2 (2): 577–84
View details for DOI 10.1109/LRA.2016.2645124
View details for Web of Science ID 000413736600027
-
On the relevance of grasp metrics for predicting grasp success
IEEE. 2017: 265–72
View details for Web of Science ID 000426978200038
-
Big Data on Robotics.
Big data
2016; 4 (4): 195-196
View details for DOI 10.1089/big.2016.29013.rob
View details for PubMedID 27992266
-
Automatic LQR Tuning Based on Gaussian Process Global Optimization
IEEE. 2016: 270–77
View details for Web of Science ID 000389516200035
-
Optimizing for what matters: The Top Grasp Hypothesis
IEEE. 2016: 2167–74
View details for Web of Science ID 000389516201130
-
Exemplar-based Prediction of Global Object Shape from Local Shape Similarity
IEEE. 2016: 3398–3405
View details for Web of Science ID 000389516202137
-
Learning Where to Search Using Visual Attention
IEEE. 2016: 5238–45
View details for Web of Science ID 000391921705041
-
Robust Gaussian Filtering using a Pseudo Measurement
IEEE. 2016: 3606–13
View details for Web of Science ID 000388376103108
-
Robot Arm Pose Estimation by Pixel-wise Regression of Joint Angles
IEEE. 2016: 616–23
View details for Web of Science ID 000389516200076
-
Depth-Based Object Tracking Using a Robust Gaussian Filter
IEEE. 2016: 608–15
View details for Web of Science ID 000389516200075
-
Leveraging Big Data for Grasp Planning
IEEE COMPUTER SOC. 2015: 4304–11
View details for Web of Science ID 000370974904036
-
The Coordinate Particle Filter - A novel Particle Filter for High Dimensional Systems
IEEE COMPUTER SOC. 2015: 2454–61
View details for Web of Science ID 000370974902067
-
Data-Driven Grasp Synthesis-A Survey
IEEE TRANSACTIONS ON ROBOTICS
2014; 30 (2): 289–309
View details for DOI 10.1109/TRO.2013.2289018
View details for Web of Science ID 000334596700001
-
Three-dimensional object reconstruction of symmetric objects by fusing visual and tactile sensing
INTERNATIONAL JOURNAL OF ROBOTICS RESEARCH
2014; 33 (2): 321–41
View details for DOI 10.1177/0278364913497816
View details for Web of Science ID 000333552500006
-
Robot Arm Pose Estimation through Pixel-Wise Part Classification
IEEE. 2014: 3143–50
View details for Web of Science ID 000377221103023
-
Dual Execution of Optimized Contact Interaction Trajectories
IEEE. 2014: 47–54
View details for Web of Science ID 000349834600008
-
Fusing Visual and Tactile Sensing for 3-D Object Reconstruction While Grasping
IEEE. 2013: 3547–54
View details for Web of Science ID 000337617303083
-
Probabilistic Object Tracking using a Range Camera
IEEE. 2013: 3195–3202
View details for Web of Science ID 000331367403039
-
Visual servoing on unknown objects
MECHATRONICS
2012; 22 (4): 423–35
View details for DOI 10.1016/j.mechatronics.2011.09.009
View details for Web of Science ID 000304847300007
-
Mind the Gap - Robotic Grasping under Incomplete Observation
IEEE International Conference on Robotics and Automation
2011
View details for DOI 10.1109/ICRA.2011.5980354
-
Enhanced Visual Scene Understanding through Human-Robot Dialog
IEEE. 2011: 3342–48
View details for Web of Science ID 000297477503104
-
Learning grasping points with shape context
ROBOTICS AND AUTONOMOUS SYSTEMS
2010; 58 (4): 362–77
View details for DOI 10.1016/j.robot.2009.10.003
View details for Web of Science ID 000276666100003
-
OpenGRASP: A Toolkit for Robot Grasping Simulation
SPRINGER-VERLAG BERLIN. 2010: 109–20
View details for Web of Science ID 000329156800013
-
Strategies for Multi-Modal Scene Exploration
IEEE. 2010: 4509–15
View details for DOI 10.1109/IROS.2010.5652967
View details for Web of Science ID 000287672003149
-
Attention-based Active 3D Point Cloud Segmentation
IEEE. 2010: 1165–70
View details for DOI 10.1109/IROS.2010.5649872
View details for Web of Science ID 000287672003158
-
TOWARDS GRASP-ORIENTED VISUAL PERCEPTION FOR HUMANOID ROBOTS
WORLD SCIENTIFIC PUBL CO PTE LTD. 2009: 387–434
View details for DOI 10.1142/S0219843609001796
View details for Web of Science ID 000270041900004
-
Integration of Visual Cues for Robotic Grasping
SPRINGER-VERLAG BERLIN. 2009: 245–54
View details for Web of Science ID 000274012700025