Spot Sim2Real Release Note
This release aims to reproduce the demos of Adaptive Skill Coordination ASC and Language-guided Skill Coordination LSC. We open-source the weights of the navigation, picking, and place skills, and the infrastructure to use Large Language Models (LLMs) to interact with those skills.
The codebase consists of two major components:
- bd_spot_wrapper: It provides a Python interface for using Boston Dynamics API to control the robots. We expect users will use this wrapper to design their own custom pipeline to control robots.
- spot_rl_experiments: It provides an example and demo code for using
bd_spot_wrapperto reproduce ASC and LSC.
Major Features and Improvements:
-
Designed Boston Dynamics API wrapper by @naokiyokoyama and @akshararai in this repo
-
Designed Adaptive Skill Coordination demo by @naokiyokoyama and @akshararai in this repo
- @aclegg3 and @eundersander supported the Spot simulation infra
-
Designed the Language-guided Skill Coordination demo by @jimmytyyang, @SergioArnaud, @akshararai, @KavitShah1998 in this branch
- @jimmytyyang developed and designed first prototype of LSC demo using LLM and Whisper APIs from @SergioArnaud
- @SergioArnaud and @akshararai further greatly improved the LSC robustness and demo quality by developing a standard procedure to record the waypoints, fixing edge cases, adding third party models (e.g., owlvit model), and simplifying the demo preparation
- @KavitShah1998 then joined the team, refactored the code to make it modular and maintainable, and designed the hardware test with @jimmytyyang
-
Integrated OpenAI's whisper with LLM (lang_env) by @SergioArnaud and @KavitShah1998 in #7
-
Integrated ASC with Owlvit model by @SergioArnaud and @KavitShah1998 in #10
-
Designed documentation and cleanup unnecessary files by @KavitShah1998 in #50
Other Improvements and Bug Fixes:
- Merged lang_pipeline into bounding_box_detector by @SergioArnaud in #8
- Enhanced the robustness of the place skills by @KavitShah1998 in #13
- Ensured Owlvit model returns right bounding box after locking on the target by @KavitShah1998 in #14
- Improved the demo robustness by @KavitShah1998 in #18
- Searched for correct sound device and update device ID in whisper translator by @KavitShah1998 in #24
- Added return_to_base flag to trigger/bypass docking behavior by @KavitShah1998 in #29
- Added circle ci by @jimmytyyang in #32
- Updated circle ci by @jimmytyyang in #35
- Updated pre-commit to avoid checking the third party by @jimmytyyang in #40
- Fixed mypy checking by @jimmytyyang in #41
