v0.6.0 Alpha Release: Sapien 3 integration

[StoneT2000]

Current work on a v0.6.0 Alpha release which integrates the new SAPIEN 3 into the Maniskill benchmark. A number of new features and code refactors are also being done here in order to support some large-scale new projects coming to Maniskill for enabling better evaluation at scale and robot learning at scale.

Upcoming Changes

The biggest change is now SAPIEN 3 is now used instead of SAPIEN 2. This comes with a lot of nice upgrades to the GUI, from turning on ray tracing to make it look more realistic, to better pre-built scenes via a new scene builder API.

Preview of some of the new prebuilt scenes with some including existing tasks:

To play around with the new setup, if you have a GUI you can run

python mani_skill2/examples/demo_random_action.py -e "PegInsertionSide-v0" --render-mode="human" shader_dir "rt-fast"

which will run the PegInsertion task and open a viewer and turn on ray-tracing.

In the future, there will also be another large change (shouldn't affect users who don't need to write new tasks and just benchmark) where a lot of functions (building a scene, reward generation, getting observations) may be converted to be functional to support some new sapien features in the future. These planned changes are not in this alpha release. Moreover, demonstrations need to be regenerated, so the current ones do not work at the moment, this will be added later.

High-level Overview

• SAPIEN 3 is now used instead of SAPIEN 2.
• SAPIEN 3 uses an entity component system (ECS) which Maniskill builds upon now. To learn how ECS works see: https://github.com/SanderMertens/ecs-faq#what-is-ecs
• New Scene Builder class which simplifies any code for building scenes. Maniskill default provides a few simple scenes like a table-top scene (instead of that gray floor), as well as a prebuilt class that supports loading scenes from AI2-Thor (via the HSSD dataset: https://huggingface.co/datasets/hssd/)
• New Agent/Robot API. This enforces one robot class maps to exactly one URDF file and removes the need to define robot configurations, keeping code more readable, less scattered. Each robot class defines its own uid, urdf file, urdf_configs, controller configs, and sensor configs (all of which may be defined at runtime via the python @property decorator). Many folders and files have been reorganized as well. See mani_skill2/agents/robots/panda/panda.py for a good template to understand how the new API works.
• Cameras are no longer the top of the hierarchy, they now inherit BaseSensor and BaseSensorConfig classes, allowing users to define custom sensors (e.g. Lidar).
• A number of APIs that used be SAPIEN wrappers over the PhysX engine are now deprecated e.g sapien.Link should now be sapien.physx.PhysxArticulationLinkComponent
• New environment state parameterization
• Minimal environment class template is given. (Only initializes a robot).
• Some motion planning solution examples have been provided

More Details

SAPIEN 2 -> SAPIEN 3 Changes relevant to ManiSkill

• sapien.Link, sapien.Joint, sapien.Articulation now map to sapien.physx.Physx* components. In code we typically do import sapien.physx as physx to access these components now.
  In particular, the following have moved as so:

sapien.ActorBase -> physx.PhysxBaseComponent
sapien.Actor -> sapien.Entity
sapien.Link -> physx.PhysxArticulationLinkComponent
sapien.Joint -> physx.PhysxArticulationJoint
sapien.Contact -> physx.PhysxContact

• Transforming sapien poses is now done as pose_1 * pose_2 instead of pose_1.transform(pose_2)
• articulation.get_qlimit is now articulation.get_qlimit
• joint.get_limits is now joint.get_limit
• joint.set_limits is now joint.set_limit
• import sapien instead of import sapien.core as sapien
• builder.add_multiple_collisions_from_file is now builder.add_multiple_convex_collisions_from_file
• builder.add_collision_from_file is now builder.add_convex_collision_from_file
• sapien.RenderMaterial is now sapien.render.RenderMaterial
• actor.hide_visual is no longer a function. ManiSkill now out of the box provides a hide_entity and show_entity function which disables the first render body component of a given entity.
• sapien.Actor is now sapien.Entity. In general, everything is now an entity or component. In code, actor usually refers to entities that impact physical simulation (e.g. a static/dynamic object, the robot articulation). Entities can include even things such as the viewer window itself.
• A number of properties of sapien.Actor are now found in components physx.PhysxRigidDynamicComponent and sapien.render.RenderBodyComponent. Use entity.find_component_by_type(component_class_type) to find these and retrieve the relevant properties.
• global friction is now set via the sapien.physx module, and the default is 0.3 for all and 0 for restitution (matching unity.)
• urdf loaders no longer accept urdf_config as an argument. ManiSkill itself now provides an apply_urdf_config function to do the same thing.
• velocity is now called linear_velocity. physx.PhysxArticulationLinkComponent and physx.PhysxRigidDynamicComponent both support getting and setting linear and angular velocities.
• visual bodies no longer exist. Visual data is now in the form of a list of render shapes. To get the old visual bodies, find the RenderBodyComponent and loop over the render_shapes

for render_shape in entity.find_component_by_type(sapien.render.RenderBodyComponent).render_shapes:
    ...

• collision_shape.set_collision_groups does not accept *args, it accepts a single list of groups now.
• entity.set_damping is now deprecated. Instead you must set linear and angular damping on the physx.PhysxRigidDynamicComponent component of an entity. E.g.

obj_comp = entity.find_component_by_type(physx.PhysxRigidDynamicComponent)
obj_comp.set_linear_damping(0.1)
obj_comp.set_angular_damping(0.1)

• entity.lock_motion is now obj_comp.set_locked_motion_axes and the obj_comp is the physx.PhysxRigidDynamicComponent.
• sapien.PhysicalMaterial is now physx.PhysxMaterial
• The default sapien viewer and camera shader dir is called "default" not "ibl" now.
• scene.add_directional_light accepts shadow_scale instead of scale as argument

General Environment Changes

• Environments now have different environment state parameterizations.
• All original demonstrations (stored in the demos/v0 folder) are generally unusable due to using SAPIEN 3 now. These will be regenerated and uploaded online. When downloaded again the new demos in versions v0.6.x will be stored in demos/v1.
• Collision meshes of various assets will be regenerated with CoACD so they will be slightly different than before.
• SUPPORTED_ROBOTS is no longer something defined in environments (It was never used before)
• Deprecated the bg_name property of all base environments, which was a simple short solution to load scene files.
• With the new robot/agent API, the _configure_agent function is now deprecated.
• Environment first reset may be a little bit slower than previously. This is due to calling _load_agent an extra time to obtain run-time defined sensor and controller configurations to then correctly determine how observations generated
• Small optimization to reduce number of calls to env.evaluate function when computing reward.

Agent / Robot changes

• Each robot type has it's own folder. In each folder there are files defining the robot class for each URDF variant (e.g. Panda Arm with real sense camera on it). New API now means the agent configs folder is now removed.
• Moved base_controller.py to one folder above.
• Each robot class defines sensor_configs and controller_configs itself.
• Assets/robot descriptions have been reorganized. They are now in mani_skill2/assets/robots folder.
• Tasks that use specific robot like the real sense equipped panda arm explicitly use that robot class specified via robot uid or the robot class itself as the default robot to use instead of overloading the panda robot uid with different configurations.

Other

• All paths generally use python format string to pass in constants like PACKAGE_ASSET_DIR in file as opposed in another file
• get_actor_mesh is now get_component_mesh which gets the mesh of a given component such as a physx.PhysxArticulationLinkComponent component.
• get_actor_meshes is now get_component_meshes
• get/set_actor_visibility is now get/set_entity_visibility
• get_visual_body_meshes is now get_render_body_meshes
• RecordEpisodeWrapper now accepts a video_fps argument