Robot Segmentation
Steps in filtering a robot using cuRobo’s robot segmentation. The scene shown in left is seen by a depth camera. The depth image is converted to a pointcloud and transformed to the robot base frame as shown in yellow (second from left). The current joint configuration is used in cuRobo’s kinematics to obtain the position of robot’s spheres as seen by the orange spheres (third from left). Points within these spheres are subtracted out (fourth image from left) and the final pointcloud is obtained with no robot visible (rightmost image). This pointcloud is converted back to a depth image and returned, along with a mask label.
The geometry of a robot in a depth image can be filtered/segmented using cuRobo’s kinematics and
geometry distance queries. This is available through
curobo.wrap.model.robot_segmenter.RobotSegmenter which can be initialized with a robot
configuration file. Once initialzed, you can pass in curobo.types.camera.CameraObservation
with depth_image, intrinsics, and pose (with pose of the camera w.r.t. robot’s base frame), and
the robot’s joint configuration as a curobo.types.state.JointState. Internally, the
function first generates a pointcloud from the depth image using the intrinsics and then transforms
the pointcloud to the robot’s base frame. Then position of the spheres representing the robot geometry
are computed using cuRobo’s kinematics. Finally distance between the pointcloud and spheres are
computed and any point within a distance threshold are treated as robot points and filtered.
Check examples/robot_image_segmentation_example.py for reference. cuRobo’s implementation wraps
the operations in a CUDA graph to reduce python overheads, achieving 3000hz on a 4090 GPU for
480x480 images.
The depth input depth image is shown in the left, followed by the segmented robot in middle and world without robot in right.