The geometry of a robot in a depth image can be filtered/segmented using cuRobo’s kinematics and
geometry distance queries. This is available through
curobo.wrap.model.robot_segmenter.RobotSegmenter which can be initialized with a robot
configuration file. Once initialzed, you can pass in curobo.types.camera.CameraObservation
with depth_image, intrinsics, and pose (with pose of the camera w.r.t. robot’s base frame), and
the robot’s joint configuration as a curobo.types.state.JointState. Internally, the
function first generates a pointcloud from the depth image using the intrinsics and then transforms
the pointcloud to the robot’s base frame. Then position of the spheres representing the robot geometry
are computed using cuRobo’s kinematics. Finally distance between the pointcloud and spheres are
computed and any point within a distance threshold are treated as robot points and filtered.
Check examples/robot_image_segmentation_example.py for reference. cuRobo’s implementation wraps
the operations in a CUDA graph to reduce python overheads, achieving 3000hz on a 4090 GPU for
480x480 images.