Collision-free planning with the world represented by camera perception is an open research problem. cuRobo provides integration with nvblox for querying closest points between the robot and a voxelized map. We found that this works well in sparse obstacle environments. But as we increased the density of obstacles, the occlusions in perception can cause many failures.
pytest==6.2.5 has a conflict with typeguard which is installed with torchtyping. This will cause the repo to not work with a pytorch container <22.09.
Once a cuda graph is generated, we cannot change the dimensions of any of the tensors (e.g., dof, timesteps, number of seeds). While cuRobo library is designed to regenerate the cuda graph if a change in dimension is detected, pytorch crashes when CUDAGraph.reset() is called.
Compiling nvblox as redistributable with
cxx11_ABI=1does not work as cmake links to global glog library instead of the local one. Hence compile nvblox as standard when using with cxx11_ABI=1 as apt install glog is also compiled with this flag.
Voxel sizes less then 1 cm (
<0.01) can hit GPU memory limits very quickly.
Continuous collision checking is not implemented, so success rate can be lower in environments with very thin obstacles compared to using meshes or cuboids as obstacles.
Reinitializing nvblox inside a python script can cause CUDA errors. Instead of reinitializing nvblox mapper instance, clear the existing map.
Headless mode works reliably with Omniverse Streaming Client by setting
--headless_mode=nativefor examples in Using with Isaac Sim. WebSocket and WebRTC can have issues.
When using development docker, webrtc headless mode does not work. Use
Mimic joints cannot be loaded from a urdf in some kinematic chain instances. cuRobo also does not support mimic joints, hence convert your robot’s mimic joints as independently actuated joints or fixed joints. This is a limitation of cuRobo.