Benchmarks & Profiling¶
Benchmarks¶
Results in the arxiv paper were obtained from v0.6.0.
v0.6.2 has significant changes to improve motion quality with lower motion time, lower path length, higher pose accuracy (<1mm) while taking 15ms (RTX 4090) longer to compute solutions.
v0.7 improves signed distance calculations for cuboids and meshes. Also, leverages torch.compile
for better kernel fusion. These improvements result in higher quality solutions in lower time (10ms
faster than arxiv report on average).
Latest Motion Generation Results¶
Results obtained on November 22 2024 (v0.7.6).
Motion Generation on 2600 problems from motion benchmaker and motion policy networks, on a RTX 6000 Ada:
Metric |
Value |
|---|---|
Success % |
99.85 |
Plan Time (s) |
mean: 0.039 ± 0.043 median:0.031 75%: 0.034 98%: 0.125 |
Motion Time(s) |
mean: 1.181 ± 0.329 median:1.181 75%: 1.368 98%: 2.049 |
Path Length (rad.) |
mean: 3.144 ± 1.015 median:3.245 75%: 3.776 98%: 5.074 |
Jerk |
mean: 148.392 ± 63.158 median:138.790 75%: 186.254 98%: 292.364 |
Position Error (mm) |
mean: 0.008 ± 0.089 median:0.004 75%: 0.007 98%: 0.032 |
Motion Benchmaker (800 problems):
Metric |
Value |
|---|---|
Success % |
100.00 |
Plan Time (s) |
mean: 0.042 ± 0.041 median:0.031 75%: 0.032 98%: 0.161 |
Motion Time(s) |
mean: 1.361 ± 0.277 median:1.330 75%: 1.420 98%: 2.292 |
Path Length (rad.) |
mean: 3.815 ± 0.669 median:3.667 75%: 4.218 98%: 5.295 |
Jerk |
mean: 118.975 ± 39.765 median:121.938 75%: 144.564 98%: 197.375 |
Position Error (mm) |
mean: 0.013 ± 0.150 median:0.004 75%: 0.007 98%: 0.048 |
Motion Policy Networks (1800 problems):
Metric |
Value |
|---|---|
Success % |
99.78 |
Plan Time (s) |
mean: 0.038 ± 0.045 median:0.031 75%: 0.034 98%: 0.092 |
Motion Time(s) |
mean: 1.101 ± 0.318 median:1.066 75%: 1.301 98%: 1.884 |
Path Length (rad.) |
mean: 2.843 ± 0.999 median:2.862 75%: 3.480 98%: 4.907 |
Jerk |
mean: 161.566 ± 67.128 median:153.940 75%: 200.708 98%: 296.889 |
Position Error (mm) |
mean: 0.006 ± 0.036 median:0.004 75%: 0.007 98%: 0.015 |
World Represented as meshes (2600 problems), python benchmark/curobo_benchmark.py --mesh:
Metric |
Value |
|---|---|
Success % |
99.96 |
Plan Time (s) |
mean: 0.063 ± 0.045 median:0.052 75%: 0.064 98%: 0.167 |
Motion Time(s) |
mean: 1.179 ± 0.328 median:1.184 75%: 1.370 98%: 1.998 |
Path Length (rad.) |
mean: 3.131 ± 1.011 median:3.235 75%: 3.771 98%: 5.060 |
Jerk |
mean: 148.110 ± 62.141 median:139.333 75%: 185.350 98%: 287.974 |
Position Error (mm) |
mean: 0.009 ± 0.113 median:0.004 75%: 0.007 98%: 0.025 |
Latest Inverse Kinematics Results¶
Results obtained on April 11 2024.
Reported errors are 98th percentile. You can run this with python benchmark/ik_benchmark.py.
robot |
IK-time(ms) |
Collision-Free-IK-time(ms) |
Batch-Size |
Success-IK |
Success-Collision-Free-IK |
Position-Error(mm) |
Orientation-Error |
Position-Error-Collision-Free-IK(mm) |
Orientation-Error-Collision-Free-IK |
|
|---|---|---|---|---|---|---|---|---|---|---|
0 |
franka.yml |
0.0234922 |
0.110355 |
2000 |
99.95 |
99.7 |
0.00431221 |
6.96271e-06 |
0.00606049 |
6.98038e-06 |
1 |
ur5e.yml |
0.0210146 |
0.0525328 |
2000 |
99.85 |
99.85 |
0.00425549 |
6.81198e-06 |
0.00517626 |
7.01627e-06 |
2 |
ur10e.yml |
0.0210209 |
0.044765 |
2000 |
99.8 |
99.4 |
0.00373272 |
6.88523e-06 |
0.00616951 |
6.92548e-06 |
3 |
tm12.yml |
0.021102 |
0.0523754 |
2000 |
99.9 |
99.25 |
0.00452017 |
6.97735e-06 |
0.00859144 |
7.05593e-06 |
4 |
jaco7.yml |
0.0226465 |
0.0819137 |
2000 |
100 |
100 |
0.00271553 |
6.85656e-06 |
0.00341237 |
6.87208e-06 |
5 |
kinova_gen3.yml |
0.0227171 |
0.0601847 |
2000 |
100 |
100 |
0.00268009 |
6.91336e-06 |
0.00290017 |
6.93216e-06 |
6 |
iiwa.yml |
0.0225462 |
0.0478102 |
2000 |
100 |
100 |
0.00281161 |
6.85837e-06 |
0.00326498 |
6.89754e-06 |
Running Benchmarks¶
We use robometrics to run some of the benchmarks and also provide csv export with pandas. Install the following packages before running the benchmarks:
Install robometrics with
pip install "robometrics[evaluator] @ git+https://github.com/fishbotics/robometrics.git"Optionally, install pandas if you want csv files with
pip install pandas
Kinematics & Collision Checking¶
To measure compute time for kinematics and collision checking, run the below script which will execute for all supported robots,
python benchmark/kinematics_benchmark.py --save_path=. --file_name=kinematics
This will save the results to kinematics.yml
Inverse Kinematics¶
To measure success metrics and compute time for inverse kinematics and collision-free inverse kinematics, run
python benchmark/ik_benchmark.py --save_path=. --file_name=ik
This will save the results to ik.yml.
Motion Generation & Geometric Planning¶
To run benchmarks for motion generation:
python benchmark/curobo_benchmark.py
Pass --mesh to run benchmarks with obstacles represented as meshes.
To run benchmarks for motion generation with obstacles represented using rendered depth images in nvblox:
1. Install nvblox_torch, following instructions in Using with Depth Camera. You
would also need pyrender, install with pip install pyrender
2. Generate sensor images by rendering using: python benchmark/generate_nvblox_images.py
3. Run the benchmark with python benchmark/curobo_nvblox_benchmark.py
To evaluate the motions with the robometrics evaluator:
python benchmark/robometrics_benchmark.py
Passing --graph in the above commands will run the benchmarks with only geometric planning.
Profiling¶
For users who want to understand the computational graph and analyze compute bottlenecks, we provide scripts that run the motion generation pipeline and dump torch profiling to a json file. This file can be loaded in chrome://tracing
Before running these scripts, cd benchmark && mkdir -p log/trace,
Run python curobo_profile.py to run motion generation profiling.
Run python curobo_nvblox_profile.py to run motion generation profiling with nvblox for collision avoidance.
Run python curobo_python_profile.py to profile the code using cProfile.