Benchmarks & Profiling¶
Benchmarks¶
Results in the arxiv paper were obtained from v0.6.0.
v0.6.2 has significant changes to improve motion quality with lower motion time, lower path length, higher pose accuracy (<1mm) while taking 15ms (RTX 4090) longer to compute solutions.
v0.7 improves signed distance calculations for cuboids and meshes. Also, leverages torch.compile
for better kernel fusion. These improvements result in higher quality solutions in lower time (10ms
faster than arxiv report on average).
Latest Motion Generation Results¶
Results obtained on July 19 2024.
Motion Generation on 2600 problems from motion benchmaker and motion policy networks, on a RTX 4090:
Metric |
Value |
|---|---|
Success % |
99.81 |
Plan Time (s) |
mean: 0.036 ± 0.028 median:0.030 75%: 0.033 98%: 0.095 |
Motion Time(s) |
mean: 1.184 ± 0.357 median:1.174 75%: 1.367 98%: 2.165 |
Path Length (rad.) |
mean: 3.157 ± 1.030 median:3.261 75%: 3.821 98%: 5.155 |
Jerk |
mean: 94.885 ± 39.603 median:91.513 75%: 122.434 98%: 176.207 |
Position Error (mm) |
mean: 0.122 ± 0.344 median:0.045 75%: 0.101 98%: 0.811 |
Motion Benchmaker (800 problems):
Metric |
Value |
|---|---|
Success % |
100.00 |
Plan Time (s) |
mean: 0.039 ± 0.037 median:0.030 75%: 0.032 98%: 0.131 |
Motion Time(s) |
mean: 1.384 ± 0.318 median:1.331 75%: 1.436 98%: 2.441 |
Path Length (rad.) |
mean: 3.845 ± 0.684 median:3.703 75%: 4.250 98%: 5.473 |
Jerk |
mean: 73.223 ± 29.048 median:73.171 75%: 93.198 98%: 134.325 |
Position Error (mm) |
mean: 0.100 ± 0.236 median:0.055 75%: 0.108 98%: 0.441 |
Motion Policy Networks (1800 problems):
Metric |
Value |
|---|---|
Success % |
99.72 |
Plan Time (s) |
mean: 0.035 ± 0.023 median:0.030 75%: 0.033 98%: 0.086 |
Motion Time(s) |
mean: 1.094 ± 0.337 median:1.048 75%: 1.296 98%: 1.957 |
Path Length (rad.) |
mean: 2.849 ± 1.009 median:2.846 75%: 3.542 98%: 4.886 |
Jerk |
mean: 104.592 ± 39.858 median:103.096 75%: 134.223 98%: 178.655 |
Position Error (mm) |
mean: 0.132 ± 0.383 median:0.041 75%: 0.098 98%: 0.959 |
World Represented as meshes (2600 problems), python benchmark/curobo_benchmark.py --mesh:
Metric |
Value |
|---|---|
Success % |
99.88 |
Plan Time (s) |
mean: 0.066 ± 0.051 median:0.055 75%: 0.069 98%: 0.171 |
Motion Time(s) |
mean: 1.177 ± 0.349 median:1.170 75%: 1.366 98%: 2.080 |
Path Length (rad.) |
mean: 3.139 ± 1.024 median:3.252 75%: 3.772 98%: 5.110 |
Jerk |
mean: 95.720 ± 39.269 median:92.427 75%: 122.649 98%: 176.632 |
Position Error (mm) |
mean: 0.107 ± 0.294 median:0.043 75%: 0.093 98%: 0.634 |
Latest Inverse Kinematics Results¶
Results obtained on April 11 2024.
Reported errors are 98th percentile. You can run this with python benchmark/ik_benchmark.py.
robot |
IK-time(ms) |
Collision-Free-IK-time(ms) |
Batch-Size |
Success-IK |
Success-Collision-Free-IK |
Position-Error(mm) |
Orientation-Error |
Position-Error-Collision-Free-IK(mm) |
Orientation-Error-Collision-Free-IK |
|
|---|---|---|---|---|---|---|---|---|---|---|
0 |
franka.yml |
0.0242625 |
0.106412 |
2000 |
100 |
99.8 |
0.00473177 |
2.03276e-05 |
0.00717625 |
2.0525e-05 |
1 |
ur5e.yml |
0.0218673 |
0.0498319 |
2000 |
99.85 |
99.8 |
0.00440966 |
2.00112e-05 |
0.00601756 |
2.03599e-05 |
2 |
ur10e.yml |
0.0214335 |
0.0427507 |
2000 |
99.85 |
99.45 |
0.00433393 |
2.03003e-05 |
0.00817259 |
2.05415e-05 |
3 |
tm12.yml |
0.0212737 |
0.0502136 |
2000 |
99.9 |
99.6 |
0.00535642 |
2.04375e-05 |
0.00945224 |
2.90698e-05 |
4 |
jaco7.yml |
0.0226934 |
0.0791062 |
2000 |
100 |
100 |
0.00285595 |
2.02789e-05 |
0.00367381 |
2.04645e-05 |
5 |
kinova_gen3.yml |
0.0228323 |
0.0568765 |
2000 |
100 |
100 |
0.0028806 |
2.02274e-05 |
0.00339444 |
2.05075e-05 |
6 |
iiwa.yml |
0.0226858 |
0.0452908 |
2000 |
100 |
100 |
0.00309907 |
2.01237e-05 |
0.00347772 |
2.03777e-05 |
Running Benchmarks¶
We use robometrics to run some of the benchmarks and also provide csv export with pandas. Install the following packages before running the benchmarks:
Install robometrics with
pip install "robometrics[evaluator] @ git+https://github.com/fishbotics/robometrics.git"Optionally, install pandas if you want csv files with
pip install pandas
Kinematics & Collision Checking¶
To measure compute time for kinematics and collision checking, run the below script which will execute for all supported robots,
python benchmark/kinematics_benchmark.py --save_path=. --file_name=kinematics
This will save the results to kinematics.yml
Inverse Kinematics¶
To measure success metrics and compute time for inverse kinematics and collision-free inverse kinematics, run
python benchmark/ik_benchmark.py --save_path=. --file_name=ik
This will save the results to ik.yml.
Motion Generation & Geometric Planning¶
To run benchmarks for motion generation:
python benchmark/curobo_benchmark.py
Pass --mesh to run benchmarks with obstacles represented as meshes.
To run benchmarks for motion generation with obstacles represented using rendered depth images in nvblox:
1. Install nvblox_torch, following instructions in Using with Depth Camera. You
would also need pyrender, install with pip install pyrender
2. Generate sensor images by rendering using: python benchmark/generate_nvblox_images.py
3. Run the benchmark with python benchmark/curobo_nvblox_benchmark.py
To evaluate the motions with the robometrics evaluator:
python benchmark/robometrics_benchmark.py
Passing --graph in the above commands will run the benchmarks with only geometric planning.
Profiling¶
For users who want to understand the computational graph and analyze compute bottlenecks, we provide scripts that run the motion generation pipeline and dump torch profiling to a json file. This file can be loaded in chrome://tracing
Before running these scripts, cd benchmark && mkdir -p log/trace,
Run python curobo_profile.py to run motion generation profiling.
Run python curobo_nvblox_profile.py to run motion generation profiling with nvblox for collision avoidance.
Run python curobo_python_profile.py to profile the code using cProfile.