Benchmarks & Profiling
Benchmarks
Results in the arxiv paper were obtained from v0.6.0.
v0.6.2 has significant changes to improve motion quality with lower motion time, lower path length, higher pose accuracy (<1mm) while taking 15ms (RTX 4090) longer to compute solutions.
v0.7 improves signed distance calculations for cuboids and meshes. Also, leverages torch.compile
for better kernel fusion. These improvements result in higher quality solutions in lower time (10ms
faster than arxiv report on average).
Latest Motion Generation Results
Results obtained on April 25 2024.
Motion Generation on 2600 problems from motion benchmaker and motion policy networks, on a RTX 4090:
Metric |
Value |
---|---|
Success % |
99.81 |
Plan Time (s) |
mean: 0.038 ± 0.035 median:0.029 75%: 0.032 98%: 0.158 |
Motion Time(s) |
mean: 1.109 ± 0.304 median:1.096 75%: 1.261 98%: 1.966 |
Path Length (rad.) |
mean: 3.151 ± 1.010 median:3.251 75%: 3.782 98%: 5.122 |
Jerk |
mean: 123.752 ± 38.177 median:126.037 75%: 146.268 98%: 191.507 |
Position Error (mm) |
mean: 0.255 ± 0.584 median:0.078 75%: 0.192 98%: 2.571 |
Motion Benchmaker (800 problems):
Metric |
Value |
---|---|
Success % |
100.00 |
Plan Time (s) |
mean: 0.042 ± 0.044 median:0.029 75%: 0.030 98%: 0.192 |
Motion Time(s) |
mean: 1.274 ± 0.273 median:1.233 75%: 1.309 98%: 2.237 |
Path Length (rad.) |
mean: 3.814 ± 0.667 median:3.698 75%: 4.229 98%: 5.402 |
Jerk |
mean: 106.040 ± 33.414 median:116.845 75%: 126.770 98%: 151.311 |
Position Error (mm) |
mean: 0.183 ± 0.345 median:0.088 75%: 0.185 98%: 1.117 |
Motion Policy Networks (1800 problems):
Metric |
Value |
---|---|
Success % |
99.72 |
Plan Time (s) |
mean: 0.037 ± 0.030 median:0.029 75%: 0.032 98%: 0.146 |
Motion Time(s) |
mean: 1.035 ± 0.288 median:0.998 75%: 1.196 98%: 1.720 |
Path Length (rad.) |
mean: 2.854 ± 0.997 median:2.863 75%: 3.505 98%: 4.974 |
Jerk |
mean: 131.689 ± 37.506 median:134.606 75%: 156.066 98%: 194.174 |
Position Error (mm) |
mean: 0.287 ± 0.661 median:0.072 75%: 0.194 98%: 2.914 |
World Represented as meshes (2600 problems), python benchmark/curobo_benchmark.py --mesh
:
Metric |
Value |
---|---|
Success % |
99.88 |
Plan Time (s) |
mean: 0.065 ± 0.046 median:0.054 75%: 0.069 98%: 0.170 |
Motion Time(s) |
mean: 1.102 ± 0.291 median:1.100 75%: 1.256 98%: 1.836 |
Path Length (rad.) |
mean: 3.136 ± 1.003 median:3.243 75%: 3.782 98%: 5.102 |
Jerk |
mean: 124.223 ± 37.793 median:126.219 75%: 147.475 98%: 190.144 |
Position Error (mm) |
mean: 0.245 ± 0.564 median:0.077 75%: 0.178 98%: 2.376 |
Latest Inverse Kinematics Results
Results obtained on April 11 2024.
Reported errors are 98th percentile. You can run this with python benchmark/ik_benchmark.py
.
robot |
IK-time(ms) |
Collision-Free-IK-time(ms) |
Batch-Size |
Success-IK |
Success-Collision-Free-IK |
Position-Error(mm) |
Orientation-Error |
Position-Error-Collision-Free-IK(mm) |
Orientation-Error-Collision-Free-IK |
|
---|---|---|---|---|---|---|---|---|---|---|
0 |
franka.yml |
0.0239108 |
0.105928 |
2000 |
100 |
99.75 |
0.00567964 |
2.03334e-05 |
0.0079604 |
2.04873e-05 |
1 |
ur5e.yml |
0.0214932 |
0.0488772 |
2000 |
99.85 |
99.65 |
0.00416377 |
2.00176e-05 |
0.00489288 |
2.03361e-05 |
2 |
ur10e.yml |
0.0215468 |
0.043004 |
2000 |
99.85 |
99.35 |
0.00431679 |
2.03305e-05 |
0.00866652 |
2.13802e-05 |
3 |
tm12.yml |
0.0215514 |
0.0503777 |
2000 |
99.9 |
99.35 |
0.00535555 |
2.04348e-05 |
0.0112364 |
3.45993e-05 |
4 |
jaco7.yml |
0.0231285 |
0.0833737 |
2000 |
100 |
100 |
0.00291714 |
2.02833e-05 |
0.00411957 |
2.04493e-05 |
5 |
kinova_gen3.yml |
0.0230068 |
0.0570858 |
2000 |
100 |
99.9 |
0.00309421 |
2.03153e-05 |
0.0034506 |
2.04602e-05 |
6 |
iiwa.yml |
0.0229251 |
0.0454897 |
2000 |
100 |
100 |
0.00305969 |
2.01236e-05 |
0.0032995 |
2.0446e-05 |
7 |
iiwa_allegro.yml |
0.0648178 |
0.208883 |
2000 |
99.95 |
99.8 |
0.00330207 |
2.0406e-05 |
0.00525135 |
2.04795e-05 |
8 |
dual_ur10e.yml |
0.0393165 |
0.094734 |
2000 |
99.9 |
98.3 |
0.0048302 |
2.03297e-05 |
0.0121227 |
5.89246e-05 |
Running Benchmarks
We use robometrics to run some of the benchmarks and also provide csv export with pandas. Install the following packages before running the benchmarks:
Install robometrics with
pip install "robometrics[evaluator] @ git+https://github.com/fishbotics/robometrics.git"
Optionally, install pandas if you want csv files with
pip install pandas
Kinematics & Collision Checking
To measure compute time for kinematics and collision checking, run the below script which will execute for all supported robots,
python benchmark/kinematics_benchmark.py --save_path=. --file_name=kinematics
This will save the results to kinematics.yml
Inverse Kinematics
To measure success metrics and compute time for inverse kinematics and collision-free inverse kinematics, run
python benchmark/ik_benchmark.py --save_path=. --file_name=ik
This will save the results to ik.yml
.
Motion Generation & Geometric Planning
To run benchmarks for motion generation:
python benchmark/curobo_benchmark.py
Pass --mesh
to run benchmarks with obstacles represented as meshes.
To run benchmarks for motion generation with obstacles represented using rendered depth images in nvblox:
1. Install nvblox_torch, following instructions in Using with Depth Camera. You
would also need pyrender, install with pip install pyrender
2. Generate sensor images by rendering using: python benchmark/generate_nvblox_images.py
3. Run the benchmark with python benchmark/curobo_nvblox_benchmark.py
To evaluate the motions with the robometrics evaluator:
python benchmark/robometrics_benchmark.py
Passing --graph
in the above commands will run the benchmarks with only geometric planning.
Profiling
For users who want to understand the computational graph and analyze compute bottlenecks, we provide scripts that run the motion generation pipeline and dump torch profiling to a json file. This file can be loaded in chrome://tracing
Before running these scripts, cd benchmark && mkdir -p log/trace
,
Run python curobo_profile.py
to run motion generation profiling.
Run python curobo_nvblox_profile.py
to run motion generation profiling with nvblox for collision avoidance.
Run python curobo_python_profile.py
to profile the code using cProfile.