Benchmarks & Profiling

Benchmarks

Results in the arxiv paper were obtained from v0.6.0.

v0.6.2 has significant changes to improve motion quality with lower motion time, lower path length, higher pose accuracy (<1mm) while taking 15ms (RTX 4090) longer to compute solutions.

v0.7 improves signed distance calculations for cuboids and meshes. Also, leverages torch.compile for better kernel fusion. These improvements result in higher quality solutions in lower time (10ms faster than arxiv report on average).

Latest Motion Generation Results

Results obtained on April 25 2024.

Motion Generation on 2600 problems from motion benchmaker and motion policy networks, on a RTX 4090:

Metric

Value

Success %

99.81

Plan Time (s)

mean: 0.038 ± 0.035 median:0.029 75%: 0.032 98%: 0.158

Motion Time(s)

mean: 1.109 ± 0.304 median:1.096 75%: 1.261 98%: 1.966

Path Length (rad.)

mean: 3.151 ± 1.010 median:3.251 75%: 3.782 98%: 5.122

Jerk

mean: 123.752 ± 38.177 median:126.037 75%: 146.268 98%: 191.507

Position Error (mm)

mean: 0.255 ± 0.584 median:0.078 75%: 0.192 98%: 2.571

Motion Benchmaker (800 problems):

Metric

Value

Success %

100.00

Plan Time (s)

mean: 0.042 ± 0.044 median:0.029 75%: 0.030 98%: 0.192

Motion Time(s)

mean: 1.274 ± 0.273 median:1.233 75%: 1.309 98%: 2.237

Path Length (rad.)

mean: 3.814 ± 0.667 median:3.698 75%: 4.229 98%: 5.402

Jerk

mean: 106.040 ± 33.414 median:116.845 75%: 126.770 98%: 151.311

Position Error (mm)

mean: 0.183 ± 0.345 median:0.088 75%: 0.185 98%: 1.117

Motion Policy Networks (1800 problems):

Metric

Value

Success %

99.72

Plan Time (s)

mean: 0.037 ± 0.030 median:0.029 75%: 0.032 98%: 0.146

Motion Time(s)

mean: 1.035 ± 0.288 median:0.998 75%: 1.196 98%: 1.720

Path Length (rad.)

mean: 2.854 ± 0.997 median:2.863 75%: 3.505 98%: 4.974

Jerk

mean: 131.689 ± 37.506 median:134.606 75%: 156.066 98%: 194.174

Position Error (mm)

mean: 0.287 ± 0.661 median:0.072 75%: 0.194 98%: 2.914

World Represented as meshes (2600 problems), python benchmark/curobo_benchmark.py --mesh:

Metric

Value

Success %

99.88

Plan Time (s)

mean: 0.065 ± 0.046 median:0.054 75%: 0.069 98%: 0.170

Motion Time(s)

mean: 1.102 ± 0.291 median:1.100 75%: 1.256 98%: 1.836

Path Length (rad.)

mean: 3.136 ± 1.003 median:3.243 75%: 3.782 98%: 5.102

Jerk

mean: 124.223 ± 37.793 median:126.219 75%: 147.475 98%: 190.144

Position Error (mm)

mean: 0.245 ± 0.564 median:0.077 75%: 0.178 98%: 2.376

Latest Inverse Kinematics Results

Results obtained on April 11 2024.

Reported errors are 98th percentile. You can run this with python benchmark/ik_benchmark.py.

robot

IK-time(ms)

Collision-Free-IK-time(ms)

Batch-Size

Success-IK

Success-Collision-Free-IK

Position-Error(mm)

Orientation-Error

Position-Error-Collision-Free-IK(mm)

Orientation-Error-Collision-Free-IK

0

franka.yml

0.0239108

0.105928

2000

100

99.75

0.00567964

2.03334e-05

0.0079604

2.04873e-05

1

ur5e.yml

0.0214932

0.0488772

2000

99.85

99.65

0.00416377

2.00176e-05

0.00489288

2.03361e-05

2

ur10e.yml

0.0215468

0.043004

2000

99.85

99.35

0.00431679

2.03305e-05

0.00866652

2.13802e-05

3

tm12.yml

0.0215514

0.0503777

2000

99.9

99.35

0.00535555

2.04348e-05

0.0112364

3.45993e-05

4

jaco7.yml

0.0231285

0.0833737

2000

100

100

0.00291714

2.02833e-05

0.00411957

2.04493e-05

5

kinova_gen3.yml

0.0230068

0.0570858

2000

100

99.9

0.00309421

2.03153e-05

0.0034506

2.04602e-05

6

iiwa.yml

0.0229251

0.0454897

2000

100

100

0.00305969

2.01236e-05

0.0032995

2.0446e-05

7

iiwa_allegro.yml

0.0648178

0.208883

2000

99.95

99.8

0.00330207

2.0406e-05

0.00525135

2.04795e-05

8

dual_ur10e.yml

0.0393165

0.094734

2000

99.9

98.3

0.0048302

2.03297e-05

0.0121227

5.89246e-05

Running Benchmarks

We use robometrics to run some of the benchmarks and also provide csv export with pandas. Install the following packages before running the benchmarks:

  1. Install robometrics with pip install "robometrics[evaluator] @ git+https://github.com/fishbotics/robometrics.git"

  2. Optionally, install pandas if you want csv files with pip install pandas

Kinematics & Collision Checking

To measure compute time for kinematics and collision checking, run the below script which will execute for all supported robots,

python benchmark/kinematics_benchmark.py --save_path=. --file_name=kinematics

This will save the results to kinematics.yml

Inverse Kinematics

To measure success metrics and compute time for inverse kinematics and collision-free inverse kinematics, run

python benchmark/ik_benchmark.py --save_path=. --file_name=ik

This will save the results to ik.yml.

Motion Generation & Geometric Planning

To run benchmarks for motion generation:

python benchmark/curobo_benchmark.py

Pass --mesh to run benchmarks with obstacles represented as meshes.

To run benchmarks for motion generation with obstacles represented using rendered depth images in nvblox:

1. Install nvblox_torch, following instructions in Using with Depth Camera. You would also need pyrender, install with pip install pyrender 2. Generate sensor images by rendering using: python benchmark/generate_nvblox_images.py 3. Run the benchmark with python benchmark/curobo_nvblox_benchmark.py

To evaluate the motions with the robometrics evaluator:

python benchmark/robometrics_benchmark.py

Passing --graph in the above commands will run the benchmarks with only geometric planning.

Profiling

For users who want to understand the computational graph and analyze compute bottlenecks, we provide scripts that run the motion generation pipeline and dump torch profiling to a json file. This file can be loaded in chrome://tracing

Before running these scripts, cd benchmark && mkdir -p log/trace,

Run python curobo_profile.py to run motion generation profiling.

Run python curobo_nvblox_profile.py to run motion generation profiling with nvblox for collision avoidance.

Run python curobo_python_profile.py to profile the code using cProfile.