Benchmarks & Profiling

Benchmarks

Results in the arxiv paper were obtained from v0.6.0.

v0.6.2+ has significant changes to improve motion quality with lower motion time, lower path length, higher pose accuracy (<1mm) while taking 15ms (RTX 4090) longer to compute solutions. The new results are yet to be added to the technical report, see below for the exact results.

Latest Motion Generation Results

Results obtained on February 20 2024.

Motion Generation on 2600 problems from motion benchmaker and motion policy networks, on a RTX 4090:

Metric

Value

Success %

99.84

Plan Time (s)

mean: 0.068 ± 0.158 median:0.042 75%: 0.055 98%: 0.246

Motion Time (s)

mean: 1.169 ± 0.360 median:1.140 75%: 1.381 98%: 2.163

Path Length (rad.)

mean: 3.177 ± 1.072 median:3.261 75%: 3.804 98%: 5.376

Jerk

mean: 97.700 ± 48.630 median:88.993 75%: 126.092 98%: 199.540

Position Error (mm)

mean: 0.119 ± 0.341 median:0.027 75%: 0.091 98%: 1.039

Motion Benchmaker (800 problems):

Metric

Value

Success %

100

Plan Time (s)

mean: 0.063 ± 0.137 median:0.042 75%: 0.044 98%: 0.206

Motion Time (s)

mean: 1.429 ± 0.330 median:1.392 75%: 1.501 98%: 2.473

Path Length (rad.)

mean: 3.956 ± 0.783 median:3.755 75%: 4.352 98%: 6.041

Jerk

mean: 67.284 ± 27.788 median:61.853 75%: 83.337 98%: 143.118

Position Error (mm)

mean: 0.079 ± 0.139 median:0.032 75%: 0.077 98%: 0.472

Motion Policy Networks (1800 problems):

Metric

Value

Success %

99.77

Plan Time (s)

mean: 0.068 ± 0.117 median:0.042 75%: 0.059 98%: 0.243

Motion Time (s)

mean: 1.051 ± 0.306 median:1.016 75%: 1.226 98%: 1.760

Path Length (rad.)

mean: 2.829 ± 1.000 median:2.837 75%: 3.482 98%: 4.905

Jerk

mean: 110.610 ± 47.586 median:105.271 75%: 141.517 98%: 217.158

Position Error (mm)

mean: 0.122 ± 0.343 median:0.024 75%: 0.095 98%: 1.114

Running Benchmarks

We use robometrics to run some of the benchmarks and also provide csv export with pandas. Install the following packages before running the benchmarks:

  1. Install robometrics with pip install "robometrics[evaluator] @ git+https://github.com/fishbotics/robometrics.git"

  2. Optionally, install pandas if you want csv files with pip install pandas

Kinematics & Collision Checking

To measure compute time for kinematics and collision checking, run the below script which will execute for all supported robots,

python benchmark/kinematics_benchmark.py --save_path=. --file_name=kinematics

This will save the results to kinematics.yml

Inverse Kinematics

To measure success metrics and compute time for inverse kinematics and collision-free inverse kinematics, run

python benchmark/ik_benchmark.py --save_path=. --file_name=ik

This will save the results to ik.yml.

Motion Generation & Geometric Planning

To run benchmarks for motion generation:

python benchmark/curobo_benchmark.py

Pass --mesh to run benchmarks with obstacles represented as meshes.

To run benchmarks for motion generation with obstacles represented using rendered depth images in nvblox:

1. Install nvblox_torch, following instructions in Using with Depth Camera. You would also need pyrender, install with pip install pyrender 2. Generate sensor images by rendering using: python benchmark/generate_nvblox_images.py 3. Run the benchmark with python benchmark/curobo_nvblox_benchmark.py

To evaluate the motions with the robometrics evaluator:

python benchmark/robometrics_benchmark.py

Passing --graph in the above commands will run the benchmarks with only geometric planning.

Profiling

For users who want to understand the computational graph and analyze compute bottlenecks, we provide scripts that run the motion generation pipeline and dump torch profiling to a json file. This file can be loaded in chrome://tracing

Before running these scripts, cd benchmark && mkdir -p log/trace,

Run python curobo_profile.py to run motion generation profiling.

Run python curobo_nvblox_profile.py to run motion generation profiling with nvblox for collision avoidance.

Run python curobo_python_profile.py to profile the code using cProfile.