TensorRT explores alternative compilation settings

NVIDIA’s October 23 explanation describes the tuner introduced in TensorRT 11.1. TensorRT compiles a neural network into an executable engine, choosing how operations are combined, implemented and placed in memory. These decisions interact: a faster individual calculation need not produce a faster complete engine. Exploring alternative compilation settings addresses that model-dependent problem, while numerical tolerances constrain which results are acceptable.

The tuning manual describes a trtexec command-line search that builds and times candidate engines. Accuracy checking is optional: supply input data, reference outputs and an error threshold, with an engine-saving destination required for this mode. Candidates exceeding the threshold are excluded from selection. The default compilation route may already be optimal. Even repeating a route need not create identical engine bytes, so retain the actual saved engine, not just its settings. NVIDIA also calls for retuning after target hardware or TensorRT/tuner version changes.

The measurement guide explains why measurement conditions matter. For discrete GPUs, record and monitor clock frequency, power and temperature alongside inference; its nvidia-smi monitoring commands do not apply to Jetson or integrated GPUs. Use inputs representative of the real data distribution. By default, trtexec excludes host-to-device and device-to-host transfer times. Its includeDataTransfers option includes them when assessing whether moving data constrains the workload.