Support data dependent shapes (no fallback)

[bowang007]

TL;DR

TensorRT 8.5 supports dynamically shaped output feature, this helps support cases that shapes depend on input data. We should integrate this new feature.

Goal(s)

Allow users to use data dependent shapes in their model. Currently models which depend on input data are producing incorrect results when they are converted to TRT engine.

Usecases

• Allow data dependent shapes in the model
• No dependency on fallback

Proposed APIs/UX

Users might need to specify that the model uses data dependent shapes since TensorRT now needs to explicitly use IExecutionContext::getTensorShape() for dimensions of the output if they are computable from the dimensions of inputs, or even use IOutputAllocator otherwise.

It's also possible that TorchTRT introduces some kind of mechanism which could detect if the output shape is dependent on input data automatically. In this case, there won't be any APIs/UX changes.

Internal Implementation

Design

From the first glance it seems that we need to use IExecutionContext::getTensorShape() or IOutputAllocator for different scenarios. Specifically, these APIs might need to be explicitly used when constructing the TensorRT network. However, I didn't find any examples about how these APIs can actually be used in TensorRT.

[narendasan]

@bowang007 Can you add estimates of effort and phases to this? See this: https://github.com/pytorch/TensorRT/wiki/RFC-Template

[ncomly-nvidia]

@bowang007 is this for TS, FX or both?

[lix19937]

data-dependent output shapes differs from the static or dynamic shape cases.

[lix19937]

https://forums.developer.nvidia.com/t/data-dependent-tensor-shapes-in-tensorrt/194988