Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Improvements to HEIC and AVIF decoders #8810

Open
NicolasHug opened this issue Dec 13, 2024 · 0 comments
Open

Improvements to HEIC and AVIF decoders #8810

NicolasHug opened this issue Dec 13, 2024 · 0 comments

Comments

@NicolasHug
Copy link
Member

HEIC and AVIF decoders are now publicly available. Some improvements we can make are:

  • Better torchscript support (meh...)
  • support for decode_image()
  • Robust file format detection logic (espcially for HEIC).

More of that is documented here:

# TODO_AVIF_HEIC: Better support for torchscript. Scripting decode_avif of
# decode_heic currently fails, mainly because of the logic
# _load_extra_decoders_once() (using global variables, try/except statements,
# etc.).
# The ops (torch.ops.extra_decoders_ns.decode_*) are otherwise torchscript-able,
# and users who need torchscript can always just wrap those.
# TODO_AVIF_HEIC: decode_image() should work for those. The key technical issue
# we have here is that the format detection logic of decode_image() is
# implemented in torchvision, and torchvision has zero knowledge of
# torchvision-extra-decoders, so we cannot call the AVIF/HEIC C++ decoders
# (those in torchvision-extra-decoders) from there.
# A trivial check that could be done within torchvision would be to check the
# file extension, if a path was passed. We could also just implement the
# AVIF/HEIC detection logic in Python as a fallback, if the file detection
# didn't find any format. In any case: properly determining whether a file is
# HEIC is far from trivial, and relying on libmagic would probably be best

Torchscript support isnt' that important especially consider the op themselves support torchscript. Support in decode_image() is more important. Some update since the comment was written: there might be a way to call the C++ AVIF/HEIC decoders from within the C++ implementation of decode_image: simply calling the dispatcher like

  auto dispatcher = torch::Dispatcher::singleton();
  dispatcher.call("extra_decoders_ns::decode_heic", ...);

Need to investigate this.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant