Maya-to-Hydra crash with maya-2019 + usd-20.08/20.11 (fixed in 21.02)

[pmolodo]

It seems that mtoh in maya-2019, with a certain range of USD versions, will crash.

Specifically, it seems usd-20.08 and 20.11 crash with maya-2019. The latest tip on the USD dev branch, however, works fine (proto 21.02 - PixarAnimationStudios/OpenUSD@e9172af).

I tested using what's essentially the latest tip of maya_usd's dev branch - e7ae8b8, which is nearly identical to the current tip, 1c51229.

Mtoh always seemed to work with maya-2020, regardless of usd-version. Specifically, I tried these combinations:

maya-2019.2 + usd-20.05 - WORKS
maya-2019.2 + usd-20.08 - FAILS
maya-2019.2 + usd-20.11 - FAILS
maya-2019.2 + usd-21.02 - WORKS
maya-2020.0 + usd-20.08 - WORKS
maya-2020.0 + usd-20.11 - WORKS
maya-2020.3 + usd-20.11 - WORKS

All my tests were run on a Linux workstation - I haven't check behavior on other OSes.

In some situations, it fails a TF_VERIFY in hdSt/resourceRegistry.cpp before crashing:

Warning: in _Commit at line 791 of USD_SRC/pxr/imaging/hdSt/resourceRegistry.cpp -- Too many iterations in resolving buffer source. It's likely due to incosistent dependency.
Coding Error: in _Commit at line 796 of USD_SRC/pxr/imaging/hdSt/resourceRegistry.cpp -- Failed verification: ' numBufferSourcesResolved == _numBufferSourcesToResolve '

I'm not sure what's going wrong here, but given that EITHER using maya-2020+ OR using the latest USD will fix it, I'm not sure I want to investigate. I'm thinking of just force disabling BUILD_HDMAYA for the affected versions, and moving on.

Thoughts?

[mattyjams]

Yeah, I don't really have a good guess as to what's going on either. One wild shot in the dark is that the Hydra GLEW-ectomy is completed in 21.02, but up until then, we had been LD_PRELOAD-ing our own version of GLEW to avoid using Maya's super-old one. I'd be kind of surprised if that's related though, as I'd expect it to have crashed much earlier with basic GL symbols being invalid.

Since we've moved past Maya 2019 at this point, no objection on this end to simply disabling BUILD_HDMAYA for that Maya version.

[pmolodo]

Well, since we're still supposed to support 2019, and I had a very reproducible test case, I decided to set up a git bisect run to find which usd-commit "fixes" it. I'm going to let it run overnight (since it has to do a usd + maya_usd build for each checkout), so hopefully I can report something in the morning...

[pmolodo]

Hmm... well, wasn't totally automated, as I also needed to apply reverts to maya_usd to make sure the API lined up with exact commits... but did eventually figure out the exact USD commit that fixes the crash:

PixarAnimationStudios/OpenUSD@5db1734
[Pxr] Replaced the use of GLEW with garch/glApi

So... @mattyjams, good guess that it was GLEW-related! This also provides a good explanation for why it's fixed in 2020 - it has a different version of libglew.so. (I don't know the exact versions, but it went from ~600K in size to ~800K - so definitely different!)

Tried your LD_PRELOAD trick... and sure enough, that fixed it too. Should have listened to you, and tried that straight off!

There are a few remaining questions - such, as why did it work with usd<=20.05... but I don't think we need to answer that.

What we do need to answer is: do we disable building altogether, or just the tests?

I think the ideal would be to disable building it by default for 2019 + < (20.08|20.11), but if the user has explicitly requested BUILD_HDMAYA=ON, allow it.

[mattyjams]

Ha! Lucky guess. You were wise to be skeptical. :)

I may have a slightly more educated one for why 20.08 is problematic. I went spelunking through some of our internal issues, as I remember we had a version of a test that we ran with and without the LD_PRELOAD to help track whether we still supported Maya 2019's old GLEW. I think we finally ended up adding OpenGL 4.5+ API (glCreateBuffers() for one, if I'm remembering correctly) around June-ish with PixarAnimationStudios/OpenUSD@ed0a4e1, as that's where we started seeing failures with the no-LD_PRELOAD flavor of the test. With the removal of GLEW well underway at that point, we ended up just disabling the no-LD_PRELOAD test. That core USD commit made it out as part of the 20.08 release, so that might explain why you're seeing the breakage start there.

Here's a hacky attempt at figuring out the version change, so I'm not sure this is exactly accurate, but it looks like Maya 2019 (Update 2 for us) was GLEW 1.1.0 while Maya 2020 Update 3 is 2.1.0:

> strings <maya 2019 installation>/lib/libglew.so | grep '[0-9]\.'
...
1.10.0
...
> strings <maya 2020 installation>/lib/libglew.so | grep '[0-9]\.'
...
2.1.0
...

I think your suggestion of disabling the build for the problematic combinations is a good one, if that's not too onerous to setup in CMake. I guess I'd go with disabling altogether versus only disabling the tests, since the combination of recent OpenGL API and the old GLEW has proven pretty unstable.

[pmolodo]

Not hard to do, just a question of whether it will be... unsightly...

[pmolodo]

#1023

[pmolodo]

#1023