nv2a: Handle NaN values to be similar to HW

[antoniodesousa]

This PR is based on @abaire work at #913. So big thanks to him for his research 🙏

Due to his implementation not working on my PC, I decided to do my own research and hopefully came up with a solution that works on any hardware. The solution was tested on 2 PCs: one with an AMD Radeon RX 6600M GPU, and the other with a NVIDIA GeForce GTX 960 GPU. Both running on Windows 11 23H2, 64 GB RAM with the latest drivers installed as of today.

HW test on the left side - PC test on the right side

Otogi: Myth of Demons

Otogi 2: Immortal Warriors

Pro Cast: Sports Fishing Game

Trigger Man

Thousand Land

[antangelo]

I ran the pgraph tests on my Linux workstation with a 3060 Ti and the NaN columns are black across all tests.

[antoniodesousa]

After giving it some thought and testing I concluded that HW handles -NaN/+NaN the same. It just defaults to 1.0. I changed the implementation and retest everything again based on this assumption. I got the same results as before, but I was already expecting that. I just wanted to double check.

@antangelo can you please retest the build on your PC? Thanks.

[medievil1]

works on nvidia too (windows 11, my own build of the PR)

[Ernegien]

I agree with the .gitignore additions but they should probably be included in a separate PR.

[antoniodesousa]

I agree with the .gitignore additions but they should probably be included in a separate PR.

Yeah, I was thinking about it. I gonna wait for @mborgerson to review this PR and create another PR for it if needed.

[Ernegien]

For those following along willing to test and share their results here, you must first be signed into GitHub and can then download the xemu build for your specific system. For example, Windows users can directly use this.

You'll also need the pgragh test ISO to run within xemu. After you've loaded the pgraph test ISO within xemu it will auto-run (it may assert/crash which is likely fine) all tests. The results are stored on your Xbox HDD at E:\nxdk_pgraph_tests under the test name and case. You can retrieve these via FTP by running the LithiumX dashboard ISO.

Alternatively, you can easily manually run the ATTRIB FLOAT tests and screenshot those that differ from previous user reports. You can do this within xemu via F12; they're stored in the directory specified underneath xemu->Machine->Settings->General->Screenshot output directory.

When reporting test results within xemu, be sure to grab a copy of your build and system info via Help->About which should look similar to the following (my specific hardware config for the tests below).

Version:      0.7.132-3-g1caaf64c84
Branch:       
Commit:       1caaf64c8466b6e8f777f84c28edeb7dbcad6e11
Date:         Sun Oct  6 20:10:34 UTC 2024

CPU:          AMD Ryzen 7 4800H with Radeon Graphics         
OS Platform:  Windows
OS Version:    23H2
Manufacturer: NVIDIA Corporation
GPU Model:    NVIDIA GeForce RTX 2060/PCIe/SSE2
Driver:       4.0.0 NVIDIA 551.23
Shader:       4.00 NVIDIA via Cg compiler

My xemu test results (internal resolution scale set to 1) which do not match real hardware are as follows:

[antoniodesousa]

That's interesting, I wonder why in your PC is black @Ernegien and for @medievil1 seems to be working fine. Considering you both are on Windows 11 and using nvidia GPUs.

[medievil1]

mine do not match hardware either, but otagi definitely does work with this pr

[mborgerson]

Moved to draft state because although it may fix issues in some games, I'm not convinced yet that this is ready for merge.

I'd like to know specifically, for starters:

• What input/computation results in NaN values in games and why (e.g. shader code and actual values please)
• Are NaNs actually coming from attribute inputs or are they the result of an undefined operation, or indicative of faulty emulation elsewhere (e.g FPU), etc
• Which of these unit test results differ among different hardware vendors and why

[NZJenkins]

After giving it some thought and testing I concluded that HW handles -NaN/+NaN the same. It just defaults to 1.0.

Based on the tests HW maps -NaN to 0, +NaN to 1.
Note if you multiply -NaN by 1 in the VS, it ends up as 1 (instead of 0).
The first column in the test IIRC is the raw value passed into the vertex shader, converted to a pixel colour.

I am not sure GLSL provides any way to distinguish between +NaN and -NaN.
I found this in the GLSL 4.6 spec (no idea if xemu uses that version) which suggests built-in functions can return anything they want when given a NaN?

built-in functions that operate on a NaN are not required to return a NaN as the result.
However if NaNs are generated, isnan() must return the correct value.

[antoniodesousa]

Based on the tests HW maps -NaN to 0, +NaN to 1.

I don't know what HW tests are you referring to. My HW tests clearly shows the same result for both values, whether the value is -NaN or +NaN doesn't matter. If what you are saying was true, they wouldn't match in most of the tests.

I am not sure GLSL provides any way to distinguish between +NaN and -NaN.
I found this in the GLSL 4.6 spec (no idea if xemu uses that version) which suggests built-in functions can return anything they want when given a NaN?

Xemu uses GLSL 4.0. And yes, there's a way to distinguish between +NaN and -NaN using the sign function.

[NZJenkins]

I don't know what HW tests are you referring to. My HW tests clearly shows the same result for both values.

I'm talking about the pgraph test, running on Xbox hardware.
I didn't realise you were talking about testing on your HW. Nevermind then.

And yes, there's a way to distinguish between +NaN and -NaN using the sign function.

I don't think this is part of the GLSL spec.
The "correct" result of sign(-NaN) would be NaN, I suppose.

Ideally there is a way to get the correct behaviour programming to the spec instead of relying on GPU quirks and vendor specific behaviour.
As far as I can see, sign(floatBitsToInt(x)) should allow getting the sign bit reliably?

[medievil1]

Based on the tests HW maps -NaN to 0, +NaN to 1. Note if you multiply -NaN by 1 in the VS, it ends up as 1 (instead of 0).

as I understand it, the issue seems to be that PC gpu hardware don't behave the same...for example on AMD it matches hardware

but on nvidia it does not

I need to look at the tests source and see what it is sending to see if I can help identify the solution

[medievil1]

I wonder about this in the pgraph tests code
static float f(uint32_t v) { return *((float *)&v); }
and
"-NaNq_NaNq", "-NaN to +NaN (quiet)", {f(negNanQ), f(posNanQ)}},
as example...
NaN is already float by nature