nv2a: Adjust NaN handling to be similar to HW #913

abaire · 2022-05-11T03:01:24Z

Fixes #365 by forcing treatment of NaN values generated in the vertex shader to follow the same pattern as HW.

Unfortunately this does not address the fact that on HW, +/- NaN * 0 (or very close to 0) should be 0. This difference in behavior is also responsible for #1008.

This also does not address the same issue for +/- inf and the shader version currently used by xemu explicitly notes that operations on inf/NaN's are undefined (which may explain why Otogi textures are not blacked out on M1 mac).

Tests
HW results

abaire · 2022-05-11T03:06:12Z

Hey @NZJenkins I wasn't sure if you were already looking at a fix, but I started to suspect this might be partly to blame for #801 so I threw together a partial fix to unblock my debugging there.

This is mostly a proof of concept at the moment as it doesn't fully match the HW behavior and because some investigation will need to be done on the performance impact (and possibly some optimization)

abaire · 2022-05-11T03:22:11Z

hw/xbox/nv2a/shaders.c

+                   "    return src;\n"
+                   "  }\n"
+                   "  ivec4 signs = floatBitsToInt(src);\n"
+                   "  vec4 dest;\n"


This stuff can be vectorized, assuming the logic will stay the same:

" return mix(src, mix(vec4(1.0), vec4(0.0), lessThan(signs, ivec4(0))), nans);\n"

ko81e24wy · 2022-05-11T03:55:42Z

glitch in NGB this PR
may relate to issue #783

same glitch in DOA2U

abaire · 2022-05-11T04:05:11Z

@ko81e24wy where is the nearest save point so I can reproduce that?

ko81e24wy · 2022-05-11T04:06:50Z

@ko81e24wy where is the nearest save point so I can reproduce that?

I think you just need to start a new game

NZJenkins · 2022-05-11T07:30:04Z

Hey @NZJenkins I wasn't sure if you were already looking at a fix, but I started to suspect this might be partly to blame for #801 so I threw together a partial fix to unblock my debugging there.

No not looking at a fix.
Did something similar for cxbxr where all outputs are de-NaN-ified, but probably need more tests to determine the correct behaviour.

For texcoord and fog outputs, additional processing happens on the VS output values before they are used in the PS, which may interpret NaN, INF, etc. differently to the mapping that happens when they are passed directly from the VS to the PS.
Not sure how Xemu shaders are set up but that might need to be accounted for here.

Unfortunately this does not address the fact that on HW, +/- NaN * 0 (or very close to 0) should be 0.

The Min test is using "subnormal" float values- it might be interesting to try it with the minimum "normal" values instead, and see if they behave differently to 0.
Apparently it's more complex to support subnormal numbers correctly in hardware, so for a GPU it's likely not implemented
(added test cases - subnormals appear to be treated as 0 abaire/nxdk_pgraph_tests#65)

abaire · 2022-05-16T23:32:01Z

@ko81e24wy where is the nearest save point so I can reproduce that?

I think you just need to start a new game

There was a stupid mistake in my code that is now fixed, can you test again when you have a chance?

ko81e24wy · 2022-05-16T23:47:43Z

I am just going to work and test with a combine version with this PR merged
that white glitch was gone,and I dont know if this PR relate to #783

abaire · 2022-05-17T00:05:01Z

@ko81e24wy cool, thanks for double checking!

I have not tried to debug 783 yet, so if this PR happens to fix the missing lighting that'd be a nice bonus. I'm not exactly sure what that screenshot should look like compared to HW though; are you saying it looks more correct with this PR than before?

ko81e24wy · 2022-05-17T00:59:16Z

@ko81e24wy cool, thanks for double checking!

I have not tried to debug 783 yet, so if this PR happens to fix the missing lighting that'd be a nice bonus. I'm not exactly sure what that screenshot should look like compared to HW though; are you saying it looks more correct with this PR than before?

that white issue remind me of that light effect,the last PR just as before with that effect missing.

mysteria25 · 2022-05-17T22:27:35Z

Was hoping this would fix the black graphics of Inside Pitch 2003 but guess that is a separate issue from this pr since that game still looks like this:

abaire · 2022-05-17T23:58:17Z

Was hoping this would fix the black graphics of Inside Pitch 2003 but guess that is a separate issue from this pr since that game still looks like this:

@mysteria25 as far as I can see from the compatibility page there is no bug tracking this problem, please file one and it'll get looked into eventually (PRs are definitely not the place to report untracked issues). Grabbed by the Ghoulies has a black texture problem related to texture compression; there are many ways for things to fail so it's entirely possible what you're seeing with that game is something novel that will need a specific fix.

abaire · 2022-05-23T03:36:27Z

This doesn't entirely match HW behavior on my GTX 1070 but it is much closer and fixes #365 despite the remaining departure around infinite values and zeros.

mborgerson · 2022-05-25T10:17:26Z

hw/xbox/nv2a/shaders.c

+                   "    return src;\n"
+                   "  }\n"
+                   "  ivec4 signs = floatBitsToInt(src);\n"
+                   "  return mix(src, mix(vec4(1.0), vec4(0.0), lessThan(signs, ivec4(0))), nans);\n"


The sign bit vectorization is neat, nice :)

So "negative NaN" (msb set) gets 0 and "positive NaN" gets 1.0?

Seems odd because you mentioned 0*NaN should == 0? ~~So under what conditions do you expect it to be 1?~~(nvm, I see your comment about it) I don't think this it makes sense to generalize this based on sign bit alone here. More about this thought below.

It looks like mix(genBType) was added in GL4.5 so this might fail in a 4.0 context. Does it work on your M1 Mac? You could cast the boolean to a float

As far as I can see in the general case, I think this NaN handling is not more correct than just using 0.0 in the event we do get a NaN (which would force all platforms to behave the same at least, but might be unnecessary if we don't have issues cataloged for it). Is there another case in vsh output that this solves for, beyond fog?

To fix this fog issue for Otogi (and now at least 1 other game (Triggerman) that is affected in the same way that I've seen), I do think it's a fair compromise to have specialized fog NaN handling (like we did with Inf previously); and if this is only intended to fix the fog issue, I think we should just stick with the specialized handling for that case.

The fishing game is a different, interesting case that appears to be setting light colors to NaN, so for this second specialized case I think it makes sense to interpret NaN as 1.0 before performing lighting calculations, so lighting is calculated correctly. Perhaps it makes sense to generalize NaNs in the uniform spaces to 1.0 for such cases but I'd have to think more about it. Needs more investigation to understand how hardware is handling that case.

Partial response:

As far as I can see in the general case, I think this NaN handling is not more correct than just using 0.0 in the event we do get a NaN (which would force all platforms to behave the same at least, but might be unnecessary if we don't have issues cataloged for it). Is there another case in vsh output that this solves for, beyond fog?

The tester itself is using the diffuse channel and the xemu output does not match HW, if that's what you're asking. I don't know of any game where this makes a difference offhand, but that doesn't mean there isn't one with graphics glitching attributable to this handling.

An example difference with all three cases in one place:

master:

This branch:

HW:

To fix this fog issue for Otogi (and now at least 1 other game (Triggerman) that is affected in the same way that I've seen), I do think it's a fair compromise to have specialized fog NaN handling (like we did with Inf previously); and if this is only intended to fix the fog issue, I think we should just stick with the specialized handling for that case.

Otogi's fog is the only case I know of (I assume someone has already looked into Triggerman and determined that just applying the fix to the fog fixes that game? I don't own that one to test). I guess the issue is balancing the performance impact of doing this check against everything in an attempt to be closer to HW behavior versus the time a contributor might end up spending tracing some glitch back to a FIXME in the shader code.

The fishing game is a different, interesting case that appears to be setting light colors to NaN, so for this second specialized case I think it makes sense to interpret NaN as 1.0 before performing lighting calculations, so lighting is calculated correctly. Perhaps it makes sense to generalize NaNs in the uniform spaces to 1.0 for such cases but I'd have to think more about it. Needs more investigation to understand how hardware is handling that case.

Hopefully @Triticum0 wouldn't mind splitting Lakemasters into a new issue given that it sounds like it's a similar symptom due to a different bug?

Verified that the shader works on macOS 12.4 (M1)

@abaire Lakemaster is fixed with this PR

#1008 is due to a similar cause, INF * 0 should == 0 but is instead NaN (at least on my machine and presumably other nvidia systems since the original report was on a 3060).

In this case the incorrect number causes a chain of errors and catching it at the very end where the outputs are assigned would also lead to incorrect behavior; in the test cases I examined for #1008 the output diffuse should be set to values between 0 and 1 so blindly detecting the NaN and setting to either would do the wrong thing.

Fabxx · 2022-06-02T22:01:44Z

current handling causes this bug where the character is highlighted in proximity of lights

abaire · 2022-06-03T21:12:51Z

current handling causes this bug where the character is highlighted in proximity of lights

Looks like limiting the NaN fix to fog will fix that regression and align with Matt's suggestion, will update and re-push in a bit.

Fabxx · 2022-06-09T14:32:18Z

The highlight problem seems to be fixed with #1045

ko81e24wy · 2022-06-29T10:40:09Z

I test the lastest build of this PR,and it crash when I run xemu.
xemu.log

abaire · 2022-06-29T13:33:46Z

I test the lastest build of this PR,and it crash when I run xemu. xemu.log

Bad merge, should be fixed now. Thanks for letting me know!

Triticum0 · 2022-07-23T17:45:46Z

@abaire could you rebase this pr with the work you did on Match inv_w qualifier to attribute qualifier, Thanks

Kohryujin · 2023-01-02T21:56:23Z

Can we get another pull request or rebase or however that works so there is a working download available for this branch for testing? The one from July is expired and no longer available.

TheHollows · 2024-03-04T22:51:29Z

Any update on when this will be merged?

mborgerson · 2024-03-04T22:52:56Z

@TheHollows

Any update on when this will be merged?

When it's ready

antoniodesousa · 2024-10-03T17:05:11Z

hw/xbox/nv2a/shaders.c

+                   "  bvec4 nans = isnan(src);\n"
+                   "  if (!any(nans)) {\n"
+                   "    return src;\n"
+                   "  }\n"
+                   "  ivec4 signs = floatBitsToInt(src);\n"
+                   "  vec4 negative = vec4(lessThan(signs, ivec4(0)));"
+                   "  return mix(src, mix(vec4(1.0), vec4(0.0), negative), vec4(nans));\n"


Suggested change

" bvec4 nans = isnan(src);\n"

" if (!any(nans)) {\n"

" return src;\n"

" }\n"

" ivec4 signs = floatBitsToInt(src);\n"

" vec4 negative = vec4(lessThan(signs, ivec4(0)));"

" return mix(src, mix(vec4(1.0), vec4(0.0), negative), vec4(nans));\n"

" if (isnan(src.x)) {\n"

" src.x = step(0.0, sign(src.x));\n"

" }\n"

" if (isnan(src.y)) {\n"

" src.y = step(0.0, sign(src.y));\n"

" }\n"

" if (isnan(src.z)) {\n"

" src.z = step(0.0, sign(src.z));\n"

" }\n"

" if (isnan(src.w)) {\n"

" src.w = step(0.0, sign(src.w));\n"

" }\n"

" return src;\n"

Hi @abaire, may a suggest to change the code to what I wrote? I tried your current solution but it didn't fix anything on my PC. It behaves exactly as master. From my tests, my change behaves almost exactly as real hardware. Almost because there's a slight difference in the -NaNQ_NaNQ and -NaNS_NaNS tests cases in the N/A column, but I'm pretty sure that it's safe to ignore.

I tested a lot of games I couldn't find any regressions in any of them. Games that are fixed by this change:

Otogi

Otogi 2

Pro Cast: Sports Fishing Game

Trigger Man

HW - PC (AMD RX 6600M )

You're very welcome to take this one over, I've got some other projects going at the moment and don't have much time for xemu. It'd be good to test on an nvidia card as well, I'm not sure if glsl guarantees NaN behavior so it may be manufacturer or device specific.

I have a gtx 3060 and can test nvidia, just need the build and the test program...

That's too bad to heard. I will need to open a new PR then but I will link it to this one. Once is merged we can close both. Thanks for your research and I hope that you are back to xemu in the future.

@medievil1 you can download the build once I open the new PR.

mborgerson · 2024-10-08T21:16:15Z

Thank you for the PR! Since this work is continuing in #1780 with original author sign off, I'll close this PR now

abaire commented May 11, 2022

View reviewed changes

abaire force-pushed the fix/vsh_nan_handling branch from 415a071 to b210268 Compare May 16, 2022 23:28

abaire force-pushed the fix/vsh_nan_handling branch from b210268 to 24c5c9b Compare May 18, 2022 16:07

abaire marked this pull request as ready for review May 23, 2022 03:35

mborgerson reviewed May 25, 2022

View reviewed changes

ko81e24wy mentioned this pull request May 28, 2022

Marvel Nemesis Glitches/Bugs #986

Open

abaire marked this pull request as draft June 3, 2022 02:26

abaire force-pushed the fix/vsh_nan_handling branch from 24c5c9b to dae6f37 Compare June 3, 2022 02:31

abaire mentioned this pull request Jun 12, 2022

Metal Arms: Glitch in the System Shadow Transparency/Fade Issue #1060

Open

abaire force-pushed the fix/vsh_nan_handling branch from dae6f37 to e272865 Compare June 27, 2022 16:48

abaire force-pushed the fix/vsh_nan_handling branch from e272865 to 9853c97 Compare June 29, 2022 13:31

abaire force-pushed the fix/vsh_nan_handling branch from 9853c97 to e0de81e Compare July 23, 2022 20:51

hooby3dfx mentioned this pull request Jan 2, 2023

Blackened Texture Issues and Documentation #365

Open

xemu-project deleted a comment from washingleavesandthensome23-434 Jan 19, 2023

nv2a: Adjust NaN handling to be similar to HW

8b62926

abaire force-pushed the fix/vsh_nan_handling branch from e0de81e to 8b62926 Compare May 31, 2023 04:59

antoniodesousa reviewed Oct 3, 2024

View reviewed changes

antoniodesousa mentioned this pull request Oct 5, 2024

nv2a: Handle NaN values to be similar to HW #1780

Draft

mborgerson closed this Oct 8, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

nv2a: Adjust NaN handling to be similar to HW #913

nv2a: Adjust NaN handling to be similar to HW #913

abaire commented May 11, 2022 •

edited

Loading

abaire commented May 11, 2022

abaire May 11, 2022

ko81e24wy commented May 11, 2022 •

edited

Loading

abaire commented May 11, 2022

ko81e24wy commented May 11, 2022

NZJenkins commented May 11, 2022 •

edited

Loading

abaire commented May 16, 2022

ko81e24wy commented May 16, 2022

abaire commented May 17, 2022

ko81e24wy commented May 17, 2022

mysteria25 commented May 17, 2022

abaire commented May 17, 2022

abaire commented May 23, 2022

mborgerson May 25, 2022 •

edited

Loading

abaire May 25, 2022

abaire May 28, 2022

Triticum0 Jun 2, 2022

abaire Jun 8, 2022

Fabxx commented Jun 2, 2022

abaire commented Jun 3, 2022

Fabxx commented Jun 9, 2022

ko81e24wy commented Jun 29, 2022

abaire commented Jun 29, 2022

Triticum0 commented Jul 23, 2022

Kohryujin commented Jan 2, 2023

TheHollows commented Mar 4, 2024

mborgerson commented Mar 4, 2024

antoniodesousa Oct 3, 2024

abaire Oct 3, 2024

medievil1 Oct 3, 2024

antoniodesousa Oct 4, 2024

mborgerson commented Oct 8, 2024

-                   "  bvec4 nans = isnan(src);\n"
-                   "  if (!any(nans)) {\n"
-                   "    return src;\n"
-                   "  }\n"
-                   "  ivec4 signs = floatBitsToInt(src);\n"
-                   "  vec4 negative = vec4(lessThan(signs, ivec4(0)));"
-                   "  return mix(src, mix(vec4(1.0), vec4(0.0), negative), vec4(nans));\n"
+                   "  if (isnan(src.x)) {\n"
+                   "    src.x = step(0.0, sign(src.x));\n"
+                   "  }\n"
+                   "  if (isnan(src.y)) {\n"
+                   "    src.y = step(0.0, sign(src.y));\n"
+                   "  }\n"
+                   "  if (isnan(src.z)) {\n"
+                   "    src.z = step(0.0, sign(src.z));\n"
+                   "  }\n"
+                   "  if (isnan(src.w)) {\n"
+                   "    src.w = step(0.0, sign(src.w));\n"
+                   "  }\n"
+                   "  return src;\n"

nv2a: Adjust NaN handling to be similar to HW #913

nv2a: Adjust NaN handling to be similar to HW #913

Conversation

abaire commented May 11, 2022 • edited Loading

abaire commented May 11, 2022

Choose a reason for hiding this comment

ko81e24wy commented May 11, 2022 • edited Loading

abaire commented May 11, 2022

ko81e24wy commented May 11, 2022

NZJenkins commented May 11, 2022 • edited Loading

abaire commented May 16, 2022

ko81e24wy commented May 16, 2022

abaire commented May 17, 2022

ko81e24wy commented May 17, 2022

mysteria25 commented May 17, 2022

abaire commented May 17, 2022

abaire commented May 23, 2022

mborgerson May 25, 2022 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Fabxx commented Jun 2, 2022

abaire commented Jun 3, 2022

Fabxx commented Jun 9, 2022

ko81e24wy commented Jun 29, 2022

abaire commented Jun 29, 2022

Triticum0 commented Jul 23, 2022

Kohryujin commented Jan 2, 2023

TheHollows commented Mar 4, 2024

mborgerson commented Mar 4, 2024

Choose a reason for hiding this comment

HW - PC (AMD RX 6600M )

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mborgerson commented Oct 8, 2024

abaire commented May 11, 2022 •

edited

Loading

ko81e24wy commented May 11, 2022 •

edited

Loading

NZJenkins commented May 11, 2022 •

edited

Loading

mborgerson May 25, 2022 •

edited

Loading