GS: Fix huge ST coordinates in input vertices.

[TJnotJT]

Description of Changes

In the vertex trace, clamp floating point UV coordinates to valid UV ranges [-2047, 2047]. Add a pass to GSState::FlushPrims() that iterates through the primitivies about to be drawn and culls any that may be impossible to draw correctly (i.e., NaN values in ST coordinates) or replaces primitives with huge ST coordinates with a new primitive with clamped coordinates.

Rationale behind Changes

The input texture ST coordinates can sometimes be buggy (for reasons not totally clear, it might be emulation or game bugs), so they might be outside the valid ranges that the GS can process. Casting such values to integer can causes issues down stream with functions that use the vertex trace min/max texture coordinates, so this should fix some of those issues. In all the example that were fixes, the issue was with SW renderer not loading a full texture (because it depends on vertex trace calulations) and thus sampling from garbage values.

Suggested Testing Steps

Testing on GS dumps/games that are known to have huge ST coordinates as described above. I didn't find anything that was obviously broken in testing, though any help testing on more games/dumps or feedback on the changes would be greatly appreciated.

Here are the most visible fixes that I could find in testing (before is left and after is right)

Should be a slight blurring effect. Unfortunately, fixing this issue causes another issue with the black patches. This should be "accurate" on the GS side but the input coordinates from the EE may not be in this dump. (Counter_Terrorist_Special_Forces_-_Fire_for_Effect_SLES-53046_20230830195729.gs.xz).

Should not have double eyebrows (Bakugan_Battle_Brawlers_shadows.gs.xz).

Should have a faint star-like patterns (Psychonauts_SLUS-21120_20231013005852.gs.xz).

Should have character's shadow (Tomb_Raider_-_Legend_SLUS-21203_20230905130255.gs.xz).

Should not have dark road (testdriveswminmax.gs.xz).

More shadows in the face (not necessarily desirable and may be due to bad input coordinates from the EE; Futurama__NTSC-U__SLUS-20439.gs.xz).

[lightningterror]

TODO: On the dumps that I saw a difference should be checked on a ps2 to compare how they should look like.

[TJnotJT]

TODO: On the dumps that I saw a difference should be checked on a ps2 to compare how they should look like.

Lightning, would you be able to provide the dump name (if it is in gspack-10-july-2024)? I will try to work on this further given the new info from the hardware tests Ziemas ran.

[lightningterror]

007 Agent Under Fire - door blend
187 - Ride or Die_SLUS-21116_In_Game
2002 FIFA World Cup
50 Cent - Bulletproof_SLUS-21315_20230329231035
Ace Combat 04 - Shattered Skies_SLUS-20152_20230511225232
Ace_Combat_04_-_Shattered_Skies_SLUS-20152_20230502010138
AFL Premiership 2007_SCES-54639
Alfa Romeo
Ape Escape 2_SLUS-20685_20220914200448
Dave_Mirra_Freestyle_BMX_2_SLES-50217_20230213220044

There's more but the list is huge, ace combat 4, ape escape 2 dave mirra seem broken.

[TJnotJT]

Thanks much, this should keep me busy for some time

[TJnotJT]

I've fixed a few issues and the dump run seems to look good now on my end. Edit: Actually, there was a big error in the last force push that I didn't test. Should be fixed now hopefully.

[lightningterror]

What's the status on this, is it still worked on or ready for merge?

[TJnotJT]

What's the status on this, is it still worked on or ready for merge?

@lightningterror Thanks for checking, I actually noticed a glitch with Counter_Terrorist_Special_Forces_-_Fire_for_Effect_SLES-53046_20230830195729.gs.xz. This is because we currently don't have handling of infinite STs in the rasterizer (the incorrect texture load was masking the issue at the expense of the missing haze effect; see image below). Unfortunately, even if we emulate this accurately, I think we would end up with the same or similar image. The issue may be further upstream with incorrect floating point calculations in VU or EE.

The area outlined has a flat patch from sampling the corner of the texture.

So the current status is I need to add infinite ST handling to the rasterizer and see if it fixes the above glitch.

Also, I thought you might want to run a dump run on your end to double check me since you may have additional dumps in your collection. If you have the time that would be super helpful, or if you think the testing I'm doing is sufficient, that's ok also.