Usage of two unused render states for special game scenarios (force material hash and texture categories)

[xoxor4d]

This PR implements logic allowing the usage of two unused d3d9 render states to aid with special game scenarios whilst having game code access.

The specific purpose of this PR

I'm currently working on a portal 2 - remix compatibility mod (via reverse engineering - asi loading) and faced an issue with how the game is rendering the speed/jump gel's.

The game is building a dynamic texture that keeps track of which surfaces on the map are covered by gel. It is very similar to a lightmap texture and because it is a dynamic texture, the hash is dynamically changing as well. This prevents me from remixing it in the toolkit.

The gel texture also requires two remix texture categories to be set (IgnoreOpacityMicromap + Decal) for it to look correct.
This is not possible with the hash constantly changing.

Footage of gel being applied to surfaces (Showcase Discord):
https://discord.com/channels/1028444667789967381/1028600697463263314/1294618442346926152

In short:

When the hidden remix option 'useUnusedRenderstates' is enabled via the rtx.conf:

1. Unused render state 42 can be used to assign a remix texture category for the next surface like so:
   device->SetRenderState((D3DRENDERSTATETYPE)42, IgnoreOpacityMicromap | DecalStatic);

This does not add the hash to the specified remix option texture list as that is unnecessary
and would bloat it over time.

The code is checking bit flags. The enum I'm using matches the InstanceCategories enum class:

enum RemixInstanceCategories : uint32_t
{
	WorldUI			= 1 << 0,
	WorldMatte		= 1 << 1,
	Sky			= 1 << 2,
	Ignore			= 1 << 3,
	IgnoreLights		= 1 << 4,
	IgnoreAntiCulling	= 1 << 5,
	IgnoreMotionBlur	= 1 << 6,
	IgnoreOpacityMicromap	= 1 << 7,
	IgnoreAlphaChannel 	= 1 << 8,
	Hidden			= 1 << 9,
	Particle		= 1 << 10,
	Beam			= 1 << 11,
	DecalStatic		= 1 << 12,
	DecalDynamic		= 1 << 13,
	DecalSingleOffset	= 1 << 14,
	DecalNoOffset		= 1 << 15,
	AlphaBlendToCutout	= 1 << 16,
	Terrain			= 1 << 17,
	AnimatedWater		= 1 << 18,
	ThirdPersonPlayerModel	= 1 << 19,
	ThirdPersonPlayerBody	= 1 << 20,
	IgnoreBakedLighting	= 1 << 21,
};

2. Unused render state 150 can be used to set a custom material hash for the next surface like so:
   device->SetRenderState((D3DRENDERSTATETYPE)150, 0x1337);

Thoughts

• Rendering the gel using the remixAPI is currently not possible as the api only works with paths to actual dds textures on disk and does not expose a way to use "texture data" by providing a IDirect3DBaseTexture9 pointer. I'll create a separate feature request for that. (Even if that would work, I'm not entirely sure if I could render the gel using the api)

• This PR is only useful for people working on compatibility mods for shader heavy game with game code access. It should not affect or change anything for other remix supported games as the render states utilized are unused since ... ?

[xoxor4d]

@MarkEHenderson Thanks for your review! I've implemented the suggestions and the code is much cleaner now. Let me know if there is anything else ✌️

[xoxor4d]

@MarkEHenderson Just pushed the changes and I hope this is what you meant?

• added private member m_overrideHash which setHashOverride() sets to true whenever it is called
• this prevents updateCachedHash() from overriding the hash with the image hash
• created a new method: wasHashOverridden() because I need to skip the assert check in rtx_types.h

Note that I have not squashed the commits yet.