Remove magic debug color from RayCast3D and ShapeCast3D

[Scrawach]

Describe the project you are working on

Describe the problem or limitation you are having in your project

Describe the feature / enhancement and how it helps to overcome the problem or limitation

RayCast3D and ShapeCast3D use magic Color(0, 0, 0, 1) as default. Every time, when you set up this color as debug custom color, RayCast3D and ShapeCast3D ignore it and use debug color from ProjectSettings.

Code from RayCast3D and ShapeCast3D:

Color color = debug_shape_custom_color;
if (color == Color(0.0, 0.0, 0.0)) {
	// Use the default debug shape color defined in the Project Settings.
	color = get_tree()->get_debug_collisions_color();
}

As result, in inspector we see black color, but in editor window - another, from ProjectSettings. At first time it's looks like a bug (archieved issue about it: godotengine/godot#102611)

Yes, It's has extra note in documentation, but it's a bit confuse anyway and I'm sure, that we can easy avoid it.

Another point is that other nodes with debug color not have a magic color (as example, CollisionShape3D, CollisionPolygon3D, CollisionShape2D). And it's work fine. I think this small enhancement in QoL is worth it.

Describe how your proposal will work, with code, pseudo-code, mock-ups, and/or diagrams

Instead of check magic color every time, we can just re-use code from CollisionShape3D (or another debug shapes, it's similar) and cached debug color from ProjectSettings in constructor:

Color RayCast3D::_get_default_debug_color() const {
	const SceneTree *st = SceneTree::get_singleton();
	return st ? st->get_debug_collisions_color() : Color(0.0, 0.0, 0.0, 0.0);
}

RayCast3D::RayCast3D() {
	debug_shape_custom_color = _get_default_debug_color();
}

It will also require some additional property validation work.

If this enhancement will not be used often, can it be worked around with a few lines of script?

No.

Is there a reason why this should be core and not an add-on in the asset library?

Debug color is set from core.

[dalexeev]

Another point is that other nodes with debug color not have a magic color (as example, CollisionShape3D, CollisionPolygon3D, CollisionShape2D). And it's work fine. I think this small enhancement in QoL is worth it.

I implemented this behavior in 2D (CollisionShape2D.debug_color) for usability (you don't need to edit the alpha channel every time) and consistency (the Color.BLACK value has no special meaning and can be used literally, not in a special sense). Since Godot doesn't support nullable/union types, I couldn't use null as a default value, so the only natural option was to use the current value of the project setting. Later, another contributor (see godotengine/godot#90644) adapted this for 3D (CollisionShape3D.debug_color).

RayCast3D (and later ShapeCast3D) had different logic initially (godotengine/godot#46529 was merged before godotengine/godot#39072). 3D wasn't my area of ​​interest, so I didn't notice this inconsistency. After the 4.0 release, no one likely dared to change this for compatibility reasons.

I could be mistaken, but I think I once suggested using my logic for RayCast/ShapeCast, but my suggestion was criticized, and we were left with this inconsistency in the engine. I couldn't find that discussion, so perhaps I'm mistaken. I only found the following bug report (but as I replied, it's a fairly minor issue; you typically don't change the relevant project settings often, if at all):

• Changes to CollisionShape2D debug color project setting are not applied due to the previous default color being considered as custom godot#90965

Accordingly, I personally support your proposal to change the behavior of {Ray,Shape}Cast3D to be consistent with CollisionShape{2D,3D}. However, please note that this change will likely be considered a compatibility breaker (albeit a minor one). We need to study this issue more closely to better understand the compatibility risks this change poses.