Mesh Shaders Tracking Issue

[SupaMaggie70Incorporated]

Should replace #3018.

Progress

Spec: todo, currently based on this comment.

Current open PR(s)

#7345

naga

• Add mesh, task shaders to naga shader types so wgpu-hal can stop pretending they are compute shaders - Add mesh shader stages to wgt::ShaderStages and naga::ShaderStage #7292
• Support in WGSL frontend
• Validation
• Support in SPIRV writer
• Support in hlsl/msl writer
• Reflection info for pipeline validation

wgpu-hal backends

• Vulkan - Mesh shaders - initial wgpu hal changes #7089
• DX12 - Mesh Shading for DirectX 12 #7219
• Metal

Other

• Formal spec
• wgpu API - Add mesh shading api to wgpu & wgpu-core #7345
• wgpu-core pipeline validation - Add mesh shading api to wgpu & wgpu-core #7345 (?)
• Tests - Add mesh shading api to wgpu & wgpu-core #7345
• Examples - Add mesh shading api to wgpu & wgpu-core #7345

Features

• Multiview mesh shaders #7262
• Queries
• Point primitives(at least in vulkan)

Current Priorities

• Multiview in wgpu-hal - very simple - Add multiview mesh shaders to wgpu-hal #7278
• Naga types - needed for wgpu-hal completeness(current code pretends they are compute shaders) - Add mesh shader stages to wgt::ShaderStages and naga::ShaderStage #7292
• Formal spec created and added to trunk
• Experimental wgpu API - blocking many things - Add mesh shading api to wgpu & wgpu-core #7345
• Testing for wgpu - Add mesh shading api to wgpu & wgpu-core #7345
• Naga WGSL frontend - needed for proper wgpu implementation
• Naga SPIRV backend - needed for proper wgpu implementation
• Naga reflection info - needed for proper wgpu implementation
• DX12 in wgpu-hal - desired

Issues to work out

• How to implement multiview, what features, etc
• What information needs to be exposed through adapter limits
• How should point primitives be handled(probably not possible on directX?)

API differences

Task payload handling

• GLSL - only a single variable with the taskPayloadSharedEXT storage class per compilation unit(or none). This is then automatically passed to the entry point.
• SPIR-V - Variables can have the TaskPayloadWorkgroupEXT storage class. This is then optionally passed as an argument to OpEmitMeshTasksEXT. For mesh shaders the payload variable is inferred.
• HLSL - For mesh shaders, input is taken as in payload MeshPayloadStruct MeshPayload in function signature. groupshared payload_t MeshPayload is taken as an argument to DispatchMesh in task shaders.
• MSL - Task shaders write the payload using object_data Payload& outPayload [[payload]] in the function parameters. Mesh shaders read them with object_data const Payload& payload [[payload]].

Current proposed method: in WGSL, an @payload(global_var) attribute for entry points. Potential drawbacks:

• May not be able to properly parse all SPIR-V shaders. This is a tradeoff, as HLSL/SPIRV have the potential payload output specified in the emitMeshTasks function, while MSL/GLSL has it specified in the entry signature. This should be fine anyway, but may require smarter parsing for SPIRV.

[SupaMaggie70Incorporated]

@cwfitzgerald
I’ve been thinking, we probably ought to have some way to indicate the “payload” for task and mesh shaders. I’m updating the original comment to add an attribute, @payload(some global variable). This would also probably mean we don’t need a new storage class. Let me know if you think there is a better option.

[cwfitzgerald]

Sounds fine - one thing to keep in mind that we want the IR to be compatible enough with how SPIR-V does stuff, such that spirv-in can eventually support mesh shaders. I don't know specifics about how mesh shaders work in spirv, but we should be careful to not require things that are too hard to implement in that frrontend. We' already require heroics in the spirv frontend for atomics.

[SupaMaggie70Incorporated]

@cwfitzgerald Thanks for the response. I'll try to keep a list of things that might be hard to implement in certain environments.

[SupaMaggie70Incorporated]

@cwfitzgerald Should we wait for MVP before creating a wgpu API? Or should we add a wgpu API before it works properly/is validated? I think working on it right now would make tests, examples, etc far easier, as currently those live in my own branch.

[cwfitzgerald]

I think working on it right now would make tests, examples, etc far easier, as currently those live in my own branch.

10000% - We can add the feature to Features with the experimental prefix (so EXPERIMENTAL_MESH_SHADERS; look at raytracing for something to model this off of) so that people know it's incomplete and may have validation issues. The sooner we can get CI running mesh shaders, the better!

As a separate PR, it also would be good to add the spec as you laid out in the last issue into the repo under docs/specs and linked in the README (you can also look to how RT did this).

[SupaMaggie70Incorporated]

@cwfitzgerald What's the guidance on creating tests that require spirv-passthrough? Should I just include the spirv binaries, or some assembly to go along with them, or the original source code?

On an unrelated note, currently I am going with a unified create_render_pipeline in wgpu_core, but separated in wpgu and wgpu-hal. Do you think we should switch to a unified approach in wgpu-hal? There is a decent amount of code duplication. Also, my current approach simply overhauled the pipeline creation apis, but I know wgpu-native uses wgpu-core, and it also broke the player tests. It should be fairly simple to refactor this to have the logic for pipeline creation stored in a single function and then a function to create each of the two types of pipelines, which I suspect is what you'll recommend.

Thanks!

[SupaMaggie70Incorporated]

@cwfitzgerald More questions! No rush to answer of course. But I worry that if we extend the WGSL language too much, IDE support will completely fail, and so developers will have a rough time. For example, with the functions that must be "generic", they could be called outside of the main entry point, in which point it may not be clear which function should use which vertex/primitive output type. The way we'd work with that in naga is just detecting which entry point a given function is called in. But, and this is really a rather small issue, IDEs might not have such a good way of detecting this. But beyond IDEs, this can cause confusing issues, like changing one function invalidating another.

A similar point is that only one payload is allowed per entry point, so this must also be cleverly determined.

I can't really think of any desirable workarounds for this, since generics are pretty much out of the question for now.

Like I said, not a super huge issue, no rush for responses, I haven't even come up with a spec yet much less started implementing anything for WGSL.

[SupaMaggie70Incorporated]

Update for anyone following at home: there is only one PR until full support + testing + examples in wgpu. I will have a break in roughly a week(the week of March 24th), I expect this PR to be mostly completed by that point(by that I mean the PR will open and the remaining issues will be from reviews. It may still take longer to merge depending on wgpu maintenance). Currently the only issues that need to be resolved are with limits, and validation relating to them.

So, #7345 will be the PR to watch for now.

After that, the plan for naga is roughly

• Put together a spec for the WGSL changes
• Implement a wgsl parser & SPIR-V writer.
• Implement a SPIR-V parser(this should solidify the IR representation more than anything)
• Implement reflection for shaders
• Add support for naga shaders and validation to wgpu-core
• Implement more naga validation. This will likely need more flags for in-shader correctness checks
• Implement naga tests & wgpu tests relying on naga
• Implement naga writer for HLSL
• Work on DirectX implementation
• Then maybe look at Metal/MSL. I may get a MacBook Air sometime soon, in which case this could happen sooner.

In particular, I also plan on getting a spec created by or during my break, as well as starting work on the core naga changes.

[cwfitzgerald]

But I worry that if we extend the WGSL language too much, IDE support will completely fail, and so developers will have a rough time.

I wouldn't worry too much about this, there are already various static-use rules within wgsl that need to be minded.

Helper functions writing to payloads is an interesting conundrum. I don't have a great answer offhand.