Improved support for modifying glTF models

[javagl]

*looks left*
*looks right*
Yeah, I'm still working on this, occasionally.

I'm opening this as a draft for now, just to track the process. It is part of (and built on top of) the general extension support efforts, aiming at JglTF 3.0.0.

This specific branch/PR mainly addresses the goal to simplify reading, modifying, and writing glTF models .

@CloseFile and @LocutusV0nB0rg : You 👍 'ed this issue. You might not be interested in the "Summary". But maybe you want to have a look at the "Example" below, to maybe get an idea where this might be going.

Summary

The reason why it is built on top of the "extension support" PR is that a really convenient layer for reading/modifying/writing models requires a "model-level" understanding and awareness of extensions.

A very short summary of the approach (with some overlap to what I already wrote in #135):

• There is the ModelElement interface that is implemented by all elements of a glTF model (AccessorModel, SceneModel, AnimationModel etc).
• This interface is also implemented by all models that are defined as part of extensions. One example, just for illustration, is the DefaultMaterialsClearcoatModel for KHR_materials_clearcoat.
• One direction of the "understanding" of extensions is that each ModelElement must provide a mechanism to determine which other model elements are needed/used by this model element.
  • This is accomplished with the getReferencedModelElements function. For example, the clearcoat implementation returns the clearcoat/roughness/normal TextureModel objects.
• The other direction is that it must be possible to remove model elements from other model elements.
  • This is accomplished with the removeModelElements method. For the clearcoat implementation, this sets all TextureModel elements to null when they should be removed

With these operations, it is possible to perform certain operations that, at their core, boil down to graph operations.

For example, it is possible to build a "graph" of model elements, starting at the SceneModel/AnimationModel objects, and traversing through the reachable (referenced) model elements. Model elements that are not reachable are disconnected from the rest, and can be removed. The removal methods, in turn, indicate whether an element became "invalid" due to the removal, and have to be removed as well. Once all "unused/invalid" elements have been removed, the model can (or rather: has to be) "validated" in terms of its buffer structure.

(I'm aware of some caveats of all this. This is still a draft PR, and still requires extensive testing)

This ~"graph stuff" is currently contained in a first shot of a package called gltf-model-transform, added via 99488f6 . (Of course I won't dare to claim that it is nearly as clean and powerful as glTF-Transform, but ... I needed a name, and that's it for now...)

Example

To illustrate what already works in the state of this PR, consider the following example

DefaultGltfModel gltfModel = createExampleModel();

Set<ModelElement> toRemove = new LinkedHashSet<ModelElement>();

// Magic!!! (See below)

GltfModelTransforms.removeAll(gltfModel, toRemove);

GltfModelWriter w = new GltfModelWriter();
w.writeBinary(gltfModel, new File("result.glb"));

The goal is to create (or load) an example model, do the Magic, and then write out a valid (!) glTF model of the result.

And Magic can currently be two things:

Modifying arbitrary elements:

Imagine the model contains a material with a texture. It is then possible to do the following:

PbrMaterialModel materialModel = (PbrMaterialModel)gltfModel.getMaterialModel(0);
 DefaultPbrMetallicRoughnessModel pbr = 
  (DefaultPbrMetallicRoughnessModel) materialModel.getPbrMetallicRoughnessModel();
pbr.setBaseColorTextureInfoModel(null);

^{(Yeah, the casts are ugly - let's ignore that for now...)}

The base color texture is set to null, so the texture becomes unused. The result will be a glTF model that does no longer have the texture (and does no longer contain the ImageModel that was used by this texture).

Removing arbitrary elements:

Imagine the model contains an animation. It is then possible to do this:

ModelElement timeAccessor = gltfModel.getAnimationModel(0).getChannels().get(0).getSampler().getInput();
toRemove.add(timeAccessor);

The AccessorModel that stores the times (key frames) of the AnimationModel is explcitly removed. (Note that the animation model itself is not explicitly modified. This just removes the accessor from the glTF, regardess of where it is used!) This implies the removal of the AnimationModel, which, in turn, implies the removal of other AccessorModel instances that have been used by the animation. The result will be a glTF model that does no longer have this animation (and none of the data that was used for that animation).

---

All this is pretty preliminary. There are several TODOs and things that have to be tested more thoroughly. But it looks like this could be one way to simplify reading, modifying, and writing glTF models .

[LocutusV0nB0rg]

I like what I am reading! Keep it up.

I have modified my use cases that aim at replacing components by simply fixing up my geometry compilation before I create the actual models. This solved a lot of my problems, at least temporary. And nothing is as permanent as a temporary solution, which is fine by me in this case.

The only thing that it didn't solve is that my big models must replace some objects with animated counterparts that were manually edited using Blender and can therefore not be created from scratch by me in the code. Therefore I just keep a list of all the objects that are animated, eliminate them from the big model and tell my rendering client to add the animated objects to the view. Not the best solution, but one that works very well without me having to deep dive too much into this topic.

I am curious about any progress on this. Bon courage!

[javagl]

The points of "Modifying arbitrary elements" (by setting some property to null) and "Removing arbitrary elements" are pretty similar on the implementation level (in fact, they are calling the same function internally, which could just be called cleanUp or so).

The point to "replace some objects with animated counterparts" is a natural next step. And I don't see major hurdles for that right now. The jgltf-model-builder contains much of the structures for conveniently creating models. And adding new elements to an existing model is (in many ways) only a small step beyond that.

The main goal for allowing modifications of the model is to ensure "structural consistency". You can manually create inconsistent/invalid structures and add them to the model. Or you can make structures "invalid" by removing elements. One important aspect of that is the validity and consistency of the buffer structures (accessor->bufferView->buffer). This is the complicated part. Nearly every modification will imply changes in this structure, and there has to be a mechanism to ensure that it remains valid. Clients should not have to twiddle with that on their own.

On the implementation side, most of what is required here already exists (and already existed for a long time in JglTF, with the first versions of the BufferStructureBuilder having existed since ~2017). But applying this properly, in the context of ~"arbitrary" modifications, can be tricky.

For now, I preliminarily and hesitatingly resorted to what could be seen as a "sledgehammer method": There is an internal rebuildBufferStructure method that rebuilds this structure, from scratch. But I think that this is the only thing that makes sense. Deleting one accessor can remove one slice of a bufferView, which is just one slice of a buffer. Trying to surgically "cut this out", while still keeping the structures valid (in terms of offsets and alignments) is hardly feasible.

All this will require many more tests and validations. And I hope that the current approaches (of using that "Graph" for reachability computations, and that rebuildBufferStructure) do not turn out to become performance bottlenecks. So some tests will have to include some artificial examples, like using 10000 accessors and iteratively removing every second one or so. Some tests already exist locally, but not as unit tests. Quite some work remains to be done here.

(But... somewhat unrelated: There are currently three open draft PRs, #134 , #135 and this one. The lower-level ones (like the material refactoring) are largely "consolidated" - this is necessary for all PRs that build on top of them. At some point, I'll try to "flatten" this into the v3.0.0-dev branch)