Text-to-level should allow latent input

[schrum2]

This will allow both interactive evolution #11 and set the stage for stitching adjacent scenes together into larger levels

[schrum2]

Latent input is allowed now, but need to expand to allow a scene input rather than just latent tensor. Should also allow some parts to be frozen, or specify some parts as noise and others unchanged.

[schrum2]

Allowing a scene input seems to work in the sense that the code runs and generates an image. This is tested from the ascii data browser. However, the benefit of starting with a scene is unclear. If the number of inference steps is set to 1, then the original scene is returned, which I think confirms that the input scene is actually being used as the diffusion starting point. However, if the number of inference steps is 2, then complete garbage is generated, and if it is a higher number then I get a valid scene, but one that seems completely unrelated to the scene input.

Does this mean something is wrong with the code, or is providing a scene just like providing any old random input?

[schrum2]

Claude indicated that I could change the noise schedule, but I'm skeptical about those recommendations. It also mentioned increasing "guidance" by adding an additional loss term. I already have a guidance scale, but I don't have a "strength" value, and I recall stable diffusion having a parameter with that name which might be relevant here.

However, the first thing I'm going to try is adding skip connections to the model.

[schrum2]

Ok, so UNets already have skip connections by default as part of the ResNet blocks, so there is no change here.

I might investigate a "strength" parameter, but I'm not sure that there is anything useful that can be done with a strength input. I should still output the tensors at early/intermediate stages to investigate a bit, and I should probably compare with how Stable Diffusion behaves when it has an image input in order to see what it does that my code is not doing.

[schrum2]

The last remaining component of this issue can be demonstrated as follows. Run the ascii data browser on any data set. Then click the Load Model button at the top and select a trained model's directory: SMB1-conditional-regular. Then click the Generate From Scene button at the bottom. An image will pop up.

I was expecting the images produced from doing this to be similar to the scene they are based off of, but they just seem very random. I'm not sure if the problem is with the code, the model, or my expectations.