Choice of convnext

[smileyenot983]

Hello! Thanks for this wonderful work.
I am wondering about the choice of convnext for reference views features extraction. Why didn't you try to use VAE's encoder as feature extractor? Was it giving bad quality?

[lorenmt]

Hi, the quality is good. The decision to swap to ConvNext was simply to save the memory.

[smileyenot983]

Hi, the quality is good. The decision to swap to ConvNext was simply to save the memory.

I see, thank you!
But have you tried to run any experiment with frozen VAE encoder? Intuitively it should give a good tradeoff between the compression/representation quality

[lorenmt]

I think we did. But with the frozen encoder, the model couldn't reconstruct the reference images with an identity pose. As for now the design of EscherNet is a bit outdated, I recommend checking out our latest works: Kaleido (from me) and CausNVS (from Xin).