There should be a way to do Direct Preference Optimization with diffusion models. Ryan Murdock already has it working: https://twitter.com/advadnoun/status/1677479082752364546 Requires further investigation.