With HistoricForeignKey prefetch_related doesn't return all related objects in some situations

[duebbert]

Describe the bug

There is an issue when using HistoricForeignKey and a "bare" prefetch_related. It doesn't return all results when calling the related set, for example self.some_set.all()

This happens when doing a .prefetch_related("some_set") in the queryset.

It does NOT happen when:

1. doing a .prefetch_related(Prefetch("some_set", queryset=Some.objects.all()))
2. or not doing a prefetch_related (which obviously leads to N+1 then)
3. or changing back to the normal models.ForeignKey.

This seems to happen when prefetching related objects more than one level down.

Reproduction

Models:

from django.db import models
from simple_history.models import HistoricForeignKey, HistoricalRecords


class Main(models.Model):
    name = models.CharField(max_length=100)
    history = HistoricalRecords()


class Sub(models.Model):
    main = HistoricForeignKey(Main, on_delete=models.CASCADE)
    name = models.CharField(max_length=100)
    history = HistoricalRecords()

    def __str__(self):
        return f"linked to {self.main} ({self.pk})"


class SubSub(models.Model):
    sub = HistoricForeignKey(Sub, on_delete=models.CASCADE)
    name = models.CharField(max_length=100)
    history = HistoricalRecords()

    def __str__(self):
        return f"linked to {self.sub} ({self.pk})"

Code:

from testing.models import Main, Sub, SubSub
from django.db.models import Prefetch

main = Main.objects.create(name="Main 1")

sub1 = Sub.objects.create(name="Sub 1", main=main)
sub2 = Sub.objects.create(name="Sub 2", main=main)

subsub1 = SubSub.objects.create(name="SubSub 1", sub=sub1)
subsub2 = SubSub.objects.create(name="SubSub 2", sub=sub2)

# Correct behaviour
main = Main.objects.get(name="Main 1")
for sub in main.sub_set.all():
    print(sub, sub.subsub_set.all())

# Correct behaviour
main = Main.objects.prefetch_related("sub_set").get(name="Main 1")
for sub in main.sub_set.all():
    print(sub, sub.subsub_set.all())

# Correct behaviour
main = Main.objects.prefetch_related(
    Prefetch("sub_set__subsub_set", queryset=SubSub.objects.all())
).get(name="Main 1")
for sub in main.sub_set.all():
    print(sub, sub.subsub_set.all())

# Correct behaviour
main = Main.objects.prefetch_related(
    Prefetch(
        "sub_set",
        queryset=Sub.objects.prefetch_related(
            Prefetch("subsub_set", queryset=SubSub.objects.all())
        ).all(),
    )
).get(name="Main 1")
for sub in main.sub_set.all():
    print(sub, sub.subsub_set.all())

# INCORRECT behaviour. Not showing related objects of second "Sub".
main = Main.objects.prefetch_related("sub_set__subsub_set").get(name="Main 1")
for sub in main.sub_set.all():
    print(sub, sub.subsub_set.all())

# INCORRECT behaviour. Not showing related objects of second "Sub".
main = Main.objects.prefetch_related(
    Prefetch("sub_set", queryset=Sub.objects.prefetch_related("subsub_set").all())
).get(name="Main 1")
for sub in main.sub_set.all():
    print(sub, sub.subsub_set.all())

Expected behavior

Calling sub.subsubset.all() should return all related objects.

Environment (please complete the following information):

• Django Simple History Version: 3.3.0
• Django Version: 4.1.7

[ddabble]

Thank you for the very clear and detailed bug report! 😊