Fixed relative cursors to handle nullable types.

[sleepertassly]

🧭 Paging Cursor Enhancements: Nullable Keys and Null Ordering Support

Summary

This PR introduces enhancements to the paging logic to correctly handle sorting and filtering when cursor keys are nullable. The main change is that the WHERE condition used in pagination must now adapt based on three factors:

1. Whether the cursor key is nullable.
2. Whether the actual cursor value is null or not.
3. Whether the database sorts null values first or last.

---

🔍 Why This Is Needed

When performing cursor-based pagination on fields that can be null, it's essential to construct the WHERE clause correctly. Incorrect assumptions about null ordering or the presence of null values can result in missing or duplicated records across pages.

---

🔧 Implementation Details

To support this behavior, the following changes were made:

• Cursor Key Metadata

  • Cursor keys now include a new property (IsNullable) to indicate whether they are nullable.

• Null Ordering in Cursor

  • Cursors now carry a NullsFirst flag to indicate how null values are ordered.
  • For subsequent pages, this flag is inherited from the previous cursor.
  • For the first(last) page, the flag is calculated based on whether the first(last) item contains a null value:
    • If a null is found → we can determine the null ordering: nulls first(nulls last).
    • If not → we assume the opposite ordering.
      • If this assumption is incorrect, it does not affect correctness, because it implies there are no null values in the dataset, making null ordering irrelevant.

The concept of handling nullable types in relative cursor-based pagination is inspired by [this StackOverflow discussion].

---

✅ Benefits

• Correct pagination over nullable fields.
• Eliminates edge cases where null records might be skipped or duplicated.
• Supports both nulls first and nulls last configurations in the database.

---

⚠️ Side Effects / Limitations

If our assumption about null ordering is incorrect, secondary ordering keys may appear to be sorted in reverse (opposite to the database's actual null handling behavior), particularly when paginating backward from the last page.

This only affects the visual or logical order of null values relative to each other and does not break pagination—all items will still be included in the correct pages.

[CLAassistant]

All committers have signed the CLA.

[michaelstaib]

The PR looks very good.

I'm planning to add this extra ORDER BY condition before all nullable cursor key, but that hasn't been implemented yet.
I'm still thinking about the performance impact. The computed orderby expression OrderBy(t => t.FoundedDate == null) which moves all null values to the beginning can significantly slow down sorting, but I believe it's still faster than OFFSET/LIMIT pagination.

This is another aspect we were also looking into ... especially as with postgres null order can be configured to be at the beginning of a set or at the and of a set.

I will copy over some new tests that we had in our work branch for nullable cursor values. Ping me on slack once you have solved the null ordering issue and we can see to get this one merged.

[michaelstaib]

Ping me once you need help, want another review or are ready to merge.

[Copilot]

Copilot reviewed 34 out of 34 changed files in this pull request and generated no comments.

Comments suppressed due to low confidence (1)

src/GreenDonut/src/GreenDonut.Data.EntityFramework/Expressions/ExpressionHelpers.cs:73

• The variable 'forward' is not defined in the current scope; please define it or pass it as a parameter to ensure correct directional logic in the expression building.

var greaterThan = forward ? key.Direction == CursorKeyDirection.Ascending : key.Direction == CursorKeyDirection.Descending;