Add sorting by custom fields

[mattmatters]

Initial implementation of custom field sorting as describe in 511. This allows users to define fields they want to do arbitrary sorting without having to alias a field.

Another bonus is you can technically pass in variables to do some pretty fun dynamic sorting. Right now I'm using them to surface relavent information to a user (PostGIS coordinate compared to user location).

For now we handle those custom variables in our code and pass them in. In the future it would be cool to see that a bit more formalized since we have to basically repeat some boilerplate for verifying some of these custom sort variables.

One additional sticky situation I found while doing this implementation is cursor fields, given other complex fields are not supported I opted to do the same here. However in theory it wouldn't be much work to support, it really just requires that a get_field/1 callback can execute at runtime (a bit weird though if you rely on runtime options like I'm doing above), then we would just need to validate that a filter function is set in addition to the sorter function for the custom field.

[mattmatters]

I haven't added any documentation yet, just wanted to make sure this was the direction you were thinking first.

[woylie]

Hi @mattmatters, thanks for opening the PR. Looks good to me! Once you add documentation and fix the doc tests, we can merge this. If you want to add support for cursor pagination on custom fields, feel free to do so, but we can also deal with this later.

[bjoern-kahle]

@woylie Your work on this PR is awesome — great job!
I could help complete so we can get this merged urgently. I really need it, haha. Would it be okay with you if I push it to your PR?

[woylie]

@woylie Your work on this PR is awesome — great job! I could help complete so we can get this merged urgently. I really need it, haha. Would it be okay with you if I push it to your PR?

I think you meant to ping @mattmatters.

[mattmatters]

@woylie Your work on this PR is awesome — great job!
I could help complete so we can get this merged urgently. I really need it, haha. Would it be okay with you if I push it to your PR?

Of course! I will not have major time to work on this until next week or so. Absolutely fine if you want to use this as a launch point or if you want to clean up the tests.

[bjoern-kahle]

Thank you both for quick responses. I am not allowed to push to your branch and before I fork also myself, it is actually just two lines to be changed:

lib/flop/schema.ex
line 61: [validation: :subset, enum: [:name, :age, :owner_name, :owner_age, :dog_age]]}
line 756: [:name, :age, :owner_name, :owner_age, :dog_age]

After that these commands run fine:

mix test
mix coveralls.json.all --warnings-as-errors

[woylie]

continuing in #590

[woylie]

I finished this up in #590. I'll add support for cursor pagination on custom fields before a release.

[woylie]

One additional sticky situation I found while doing this implementation is cursor fields, given other complex fields are not supported I opted to do the same here. However in theory it wouldn't be much work to support, it really just requires that a get_field/1 callback can execute at runtime (a bit weird though if you rely on runtime options like I'm doing above), then we would just need to validate that a filter function is set in addition to the sorter function for the custom field.

This is maybe not as easy as it seems. Custom filter functions currently take the query as an argument and return a query, but the where clauses for cursor pagination are based on dynamic. We'd have to change custom field filter functions to return a dynamic expression for this to work, which would be a big breaking change. I'll need to do some experimenting.

[woylie]

I changed the initial implementation to return a dynamic expression in the sorter function in #591. #592 adds support for cursor pagination on custom fields using the sorter dynamic. I think I'll rename the sorter option to dynamic or field_dynamic. The function could also be used for filtering if no filter function is configured, which would have the advantage that the user would not have to deal with operators when implementing custom fields, but you'd lose the ability to manipulate the filter value. So in summary, for sorting and cursor pagination, you'd need to implement the field_dynamic function, and for filtering, you can still optionally implement a filter function, but if none is configured, Flop would fall back on the field_dynamic` function.

[mattmatters]

I changed the initial implementation to return a dynamic expression in the sorter function in #591. #592 adds support for cursor pagination on custom fields using the sorter dynamic. I think I'll rename the sorter option to dynamic or field_dynamic. The function could also be used for filtering if no filter function is configured, which would have the advantage that the user would not have to deal with operators when implementing custom fields, but you'd lose the ability to manipulate the filter value. So in summary, for sorting and cursor pagination, you'd need to implement the field_dynamic function, and for filtering, you can still optionally implement a filter function, but if none is configured, Flop would fall back on the field_dynamic` function.

This is incredible. Thank you for taking over on this. I kept meaning to return but other work got in the way and my fork was doing the job for then. Giving your newest changes a spin!

Once again thank you for writing and maintaining this library @woylie. I real can't gush enough about how it's become the lynchpin of all our public api list operations. I'm even playing around with using it for aggregate queries where I still want to have pagination and some filters (time series graphs for example).