feat: add migration script to/from legacy history to generic history

[omkar-ethz]

Description

Depends on #2282

Motivation

With the above PR, history for new records will be included in the datasets response. The migration script is needed to backfill the older history records as well.

Fixes

• Migration script for HistoryClass #2247

Changes:

• Added up and down migration function for legacy history <-> generic history
• Added a util function to convert obsolete history to generic history, used in up migration
• Modified migrate:db:[up|down] to include npm run build, as the migration script depends on compiled dataset/utils module

Tests included

• Included for each change/fix?
• Passing?

Documentation

• swagger documentation updated (required for API changes)
• official documentation updated

Testing

Tested locally that up / down scripts preserve records across migration.

[minottic]

is there a way to do this with an aggregation pipeline (or at least with a cursor)? I know it might duplicate logic, but I think to find and then loop is fairly inefficient

[omkar-ethz]

is there a way to do this with an aggregation pipeline (or at least with a cursor)? I know it might duplicate logic, but I think to find and then loop is fairly inefficient

find returns a cursor. It would have been memory inefficient if I did a toArray on the cursor.
As for doing the entire migration with an aggregation pipeline, i think the logic is complex enough, (e.g. in the up case to compute the intersection of fields in datasetlifecycle.previous/current value that have changed; or in the down case to reconstruct the previousValue using datasetSnapshot), that it might be too complex to express this using mongodb operators.. do you think it's worth trying?
As a small optimization, i replaced the insertOne with insertMany in the up case.

[minottic]

find returns a cursor

ah, you are right, since this is not using mongoose but the mongo driver, my bad.

I gave a quick try to see how the aggregation pipeline would look, and it's indeed a bit complex and prone to typos. Happy to use this then, let's reconsider if necessary after we test the migration

[minottic]

minor, comment, happy to approve otherwise