918 ability to freeze datasets or version #941

longshuicy · 2024-02-23T01:46:03Z

This PR includes features to "release" a dataset. Once dataset is released, the dataset document will be moved to a new collection "freeze_datasets" with additional fields: frozen flag to indicate released; a version number.
Then the "dataset" and "freeze_datasets" collection are joined using mongo view to provide the combined list of datasets for client/frontend consumption. Similar concept with file, folder, metadata, visualization and thumbnails. Details please refer to the diagrams.

Backend change:

New model for frozen dataset; and new Dataset Views
New model for frozen files, folder, metadata, visualization (config & data), and thumbnail. With new views to join frozen + regular db accordingly
Change all the related "GET" endpoints to use the joined views so we can see all the versions; but all the POST, PUT, PATCH are still pointing to the original dataset to ensure frozen
Add new endpoints to freeze dataset.
- Once it's frozen, the associated file, folder (parent folders recursively), metadata, thumbnails, visualization are all properly copied and linked.
- Use a "origin_id" to establish linkage with their origin within each category (dataset, file, folder, vis, metadata, thumbnail)
- Dataset version will increment using frozen_version_num field
A few more additional endpoints to list all frozen dataset, list a specific version, list the latest version
Simple Pytest

Frontend change see screenshots:

Todos:

~~- Visualization~~
~~- Thumbnail~~
~~- Public datasets~~

List all dataset versions using collapsible, links, and add pagination Dataset Versions show in links #1039
Delete given dataset version and/or delete current dataset Delete given dataset version #1040
Need to test Admin/Auth related with dataset versioning
Mint DOI DOI support #919
Write proper documentations about dataset versioning Documentations about dataset versioning #1041

…aset and freeze dataset

… published

ddey2 · 2024-05-31T16:42:00Z

It says "Viewing dataset version -999" when I change the status to 'PUBLIC'. May be we want to say current version?

frontend/src/components/datasets/OtherMenu.tsx

frontend/src/components/datasets/DatasetVersions.tsx

backend/app/models/datasets.py

frontend/src/components/datasets/PublicDatasetVersions.tsx

lmarini

Some minor UI comments:

When looking at a version, the pill with the version number next to the title pushes the tabs to the right. Try switching between current and a version and look at the tabs.
When loading the dataset page, we always show the notification showing with version we are viewing. So by default it shows "Viewing dataset version current unreleased." Do we always want to pop this up? (Testing on the delete branch, maybe it's just there).
Want to try and remove the gray lines below every version? You have enough white space. Not sure it is needed.
When looking at a specific version there is a little arrow that shows up to the right of the tabs. It's like the tabs think there is one more tab to the right.

…sions

longshuicy · 2024-06-04T14:50:21Z

Being a little picky

I was thinking may be just putting the version name beside the dataset name?

Right now, if I move from/back to current version from a versioned one, the alignment of below menu bar for Files, Metadata etc changes.

Good catch. The alignment below has been fixed. It's not related to the version pill but related to the tabs being scrollable. There was an extra scroll button "> " that causing the alignment issue. Please check again :-)

We had a discussed about the best way of displaying dataset version and that's implementation out of that. For the current version, we display nothing; for released version, we put a tab next to the dataset name like it is.

longshuicy · 2024-06-04T15:02:05Z

Some minor UI comments:

When looking at a version, the pill with the version number next to the title pushes the tabs to the right. Try switching between current and a version and look at the tabs.

When loading the dataset page, we always show the notification showing with version we are viewing. So by default it shows "Viewing dataset version current unreleased." Do we always want to pop this up? (Testing on the delete branch, maybe it's just there).

Want to try and remove the gray lines below every version? You have enough white space. Not sure it is needed.

When looking at a specific version there is a little arrow that shows up to the right of the tabs. It's like the tabs think there is one more tab to the right.

Tab position issue is fixed now. It was due to the tab item number changed with the autoscroll in place. We never had chances to visualize the switch between an item has more tabs vs less tabs.
I change the wording to "Latest" and also turned off snack bar notification for the latest. Not sure if that's what you meant though. We can discuss more about it.
The latest version doesn't have any gray lines. Could you pull again? Note in the "delete dataset versions" PR the look of that section will be further changed. i.e. how it got highlighted
Arrow is fixed together with the tab position (1.)

longshuicy · 2024-06-04T15:59:51Z

It says "Viewing dataset version -999" when I change the status to 'PUBLIC'. May be we want to say current version?

Good catch. I fixed this bug. Now when it's latest version, I'm not showing any notification.

ddey2 · 2024-06-07T17:41:01Z

This looks good to me. @longshuicy you might want to resolve the conflicts. You can do it before emerging too. You might get some more conflicts :)

* fix bug with get/delete a specific version of dataset; frontend delete button is working * dataset versioning styling * public dataset versioning styling * delete thumbnail and vis endpoint considering versioning * might be a bug deleting thumbnails * switch the order * write utility funcitons to help manage delete * backend delete is configured * frontend delete wired in correctly * delete modal color and prompt updated * add docstrings * add last modified * add specific pytest * add visualization tests to dataset versioning * add more pytest after deleting the latest dataset

* first draft of the versioning documentation * update documentation * wording

tcnichol · 2024-06-13T18:45:36Z

A question about datasets and the dataset status.

I created a dataset and went through a few versions, each one adding file, metadata etc.

I took the latest version and I made it 'public.'

The latest version is visible on the public page, but none of the previous versions.

It also looks like once a dataset is frozen, its no longer able to be shared or have its status changed.

Does this seem like something that we might want users to be able to do? I could imagine a case where someone might decide they want to show previous versions to users. Or not?

tcnichol · 2024-06-13T18:48:23Z

Something else I noticed.

I have a dataset with 5 versions plus latest.

For the latest, I changed the dataset from 'PUBLIC' to 'PRIVATE.'

On the 'public' page, version 5 is no longer visible.

Should an older version show up on the 'public' page if it was made public, but a later version is not?

longshuicy · 2024-06-13T19:56:59Z

Something else I noticed.

I have a dataset with 5 versions plus latest.

For the latest, I changed the dataset from 'PUBLIC' to 'PRIVATE.'

On the 'public' page, version 5 is no longer visible.

Should an older version show up on the 'public' page if it was made public, but a later version is not?

I created an issue: #1110

tcnichol

Marking approved. Made a comment about public datasets and previous versions that will be resolved in a later issue, so this one is good to go.

ddey2

looks good to me

longshuicy added 19 commits February 19, 2024 16:01

initial naive approach of frozen dataset

281b800

lock works

b788d97

use the view instead of just datset db

a92eabb

all get endpoint now change to use the dataset view that has both dat…

ba3363d

…aset and freeze dataset

refactoring GET public dataset related endpoints

1e62c9e

refactoring every GET endpoint related to dataset

08e21a9

write a published wrapper to control showing of edit or not

f87b3ec

use wrapper whenever possible to avoid hardcoding criterias

26ff45f

add more publish wrapper

8a67ed0

wrap proper auth and published check to various action menu

de67fbe

add lock to file

f6b2e92

fix metadata edit auth and published

6eb5d8e

something wrong with tab and file role

2671b6b

dataset tab condition set

26ac61b

fix fileRoleType

edb66be

fix file tabs

6ed0f80

freeze button works

aa258f5

frozen button works

de7678f

codegen

1bdd136

longshuicy requested review from lmarini, ddey2 and tcnichol February 23, 2024 01:46

longshuicy linked an issue Feb 23, 2024 that may be closed by this pull request

Ability to freeze datasets or version #918

Closed

longshuicy added 7 commits March 6, 2024 11:18

list the latest version of published dataset is working

f6c0e99

update freeze endpoint to take versioning into consideration

db00e9f

enable the cability of frozen and frozen draft

34986e8

messy logic

ec1e7f1

change pipeline logic to always have both the latest draft and latest…

4fc1c1c

… published

add authorization logic

1f901a0

sorting is working now

d930904

ddey2 requested changes May 31, 2024

View reviewed changes

lmarini reviewed Jun 3, 2024

View reviewed changes

Merge branch 'main' into 918-ability-to-freeze-datasets-or-version

a6624a0

longshuicy mentioned this pull request Jun 3, 2024

Implement "more" button on versions #1074

Open

longshuicy added 3 commits June 3, 2024 16:01

add folder to action text

0e22674

wrap style into function; fix bugs

391571a

turn off scrollbutton to avoid tab position change when switching ver…

c56520d

…sions

change the wording to latest; don't show snackbar when it's latest

14dd1df

fix viewing -999 bug

6b43641

longshuicy requested review from lmarini and ddey2 June 4, 2024 16:00

longshuicy added 4 commits June 12, 2024 14:52

Merge branch 'main' into 918-ability-to-freeze-datasets-or-version

06bbca4

fix a potential bug when updating dataset

adcc421

Versioning documentation (#1044)

ffb78c5

* first draft of the versioning documentation * update documentation * wording

Merge branch 'main' into 918-ability-to-freeze-datasets-or-version

aaaccca

longshuicy mentioned this pull request Jun 13, 2024

Status change after released dataset #1110

Open

tcnichol approved these changes Jun 17, 2024

View reviewed changes

ddey2 approved these changes Jun 20, 2024

View reviewed changes

Merge branch 'main' into 918-ability-to-freeze-datasets-or-version

55d9ef5

longshuicy merged commit 001c686 into main Jun 28, 2024
6 checks passed

longshuicy deleted the 918-ability-to-freeze-datasets-or-version branch June 28, 2024 15:32

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

918 ability to freeze datasets or version #941

918 ability to freeze datasets or version #941

longshuicy commented Feb 23, 2024 •

edited

Loading

ddey2 commented May 31, 2024

lmarini left a comment

longshuicy commented Jun 4, 2024

longshuicy commented Jun 4, 2024

longshuicy commented Jun 4, 2024 •

edited

Loading

ddey2 commented Jun 7, 2024

tcnichol commented Jun 13, 2024

tcnichol commented Jun 13, 2024

longshuicy commented Jun 13, 2024

tcnichol left a comment

ddey2 left a comment

918 ability to freeze datasets or version #941

918 ability to freeze datasets or version #941

Conversation

longshuicy commented Feb 23, 2024 • edited Loading

Backend change:

Frontend change see screenshots:

Todos:

ddey2 commented May 31, 2024

lmarini left a comment

Choose a reason for hiding this comment

longshuicy commented Jun 4, 2024

longshuicy commented Jun 4, 2024

longshuicy commented Jun 4, 2024 • edited Loading

ddey2 commented Jun 7, 2024

tcnichol commented Jun 13, 2024

tcnichol commented Jun 13, 2024

longshuicy commented Jun 13, 2024

tcnichol left a comment

Choose a reason for hiding this comment

ddey2 left a comment

Choose a reason for hiding this comment

longshuicy commented Feb 23, 2024 •

edited

Loading

longshuicy commented Jun 4, 2024 •

edited

Loading