Implement "Shared cart" feature

[fedorov]

This would mirror the functionality available in TCIA Portal. Users can save the content of the cart, and share public link to that cart. Need to evaluate security implications, probably implement rate limiting per IP. I will also raise the question with TCIA how they implemented it.

[s-paquette]

@fedorov Items to address:

• Assuming we want the url to be something like /?saved-cart=idc-cart-[ID], and ID must be unique and non-guessable, what format do we want? We can do a simple hash of the database row ID (looks like this is what TCIA has there) eg. 5556217739040070284, we can use a UUID eg. 870928f7-755f-4062-bced-f7435fc842ca, we can use a base64 UUID eg. hwko93VfQGK87fdDX8hCyg.
• Do we want to put an expiration date on these and set a cron job to clean up the table, and if so, how often? 90 days, a year...
• What's a reasonable rate limit to place on bookmarks per IP per time to prevent abuse? One bookmark per minute? 2 per minute?
• Similar to cohorts, a cart's definition is related to the filters applied at the time it is saved, against the IDC version at that moment. How do we want to handle people coming back to a bookmark placed during a prior version? (Since eg. you can add 'all of collection X, Y, and Z, and the contents of that study could change).

[fedorov]

@s-paquette since here we are trying to deliver the feature that TCIA browser had, I think it is important to coordinate with Justin Kirby. I will loop him in via slack. Here are my thoughts:

Assuming we want the url to be something like /?saved-cart=idc-cart-[ID], and ID must be unique and non-guessable, what format do we want? We can do a simple hash of the database row ID (looks like this is what TCIA has there) eg. 5556217739040070284, we can use a UUID eg. 870928f7-755f-4062-bced-f7435fc842ca, we can use a base64 UUID eg. hwko93VfQGK87fdDX8hCyg.

I think shorter is better. How difficult would it be to allow users pick the name and check for clashes?

Do we want to put an expiration date on these and set a cron job to clean up the table, and if so, how often? 90 days, a year...

I would say we do not purge them at all. It shouldn't too much to keep around, no?

What's a reasonable rate limit to place on bookmarks per IP per time to prevent abuse? One bookmark per minute? 2 per minute?

Anything you think is reasonable. Based on Justin response in slack, TCIA never ran into any issues that would require to consider rate-limiting. Let's start with anything, and we can adjust based on actual usage.

Similar to cohorts, a cart's definition is related to the filters applied at the time it is saved, against the IDC version at that moment. How do we want to handle people coming back to a bookmark placed during a prior version? (Since eg. you can add 'all of collection X, Y, and Z, and the contents of that study could change).

This is a tricky one. I thought when we save a cart, we actually keep track of the individual series UIDs. I wonder how TCIA implemented this. What was TCIA browser behavior in a situation where a series was removed? Let's hear from Justin - I will ping him on slack.

[kirbyju]

Assuming we want the url to be something like /?saved-cart=idc-cart-[ID], and ID must be unique and non-guessable, what format do we want? We can do a simple hash of the database row ID (looks like this is what TCIA has there) eg. 5556217739040070284, we can use a UUID eg. 870928f7-755f-4062-bced-f7435fc842ca, we can use a base64 UUID eg. hwko93VfQGK87fdDX8hCyg.

I think shorter is better. How difficult would it be to allow users pick the name and check for clashes?

Yes, shorter is better for the randomly generated. That's a nice default thing to do for people, but we've found it to be very useful/important to let our curators generate carts with meaningful names tied to subsets within a dataset as PIs often was to group things (e.g. "this is my training set" and "this is my test set" becomes /?saved-cart=Collection1-Training and /?saved-cart=Collection1-Test.

Also, I don't think you should put "idc-cart-" as a prefix in the saved cart URLs. It just adds more length and you will already have IDC branding in the base URL, e.g. https://portal.imaging.datacommons.cancer.gov/explore/?saved-cart=CartName.

Do we want to put an expiration date on these and set a cron job to clean up the table, and if so, how often? 90 days, a year...

I would say we do not purge them at all. It shouldn't too much to keep around, no?

Agreed, keep forever if possible. Hasn't been a problem for us over a 10+ year span so far.

What's a reasonable rate limit to place on bookmarks per IP per time to prevent abuse? One bookmark per minute? 2 per minute?

Anything you think is reasonable. Based on Justin response in slack, TCIA never ran into any issues that would require to consider rate-limiting. Let's start with anything, and we can adjust based on actual usage.

+1 to this

Similar to cohorts, a cart's definition is related to the filters applied at the time it is saved, against the IDC version at that moment. How do we want to handle people coming back to a bookmark placed during a prior version? (Since eg. you can add 'all of collection X, Y, and Z, and the contents of that study could change).

This is a tricky one. I thought when we save a cart, we actually keep track of the individual series UIDs. I wonder how TCIA implemented this. What was TCIA browser behavior in a situation where a series was removed? Let's hear from Justin - I will ping him on slack.

Yes, a cart's definition is purely a list of series UIDs (no filters). In addition to the subsetting use case I mentioned above, the other main use case for this feature is to freeze a set of data at a point in time to be sure nothing about it changes for people who want to use it later. NBIA didn't fully deliver on this in the case of modified files or deletions, but I believe IDC can if you're retaining all old versions of files!

If data is changed or removed later from the "main" dataset that should not matter to the shared cart. You should retain the exact, older data that was in it on the day the cart was created.

One other thing I brought up in slack was that NBIA allows users to optionally provide 1) a short free text description and 2) a URL that is associated with the shared cart. This would keep more of the information about what the cart contains in IDC if users choose to provide it. I don't think it's critical that you take this on now, or at all, but do think it falls into the "nice to have" category.

[fedorov]

I believe IDC can if you're retaining all old versions of files!

There are several critical caveats:

1. IDC Portal only knows about the current version. IDC viewers can only show content in the latest version. We should be able to download content that refers to the older versions.
2. We still can/will have situations when data is actually removed, purged from IDC. In that case, it becomes completely inaccessible.

[kirbyju]

Your last comment also made me realize we didn't really cover what happens when someone goes to these URLs. I assume you would take someone to your "cart" screen with their series pre-loaded? Or are you building a new custom UI/screen for what to show people for these?

If reusing the existing cart page, it looks like the cart page groups things by study UID. If someone hand selects a few series from a particular study and then goes to your cart page, will the buttons on the cart page all work to only show info about the specific series in the shared cart? E.g. does clicking the button to view in OHIF only show the selected cart series or does it show all series in the study UID? Will the button to show relevant citations still work correctly?

[s-paquette]

I think shorter is better. How difficult would it be to allow users pick the name and check for clashes?

Logistically it's not difficult; security-wise it can be an issue. The problem with letting users define a name where we have no logins is it allows a way for names to be guessed. You attempt to save with a name, and if you get a collision, you know a cart has that name already, and now all you have to do is put it in to the main URL to access it and see what it has. This isn't a problem if we think guessing the names is fine for anyone to do, but we need to bear in mind our site has had issues with bad actors in the past, even when we did have accounts; they would come in and make hundreds of cohorts looking for exploits. (BillL would keep an eye out for these in the weekly audit and ban the user/IP once he found them.) @DavidPotCanuck suggested that, like tinyurl and bitly, we offer either a UID or let them do a custom name, while giving a warning a custom ID can be guessed and accessed by anyone.

I would say we do not purge them at all. It shouldn't too much to keep around, no?

This gets into an item below about how they're saved, which I'll go into there.

This is a tricky one. I thought when we save a cart, we actually keep track of the individual series UIDs.

We only do if a user actually clicked on specific IDs, or removed specific ones; if they add all of NLST, the tracking is simply 'NLST'. This is more about Solr than anything else; there's a 64k item limit on querying Solr, so we can only query 64k IDs at a time, thus getting the explorer page to load for eg. a 200k cart would require 4 sequential calls. 10, in the case of NLST.

We can track the list, but it incurs a storage issue, namely that if someone adds for example all of NLST + Analysis Results IDs, now we're looking at 590k strings to store. And that's only NLST; theoretically, someone can add much more. Realistically I don't expect a user to do this, but as our site is tested by NCI we have to account for what to do when, at a minimum, they do it, and do it repeatedly in a security-testing fashion.

There's a limit on how much can go into a given column, but it's generous (4GB) so we're not likely to have an issue there. The problem, as ever, is more about security and bad actors, or at a minimum, testing by NCI, and in the case of users, table growth over time.

@kirbyju to this question...

Your last comment also made me realize we didn't really cover what happens when someone goes to these URLs. I assume you would take someone to your "cart" screen with their series pre-loaded? Or are you building a new custom UI/screen for what to show people for these?

We're still working on the cart page itself, but my assumption was it would go to the Explorer page with that cart loaded. It could go to the cart page (which we have plans to update). Both are backed by the same searcher, which is Solr, and we can only query 64k series IDs at a time on Solr. So, once again, if a user adds eg. all of TCGA-BRCA (including analysis results) we have a possible issue.

There's a rather easy solution for Solr, which is we use the 'definition' for Solr, and keep the ID set regardless. Definitely not a problem to do this, but we do still need to think about as @fedorov said the behavior when some series are missing now, and size growth of the table over time.

If reusing the existing cart page, it looks like the cart page groups things by study UID. If someone hand selects a few series from a particular study and then goes to your cart page, will the buttons on the cart page all work to only show info about the specific series in the shared cart? E.g. does clicking the button to view in OHIF only show the selected cart series or does it show all series in the study UID? Will the button to show relevant citations still work correctly?

The OHIF buttons on the cart page are series-specific, so they only show that series. (We've discussed but not yet implemented the option to view just a specific set of series from a study in OHIF, as opposed to just one series or the whole study.) Citations are determined against the actual items in the cart, which is to say if you add one series with a given source DOI the citation will display.

[kirbyju]

Ok, well I don't think it's an issue for people to discover a cohort someone has created previously with a custom name since the only content of that is a list of scans that are already fully publicly accessible.

Sounds like some interesting design considerations on the rest. Let me know if you need any further input about anything.

[s-paquette]

Ok, well I don't think it's an issue for people to discover a cohort someone has created previously with a custom name since the only content of that is a list of scans that are already fully publicly accessible.

While I completely agree with you, it was an issue in our prior system (ISB-CGC) which also supported cohorts; users absolutely didn't want just anyone to be able to see their filters or set of IDs, or find them by guessing URLs. (If we allow for descriptions I think this would become a stronger case, since people could write just about anything.)

[fedorov]

I think should just have messaging/icons similar to how Dropbox/Google/other sharing platforms do it. The link is public with all the downstream implications.

We're still working on the cart page itself, but my assumption was it would go to the Explorer page with that cart loaded.

Yes, I do not see any benefits in investing effort to maintain and improve cart page. I would deprecate it and instead open Explorer page with the cart content.

[kirbyju]

Yes, I do not see any benefits in investing effort to maintain and improve cart page. I would deprecate it and instead open Explorer page with the cart content.

This is intriguing. So will you just replace this "view cart" button with a "URL" button like you have in the cohort section?

And would it then make sense to add the SQL button too for consistency? Then you can perform all the same operations and the only difference is that queries would update as datasets change over time whereas carts are intended to freeze a static set of specific data?

[s-paquette]

We could definitely add the SQL button though I don't know how useful it would be if only because the SQL queries to pull out these series lists can be quite dense, depending on the filters used and the cart selections made. (The BQ tables are SOPInstance level, not series level, which complicates things.) On the other hand it's not a large lift to do it, as the code to make the query is already in use on the backend for other things, so it's just a matter of making a call to the server to request it and then displaying the result.

[s-paquette]

@fedorov What did we want to do regarding carts defined in prior versions which might have now-removed (or even added but not present) series in them? (Eg. if a single series is removed, if the contents of a study, case, collection, or program are altered, etc.) What would be your preferred workflow in such a case? This is assuming we save the full ID set, which we should be able to do though we'll need to keep an eye on the database growth. There's an additional issue that Solr can only filter on 64k IDs, so if a user saves a cart larger than that we need to use the shorthand definition for any metrics.

Like under cohorts, we can snapshot the version it was saved under for comparative analysis; if there's a change in the cart's numerics, we warn the user and let them know not all the series they saved are available anymore, or that changes happened since they saved it, etc.

[fedorov]

I like the approach you seem to be suggesting - we just give a warning that what they see is not exactly what the cart contained at the time it was created.