1287 remote rsd search (RSD v3)

[dmijatovic]

Search for software in multiple RSD instances

Closes #1287

Changes proposed in this pull request:

• rsd administrators can specify the instance name to be used as "label" for the software from this instance. New section "RSD info" is created for this purpose. Initially the default "Not defined" value is added to rsd_info table.
• rsd_info table/endpoint is created to store the info about rsd instance. Basic endpoint information and remote_name property is inserted in the table. remote_name prop is used in aggregated_software_overview.
• rsd admin can add remote RSD to be included in software search on software overview page.
• all users can search and filter software in all defined RSD instances from the software overview page.
• the "remote" software opens in the new tab and the "local" software opens in the same tab. The remote software has "opens in new tab" icon at the top right of the card
• if at least one remote RSD is defined, the additional filter is shown on the software overview page.
• global search includes remote software. Under the software name the domain name is shown to indicate "remote" source
• upgraded all dependencies to latest version: MUI v5.16.4 and React v19. React upgrade to v19 is commited as "BREAKING CHANGE" and will result in major version bump of RSD images to v3.0.0

How to test:

• make start to build and generate test data
• navigate to software overview page and confirm that no RSD hosts filter is present (no remotes are defined yet).
• login as rsd-admin, navigate to Rsd info page. Change remote_name to your liking. At some other values. You can read this values from the api at: /api/v1/rsd_info
• navigate to Remotes page (Admin section)
• add Helmholtz remote: https://helmholtz.software/
• add RSD Saas remote: https://research-software-directory.org/
• add RSD dev remote: https://research-software.dev/ . RSD dev is updated to this PR version to return RSD remote name. This suggestion is shown in the form of the icon (see last image).
• try to add not existing RSD remote. The validation should raise error and prevent you from adding invalid remote.
• run scrapers manually or wait for scrapers to run. You can stop & restart RSD to see the logs docker compose down & docker compose up
• after scrapers run examine the software overview. It should include software from remote RSD's too.
• you can also filter all software, including remote software. The counts shown in the filters should reflect all software including remote software.
• confirm that additional filter is shown on the software overview page and you can use filter.
• confirm that local software is open in the same tab and remote software in the new tab.
• navigate to highlights page and confirm that additional filter "RSD Hosts" is not shown on the page and that filtering works properly

Example rsd info section

Example add remote RSD

Example remotes overview

Example software overview cards (showing source, remote icon and the additional filter)

Example software overview list (with source)

Example suggested remote RSD name (based on info received from remote)

Example global search includes domain for the remotes

PR Checklist:

• Increase version numbers in docker-compose.yml
• Link to a GitHub issue
• Update documentation
• Tests

[jmaassen]

The fetching of the remote RSD name seems to be a bit buggy?

When I add the dev server URL, the remote RSD name does not show up automatically. Instead I have to click the "remote control icon" first. Why not fetch it and allow the admin to change it later?

After fetching the name, the text box containing it seems messed up? Also, the save icon is still disabled.

When clicking on the name text box, it is cleaned up, but the save is still disabled.

Clicking on the update interval or URL box activates the save button.

[jmaassen]

When selecting by keyword, the selected keyword does not always show up. For example, when harvesting from the RSD SAAS and Helmholtz RSD, selecting "3D" works as expected:

However, if I select "Data Analysis" instead, the filter works as expected, but isn't shown in the sidebar:

All others I've tried seem to work as expected, so I'm not sure what the problem is with this one?

[dmijatovic]

@jmaassen Well spotted! I have fixed both remarks. I did force push so please remote old branch and pull again.

[cmeessen]

When adding research-software.dev as a remote I get this error message:

[dmijatovic]

When adding research-software.dev as a remote I get this error message:

@cmeessen Thanks for notifying us. It seems that rsd-dev api failed for uknown reason. I restarted docker-compose and the rsd-dev is working now.

[ewan-escience]

Nice work, I have some comments on the database setup.

[cmeessen]

Using the global search does not include the results from remote RSDs.

[jmaassen]

When looking at the NWO website, I noticed that they used the following method for indicating that clicking on a tile leads to an external website:

So a similar "exit" icon that we discussed during out meeting last friday. They do highlight this icon by surrounding it using a colored box.

[ewan-escience]

1. When there are no entires in rsd_info (because they were directly deleted in the database), the Add button is not there.
2. I would also make the remote_name entry fully mutable (don't disable the slider and delete button).
3. The database enforces the label of remote_rsd to be of at least length 3, but the frontend form doesn't show this.
4. The software overview page doesn't handle a missing remote_name gracefully: it shows a blank entry in the filter and when you click on it, it shows nothing with a null in the URL bar in the sources parameter.

[jmaassen]

The new tab to add RSD info has different capitalization from the others. It uses "Rsd" instead of "RSD":

[jmaassen]

Once the remote_name property is set, it cannot be unset, only changed. While this is not necessarily a problem, the behavior of the tab is a bit strange.

Initially the value is set:

I select it:

Press delete. The value is deleted:

Navigate away from the box, the value is restored (which is okay):

Next time I click on the value, it is highlighted dark red:

If I navigate away the highlight disappears. Next selection works normally:

[jmaassen]

During testing of the RSD info tab, I somehow lost the rights to add key value pairs:

results in:

This could only be resolved by signing out and signing in again.

I cannot reproduce this behavior however....

[jmaassen]

When I change a value in the RSD info tab, while it it red, it appears to be stored but isn't.

I start with a valid value:

Then select and delete it:

The value is restored:

When I click it, the value turns read. I change it:

The value seems stored:

However, when I click and delete it again, the original value is restored, not the changed one:

[jmaassen]

The technical documentation for the RSD info tab is still missing I think?

[sonarqubecloud]

Quality Gate passed for 'rsd-database'

Issues
4 New issues
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
8.2% Duplication on New Code

See analysis details on SonarQube Cloud

[sonarqubecloud]

Quality Gate passed for 'scrapers'

Issues
5 New issues
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube Cloud

[sonarqubecloud]

Quality Gate passed for 'rsd-frontend'

Issues
0 New issues
0 Accepted issues

Measures
0 Security Hotspots
23.3% Coverage on New Code
1.7% Duplication on New Code

See analysis details on SonarQube Cloud