Skip to content

Commit a6b04a5

Browse files
authored
Merge branch 'SciCatProject:master' into master
2 parents 9c099a7 + 6787aa4 commit a6b04a5

23 files changed

+2477
-535
lines changed

.env.example

Lines changed: 3 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -64,11 +64,12 @@ RABBITMQ_ENABLED=<"yes"|"no">
6464
RABBITMQ_HOSTNAME="localhost"
6565
RABBITMQ_USERNAME="rabbitmq"
6666
RABBITMQ_PASSWORD="rabbitmq"
67+
REGISTER_DOI_URI="https://api.test.datacite.org/dois"
6768
RABBITMQ_PORT=5672
68-
REGISTER_DOI_URI="https://mds.test.datacite.org/doi"
6969
REGISTER_METADATA_URI="https://mds.test.datacite.org/metadata"
7070
DOI_USERNAME="username"
7171
DOI_PASSWORD="password"
72+
DOI_SHORT_SUFFIX=false
7273
SITE=<SITE>
7374
EMAIL_TYPE=<"smtp"|"ms365">
7475
EMAIL_FROM=<MESSAGE_FROM>
@@ -98,6 +99,7 @@ FRONTEND_CONFIG_FILE="./src/config/frontend.config.json"
9899
FRONTEND_THEME_FILE="./src/config/frontend.theme.json"
99100
LOGGERS_CONFIG_FILE="loggers.json"
100101
PROPOSAL_TYPES_FILE="proposalTypes.json"
102+
PUBLISHED_DATA_CONFIG_FILE="publishedDataConfig.json"
101103
DATASET_TYPES_FILE="datasetTypes.json"
102104
SWAGGER_PATH="explorer"
103105
MAX_FILE_UPLOAD_SIZE="16mb"

.github/workflows/test.yml

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -135,6 +135,8 @@ jobs:
135135
ES_PASSWORD: duo-password
136136
MEM_LIMIT: 4G
137137
STACK_VERSION: 8.8.2
138+
DOI_USERNAME: ${{ secrets.DOI_TEST_USERNAME }}
139+
DOI_PASSWORD: ${{ secrets.DOI_TEST_PASSWORD }}
138140
# Start mongo container and app before running api tests
139141
run: |
140142
cp CI/ESS/docker-compose.api.yaml docker-compose.yaml

.gitignore

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -8,6 +8,7 @@ loggers.json
88
jobConfig.json
99
jobConfig.yaml
1010
metricsConfig.json
11+
publishedDataConfig.json
1112

1213
# Configs
1314
.env

CI/E2E/.env.backend-next.example

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -24,7 +24,7 @@ RABBITMQ_ENABLED="no"
2424
RABBITMQ_HOSTNAME="localhost"
2525
RABBITMQ_USERNAME="rabbitmq"
2626
RABBITMQ_PASSWORD="rabbitmq"
27-
REGISTER_DOI_URI="https://mds.test.datacite.org/doi"
27+
REGISTER_DOI_URI="https://api.test.datacite.org/dois"
2828
REGISTER_METADATA_URI="https://mds.test.datacite.org/metadata"
2929
SITE="ESS"
3030
OAI_PROVIDER_ROUTER=

README.md

Lines changed: 13 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -109,7 +109,7 @@ Follow the structure of the provided [frontend.config.json](/frontend.config.jso
109109

110110
Providing a file called _loggers.json_ at the root of the project, locally or in the container, and create an external logger class in the `src/loggers/loggingProviders/`directory will automatically create specified one or multiple loggers instances.
111111

112-
The `loggers.json.example` file in the root directory showcases the example of configuration structure for the one or multiple loggers. `logger.service.ts` file contains the configuration handling process logic, and `src/loggers/loggingProviders/grayLogger.ts` includes actual usecase of grayLogger.
112+
The `loggers.example.json` file in the root directory showcases the example of configuration structure for the one or multiple loggers. `logger.service.ts` file contains the configuration handling process logic, and `src/loggers/loggingProviders/grayLogger.ts` includes actual usecase of grayLogger.
113113

114114
### Proposal types configuration
115115

@@ -121,7 +121,13 @@ The `proposalTypes.json.example` file in the root directory showcases the exampl
121121

122122
When providing a file called _datasetTypes.json_ at the root of the project, locally or in the container, it will be automatically loaded into the application configuration service under property called `datasetTypes` and used for validation against dataset creation and update. The types `Raw` and `Derived` are always valid dataset types by default.
123123

124-
The `datasetTypes.json.example` file in the root directory showcases an example of configuration structure for dataset types.
124+
The `datasetTypes.example.json` file in the root directory showcases an example of configuration structure for dataset types.
125+
126+
### Published data configuration
127+
128+
Providing a file called _publishedDataConfig.json_ at the root of the project, locally or in the container, will be automatically loaded into the application configuration service under property called `publishedDataConfig`. It will be used for published data metadata form generation in the frotnend and metadata validation in publication and registration of the published data.
129+
130+
The `publishedDataConfig.json.example` file in the root directory showcases the example of configuration structure for published data metadata.
125131

126132
## Environment variables
127133

@@ -149,6 +155,9 @@ Valid environment variables for the .env file. See [.env.example](/.env.example)
149155
| `ACCESS_GROUPS_STATIC_VALUES` | string | Yes | Comma-separated list of access groups automatically assigned to all users. Example: "scicat, user". | |
150156
| `ACCESS_GROUPS_OIDCPAYLOAD_ENABLED` | string | Yes | Flag to enable/disable fetching access groups directly from OIDC response. Requires specifying a field via `OIDC_ACCESS_GROUPS_PROPERTY` to extract access groups. | false |
151157
| `DOI_PREFIX` | string | | The facility DOI prefix, with trailing slash. | |
158+
| `DOI_SHORT_SUFFIX` | string | | By default `uuidv4` is used to generate the DOI suffix but if this flag is `true` the shorter version of 10 random characters is used as DOI suffix. | |
159+
| `DOI_USERNAME` | string | | The facility DOI DataCite username. | |
160+
| `DOI_PASSWORD` | string | | The facility DOI DataCite password. | |
152161
| `EXPRESS_SESSION_SECRET` | string | No | Secret used to set up express session. Required if using OIDC authentication | |
153162
| `EXPRESS_SESSION_STORE` | string | Yes | Where to store the express session. When "mongo" on mongo else in memory | |
154163
| `HTTP_MAX_REDIRECTS` | number | Yes | Max redirects for HTTP requests. | 5 |
@@ -222,8 +231,8 @@ Valid environment variables for the .env file. See [.env.example](/.env.example)
222231
|`ES_REFRESH`| string | | If set to`wait_for`, Elasticsearch will wait till data is inserted into the specified index before returning a response. | false |
223232
|`STACK_VERSION` | string | Yes | Defines the Elasticsearch version to deploy | "8.8.2" |
224233
|`CLUSTER_NAME` | string | Yes | Sets the name of the Elasticsearch cluster | "es-cluster" |
225-
|`MEM_LIMIT` | string | Yes | Specifies the max memory for Elasticsearch container (or process) | "4G" |
226-
| `FRONTEND_CONFIG_FILE`| string | | The file name for frontend configuration, located in the`/src/config`directory by default. | "./src/config/frontend.config.json" |
234+
|`MEM_LIMIT`| string | Yes | Specifies the max memory for Elasticsearch container (or process) | "4G" |
235+
|`FRONTEND_CONFIG_FILE`| string | | The file name for frontend configuration, located in the`/src/config`directory by default. | "./src/config/frontend.config.json" |
227236
|`FRONTEND_THEME_FILE`| string | | The file name for frontend theme, located in the`/src/config`directory by default. | "./src/config/frontend.theme.json" |
228237
|`LOGGERS_CONFIG_FILE`| string | | The file name for loggers configuration, located in the project root directory. | "loggers.json" |
229238
|`PROPOSAL_TYPES_FILE`| string | | The file name for proposal types configuration, located in the project root directory. | "proposalTypes.json" |
File renamed without changes.
Lines changed: 106 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,106 @@
1+
module.exports = {
2+
async up(db, client) {
3+
await db
4+
.collection("PublishedData")
5+
.find({})
6+
.forEach(async (publishedData) => {
7+
const pid = publishedData._id;
8+
const metadata = {
9+
affiliation: publishedData.affiliation,
10+
downloadLink: publishedData.downloadLink,
11+
scicatUser: publishedData.scicatUser,
12+
thumbnail: publishedData.thumbnail,
13+
url: publishedData.url,
14+
creators: publishedData.creator.map((creator) => ({
15+
name: creator.trim(),
16+
affiliation: [{ name: publishedData.affiliation?.trim() || "" }],
17+
})),
18+
publisher: {
19+
name: publishedData.publisher.trim(),
20+
},
21+
publicationYear: publishedData.publicationYear,
22+
dataDescription: publishedData.dataDescription,
23+
resourceType: publishedData.resourceType,
24+
contributors:
25+
publishedData.authors?.map((author) => ({
26+
name: author.trim(),
27+
})) || [],
28+
relatedIdentifiers:
29+
publishedData.relatedPublications?.map((relatedPublication) => ({
30+
relatedIdentifier: relatedPublication,
31+
})) || [],
32+
};
33+
const datasetPids = publishedData.pidArray;
34+
const status =
35+
publishedData.status === "registered" ? "registered" : "private";
36+
37+
console.log(`Updating PublishedData (Id: ${pid})`);
38+
await db.collection("PublishedData").updateOne(
39+
{ _id: pid },
40+
{
41+
$set: {
42+
pid,
43+
metadata,
44+
datasetPids,
45+
status,
46+
},
47+
$unset: {
48+
affiliation: true,
49+
downloadLink: true,
50+
scicatUser: true,
51+
authors: true,
52+
pidArray: true,
53+
creator: true,
54+
publisher: true,
55+
publicationYear: true,
56+
dataDescription: true,
57+
relatedPublications: true,
58+
resourceType: true,
59+
thumbnail: true,
60+
url: true,
61+
},
62+
},
63+
);
64+
});
65+
},
66+
async down(db, client) {
67+
await db
68+
.collection("PublishedData")
69+
.find({})
70+
.forEach(async (publishedData) => {
71+
console.log(`Updating PublishedData (Id: ${publishedData._id})`);
72+
await db.collection("PublishedData").updateOne(
73+
{ _id: publishedData._id },
74+
{
75+
$set: {
76+
affiliation: publishedData.metadata.affiliation,
77+
downloadLink: publishedData.metadata.downloadLink,
78+
scicatUser: publishedData.metadata.scicatUser,
79+
thumbnail: publishedData.metadata.thumbnail,
80+
url: publishedData.metadata.url,
81+
creator: publishedData.metadata.creators.map(
82+
(creator) => creator.name,
83+
),
84+
publisher: publishedData.metadata.publisher.name,
85+
publicationYear: publishedData.metadata.publicationYear,
86+
dataDescription: publishedData.metadata.dataDescription,
87+
resourceType: publishedData.metadata.resourceType,
88+
authors: publishedData.metadata.contributors?.map(
89+
(contributor) => contributor.name,
90+
),
91+
relatedPublications:
92+
publishedData.metadata.relatedIdentifiers?.map(
93+
(relatedIdentifier) => relatedIdentifier.relatedIdentifier,
94+
),
95+
pidArray: publishedData.datasetPids,
96+
status:
97+
publishedData.status === "registered"
98+
? "registered"
99+
: "pending_registration",
100+
},
101+
$unset: { metadata: true, datasetPids: true, pid: true },
102+
},
103+
);
104+
});
105+
},
106+
};

package-lock.json

Lines changed: 1 addition & 0 deletions
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

package.json

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -28,7 +28,7 @@
2828
"test:api": "npm run test:api:jest --maxWorkers=50% && concurrently -k -s first \"wait-on http://localhost:3000/explorer/ && npm run test:api:mocha\" \"npm run start:test\"",
2929
"test:api:jest": "dotenv -o -e test/config/.env -e test/config/.env.override -- jest --config ./test/config/jest-e2e.json --maxWorkers=50%",
3030
"test:api:mocha": "dotenv -o -e test/config/.env -e test/config/.env.override -- mocha --config ./test/config/.mocharc.json -r chai/register-should.js",
31-
"prepare:local": "docker-compose -f CI/E2E/docker-compose-local.yaml --env-file CI/E2E/.env.elastic-search up -d && cp test/config/functionalAccounts.json functionalAccounts.json && cp proposalTypes.example.json proposalTypes.json"
31+
"prepare:local": "docker-compose -f CI/E2E/docker-compose-local.yaml --env-file CI/E2E/.env.elastic-search up -d && cp test/config/functionalAccounts.json functionalAccounts.json && cp proposalTypes.example.json proposalTypes.json && cp publishedDataConfig.example.json publishedDataConfig.json"
3232
},
3333
"dependencies": {
3434
"@casl/ability": "^6.3.2",
@@ -47,6 +47,7 @@
4747
"@nestjs/swagger": "^11.1.1",
4848
"@nestjs/terminus": "^11",
4949
"@types/json-merge-patch": "^1.0.0",
50+
"@types/jsonschema": "^1.1.1",
5051
"@user-office-software/duo-logger": "^2.1.1",
5152
"@user-office-software/duo-message-broker": "^1.4.0",
5253
"ajv": "^8.12.0",

0 commit comments

Comments
 (0)