Skip to content

Commit cff11ca

Browse files
VsevolodXVsevolodX
authored
Merge pull request #210 from Exabyte-io/update/SOF-7324
update/SOF-7324 Swap materials project to materials bank
2 parents d3f8653 + 8fe2677 commit cff11ca

6 files changed

Lines changed: 152 additions & 160 deletions

‎README.ipynb‎

Lines changed: 0 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -16,7 +16,6 @@
1616
"| [Examples/Workflow](examples/workflow/) | [Quantum Espresso Workflow and Job](examples/workflow/qe_scf_calculation.ipynb) | Create Quantum Espresso workflow starting from QE input file; create and submit job; after the job is finished, download output file, and finally perform postprocessing analysis.\n",
1717
"| [Examples/Material](examples/material/) | [Get Materials by Formula](examples/material/get_materials_by_formula.ipynb) | Shows how [queries](https://docs.mat3ra.com/rest-api/query-structure/) can be made to search for materials stored on your account by their formula. In this example, we search for a system containing Si.\n",
1818
"| [Examples/Material](examples/material/) | [Create Material](examples/material/create_material.ipynb) | Gives an overview of how materials can be generated in [JSON format](https://docs.mat3ra.com/materials/data/) and uploaded to your user account. In this example, we create an FCC Si crystal and upload it.\n",
19-
"| [Examples/Material](examples/material/) | [Import Materials from Materials Project](other/materialsproject/import_materials_from_materialsproject.ipynb) | Demonstrates how materials can be imported from [Materials Project](https://materialsproject.org/about), if their Materials Project ID is known. In this example, we import monoclinic and hexagonal SiGe cells.\n",
2019
"| [Examples/Material](examples/material/) | [Import Materials from Poscar](examples/material/upload_materials_from_file_poscar.ipynb) | Provides an example of how materials can be imported directly from Poscar files (a common chemical file format best-known [for its use in VASP](https://www.vasp.at/wiki/index.php/Input)). In this example, we import the unit cell of SiGe.\n",
2120
"| [Examples/Job](examples/job/) | [Create and Submit Job](examples/job/create_and_submit_job.ipynb) | Shows how to use the Mat3ra API to [create jobs](https://docs.mat3ra.com/jobs/data/) and run them on our cluster. In this example, we run a DFT calculation to get the total energy of an FCC Si unit cell using Quantum Espresso.\n",
2221
"| [Examples/Job](examples/job/) | [Get File from Job](examples/job/get-file-from-job.ipynb) | Uses the Mat3ra API to query for a list of files produced by a job, describes the metadata assigned to each file, and ends by demonstrating how to download any remote file generated by a job to the local disk.\n",

‎README.md‎

Lines changed: 0 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -11,7 +11,6 @@ Below, we list the contents of this repository, in roughly the order that a user
1111
| [Examples/Workflow](examples/workflow/) | [Quantum Espresso Workflow and Job](examples/workflow/qe_scf_calculation.ipynb) | Create Quantum Espresso workflow starting from QE input file; create and submit job; after the job is finished, download output file, and finally perform postprocessing analysis.
1212
| [Examples/Material](examples/material/) | [Get Materials by Formula](examples/material/get_materials_by_formula.ipynb) | Shows how [queries](https://docs.mat3ra.com/rest-api/query-structure/) can be made to search for materials stored on your account by their formula. In this example, we search for a system containing Si.
1313
| [Examples/Material](examples/material/) | [Create Material](examples/material/create_material.ipynb) | Gives an overview of how materials can be generated in [JSON format](https://docs.mat3ra.com/materials/data/) and uploaded to your user account. In this example, we create an FCC Si crystal and upload it.
14-
| [Examples/Material](examples/material/) | [Import Materials from Materials Project](other/materialsproject/import_materials_from_materialsproject.ipynb) | Demonstrates how materials can be imported from [Materials Project](https://materialsproject.org/about), if their Materials Project ID is known. In this example, we import monoclinic and hexagonal SiGe cells.
1514
| [Examples/Material](examples/material/) | [Import Materials from Poscar](examples/material/upload_materials_from_file_poscar.ipynb) | Provides an example of how materials can be imported directly from Poscar files (a common chemical file format best-known [for its use in VASP](https://www.vasp.at/wiki/index.php/Input)). In this example, we import the unit cell of SiGe.
1615
| [Examples/Job](examples/job/) | [Create and Submit Job](examples/job/create_and_submit_job.ipynb) | Shows how to use the Mat3ra API to [create jobs](https://docs.mat3ra.com/jobs/data/) and run them on our cluster. In this example, we run a DFT calculation to get the total energy of an FCC Si unit cell using Quantum Espresso.
1716
| [Examples/Job](examples/job/) | [Get File from Job](examples/job/get-file-from-job.ipynb) | Uses the Mat3ra API to query for a list of files produced by a job, describes the metadata assigned to each file, and ends by demonstrating how to download any remote file generated by a job to the local disk.

‎examples/job/create_and_submit_job.ipynb‎

Lines changed: 1 addition & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -39,11 +39,9 @@
3939
"\n",
4040
"ACCOUNT_ID and AUTH_TOKEN - Authentication parameters needed for when making requests to [Mat3ra.com's API Endpoints](https://docs.mat3ra.com/rest-api/endpoints/).\n",
4141
"\n",
42-
"MATERIALS_PROJECT_API_KEY - Authentication parameter needed for when making requests to [Material Project's API](https://materialsproject.org/open)\n",
43-
"\n",
4442
"ORGANIZATION_ID - Authentication parameter needed for when working with collaborative accounts https://docs.mat3ra.com/collaboration/organizations/overview/\n",
4543
"\n",
46-
"> <span style=\"color: orange\">**NOTE**</span>: If you are running this notebook from Jupyter, the variables ACCOUNT_ID, AUTH_TOKEN, MATERIALS_PROJECT_API_KEY, and ORGANIZATION_ID should be set in the file [settings.json](../../utils/settings.json) if you need to use these variables. To obtain API token parameters, please see the following link to the documentation explaining how to get them: https://docs.mat3ra.com/accounts/ui/preferences/api/"
44+
"> <span style=\"color: orange\">**NOTE**</span>: If you are running this notebook from Jupyter, the variables ACCOUNT_ID, AUTH_TOKEN, and ORGANIZATION_ID should be set in the file [settings.json](../../utils/settings.json) if you need to use these variables. To obtain API token parameters, please see the following link to the documentation explaining how to get them: https://docs.mat3ra.com/accounts/ui/preferences/api/"
4745
]
4846
},
4947
{
@@ -62,7 +60,6 @@
6260
"# @title Authorization Form\n",
6361
"ACCOUNT_ID = \"ACCOUNT_ID\" # @param {type:\"string\"}\n",
6462
"AUTH_TOKEN = \"AUTH_TOKEN\" # @param {type:\"string\"}\n",
65-
"MATERIALS_PROJECT_API_KEY = \"MATERIALS_PROJECT_API_KEY\" # @param {type:\"string\"}\n",
6663
"ORGANIZATION_ID = \"ORGANIZATION_ID\" # @param {type:\"string\"}\n",
6764
"\n",
6865
"import os\n",
@@ -73,7 +70,6 @@
7370
" dict(\n",
7471
" ACCOUNT_ID=ACCOUNT_ID,\n",
7572
" AUTH_TOKEN=AUTH_TOKEN,\n",
76-
" MATERIALS_PROJECT_API_KEY=MATERIALS_PROJECT_API_KEY,\n",
7773
" ORGANIZATION_ID=ORGANIZATION_ID,\n",
7874
" )\n",
7975
" )\n",
@@ -87,7 +83,6 @@
8783
" dict(\n",
8884
" ACCOUNT_ID=apiConfig.get(\"accountId\"),\n",
8985
" AUTH_TOKEN=apiConfig.get(\"authToken\"),\n",
90-
" MATERIALS_PROJECT_API_KEY=MATERIALS_PROJECT_API_KEY,\n",
9186
" ORGANIZATION_ID=apiConfig.get(\"organizationId\") or \"\",\n",
9287
" )\n",
9388
" )\n",

‎examples/job/get-file-from-job.ipynb‎

Lines changed: 44 additions & 45 deletions
Original file line numberDiff line numberDiff line change
@@ -30,7 +30,7 @@
3030
"\n",
3131
"After working through this notebook, you will be able to:\n",
3232
"\n",
33-
"1. Import [the structure of Si](https://materialsproject.org/materials/mp-149/) from Materials Project\n",
33+
"1. Import [the structure of Si](https://materialsproject.org/materials/mp-149/) from Materials Bank\n",
3434
"2. Set up and run a single-point calculation using Quantum Espresso.\n",
3535
"3. List files currently in the job's directory\n",
3636
"4. Check metadata for every file (modification date, size, etc)\n",
@@ -42,7 +42,6 @@
4242
"The explanation below assumes that the reader is familiar with the concepts used in Mat3ra platform and RESTful API. We outline these below and direct the reader to the original sources of information:\n",
4343
"\n",
4444
"- [Generating RESTful API authentication parameters](../system/get_authentication_params.ipynb)\n",
45-
"- [Importing materials from materials project](../material/import_materials_from_materialsproject.ipynb)\n",
4645
"- [Creating and submitting jobs](../job/create_and_submit_job.ipynb)"
4746
]
4847
},
@@ -61,16 +60,13 @@
6160
"\n",
6261
"ACCOUNT_ID and AUTH_TOKEN - Authentication parameters needed for when making requests to [Mat3ra.com's API Endpoints](https://docs.mat3ra.com/rest-api/endpoints/).\n",
6362
"\n",
64-
"MATERIALS_PROJECT_API_KEY - Authentication parameter needed for when making requests to [Material Project's API](https://materialsproject.org/open)\n",
65-
"\n",
6663
"ORGANIZATION_ID - Authentication parameter needed for when working with collaborative accounts https://docs.mat3ra.com/collaboration/organizations/overview/\n",
6764
"\n",
6865
"> <span style=\"color: orange\">**NOTE**</span>: If you are running this notebook from Jupyter, the variables ACCOUNT_ID, AUTH_TOKEN, MATERIALS_PROJECT_API_KEY, and ORGANIZATION_ID should be set in the file [settings.json](../../utils/settings.json) if you need to use these variables. To obtain API token parameters, please see the following link to the documentation explaining how to get them: https://docs.mat3ra.com/accounts/ui/preferences/api/"
6966
]
7067
},
7168
{
7269
"cell_type": "code",
73-
"execution_count": null,
7470
"metadata": {
7571
"cellView": "form",
7672
"colab": {
@@ -79,12 +75,10 @@
7975
"id": "nXDYU_oP8GIr",
8076
"outputId": "aee909b7-1a48-43c2-81b7-3659e0fc9c79"
8177
},
82-
"outputs": [],
8378
"source": [
8479
"# @title Authorization Form\n",
8580
"ACCOUNT_ID = \"ACCOUNT_ID\" # @param {type:\"string\"}\n",
8681
"AUTH_TOKEN = \"AUTH_TOKEN\" # @param {type:\"string\"}\n",
87-
"MATERIALS_PROJECT_API_KEY = \"MATERIALS_PROJECT_API_KEY\" # @param {type:\"string\"}\n",
8882
"ORGANIZATION_ID = \"ORGANIZATION_ID\" # @param {type:\"string\"}\n",
8983
"\n",
9084
"import os\n",
@@ -95,7 +89,6 @@
9589
" dict(\n",
9690
" ACCOUNT_ID=ACCOUNT_ID,\n",
9791
" AUTH_TOKEN=AUTH_TOKEN,\n",
98-
" MATERIALS_PROJECT_API_KEY=MATERIALS_PROJECT_API_KEY,\n",
9992
" ORGANIZATION_ID=ORGANIZATION_ID,\n",
10093
" )\n",
10194
" )\n",
@@ -109,7 +102,6 @@
109102
" dict(\n",
110103
" ACCOUNT_ID=apiConfig.get(\"accountId\"),\n",
111104
" AUTH_TOKEN=apiConfig.get(\"authToken\"),\n",
112-
" MATERIALS_PROJECT_API_KEY=MATERIALS_PROJECT_API_KEY,\n",
113105
" ORGANIZATION_ID=apiConfig.get(\"organizationId\") or \"\",\n",
114106
" )\n",
115107
" )\n",
@@ -121,7 +113,9 @@
121113
" from utils.jupyterlite import install_packages\n",
122114
"\n",
123115
" await install_packages(\"\")"
124-
]
116+
],
117+
"outputs": [],
118+
"execution_count": null
125119
},
126120
{
127121
"attachments": {},
@@ -135,14 +129,12 @@
135129
},
136130
{
137131
"cell_type": "code",
138-
"execution_count": null,
139132
"metadata": {
140133
"id": "rG9HfjnR8GIs"
141134
},
142-
"outputs": [],
143135
"source": [
144136
"# Import settings file and utils file\n",
145-
"from utils.settings import ENDPOINT_ARGS, ACCOUNT_ID, MATERIALS_PROJECT_API_KEY\n",
137+
"from utils.settings import ENDPOINT_ARGS, ACCOUNT_ID\n",
146138
"from utils.generic import (\n",
147139
" wait_for_jobs_to_finish,\n",
148140
" get_property_by_subworkflow_and_unit_indicies,\n",
@@ -154,9 +146,11 @@
154146
"from exabyte_api_client.endpoints.jobs import JobEndpoints\n",
155147
"from exabyte_api_client.endpoints.projects import ProjectEndpoints\n",
156148
"from exabyte_api_client.endpoints.materials import MaterialEndpoints\n",
157-
"from exabyte_api_client.endpoints.bank_workflows import BankWorkflowEndpoints\n",
158-
"from exabyte_api_client.endpoints.properties import PropertiesEndpoints"
159-
]
149+
"from exabyte_api_client.endpoints.bank_materials import BankMaterialEndpoints\n",
150+
"from exabyte_api_client.endpoints.bank_workflows import BankWorkflowEndpoints"
151+
],
152+
"outputs": [],
153+
"execution_count": null
160154
},
161155
{
162156
"attachments": {},
@@ -173,22 +167,20 @@
173167
"\n",
174168
"The PBE functional is used in conjunction with an ultrasoft pseudopotential and a planewave basis set.\n",
175169
"\n",
176-
"The material we will investigate is elemental [Silicon](https://materialsproject.org/materials/mp-149/), as-is from Materials Project.\n",
170+
"The material we will investigate is elemental [Silicon](https://materialsproject.org/materials/mp-149/), as-is from Materials Bank.\n",
177171
"\n",
178-
"> <span style=\"color: orange\">Note</span>: This cell uses our API to copy the unit cell of silicon from Materials Project into your account. It then copies a workflow to get the total energy of a system using Quantum Espresso to your account. Finally, a job is created using the Quantum Espresso workflow for the silicon unit cell, and the job is submitted to the cluster. For more information, please refer to our [run-simulation-and-extract-properties](./run-simulations-and-extract-properties.ipynb) notebook, located in this directory."
172+
"> <span style=\"color: orange\">Note</span>: This cell uses our API to copy the unit cell of silicon from Materials Bank into your account. It then copies a workflow to get the total energy of a system using Quantum Espresso to your account. Finally, a job is created using the Quantum Espresso workflow for the silicon unit cell, and the job is submitted to the cluster. For more information, please refer to our [run-simulation-and-extract-properties](./run-simulations-and-extract-properties.ipynb) notebook, located in this directory."
179173
]
180174
},
181175
{
182176
"cell_type": "code",
183-
"execution_count": null,
184177
"metadata": {
185178
"colab": {
186179
"base_uri": "https://localhost:8080/"
187180
},
188181
"id": "le9f28c08GIt",
189182
"outputId": "27e9b94f-29dd-4742-9448-2e1bce79cb15"
190183
},
191-
"outputs": [],
192184
"source": [
193185
"# Get some account information\n",
194186
"project_endpoints = ProjectEndpoints(*ENDPOINT_ARGS)\n",
@@ -199,13 +191,18 @@
199191
"\n",
200192
"# Get a workflow for the job from the bank, and copy it to our account\n",
201193
"bank_workflow_endpoints = BankWorkflowEndpoints(*ENDPOINT_ARGS)\n",
202-
"BANK_WORKFLOW_ID = \"84DAjE9YyTFndx6z3\"\n",
203-
"workflow_id = bank_workflow_endpoints.copy(BANK_WORKFLOW_ID, owner_id)[\"_id\"]\n",
204-
"\n",
205-
"# Get materials for the job\n",
194+
"WORKFLOW_QUERY = {\"systemName\": \"espresso-total-energy\"}\n",
195+
"bank_workflow_id = bank_workflow_endpoints.list(WORKFLOW_QUERY)[0][\"_id\"]\n",
196+
"workflow_id = bank_workflow_endpoints.copy(bank_workflow_id, owner_id)[\"_id\"]\n",
197+
"\n",
198+
"# Get materials from bank and copy one to our account\n",
199+
"material_bank_endpoints = BankMaterialEndpoints(*ENDPOINT_ARGS)\n",
200+
"MATERIAL_QUERY = {\"formula\": \"Si\"}\n",
201+
"bank_materials = material_bank_endpoints.list(MATERIAL_QUERY)\n",
202+
"bank_material_id = bank_materials[0][\"_id\"]\n",
206203
"material_endpoints = MaterialEndpoints(*ENDPOINT_ARGS)\n",
207-
"material_project_id = [\"mp-149\"] # The importer expects a list\n",
208-
"materials = material_endpoints.import_from_materialsproject(MATERIALS_PROJECT_API_KEY, material_project_id, owner_id)\n",
204+
"material_id = material_bank_endpoints.copy(bank_material_id, owner_id)[\"_id\"]\n",
205+
"materials = material_endpoints.list({\"owner._id\": owner_id})\n",
209206
"\n",
210207
"# Create the job\n",
211208
"job_endpoints = JobEndpoints(*ENDPOINT_ARGS)\n",
@@ -216,7 +213,9 @@
216213
"# Submit the job\n",
217214
"job_endpoints.submit(job[\"_id\"])\n",
218215
"wait_for_jobs_to_finish(job_endpoints, [job[\"_id\"]])"
219-
]
216+
],
217+
"outputs": [],
218+
"execution_count": null
220219
},
221220
{
222221
"attachments": {},
@@ -243,22 +242,22 @@
243242
},
244243
{
245244
"cell_type": "code",
246-
"execution_count": null,
247245
"metadata": {
248246
"colab": {
249247
"base_uri": "https://localhost:8080/"
250248
},
251249
"id": "yijRe7qp8GIv",
252250
"outputId": "19b147fc-736e-41f0-bd0f-6f69e77b0adf"
253251
},
254-
"outputs": [],
255252
"source": [
256253
"files = job_endpoints.list_files(job[\"_id\"])\n",
257254
"paths = [file[\"key\"] for file in files]\n",
258255
"for path in paths:\n",
259256
" if \"outdir\" not in path:\n",
260257
" print(path)"
261-
]
258+
],
259+
"outputs": [],
260+
"execution_count": null
262261
},
263262
{
264263
"attachments": {},
@@ -283,21 +282,21 @@
283282
},
284283
{
285284
"cell_type": "code",
286-
"execution_count": null,
287285
"metadata": {
288286
"colab": {
289287
"base_uri": "https://localhost:8080/"
290288
},
291289
"id": "U2dq0A1e8GIw",
292290
"outputId": "7b4f8c36-6342-46fa-d02e-0825b44e6c64"
293291
},
294-
"outputs": [],
295292
"source": [
296293
"for file in files:\n",
297294
" if file[\"name\"] == \"pw_scf.out\":\n",
298295
" output_file_metadata = file\n",
299296
"display_JSON(output_file_metadata)"
300-
]
297+
],
298+
"outputs": [],
299+
"execution_count": null
301300
},
302301
{
303302
"attachments": {},
@@ -313,15 +312,13 @@
313312
},
314313
{
315314
"cell_type": "code",
316-
"execution_count": null,
317315
"metadata": {
318316
"colab": {
319317
"base_uri": "https://localhost:8080/"
320318
},
321319
"id": "OpdOMuDT8GIw",
322320
"outputId": "e4e2e6b1-8431-439a-8606-ae500534e601"
323321
},
324-
"outputs": [],
325322
"source": [
326323
"import urllib\n",
327324
"\n",
@@ -336,7 +333,9 @@
336333
"lines = output_file.split(\"\\n\")\n",
337334
"for line in lines[-90:]:\n",
338335
" print(line)"
339-
]
336+
],
337+
"outputs": [],
338+
"execution_count": null
340339
},
341340
{
342341
"attachments": {},
@@ -352,46 +351,46 @@
352351
},
353352
{
354353
"cell_type": "code",
355-
"execution_count": null,
356354
"metadata": {
357355
"id": "osy2JKpB8GIx"
358356
},
359-
"outputs": [],
360357
"source": [
361358
"# We've already got an output file, so let's grab the input file we sent to Quantum Espresso\n",
362359
"for file in files:\n",
363360
" if \"pw_scf.in\" == file[\"name\"]:\n",
364361
" input_file_metadata = file\n",
365362
"server_response = urllib.request.urlopen(input_file_metadata[\"signedUrl\"])\n",
366363
"input_file_bytes = server_response.read()"
367-
]
364+
],
365+
"outputs": [],
366+
"execution_count": null
368367
},
369368
{
370369
"cell_type": "code",
371-
"execution_count": null,
372370
"metadata": {
373371
"id": "COPinBrO8GIx"
374372
},
375-
"outputs": [],
376373
"source": [
377374
"# Let's write the input file to disk. Note that we get files as a bytes string from the server, which is convenient for binaries, images, and other non-human-readable data.\n",
378375
"# Although we could decode before writing to disk, we can just write it directly with the \"wb\" (write bytes) file mode.\n",
379376
"with open(input_file_metadata[\"name\"], \"wb\") as file_descriptor:\n",
380377
" file_descriptor.write(input_file_bytes)"
381-
]
378+
],
379+
"outputs": [],
380+
"execution_count": null
382381
},
383382
{
384383
"cell_type": "code",
385-
"execution_count": null,
386384
"metadata": {
387385
"id": "rzDcWyrL8GIx"
388386
},
389-
"outputs": [],
390387
"source": [
391388
"# Now, let's write our output file to the disk. Note that because we already decoded it, we can just use the 'w' file mode.\n",
392389
"with open(output_file_metadata[\"name\"], \"w\") as file_descriptor:\n",
393390
" file_descriptor.write(output_file)"
394-
]
391+
],
392+
"outputs": [],
393+
"execution_count": null
395394
}
396395
],
397396
"metadata": {

0 commit comments

Comments
 (0)