Skip to content
This repository was archived by the owner on Dec 8, 2023. It is now read-only.

Commit 8907b33

Browse files
kabilarkabilar
authored
Merge pull request #80 from CBroz1/main
2 parents 9ba5edd + e054992 commit 8907b33

File tree

4 files changed

+318
-61
lines changed

4 files changed

+318
-61
lines changed

‎CHANGELOG.md

+3-2
Original file line numberDiff line numberDiff line change
@@ -3,9 +3,9 @@
33
Observes [Semantic Versioning](https://semver.org/spec/v2.0.0.html) standard and
44
[Keep a Changelog](https://keepachangelog.com/en/1.0.0/) convention.
55

6-
## [0.2.3] - 2022-09-20
6+
## [0.2.4] - 2022-09-20
77

8-
+ Add - Notebook for Allen Institute workshop
8+
+ Add - Add (0.2.3) and revise (0.2.4) notebook for Allen Institute workshop
99

1010
## [0.2.2] - 2022-09-18
1111

@@ -64,6 +64,7 @@ specific forks for tests.
6464

6565
+ Add - Version
6666

67+
[0.2.4]: https://github.com/datajoint/workflow-array-ephys/releases/tag/0.2.4
6768
[0.2.3]: https://github.com/datajoint/workflow-array-ephys/releases/tag/0.2.3
6869
[0.2.2]: https://github.com/datajoint/workflow-array-ephys/releases/tag/0.2.2
6970
[0.2.1]: https://github.com/datajoint/workflow-array-ephys/releases/tag/0.2.1

‎notebooks/2022-allen-institute-workshop.ipynb

+199-32
Original file line numberDiff line numberDiff line change
@@ -7,13 +7,27 @@
77
"source": [
88
"# Allen Institute Ephys Workshop\n",
99
"September 22, 2022\n",
10+
"\n",
1011
"+ In this notebook, we will show how to interact with a database in Python and how export data into a Neurodata Without Borders (NWB) file.\n",
1112
"\n",
1213
"+ Other notebooks in this directory describe the process for running the analysis steps in more detail.\n",
1314
"\n",
1415
"+ This notebook is meant to be run on CodeBook (`https://codebook.datajoint.io`) which contains example data.\n",
1516
"\n",
16-
"+ First run the `01-configure` and `04-automate` notebooks to set up your environment and load example data into the database, respectively."
17+
"First, some packages we'll use in this notebook..."
18+
]
19+
},
20+
{
21+
"cell_type": "code",
22+
"execution_count": null,
23+
"id": "e1e0295c-57bf-49d2-9a1e-931d0e53264b",
24+
"metadata": {},
25+
"outputs": [],
26+
"source": [
27+
"import datajoint as dj\n",
28+
"import numpy as np\n",
29+
"from matplotlib import pyplot\n",
30+
"import getpass"
1731
]
1832
},
1933
{
@@ -25,15 +39,11 @@
2539
]
2640
},
2741
{
28-
"cell_type": "code",
29-
"execution_count": null,
30-
"id": "e1e0295c-57bf-49d2-9a1e-931d0e53264b",
42+
"cell_type": "markdown",
43+
"id": "cb03797f",
3144
"metadata": {},
32-
"outputs": [],
3345
"source": [
34-
"import datajoint as dj\n",
35-
"import numpy as np\n",
36-
"from matplotlib import pyplot"
46+
"These steps are taken from [01-configure](01-configure.ipynb). If you've already saved a config file, you can skip to the next section."
3747
]
3848
},
3949
{
@@ -53,22 +63,178 @@
5363
},
5464
"outputs": [],
5565
"source": [
56-
"dj.config['custom'] = {'database.prefix': '<username>_allen_ephys_',\n",
57-
" 'ephys_root_data_dir': [\"/tmp/test_data/workflow_ephys_data1/\",\n",
58-
" \"/tmp/test_data/workflow_ephys_data2/\",\n",
59-
" \"/tmp/test_data/workflow_localization/\", \n",
60-
" \"/home/inbox/0.1.0a4/workflow_ephys_data1/\",\n",
61-
" \"/home/inbox/0.1.0a4/workflow_ephys_data2/\",\n",
62-
" \"/home/inbox/0.1.0a4/workflow_localization/\"\n",
63-
" ]}"
66+
"username_as_prefix = dj.config[\"database.user\"] + \"_ephys_\"\n",
67+
"dj.config['custom'] = {\n",
68+
" 'database.prefix': username_as_prefix,\n",
69+
" 'ephys_root_data_dir': [\n",
70+
" \"/home/inbox/0.1.0a4/workflow_ephys_data1/\",\n",
71+
" \"/home/inbox/0.1.0a4/workflow_ephys_data2/\",\n",
72+
" \"/home/inbox/0.1.0a4/workflow_localization/\"\n",
73+
" ],\n",
74+
" \"ephys_mode\": \"no-curation\"\n",
75+
"}"
76+
]
77+
},
78+
{
79+
"cell_type": "markdown",
80+
"id": "21a69c02",
81+
"metadata": {},
82+
"source": [
83+
"Next, we'll use a prompt to securely save your password."
84+
]
85+
},
86+
{
87+
"cell_type": "code",
88+
"execution_count": null,
89+
"id": "66bea7cb",
90+
"metadata": {},
91+
"outputs": [],
92+
"source": [
93+
"dj.config[\"database.password\"] = getpass.getpass()"
94+
]
95+
},
96+
{
97+
"cell_type": "markdown",
98+
"id": "c45a3aad",
99+
"metadata": {},
100+
"source": [
101+
"Now to save these credentials."
102+
]
103+
},
104+
{
105+
"cell_type": "code",
106+
"execution_count": null,
107+
"id": "4e4f0448",
108+
"metadata": {},
109+
"outputs": [],
110+
"source": [
111+
"dj.config.save_global()"
112+
]
113+
},
114+
{
115+
"cell_type": "markdown",
116+
"id": "a570822d",
117+
"metadata": {},
118+
"source": [
119+
"## Populating the database"
120+
]
121+
},
122+
{
123+
"cell_type": "markdown",
124+
"id": "8ee61c1d",
125+
"metadata": {},
126+
"source": [
127+
"Next, we'll populate these schema using steps from [04-automate](04-automate-optional.ipynb). If your schema are already populated, you can skip this step. For more details on each of these steps, please visit [that notebook](04-automate-optional.ipynb)."
128+
]
129+
},
130+
{
131+
"cell_type": "code",
132+
"execution_count": null,
133+
"id": "2a6388cc",
134+
"metadata": {},
135+
"outputs": [],
136+
"source": [
137+
"from workflow_array_ephys.pipeline import session, ephys # import schemas\n",
138+
"from workflow_array_ephys.ingest import ingest_subjects, ingest_sessions # csv loaders\n",
139+
"\n",
140+
"ingest_subjects(subject_csv_path=\"/home/user_data/subjects.csv\")\n",
141+
"ingest_sessions(session_csv_path=\"/home/user_data/sessions.csv\")\n",
142+
"\n",
143+
"params_ks = {\n",
144+
" \"fs\": 30000,\n",
145+
" \"fshigh\": 150,\n",
146+
" \"minfr_goodchannels\": 0.1,\n",
147+
" \"Th\": [10, 4],\n",
148+
" \"lam\": 10,\n",
149+
" \"AUCsplit\": 0.9,\n",
150+
" \"minFR\": 0.02,\n",
151+
" \"momentum\": [20, 400],\n",
152+
" \"sigmaMask\": 30,\n",
153+
" \"ThPr\": 8,\n",
154+
" \"spkTh\": -6,\n",
155+
" \"reorder\": 1,\n",
156+
" \"nskip\": 25,\n",
157+
" \"GPU\": 1,\n",
158+
" \"Nfilt\": 1024,\n",
159+
" \"nfilt_factor\": 4,\n",
160+
" \"ntbuff\": 64,\n",
161+
" \"whiteningRange\": 32,\n",
162+
" \"nSkipCov\": 25,\n",
163+
" \"scaleproc\": 200,\n",
164+
" \"nPCs\": 3,\n",
165+
" \"useRAM\": 0,\n",
166+
"}\n",
167+
"ephys.ClusteringParamSet.insert_new_params(\n",
168+
" clustering_method=\"kilosort2\",\n",
169+
" paramset_idx=0,\n",
170+
" params=params_ks,\n",
171+
" paramset_desc=\"Spike sorting using Kilosort2\",\n",
172+
")\n",
173+
"session_key = (session.Session & 'subject=\"subject6\"').fetch1(\"KEY\")\n",
174+
"ephys.ProbeInsertion.auto_generate_entries(session_key)"
175+
]
176+
},
177+
{
178+
"cell_type": "markdown",
179+
"id": "49d60c21",
180+
"metadata": {},
181+
"source": [
182+
"Next, we'll trigger the relevant `populate` commands."
183+
]
184+
},
185+
{
186+
"cell_type": "code",
187+
"execution_count": null,
188+
"id": "dbbe978e",
189+
"metadata": {},
190+
"outputs": [],
191+
"source": [
192+
"populate_settings = {\"display_progress\": True}\n",
193+
"ephys.EphysRecording.populate(**populate_settings)\n",
194+
"ephys.LFP.populate(**populate_settings)\n",
195+
"ephys.ClusteringTask.insert1(\n",
196+
" dict(\n",
197+
" session_key,\n",
198+
" insertion_number=0,\n",
199+
" paramset_idx=0,\n",
200+
" clustering_output_dir=\"subject6/session1/towersTask_g0_imec0\",\n",
201+
" ),\n",
202+
" skip_duplicates=True,\n",
203+
")\n",
204+
"ephys.Clustering.populate(**populate_settings)\n",
205+
"ephys.CuratedClustering.populate(**populate_settings)\n",
206+
"# ephys.WaveformSet.populate(**populate_settings) # Time-consuming process"
207+
]
208+
},
209+
{
210+
"cell_type": "markdown",
211+
"id": "c1c485bb",
212+
"metadata": {},
213+
"source": [
214+
"**Notes:** \n",
215+
"+ `ephys.WaveformSet.populate` takes significant time to populate with current CodeBook hardware allocations. The output will not be used directly in this notebook.\n",
216+
"+ The `process` script runs through all `populate` commands in order and could be used instead of the commands above. It could be used as follows\n",
217+
"```python\n",
218+
"from workflow_array_ephys import process; process.run(display_progress=True)\n",
219+
"```"
220+
]
221+
},
222+
{
223+
"cell_type": "markdown",
224+
"id": "07747574",
225+
"metadata": {},
226+
"source": [
227+
"## Exploring the workflow"
64228
]
65229
},
66230
{
67231
"cell_type": "markdown",
68232
"id": "7c545770",
69233
"metadata": {},
70234
"source": [
71-
"Import the workflow. The current workflow is composed of multiple database schemas, each of them corresponding to a module within the `workflow_array_ephys.pipeline` file."
235+
"### Import the workflow\n",
236+
"\n",
237+
"The current workflow is composed of multiple database schemas, each of them corresponding to a module within the `workflow_array_ephys.pipeline` file."
72238
]
73239
},
74240
{
@@ -86,9 +252,9 @@
86252
"id": "2bd6d86b",
87253
"metadata": {},
88254
"source": [
89-
"## Workflow diagram\n",
255+
"### Diagrams and table design\n",
90256
"\n",
91-
"Plot the workflow diagram. In relational databases, the entities (i.e. rows) in different tables are connected to each other. Visualization of this relationship helps one to write accurate queries. For the array ephys workflow, this connection is as follows:"
257+
"We can plot the workflow diagram. In relational databases, the entities (i.e. rows) in different tables are connected to each other. Visualization of this relationship helps one to write accurate queries. For the array ephys workflow, this connection is as follows:"
92258
]
93259
},
94260
{
@@ -139,7 +305,9 @@
139305
"id": "5ca53cb7",
140306
"metadata": {},
141307
"source": [
142-
"## Fetch data from the database and generate a raster plot"
308+
"### Fetch data\n",
309+
"\n",
310+
"Here, we fetch data from the database and generate a raster plot"
143311
]
144312
},
145313
{
@@ -190,16 +358,16 @@
190358
"y = np.hstack([np.full_like(s, u) for u, s in zip(units, unit_spiketimes)])\n",
191359
"\n",
192360
"pyplot.plot(x, y, \"|\")\n",
193-
"pyplot.set_xlabel(\"Time (s)\")\n",
194-
"pyplot.set_ylabel(\"Unit\")"
361+
"pyplot.xlabel(\"Time (s)\")\n",
362+
"pyplot.ylabel(\"Unit\")"
195363
]
196364
},
197365
{
198366
"cell_type": "markdown",
199367
"id": "68559c95-3c4e-4c6a-acbb-c06796a8399c",
200368
"metadata": {},
201369
"source": [
202-
"## Export to NWB\n",
370+
"### Export to NWB\n",
203371
"\n",
204372
"The Element's `ecephys_session_to_nwb` function provides a full export mechanism, returning an NWB file with raw data, spikes, and LFP. Optional arguments determine which pieces are exported. For demonstration purposes, we recommend limiting `end_frame`."
205373
]
@@ -229,7 +397,9 @@
229397
"id": "236ab592",
230398
"metadata": {},
231399
"source": [
232-
"Select an experimental session to export."
400+
"Note that a subset of arguments (`lab_key`, `project_key`, and `protocol_key`) take keys from upstream Elements. To populate this information, see [09-NWB-Export](09-NWB-Export.ipynb).\n",
401+
"\n",
402+
"Next, select an experimental session to export."
233403
]
234404
},
235405
{
@@ -271,12 +441,9 @@
271441
"source": [
272442
"nwbfile = ecephys_session_to_nwb(session_key=session_key,\n",
273443
" raw=True,\n",
274-
" spikes=True,\n",
444+
" spikes=False, # True requires WaveformSet.populate()\n",
275445
" lfp=\"dj\",\n",
276446
" end_frame=100,\n",
277-
" lab_key=dict(lab='LabA'),\n",
278-
" project_key=dict(project='ProjA'),\n",
279-
" protocol_key=dict(protocol='ProtA'),\n",
280447
" nwbfile_kwargs=None)"
281448
]
282449
},
@@ -316,7 +483,7 @@
316483
"id": "42e7664f",
317484
"metadata": {},
318485
"source": [
319-
"Next, the NWB file can be uploaded to DANDI. See the `09-NWB-Export` notebook for more details."
486+
"Next, the NWB file can be uploaded to DANDI. See the [09-NWB-Export](09-NWB-Export.ipynb) notebook for more details."
320487
]
321488
},
322489
{
@@ -335,7 +502,7 @@
335502
"formats": "ipynb,py"
336503
},
337504
"kernelspec": {
338-
"display_name": "Python 3.7.9 ('workflow-calcium-imaging')",
505+
"display_name": "Python 3.9.13 ('ele')",
339506
"language": "python",
340507
"name": "python3"
341508
},
@@ -349,11 +516,11 @@
349516
"name": "python",
350517
"nbconvert_exporter": "python",
351518
"pygments_lexer": "ipython3",
352-
"version": "3.7.9"
519+
"version": "3.9.13"
353520
},
354521
"vscode": {
355522
"interpreter": {
356-
"hash": "2da0f701b958ef27dbb9ddfd0ee0eb0d75664c3cb2b4bc67fd0780a51e2b3863"
523+
"hash": "d00c4ad21a7027bf1726d6ae3a9a6ef39c8838928eca5a3d5f51f3eb68720410"
357524
}
358525
}
359526
},

0 commit comments

Comments
 (0)