Skip to content
This repository was archived by the owner on Dec 8, 2023. It is now read-only.

Commit 9ba5edd

Browse files
CBroz1CBroz1
authored
Merge pull request #79 from kabilar/main
Add notebook for Allen Institute workshop
2 parents 3711aec + 0e073ae commit 9ba5edd

File tree

4 files changed

+500
-3
lines changed

4 files changed

+500
-3
lines changed

‎CHANGELOG.md

Lines changed: 10 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -3,12 +3,18 @@
33
Observes [Semantic Versioning](https://semver.org/spec/v2.0.0.html) standard and
44
[Keep a Changelog](https://keepachangelog.com/en/1.0.0/) convention.
55

6+
## [0.2.3] - 2022-09-20
7+
8+
+ Add - Notebook for Allen Institute workshop
9+
610
## [0.2.2] - 2022-09-18
11+
712
+ Update - config ephys roots in notebooks for codebook deployment
813
+ Update - requirements.txt to add element electrode localization
914
+ Update - add lab.User insert to ingest.py for users with empty element-lab schemas
1015

1116
## [0.2.1] - 2022-07-22
17+
1218
+ Add - Mention CodeBook data directory in notebooks
1319
+ Add - Mention of config items in notebook 09
1420
+ Update - Remove directory assertion in notebooks for CodeBook deployment
@@ -58,6 +64,8 @@ specific forks for tests.
5864

5965
+ Add - Version
6066

61-
[0.2.1]: https://github.com/datajoint/workflow-array-ephys/compare/0.2.1...0.2.0
62-
[0.2.0]: https://github.com/datajoint/workflow-array-ephys/compare/0.2.0...0.1.0
67+
[0.2.3]: https://github.com/datajoint/workflow-array-ephys/releases/tag/0.2.3
68+
[0.2.2]: https://github.com/datajoint/workflow-array-ephys/releases/tag/0.2.2
69+
[0.2.1]: https://github.com/datajoint/workflow-array-ephys/releases/tag/0.2.1
70+
[0.2.0]: https://github.com/datajoint/workflow-array-ephys/releases/tag/0.2.0
6371
[0.1.0]: https://github.com/datajoint/workflow-array-ephys/releases/tag/0.1.0

‎notebooks/2022-allen-institute-workshop.ipynb

Lines changed: 362 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,362 @@
1+
{
2+
"cells": [
3+
{
4+
"cell_type": "markdown",
5+
"id": "11e51a9c",
6+
"metadata": {},
7+
"source": [
8+
"# Allen Institute Ephys Workshop\n",
9+
"September 22, 2022\n",
10+
"+ In this notebook, we will show how to interact with a database in Python and how export data into a Neurodata Without Borders (NWB) file.\n",
11+
"\n",
12+
"+ Other notebooks in this directory describe the process for running the analysis steps in more detail.\n",
13+
"\n",
14+
"+ This notebook is meant to be run on CodeBook (`https://codebook.datajoint.io`) which contains example data.\n",
15+
"\n",
16+
"+ First run the `01-configure` and `04-automate` notebooks to set up your environment and load example data into the database, respectively."
17+
]
18+
},
19+
{
20+
"cell_type": "markdown",
21+
"id": "14a6ba1d",
22+
"metadata": {},
23+
"source": [
24+
"## Configuration"
25+
]
26+
},
27+
{
28+
"cell_type": "code",
29+
"execution_count": null,
30+
"id": "e1e0295c-57bf-49d2-9a1e-931d0e53264b",
31+
"metadata": {},
32+
"outputs": [],
33+
"source": [
34+
"import datajoint as dj\n",
35+
"import numpy as np\n",
36+
"from matplotlib import pyplot"
37+
]
38+
},
39+
{
40+
"cell_type": "markdown",
41+
"id": "e652916f",
42+
"metadata": {},
43+
"source": [
44+
"Enter database credentials. A DataJoint workflow requires a connection to an existing relational database. The connection setup parameters are defined in the `dj.config` python dictionary."
45+
]
46+
},
47+
{
48+
"cell_type": "code",
49+
"execution_count": null,
50+
"id": "fdc35bd8-2381-4769-9f43-33a7f3fd2332",
51+
"metadata": {
52+
"tags": []
53+
},
54+
"outputs": [],
55+
"source": [
56+
"dj.config['custom'] = {'database.prefix': '<username>_allen_ephys_',\n",
57+
" 'ephys_root_data_dir': [\"/tmp/test_data/workflow_ephys_data1/\",\n",
58+
" \"/tmp/test_data/workflow_ephys_data2/\",\n",
59+
" \"/tmp/test_data/workflow_localization/\", \n",
60+
" \"/home/inbox/0.1.0a4/workflow_ephys_data1/\",\n",
61+
" \"/home/inbox/0.1.0a4/workflow_ephys_data2/\",\n",
62+
" \"/home/inbox/0.1.0a4/workflow_localization/\"\n",
63+
" ]}"
64+
]
65+
},
66+
{
67+
"cell_type": "markdown",
68+
"id": "7c545770",
69+
"metadata": {},
70+
"source": [
71+
"Import the workflow. The current workflow is composed of multiple database schemas, each of them corresponding to a module within the `workflow_array_ephys.pipeline` file."
72+
]
73+
},
74+
{
75+
"cell_type": "code",
76+
"execution_count": null,
77+
"id": "ee08b24f",
78+
"metadata": {},
79+
"outputs": [],
80+
"source": [
81+
"from workflow_array_ephys.pipeline import lab, subject, session, probe, ephys"
82+
]
83+
},
84+
{
85+
"cell_type": "markdown",
86+
"id": "2bd6d86b",
87+
"metadata": {},
88+
"source": [
89+
"## Workflow diagram\n",
90+
"\n",
91+
"Plot the workflow diagram. In relational databases, the entities (i.e. rows) in different tables are connected to each other. Visualization of this relationship helps one to write accurate queries. For the array ephys workflow, this connection is as follows:"
92+
]
93+
},
94+
{
95+
"cell_type": "code",
96+
"execution_count": null,
97+
"id": "8dd6d3ee",
98+
"metadata": {
99+
"tags": []
100+
},
101+
"outputs": [],
102+
"source": [
103+
"dj.Diagram(lab.Lab) + dj.Diagram(subject.Subject) + dj.Diagram(session.Session) + \\\n",
104+
"dj.Diagram(probe) + dj.Diagram(ephys)"
105+
]
106+
},
107+
{
108+
"cell_type": "code",
109+
"execution_count": null,
110+
"id": "1caa7ced-93f6-4182-aa57-61bfbf961d38",
111+
"metadata": {},
112+
"outputs": [],
113+
"source": [
114+
"subject.Subject()"
115+
]
116+
},
117+
{
118+
"cell_type": "code",
119+
"execution_count": null,
120+
"id": "a110dca3-9149-40dd-8776-5b61476ccda1",
121+
"metadata": {},
122+
"outputs": [],
123+
"source": [
124+
"ephys.EphysRecording()"
125+
]
126+
},
127+
{
128+
"cell_type": "code",
129+
"execution_count": null,
130+
"id": "9a4ea243",
131+
"metadata": {},
132+
"outputs": [],
133+
"source": [
134+
"ephys.CuratedClustering.Unit()"
135+
]
136+
},
137+
{
138+
"cell_type": "markdown",
139+
"id": "5ca53cb7",
140+
"metadata": {},
141+
"source": [
142+
"## Fetch data from the database and generate a raster plot"
143+
]
144+
},
145+
{
146+
"cell_type": "code",
147+
"execution_count": null,
148+
"id": "955af65d",
149+
"metadata": {},
150+
"outputs": [],
151+
"source": [
152+
"subset=ephys.CuratedClustering.Unit & 'unit IN (\"6\",\"7\",\"9\",\"14\",\"15\",\"17\",\"19\")'\n",
153+
"subset"
154+
]
155+
},
156+
{
157+
"cell_type": "markdown",
158+
"id": "509d5335",
159+
"metadata": {},
160+
"source": [
161+
"Fetch the spike times from the database for the units above."
162+
]
163+
},
164+
{
165+
"cell_type": "code",
166+
"execution_count": null,
167+
"id": "96c1a6cd",
168+
"metadata": {},
169+
"outputs": [],
170+
"source": [
171+
"units, unit_spiketimes = (subset).fetch(\"unit\", \"spike_times\")"
172+
]
173+
},
174+
{
175+
"cell_type": "markdown",
176+
"id": "a9bc7039",
177+
"metadata": {},
178+
"source": [
179+
"Generate the raster plot."
180+
]
181+
},
182+
{
183+
"cell_type": "code",
184+
"execution_count": null,
185+
"id": "a9a2cd6a",
186+
"metadata": {},
187+
"outputs": [],
188+
"source": [
189+
"x = np.hstack(unit_spiketimes)\n",
190+
"y = np.hstack([np.full_like(s, u) for u, s in zip(units, unit_spiketimes)])\n",
191+
"\n",
192+
"pyplot.plot(x, y, \"|\")\n",
193+
"pyplot.set_xlabel(\"Time (s)\")\n",
194+
"pyplot.set_ylabel(\"Unit\")"
195+
]
196+
},
197+
{
198+
"cell_type": "markdown",
199+
"id": "68559c95-3c4e-4c6a-acbb-c06796a8399c",
200+
"metadata": {},
201+
"source": [
202+
"## Export to NWB\n",
203+
"\n",
204+
"The Element's `ecephys_session_to_nwb` function provides a full export mechanism, returning an NWB file with raw data, spikes, and LFP. Optional arguments determine which pieces are exported. For demonstration purposes, we recommend limiting `end_frame`."
205+
]
206+
},
207+
{
208+
"cell_type": "code",
209+
"execution_count": null,
210+
"id": "917eff20",
211+
"metadata": {},
212+
"outputs": [],
213+
"source": [
214+
"from workflow_array_ephys.export import ecephys_session_to_nwb, write_nwb"
215+
]
216+
},
217+
{
218+
"cell_type": "code",
219+
"execution_count": null,
220+
"id": "6b01dfad",
221+
"metadata": {},
222+
"outputs": [],
223+
"source": [
224+
"help(ecephys_session_to_nwb)"
225+
]
226+
},
227+
{
228+
"cell_type": "markdown",
229+
"id": "236ab592",
230+
"metadata": {},
231+
"source": [
232+
"Select an experimental session to export."
233+
]
234+
},
235+
{
236+
"cell_type": "code",
237+
"execution_count": null,
238+
"id": "552e03ae",
239+
"metadata": {},
240+
"outputs": [],
241+
"source": [
242+
"dj.Diagram(subject.Subject) + dj.Diagram(session.Session) + \\\n",
243+
"dj.Diagram(probe) + dj.Diagram(ephys)"
244+
]
245+
},
246+
{
247+
"cell_type": "code",
248+
"execution_count": null,
249+
"id": "e7a45887",
250+
"metadata": {},
251+
"outputs": [],
252+
"source": [
253+
"session_key=dict(subject=\"subject5\",\n",
254+
" session_datetime=\"2018-07-03 20:32:28\")"
255+
]
256+
},
257+
{
258+
"cell_type": "markdown",
259+
"id": "8d6591a6",
260+
"metadata": {},
261+
"source": [
262+
"Return the NWBFile object for the selected experimental session."
263+
]
264+
},
265+
{
266+
"cell_type": "code",
267+
"execution_count": null,
268+
"id": "5fb498b3",
269+
"metadata": {},
270+
"outputs": [],
271+
"source": [
272+
"nwbfile = ecephys_session_to_nwb(session_key=session_key,\n",
273+
" raw=True,\n",
274+
" spikes=True,\n",
275+
" lfp=\"dj\",\n",
276+
" end_frame=100,\n",
277+
" lab_key=dict(lab='LabA'),\n",
278+
" project_key=dict(project='ProjA'),\n",
279+
" protocol_key=dict(protocol='ProtA'),\n",
280+
" nwbfile_kwargs=None)"
281+
]
282+
},
283+
{
284+
"cell_type": "code",
285+
"execution_count": null,
286+
"id": "ec0841aa",
287+
"metadata": {},
288+
"outputs": [],
289+
"source": [
290+
"nwbfile"
291+
]
292+
},
293+
{
294+
"cell_type": "markdown",
295+
"id": "6836ae36",
296+
"metadata": {},
297+
"source": [
298+
"`write_nwb` can then be used to write this file to disk."
299+
]
300+
},
301+
{
302+
"cell_type": "code",
303+
"execution_count": null,
304+
"id": "637b9ec1",
305+
"metadata": {},
306+
"outputs": [],
307+
"source": [
308+
"import time\n",
309+
"nwb_filename = f\"/home/{dj.config['database.user']}/\"+time.strftime(\"_test_%Y%m%d-%H%M%S.nwb\")\n",
310+
"\n",
311+
"write_nwb(nwbfile, nwb_filename)"
312+
]
313+
},
314+
{
315+
"cell_type": "markdown",
316+
"id": "42e7664f",
317+
"metadata": {},
318+
"source": [
319+
"Next, the NWB file can be uploaded to DANDI. See the `09-NWB-Export` notebook for more details."
320+
]
321+
},
322+
{
323+
"cell_type": "markdown",
324+
"id": "4c8087a9",
325+
"metadata": {},
326+
"source": [
327+
"## Summary and next steps\n",
328+
"\n",
329+
"In this notebook we explored how to query and fetch data from the database, and export an experimental ephys session to a NWB file. Next, please explore more of the features of the DataJoint Elements in the other notebooks. Once you are ready to begin setting up your pipeline, fork this repository on GitHub and begin adapting it for your projects requirements."
330+
]
331+
}
332+
],
333+
"metadata": {
334+
"jupytext": {
335+
"formats": "ipynb,py"
336+
},
337+
"kernelspec": {
338+
"display_name": "Python 3.7.9 ('workflow-calcium-imaging')",
339+
"language": "python",
340+
"name": "python3"
341+
},
342+
"language_info": {
343+
"codemirror_mode": {
344+
"name": "ipython",
345+
"version": 3
346+
},
347+
"file_extension": ".py",
348+
"mimetype": "text/x-python",
349+
"name": "python",
350+
"nbconvert_exporter": "python",
351+
"pygments_lexer": "ipython3",
352+
"version": "3.7.9"
353+
},
354+
"vscode": {
355+
"interpreter": {
356+
"hash": "2da0f701b958ef27dbb9ddfd0ee0eb0d75664c3cb2b4bc67fd0780a51e2b3863"
357+
}
358+
}
359+
},
360+
"nbformat": 4,
361+
"nbformat_minor": 5
362+
}

0 commit comments

Comments
 (0)