Skip to content
This repository was archived by the owner on Dec 8, 2023. It is now read-only.

Commit 374b016

Browse files
authored
Merge pull request #72 from kabilar/main
Minor fixes and update changelog for release `0.2.0`
2 parents 020ce9a + 57b540b commit 374b016

27 files changed

+842
-996
lines changed

CHANGELOG.md

+24-3
Original file line numberDiff line numberDiff line change
@@ -1,9 +1,27 @@
11
# Changelog
22

3-
Observes [Semantic Versioning](https://semver.org/spec/v2.0.0.html) standard and
3+
Observes [Semantic Versioning](https://semver.org/spec/v2.0.0.html) standard and
44
[Keep a Changelog](https://keepachangelog.com/en/1.0.0/) convention.
55

6-
## 0.1.0 - 2022-06-02
6+
## [0.2.0] - 2022-07-08
7+
8+
+ Add - Adopt black formatting into code base
9+
+ Add - GitHub Issue templates
10+
+ Add - Code of conduct
11+
+ Add - Jupytext paired python scripts
12+
+ Add - Example data for trials, events, blocks, recordings, and alignments
13+
+ Add - Analysis schema
14+
+ Add - Raster and PSTH plots
15+
+ Add - Video tutorial link to README
16+
+ Add - Images for README
17+
+ Add - Jupyter notebooks for downstream analysis, electrode localization, and NWB export
18+
+ Add - Electrode localization schema activation script
19+
+ Update - README for ephys modes, element-event, and element-electrode-localization
20+
+ Update - Docker and Compose files for element-event and element-electrode-localization
21+
+ Update - Pipeline script for ephys mode and element-event
22+
+ Update - Pytests
23+
24+
## [0.1.0] - 2022-06-02
725

826
+ Update - Docker and Compose files
927
+ Add - NWB export pytests
@@ -17,7 +35,7 @@ Observes [Semantic Versioning](https://semver.org/spec/v2.0.0.html) standard and
1735
+ Update - Notebooks
1836
+ Update - Move instructions to [datajoint-elements/install.md](
1937
https://github.com/datajoint/datajoint-elements/blob/main/install.md).
20-
+ Update - Docker and Compose files for new base image and added options to install
38+
+ Update - Docker and Compose files for new base image and added options to install
2139
specific forks for tests.
2240

2341
## 0.1.0a2 - 2021-04-12
@@ -28,3 +46,6 @@ specific forks for tests.
2846
## 0.1.1 - 2021-03-26
2947

3048
+ Add - Version
49+
50+
[0.2.0]: https://github.com/datajoint/workflow-array-ephys/releases/tag/0.2.0
51+
[0.1.0]: https://github.com/datajoint/workflow-array-ephys/releases/tag/0.1.0

README.md

+28-20
Original file line numberDiff line numberDiff line change
@@ -1,27 +1,29 @@
11
# DataJoint Workflow - Array Electrophysiology
22

3-
Workflow for extracellular array electrophysiology data acquired with a polytrode probe
4-
(e.g. [Neuropixels](https://www.neuropixels.org), Neuralynx) using the [SpikeGLX](https://github.com/billkarsh/SpikeGLX) or
3+
Workflow for extracellular array electrophysiology data acquired with a polytrode probe
4+
(e.g. [Neuropixels](https://www.neuropixels.org), Neuralynx) using the [SpikeGLX](https://github.com/billkarsh/SpikeGLX) or
55
[OpenEphys](https://open-ephys.org/gui) acquisition software and processed with [MATLAB-based Kilosort](https://github.com/MouseLand/Kilosort) or [python-based Kilosort](https://github.com/MouseLand/pykilosort) spike sorting software.
66

77
A complete electrophysiology workflow can be built using the DataJoint Elements.
8+
89
+ [element-lab](https://github.com/datajoint/element-lab)
910
+ [element-animal](https://github.com/datajoint/element-animal)
1011
+ [element-session](https://github.com/datajoint/element-session)
1112
+ [element-array-ephys](https://github.com/datajoint/element-array-ephys)
1213

1314
This repository provides demonstrations for:
15+
1416
1. Set up a workflow using DataJoint Elements (see
1517
[workflow_array_ephys/pipeline.py](workflow_array_ephys/pipeline.py))
1618
2. Ingestion of data/metadata based on a predefined file structure, file naming
1719
convention, and directory lookup methods (see
1820
[workflow_array_ephys/paths.py](workflow_array_ephys/paths.py)).
1921
3. Ingestion of clustering results.
20-
4. Export of `no_curation` schema to NWB and DANDI (see (notebooks/09-NWB-export.ipynb)[notebooks/09-NWB-export.ipynb]
22+
4. Export of `no_curation` schema to NWB and DANDI (see [notebooks/09-NWB-export.ipynb](notebooks/09-NWB-export.ipynb)).
2123

2224
See the [Element Array Electrophysiology documentation](https://elements.datajoint.org/description/array_ephys/) for the background information and development timeline.
2325

24-
For more information on the DataJoint Elements project, please visit https://elements.datajoint.org. This work is supported by the National Institutes of Health.
26+
For more information on the DataJoint Elements project, please visit <https://elements.datajoint.org>. This work is supported by the National Institutes of Health.
2527

2628
## Workflow architecture
2729

@@ -31,52 +33,58 @@ The electrophysiology workflow presented here uses components from 4 DataJoint E
3133
[element-session](https://github.com/datajoint/element-session),
3234
[element-array-ephys](https://github.com/datajoint/element-array-ephys)) assembled
3335
together to form a fully functional workflow. Note that element-array-ephys offers three
34-
schema options, selected via the DataJoint config file, with
36+
schema options, selected via the DataJoint config file, with
3537
`dj.config['custom']['ephys_mode']`
38+
3639
+ `acute` probe insertion, with curated clustering
3740
+ `chronic` probe insertion, with curated clustering
3841
+ `no-curation`, acute probe insertion with kilosort triggered clustering and supported NWB export
42+
+ `precluster`, acute probe insertion with pre-processing steps prior to clustering and curated clustering
3943

40-
![element-array-ephys](images/attached_array_ephys_element.svg)
44+
![](https://raw.githubusercontent.com/datajoint/workflow-array-ephys/main/images/attached_array_ephys_element.svg)
4145

42-
Optionally, this can be used in conjunction with
46+
Optionally, this can be used in conjunction with
4347
[element-event](https://github.com/datajoint/element-event)
4448
and [element-electrode-localization](https://github.com/datajoint/element-electrode-localization/).
4549

46-
![element-event_attached](images/attached_trial_analysis.svg)
50+
![](https://raw.githubusercontent.com/datajoint/workflow-array-ephys/main/images/attached_trial_analysis.svg)
4751

48-
![element-electrode-localization_attached](images/attached_electrode_localization.svg)
52+
![](https://raw.githubusercontent.com/datajoint/workflow-array-ephys/main/images/attached_electrode_localization.svg)
4953

5054
## Installation instructions
5155

52-
The installation instructions can be found at the
56+
The installation instructions can be found at the
5357
[DataJoint Elements documentation](https://elements.datajoint.org/usage/install/).
5458

5559
## Interacting with the DataJoint workflow
5660

57-
Please refer to the workflow-specific
58-
[Jupyter notebooks](/notebooks)
61+
Please refer to the workflow-specific
62+
[Jupyter notebooks](/notebooks)
5963
for an in-depth explanation of how to ...
64+
6065
1. Run the workflow ([03-process.ipynb](notebooks/03-process.ipynb)).
6166
2. Explore the data ([05-explore.ipynb](notebooks/05-explore.ipynb)).
62-
3. Examine trialized analyses, and establish downstream analyses
67+
3. Examine trialized analyses, and establish downstream analyses
6368
([07-downstream-analysis.ipynb](notebooks/07-downstream-analysis.ipynb))
64-
4. Locate probes within the
65-
[Common Coordinate Framework](https://www.sciencedirect.com/science/article/pii/S0092867420304025)
69+
4. Locate probes within the
70+
[Common Coordinate Framework](https://www.sciencedirect.com/science/article/pii/S0092867420304025)
6671
([08-electrode-localization.ipynb](notebooks/08-electrode-localization.ipynb))
6772
5. Export to NWB and DANDI ([09-NWB-export.ipynb](notebooks/09-NWB-export.ipynb))
6873

74+
See our YouTube tutorial for a walkthrough of the schemas and functions:
75+
[![YouTube tutorial](https://img.youtube.com/vi/KQlGYOBq7ow/0.jpg)](https://www.youtube.com/watch?v=KQlGYOBq7ow)
6976

7077
## Citation
7178

7279
If your work uses DataJoint and DataJoint Elements, please cite the respective Research Resource Identifiers (RRIDs) and manuscripts.
7380

74-
+ DataJoint for Python or MATLAB
75-
+ Yatsenko D, Reimer J, Ecker AS, Walker EY, Sinz F, Berens P, Hoenselaar A, Cotton RJ, Siapas AS, Tolias AS. DataJoint: managing big scientific data using MATLAB or Python. bioRxiv. 2015 Jan 1:031658. doi: https://doi.org/10.1101/031658
7681

77-
+ DataJoint ([RRID:SCR_014543](https://scicrunch.org/resolver/SCR_014543)) - DataJoint for `<Select Python or MATLAB>` (version `<Enter version number>`)
82+
* DataJoint for Python or MATLAB
83+
+ Yatsenko D, Reimer J, Ecker AS, Walker EY, Sinz F, Berens P, Hoenselaar A, Cotton RJ, Siapas AS, Tolias AS. DataJoint: managing big scientific data using MATLAB or Python. bioRxiv. 2015 Jan 1:031658. doi: <https://doi.org/10.1101/031658>
84+
85+
+ DataJoint ([RRID:SCR_014543](https://scicrunch.org/resolver/SCR_014543)) - DataJoint for `<Select Python or MATLAB>` (version `<Enter version number>`)
7886

7987
+ DataJoint Elements
80-
+ Yatsenko D, Nguyen T, Shen S, Gunalan K, Turner CA, Guzman R, Sasaki M, Sitonic D, Reimer J, Walker EY, Tolias AS. DataJoint Elements: Data Workflows for Neurophysiology. bioRxiv. 2021 Jan 1. doi: https://doi.org/10.1101/2021.03.30.437358
88+
+ Yatsenko D, Nguyen T, Shen S, Gunalan K, Turner CA, Guzman R, Sasaki M, Sitonic D, Reimer J, Walker EY, Tolias AS. DataJoint Elements: Data Workflows for Neurophysiology. bioRxiv. 2021 Jan 1. doi: <https://doi.org/10.1101/2021.03.30.437358>
8189

82-
+ DataJoint Elements ([RRID:SCR_021894](https://scicrunch.org/resolver/SCR_021894)) - Element Array Electrophysiology (version `<Enter version number>`)
90+
+ DataJoint Elements ([RRID:SCR_021894](https://scicrunch.org/resolver/SCR_021894)) - Element Array Electrophysiology (version `<Enter version number>`)

notebooks/00-data-download-optional.ipynb

+27-36
Original file line numberDiff line numberDiff line change
@@ -4,6 +4,8 @@
44
"cell_type": "markdown",
55
"metadata": {},
66
"source": [
7+
"# Download example data\n",
8+
"\n",
79
"This workflow will need Ephys data collected from either SpikeGLX or OpenEphys and the output from kilosort2. We provided an example dataset to be downloaded to run through the pipeline. This notebook walks you through the process to download the dataset."
810
]
911
},
@@ -41,19 +43,12 @@
4143
{
4244
"cell_type": "code",
4345
"execution_count": null,
46+
"id": "e55f4f77",
4447
"metadata": {},
4548
"outputs": [],
4649
"source": [
4750
"import os\n",
48-
"import djarchive_client\n",
49-
"client = djarchive_client.client()"
50-
]
51-
},
52-
{
53-
"cell_type": "markdown",
54-
"metadata": {},
55-
"source": [
56-
"To browse the datasets that are available in djarchive:"
51+
"import djarchive_client"
5752
]
5853
},
5954
{
@@ -62,14 +57,14 @@
6257
"metadata": {},
6358
"outputs": [],
6459
"source": [
65-
"list(client.datasets())"
60+
"client = djarchive_client.client()"
6661
]
6762
},
6863
{
6964
"cell_type": "markdown",
7065
"metadata": {},
7166
"source": [
72-
"Each of the datasets have different versions associated with the version of workflow package. To browse the revisions:"
67+
"To browse the datasets that are available in djarchive:"
7368
]
7469
},
7570
{
@@ -78,14 +73,14 @@
7873
"metadata": {},
7974
"outputs": [],
8075
"source": [
81-
"list(client.revisions())"
76+
"list(client.datasets())"
8277
]
8378
},
8479
{
8580
"cell_type": "markdown",
8681
"metadata": {},
8782
"source": [
88-
"To download the dataset, let's prepare a root directory, for example in `/tmp`:"
83+
"Each of the datasets have different versions associated with the version of workflow package. To browse the revisions:"
8984
]
9085
},
9186
{
@@ -94,36 +89,23 @@
9489
"metadata": {},
9590
"outputs": [],
9691
"source": [
97-
"os.mkdir('/tmp/test_data')"
92+
"list(client.revisions())"
9893
]
9994
},
10095
{
10196
"cell_type": "markdown",
10297
"metadata": {},
10398
"source": [
104-
"Get the dataset revision with the current version of the workflow:"
99+
"To download the dataset, let's prepare a root directory, for example in `/tmp`:"
105100
]
106101
},
107102
{
108103
"cell_type": "code",
109-
"execution_count": 1,
104+
"execution_count": null,
110105
"metadata": {},
111-
"outputs": [
112-
{
113-
"data": {
114-
"text/plain": [
115-
"'0_1_0a2'"
116-
]
117-
},
118-
"execution_count": 1,
119-
"metadata": {},
120-
"output_type": "execute_result"
121-
}
122-
],
106+
"outputs": [],
123107
"source": [
124-
"from workflow_array_ephys import version\n",
125-
"revision = version.__version__.replace('.', '_')\n",
126-
"revision"
108+
"os.mkdir(\"/tmp/test_data\")"
127109
]
128110
},
129111
{
@@ -139,7 +121,11 @@
139121
"metadata": {},
140122
"outputs": [],
141123
"source": [
142-
"client.download('workflow-array-ephys-test-set', target_directory='/tmp/test_data', revision=revision)"
124+
"client.download(\n",
125+
" \"workflow-array-ephys-benchmark\",\n",
126+
" target_directory=\"/tmp/test_data\",\n",
127+
" revision='v1',\n",
128+
")"
143129
]
144130
},
145131
{
@@ -184,12 +170,12 @@
184170
],
185171
"metadata": {
186172
"jupytext": {
187-
"formats": "ipynb,py_scripts//py"
173+
"formats": "ipynb,py"
188174
},
189175
"kernelspec": {
190-
"display_name": "ephys_workflow_runner",
176+
"display_name": "Python 3.10.4 64-bit ('python3p10')",
191177
"language": "python",
192-
"name": "ephys_workflow_runner"
178+
"name": "python3"
193179
},
194180
"language_info": {
195181
"codemirror_mode": {
@@ -201,7 +187,12 @@
201187
"name": "python",
202188
"nbconvert_exporter": "python",
203189
"pygments_lexer": "ipython3",
204-
"version": "3.7.0"
190+
"version": "3.10.4"
191+
},
192+
"vscode": {
193+
"interpreter": {
194+
"hash": "ff52d424e56dd643d8b2ec122f40a2e279e94970100b4e6430cb9025a65ba4cf"
195+
}
205196
}
206197
},
207198
"nbformat": 4,

0 commit comments

Comments
 (0)