Data Science Work Organization

[sanjiv999]

Hi all, just curious if people have a data science pipeline and organization system that they like or have fleshed out in detail. I do work in python and keep varying in terms of organizing notebooks, storing raw data vs interim data vs outputs, etc. not sure if there are some general best practices around this that people are aware of. Also how do you align this with other tasks of research/operations work and those folder structures.

[drelliche]

Hi @sanjiv999! We work with a team of library scientists and they put together a project template for people who are going to work with us to structure their systems. Some of it might be specific to the way we archive data for later reuse, but a lot of it could apply to any project.

[sanjiv999]

Exactly what I was looking for. Thank you!

…

On Wed, Nov 15, 2023 at 1:08 PM drelliche ***@***.***> wrote: Hi @sanjiv999 <https://github.com/sanjiv999>! We work with a team of library scientists and they put together a project template <https://liascript.github.io/course/?https://raw.githubusercontent.com/arcus/Arcus_Labs_Orientation/main/data_contribution_2_data.md#7> for people who are going to work with us to structure their systems. Some of it might be specific to the way we archive data for later reuse, but a lot of it could apply to any project. — Reply to this email directly, view it on GitHub <#46 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ALQXKSQC2CPRSKZQVNFB2RDYEUALVAVCNFSM6AAAAAA7EUCPSWVHI2DSMVQWIX3LMV43SRDJONRXK43TNFXW4Q3PNVWWK3TUHM3TKOBQGAYTG> . You are receiving this because you were mentioned.Message ID: <arcus/DART_Community_of_Practice/repo-discussions/46/comments/7580013@ github.com>