Change Notebooks to local installation of latest dev state

[gugrimm]

Related Issue

Closes #678

Description

This changes the notebooks when executed in colab.
The motivation is to avoid errors that occur when both the latest release from pypi and the assume git repo are used in one notebook and the release is behind the git repo in terms of namings, data structure etc.

The proposed solution is to replace !pip install assume-framework by !pip install -e . in all notebooks, where we clone the git repo in order to have access to tutorial data.

Checklist

• no ducumentation required
• New unit/integration tests added (not applicable)
• Changes noted in release notes (not applicable)
• Consent to release this PR's code under the GNU Affero General Public License v3.0

[gugrimm]

Also Fixes #689

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 45.96%. Comparing base (e64e047) to head (4c3d177).
⚠️ Report is 5 commits behind head on main.

Additional details and impacted files

@@           Coverage Diff           @@
##             main     #690   +/-   ##
=======================================
  Coverage   45.96%   45.96%           
=======================================
  Files          55       55           
  Lines        8012     8012           
=======================================
  Hits         3683     3683           
  Misses       4329     4329           

Flag	Coverage Δ
pytest	45.96% <ø> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[kim-mskw]

On the review @jrasko and @maurerle

[kim-mskw]

Should we not do this consistently, then, to avoid future errors? If we decide to install the dev state, do we always?

[kim-mskw]

It is weird that we do not install assume in the section that is called installation process. I would suggest restructuring the chapter as well

[kim-mskw]

The installation command in the markdown cell still references the normal installation, there also with ". -e" then?

[kim-mskw]

This happens in multiple tutorials. Please make changes accordingly.

[kim-mskw]

Thank you for fixing this. I have a few remarks regarding consistency:

• I would suggest using the dev state in all tutorials, regardless of whether it is currently required. This helps avoid future issues and keeps everything consistent.
• Related to consistency: I know it’s a bit tedious, but please also update the respective markdowns in the tutorials.
  Some of them still do not reference the dev state, and they mention that installation cells do not need to be executed if you already have the package installed.
  However, with only the latest release installed, you do need to run the installation cells again to obtain the dev version.

[kim-mskw]

This happens in multiple tutorials. Please make changes accordingly.

[kim-mskw]

Adjust the comment above that the code does not need to be executed when one has the current dev state installed, not the release.

[kim-mskw]

clear outputs please and push again :)

[gugrimm]

@assume-framework/core-devs This is bigger than initially thought and made me think of our approach to versioning:
With the last commit, I added cases to the installation part in notebook 03 that would

• in colab: clone and pip install -e .
• locally: clone and pip install -e . when repository "assume" and "assume-repo" dont exist
• locally: git pull and pip install -e . when the repo exists
  This is a little bit long. But the only thing I could think of. However it would not work for users without admin rights, the tests fail, and most importantly we force users to install things on their local PC without asking or the option to refuse. (It is also unclear to me what happens if I am in a local branch with local changes and then inside a ipynb notebook cell do !git pull. What does this pull? Main? Interestingly this does not overwrite my local changes, but also does not abort and not tells me to stash my work first. Another thing in .ipynb that I don't fully understand...)

After discussing this with Ramiz, we came up with 3 potential solutions:

1. introduce a dev branch and keep main same as the pypi release until the next release. We would still do developments on small branches, but then merge them into dev. When we do a release, we merge dev into main. The version and therefore the versioning number of pypi and github main would then always be the same.
2. with each PR update the version number in github main. This is currently not done (we have pypi 0.5.5 and github 0.5.5, although the versions are clearly NOT the same.) This would then allow in the notebooks (and anywhere else) to check versions and print warnings, as it is for example done in PyPSA (e.g. if importing an old network file to a new PyPSA version).
3. give responsibility to the user (notebooks not runnable out of the box): We could git clone and pip install -e if working in colab. If users work locally, they must make sure for themselves to use tha latest dev state from main. We would indicate this more clearly in the notebooks.

I would be in favor of 1) but thinking about consistent versioning seems to be a good idea to me anyway. What are your thoughts?

[maurerle]

@gugrimm I think you mentioned some additional constraints, which makes this problem look larger than I hope it is :D

If users installed assume with pip installe -e ., then any checkout should have the matching .ipynb version.
git pull should not be executed by scripts - and should not be needed.

btw when do you need admin rights to run the tests or something?

This is currently not done (we have pypi 0.5.5 and github 0.5.5, although the versions are clearly NOT the same.)

they are the same tag, same code, published through and from GitHub Actions?

I would go with 3. - we take care of pip install -e . everywhere and additionally clone the repo in COLAB:

give responsibility to the user (notebooks not runnable out of the box): We could git clone and pip install -e if working in colab. If users work locally, they must make sure for themselves to use the latest dev state from main. We would indicate this more clearly in the notebooks.

as the local installed version should always match the notebook, this should always result in a working state (even if not the latest state - but well, running git pull to update a git repository can be expected?

[gugrimm]

Now that we have setuptools-scm with dynamic versioning included (#701), it should be possible to print warnings if the installed assume version on users local machines does not match the repository straucture of the latest dev state. This would avoid users to trap into problems.
However I don't understand how this versioning works.
I suggested the general structure to check in Notebooks 03 and 04a - if you @maurerle could have a look, give advice or make a suggestion? @jrasko