Open
Description
Clone repo at https://github.com/davidADSP/Generative_Deep_Learning_2nd_Edition.git
get Kaggle account - use old google account
https://www.kaggle.com
Get a token - profile | account
Run an example notebook
https://www.kaggle.com/code/galalqassas/analysis-of-tornado-data-since-1950
rename/edit .env file
build container (download 3G)
docker compose -f docker-compose.gpu.yml build
$ docker compose -f docker-compose.gpu.yml build
[+] Building 415.6s (17/17) FINISHED
=> [app internal] load .dockerignore 0.0s
=> => transferring context: 52B 0.0s
=> [app internal] load build definition from Dockerfile.gpu 0.0s
=> => transferring dockerfile: 587B 0.0s
=> [app internal] load metadata for docker.io/tensorflow/tensorflow:2.10.1-gpu 0.3s
=> [app 1/12] FROM docker.io/tensorflow/tensorflow:2.10.1-gpu@sha256:087210b1d2537872f44c82e6c941e0c83a9a7c4e156873b9e1088918318accc5 275.9s
=> => resolve docker.io/tensorflow/tensorflow:2.10.1-gpu@sha256:087210b1d2537872f44c82e6c941e0c83a9a7c4e156873b9e1088918318accc5 0.0s
=> => sha256:eaead16dc43bb8811d4ff450935d607f9ba4baffda4fc110cc402fa43f601d83 28.58MB / 28.58MB 5.6s
=> => sha256:2798fbbc3b3bc018c0c246c05ee9f91a1ebe81877940610a5e25b77ec5d4fe24 7.93MB / 7.93MB 2.1s
=> => sha256:087210b1d2537872f44c82e6c941e0c83a9a7c4e156873b9e1088918318accc5 3.26kB / 3.26kB 0.0s
=> => sha256:6aafbce389f98e508428ecdf171fd6e248a9ad0a5e215ec3784e47ffa6c0dd3e 11.04MB / 11.04MB 3.7s
=> => sha256:f058b345b1f90b59372febfa19b194721beeeca368148e23917ae64fd57d7597 12.71kB / 12.71kB 0.0s
=> => sha256:4d8ee731d34ea0ab8f004c609993c2e93210785ea8fc64ebc5185bfe2abdf632 188B / 188B 2.3s
=> => sha256:92d2e1452f727e063220a45c1711b635ff3f861096865688b85ad09efa04bd52 6.43kB / 6.43kB 2.4s
=> => sha256:7bf023eb24e1707dc56a984a7b2fb61a6fd96c1baedc6c05d2e63880599227f7 1.91GB / 1.91GB 246.9s
=> => sha256:86ddbb160b1d4fdab349fc3f2a3f4f61473473cd460f4f9855523c777d6a7788 206.00MB / 206.00MB 57.8s
=> => extracting sha256:eaead16dc43bb8811d4ff450935d607f9ba4baffda4fc110cc402fa43f601d83 0.7s
=> => sha256:99dc85f60e170e908b247f63c8c0392df086a2014bea330994f7231f4c55c80a 7.86kB / 7.86kB 6.0s
=> => sha256:34eb1a77b7bf36a98c3c16eb1cb1f5815216ac5e433e9efb30c3474d7e0ce1a0 41.48MB / 41.48MB 15.4s
=> => extracting sha256:2798fbbc3b3bc018c0c246c05ee9f91a1ebe81877940610a5e25b77ec5d4fe24 0.2s
=> => extracting sha256:6aafbce389f98e508428ecdf171fd6e248a9ad0a5e215ec3784e47ffa6c0dd3e 0.1s
=> => extracting sha256:4d8ee731d34ea0ab8f004c609993c2e93210785ea8fc64ebc5185bfe2abdf632 0.0s
=> => extracting sha256:92d2e1452f727e063220a45c1711b635ff3f861096865688b85ad09efa04bd52 0.0s
=> => sha256:f95ddf392b1285190dbbd2bab2af614eede8e206ed82d3459cb39fb3922db9d1 4.74MB / 4.74MB 16.7s
=> => sha256:513131f5ea46b1700023ad97fa38b8193c3994b920e11a82b932691e211de65d 169B / 169B 16.8s
=> => sha256:00109ab89b686cd25ab053f6442c3b16546b9a04fb8a0c0788168634e88f69c9 650.55MB / 650.55MB 152.3s
=> => sha256:f96cf07c18482a395382de21e20391e7622968694ca54f85b76ca8bf13ba8d10 1.09kB / 1.09kB 58.1s
=> => sha256:0be56d5808079d1724f54f4cfb3390d354989691cbb98768c6b6fd4f06ced472 1.09kB / 1.09kB 58.6s
=> => extracting sha256:7bf023eb24e1707dc56a984a7b2fb61a6fd96c1baedc6c05d2e63880599227f7 16.7s
=> => extracting sha256:86ddbb160b1d4fdab349fc3f2a3f4f61473473cd460f4f9855523c777d6a7788 2.0s
=> => extracting sha256:99dc85f60e170e908b247f63c8c0392df086a2014bea330994f7231f4c55c80a 0.0s
=> => extracting sha256:34eb1a77b7bf36a98c3c16eb1cb1f5815216ac5e433e9efb30c3474d7e0ce1a0 0.4s
=> => extracting sha256:f95ddf392b1285190dbbd2bab2af614eede8e206ed82d3459cb39fb3922db9d1 0.2s
=> => extracting sha256:513131f5ea46b1700023ad97fa38b8193c3994b920e11a82b932691e211de65d 0.0s
=> => extracting sha256:00109ab89b686cd25ab053f6442c3b16546b9a04fb8a0c0788168634e88f69c9 8.9s
=> => extracting sha256:f96cf07c18482a395382de21e20391e7622968694ca54f85b76ca8bf13ba8d10 0.0s
=> => extracting sha256:0be56d5808079d1724f54f4cfb3390d354989691cbb98768c6b6fd4f06ced472 0.0s
=> [app internal] load build context 0.0s
=> => transferring context: 11.89kB 0.0s
=> [app 2/12] RUN rm /etc/apt/sources.list.d/cuda.list 1.1s
=> [app 3/12] RUN apt-get update 5.0s
=> [app 4/12] RUN apt-get install -y unzip graphviz curl musescore3 78.7s
=> [app 5/12] RUN pip install --upgrade pip 1.8s
=> [app 6/12] WORKDIR /app 0.0s
=> [app 7/12] COPY ./requirements.txt /app 0.1s
=> [app 8/12] RUN pip install -r /app/requirements.txt 44.1s
=> [app 9/12] RUN pip install tensorflow-io 4.8s
=> [app 10/12] RUN pip uninstall -y tensorflow-io 0.6s
=> [app 11/12] COPY /notebooks/. /app/notebooks 0.1s
=> [app 12/12] COPY /scripts/. /app/scripts 0.1s
=> [app] exporting to image 3.0s
=> => exporting layers 3.0s
=> => writing image sha256:1cdfa3377e2e7948aa62dd0b0f9b70bb3ae14dadb73df2210e19cf7f247d5594 0.0s
=> => naming to docker.io/library/generative_deep_learning_2nd_edition-app
Bring the jupyter notebook up
michael@13900b MINGW64 /c/wse_github/davidASSP/Generative_Deep_Learning_2nd_Edition (main)
$ docker compose -f docker-compose.gpu.yml up
[+] Running 2/0
✔ Network generative_deep_learning_2nd_edition_default Created 0.0s
✔ Container generative_deep_learning_2nd_edition-app-1 Created 0.0s
Attaching to generative_deep_learning_2nd_edition-app-1
generative_deep_learning_2nd_edition-app-1 | [I 2023-09-24 14:58:42.670 ServerApp] Package jupyterlab took 0.0000s to import
generative_deep_learning_2nd_edition-app-1 | [I 2023-09-24 14:58:42.672 ServerApp] Package jupyter_server_terminals took 0.0020s to import
generative_deep_learning_2nd_edition-app-1 | [I 2023-09-24 14:58:42.672 ServerApp] Package nbclassic took 0.0000s to import
generative_deep_learning_2nd_edition-app-1 | [W 2023-09-24 14:58:42.673 ServerApp] A `_jupyter_server_extension_points` function was not found in nbclassic. Instead, a `_jupyter_server_extension_paths` function was found and will be used for now. This function name will be deprecated in future releases of Jupyter Server.
generative_deep_learning_2nd_edition-app-1 | [I 2023-09-24 14:58:42.674 ServerApp] Package notebook_shim took 0.0000s to import
generative_deep_learning_2nd_edition-app-1 | [W 2023-09-24 14:58:42.674 ServerApp] A `_jupyter_server_extension_points` function was not found in notebook_shim. Instead, a `_jupyter_server_extension_paths` function was found and will be used for now. This function name will be deprecated in future releases of Jupyter Server.
generative_deep_learning_2nd_edition-app-1 | [I 2023-09-24 14:58:42.676 ServerApp] jupyter_server_terminals | extension was successfully linked.
generative_deep_learning_2nd_edition-app-1 | [I 2023-09-24 14:58:42.680 ServerApp] jupyterlab | extension was successfully linked.
generative_deep_learning_2nd_edition-app-1 | [I 2023-09-24 14:58:42.682 ServerApp] nbclassic | extension was successfully linked.
generative_deep_learning_2nd_edition-app-1 | [I 2023-09-24 14:58:42.683 ServerApp] Writing Jupyter server cookie secret to /root/.local/share/jupyter/runtime/jupyter_cookie_secret
generative_deep_learning_2nd_edition-app-1 | [I 2023-09-24 14:58:42.778 ServerApp] notebook_shim | extension was successfully linked.
generative_deep_learning_2nd_edition-app-1 | [I 2023-09-24 14:58:42.886 ServerApp] notebook_shim | extension was successfully loaded.
generative_deep_learning_2nd_edition-app-1 | [I 2023-09-24 14:58:42.887 ServerApp] jupyter_server_terminals | extension was successfully loaded.
generative_deep_learning_2nd_edition-app-1 | [I 2023-09-24 14:58:42.887 LabApp] JupyterLab extension loaded from /usr/local/lib/python3.8/dist-packages/jupyterlab
generative_deep_learning_2nd_edition-app-1 | [I 2023-09-24 14:58:42.887 LabApp] JupyterLab application directory is /usr/local/share/jupyter/lab
generative_deep_learning_2nd_edition-app-1 | [I 2023-09-24 14:58:42.889 ServerApp] jupyterlab | extension was successfully loaded.
generative_deep_learning_2nd_edition-app-1 |
generative_deep_learning_2nd_edition-app-1 | _ _ _ _
generative_deep_learning_2nd_edition-app-1 | | | | |_ __ __| |__ _| |_ ___
generative_deep_learning_2nd_edition-app-1 | | |_| | '_ \/ _` / _` | _/ -_)
generative_deep_learning_2nd_edition-app-1 | \___/| .__/\__,_\__,_|\__\___|
generative_deep_learning_2nd_edition-app-1 | |_|
generative_deep_learning_2nd_edition-app-1 |
generative_deep_learning_2nd_edition-app-1 | Read the migration plan to Notebook 7 to learn about the new features and the actions to take if you are using extensions.
generative_deep_learning_2nd_edition-app-1 |
generative_deep_learning_2nd_edition-app-1 | https://jupyter-notebook.readthedocs.io/en/latest/migrate_to_notebook7.html
generative_deep_learning_2nd_edition-app-1 |
generative_deep_learning_2nd_edition-app-1 | Please note that updating to Notebook 7 might break some of your extensions.
generative_deep_learning_2nd_edition-app-1 |
generative_deep_learning_2nd_edition-app-1 | [I 2023-09-24 14:58:42.891 ServerApp] nbclassic | extension was successfully loaded.
generative_deep_learning_2nd_edition-app-1 | [I 2023-09-24 14:58:42.891 ServerApp] Serving notebooks from local directory: /app
generative_deep_learning_2nd_edition-app-1 | [I 2023-09-24 14:58:42.891 ServerApp] Jupyter Server 2.7.3 is running at:
generative_deep_learning_2nd_edition-app-1 | [I 2023-09-24 14:58:42.891 ServerApp] http://bfa4c2361f51:8888/lab?token=f826f4718483b3249a2d28ba42ee37c9385f1b03aa1290b0
generative_deep_learning_2nd_edition-app-1 | [I 2023-09-24 14:58:42.891 ServerApp] http://127.0.0.1:8888/lab?token=f826f4718483b3249a2d28ba42ee37c9385f1b03aa1290b0
generative_deep_learning_2nd_edition-app-1 | [I 2023-09-24 14:58:42.891 ServerApp] Use Control-C to stop this server and shut down all kernels (twice to skip confirmation).
generative_deep_learning_2nd_edition-app-1 | [C 2023-09-24 14:58:42.893 ServerApp]
generative_deep_learning_2nd_edition-app-1 |
generative_deep_learning_2nd_edition-app-1 | To access the server, open this file in a browser:
generative_deep_learning_2nd_edition-app-1 | file:///root/.local/share/jupyter/runtime/jpserver-1-open.html
generative_deep_learning_2nd_edition-app-1 | Or copy and paste one of these URLs:
generative_deep_learning_2nd_edition-app-1 | http://bfa4c2361f51:8888/lab?token=f826f4718483b3249a2d28ba42ee37c9385f1b03aa1290b0
generative_deep_learning_2nd_edition-app-1 | http://127.0.0.1:8888/lab?token=f826f4718483b3249a2d28ba42ee37c9385f1b03aa1290b0
Launch the notebook
Download a model (with the notebook up) - separate terminal - 202k images - takes at least 30 min to extract (single vCore)
michael@13900b MINGW64 /c/wse_github/davidASSP/Generative_Deep_Learning_2nd_Edition (main)
$ scripts/download.sh faces
Downloading celeba-dataset.zip to /app/data
6%|█████████▋ | 85.0M/1.33G [00:07<01:53, 11.8MB/s]
Verify that your WSL 2 configuration in .wslconfig is setup to use more than the minimum vCores and ram - like 2/4. For example on a 192G 13900K with 32 threads - I set the following...
memory=96GB
processors=16
Test this by running a small benchmark docker container from
https://hub.docker.com/repository/docker/obrienlabs/collatz-se/general
The following will run 2^5 and 2^6 threads - for a 32 thread processor it will saturate the 32 threads at 32 and 64 respectively
.
docker run -it obrienlabs/collatz-se:intel
availableProc : 32
fjps threads : 5,6
