[DevX bootstrap] Benchmark the speed download of the devx closure
#22
Comments
|
I wanted to measure the time latency improvement of such a hack, so I wrote a dumb python script: import os
import timeit
DEV_SHELLS = [
"ghc8107",
"ghc902",
"ghc925",
"ghc8107-minimal",
"ghc902-minimal",
"ghc925-minimal",
"ghc8107-static-minimal",
"ghc902-static-minimal",
"ghc925-static-minimal",
]
T = {}
flake = "input-output-hk/devx" # vs. yvan-sraka/static-closure
for devShell in DEV_SHELLS:
os.system(f"nix-collect-garbage -d")
x = lambda number: round(timeit.timeit(lambda: os.system(
f'nix develop "github:{flake}#{devShell}"\
--no-write-lock-file --refresh --command true'
), number=number), 2)
T[devShell] = {"bootstrap": x(1), "reload": x(10)}
print(T)I currently exhaustively list working version of the
The hack lives in this I'll post the benchmark result I got in this thread :) |
|
This is good looking forward to the benchmarks! |
|
The awaited benchmarks that ran past night on my machine (values unit is seconds): Without the speed download hack (meaning the actual ghc8107:
bootstrap: 2155.54
reload: 4.57
ghc902: broken
ghc925:
bootstrap: 2005.21
reload: 4.43
ghc8107-minimal:
bootstrap: 1820.64
reload: 3.59
ghc902-minimal:
bootstrap: 1788.87
reload: 3.33
ghc925-minimal:
bootstrap: 1826.22
reload: 3.76
ghc8107-static-minimal:
bootstrap: 1774.84
reload: 3.46
ghc902-static-minimal:
bootstrap: 1780.89
reload: 4.75
ghc925-static-minimal:
bootstrap: 1871.04
reload: 3.4With the speed download hack of version ghc8107: broken
ghc902: broken
ghc925: broken
ghc8107-minimal:
bootstrap: 95.71
reload: 37.74
ghc902-minimal:
bootstrap: 100.58
reload: 37.6
ghc925-minimal:
bootstrap: 101.05
reload: 33.25
ghc8107-static-minimal:
bootstrap: 82.32
reload: 30.61
ghc902-static-minimal:
bootstrap: 91.1
reload: 31.1
ghc925-static-minimal:
bootstrap: 88.54
reload: 27.21First, there are few settings that are “broken” and I should investigate why … Then, as you can see, it's a big improvement in bootstrap speed (I have a quite slow internet connection so that surely helps to increase the numbers) … … but there is more work to do, as @angerman made me realize: wrapper derivation should not have to rely on |
|
@yvan-sraka can you please update the comment above with the following remarks:
If we had a canary derivation (e.g. the root of the imported closure), we could validate the existence of the closure in the store, by checking for the existence of that file; and skip the import? |
Edited :)
Yes! That's precisely what I've in mind and implemented in a new flake version that should also have fixed the broken builds. I should indeed re-run benchmark against this new flake version, which is currently |
|
On |
|
I will re-run benchmarks in a GitHub Action context to have some consistency, since my personal internet connection is right now not enough reliable to not false results … |
|
I don't think speed is the primary issue, as long as it's consistent. E.g. if you always get the same speed reliably that's going to provide good numbers. And most users won't be having 1G or 10G lines, but some XXX Mbit most likely. If we find out that for fast lines, it's even worse though, that would also be good to know. |
Yes! My current connection issues are effectively more about consistency than speed :) |
|
I've run new benchmarks on @hamishmack #! /usr/bin/env python
import os
def Dockerfile(shell, fast):
if fast:
cmd = f'./fetch-docker.sh input-output-hk/devx x86_64-linux.{shell}-env | zstd -d | nix-store --import'
else:
cmd = f'nix develop "github:input-output-hk/devx#{shell}" --command true'
return f"""FROM nixos/nix
RUN nix-channel --update
RUN echo "experimental-features = nix-command flakes" >> /etc/nix/nix.conf
RUN echo "accept-flake-config = true" >> /etc/nix/nix.conf
RUN ln -s $(which bash) /bin/bash
RUN nix profile install "nixpkgs#jq" "nixpkgs#zstd"
RUN curl -L https://raw.githubusercontent.com/input-output-hk/actions/latest/devx/support/fetch-docker.sh -o fetch-docker.sh
RUN chmod +x fetch-docker.sh
RUN time {cmd}"""
for shell in ["ghc8107-iog", "ghc962-iog", "ghc8107-static-minimal", "ghc962-static-minimal"]:
for fast in [True, False]:
with open('Dockerfile', 'w') as file:
file.write(Dockerfile(shell, fast))
os.system(f'docker build . | tee {shell}-{"fast" if fast else "slow"}.log')… and I retrieve the results with: #! /usr/bin/env bash
for file in *.log; do
grep '^real' "$file" | awk -v file="$file" '{print file " -> " $2}'
done… I run it on my legacy ThinkPad X230 on a French countryside internet connection, to display how it changes loading times in context where internet connection and computing power are precious resources, like in a heavy CI: … should I run these benchmarks with other |
yvan-sraka
changed the title
devx closure
|
So it's about 50% of the total time. That's good! Thanks for running the benchmarks! |
When using GHA to turn
iohk/devxinto a shell to run, e.g.cabal update, cabal build, we take a lot of time downloading stuff.A lot of this time comes down to nix sequentially downloading a lot of data. We should build the store and export/import it instead to speed this up.
… will give us something like
2.5G. That's a lot.We can also enter the shell using
(e.g. instead of
nix develop).We could pre-build the closure (e.g. from
result), and store that as azstdcompressed archive:And then re-import this as the first step in GHAs after setting up nix.
See for example this GHA: https://github.com/angerman/x/blob/c559ae0429bb69829a9c9cae8c21ab777461aaf2/.github/workflows/main.yml#L23-L66, which doesn't work properly yet (nix still ends up downloading stuff when trying to enter the shell; maybe this can be eliminated with the
env.shidea from above).The text was updated successfully, but these errors were encountered: