input-output-hk · yvan-sraka · Jul 14, 2023 · Nov 6, 2023 · Nov 7, 2023 · Nov 14, 2023
diff --git a/blog/2023-11-13-devx.md b/blog/2023-11-13-devx.md
@@ -0,0 +1,277 @@
+---
+slug: 2023-11-13-devx
+title: The IOG Developer's Experience Shell
+authors: [yvan]
+tags: [devx, haskell.nix, hix, iogx, nix]
+---
+
+# The IOG Developer's Experience Shell
+
+The IOG's Developer Experience Shell, or `DevX` for short, is an opinionated development environment tailored for Haskell. Built on top of Nix expressions, it provides a reproducible, seamless, and full-featured developer shell with tools such as `cabal-install`, `ghc`, `hls`, and `hlint`. `DevX` is built upon the battle-tested `haskell.nix`, a framework for building Haskell packages with Nix. Knowing Nix isn't a prerequisite to read the post except for the last `hix` section. In short, `haskell.nix` turns your Cabal or Stack project and its dependencies into a Nix expression. We will explain how to use `haskell.nix` on its own using `hix` in the last section.
+
+ But why Nix? Haskell developers often encounter the need for distinct environments per project. While solutions like stack resolvers and cabal project files address some concerns, they often fall short in terms of environment isolation. E.g., a Cabal project file could not be enough, because external (non-Haskell / system) dependencies need to be tracked too (that's exactly what Nix does), and could lead to inconsistent (non-reproducibles) builds. For instance, working on the Cardano performance regression while migrating from GHC 8.10 to 9.2, this required different toolchains. And, addressing this without `nix` would have been an absolute mess... With the `DevX` shell and `direnv`, you can achieve seamless transitions between projects. To illustrate, consider:
+```shell
+cd project1    # loading project1/.envrc
+ghc --version  # 8.10
+cd ../project2 # unloading project1/.envrc and loading project2/.envrc
+ghc --version  # 9.2
+```
+Nix achieves reproducibility by managing the complete dependencies graph (the _closure_) of a project. While this will not be discussed in this article, it makes it a great cross-compilation framework and it is worth noticing this is one of the main reason of why we use it at IOG. Other challenges solved by Nix are: How do we sync workspaces to make sure we're all dealing with the same technical stack? How do we make sure that Developer A can reuse the work of Developer B?
+
+So, to sum up, Nix is the tool that let you write reproducible, seamless and correct developer enviroments, and `DevX` shell is our opiniated definition on how to pack it with tools such as it is full-featured for Haskell development. This article aims to give you a technical deep dive on how to use the `DevX` shell (and other IOG Nix-related projects such as IOGx flakes' framework and `hix` CLI tool) in your own Haskell project. By the end of the following section you should be able to setup it in few steps, even in your Github Action CI or GitHub CodeSpace developer environment!
+
+## Getting started with `DevX` Shell
+
+The `DevX` GitHub repository provides a comprehensive [README](https://github.com/input-output-hk/devx#readme) that documents the different outputs that its `flake.nix` provides. As this article assumes no prior knowledge of Nix, we will quickly discuss three ergonomic ways of using this tool with no Nix knowledge required.
+
+We need to ensure [Nix is installed](https://nixos.org/download) and configured with `experimental-features = [ "nix-command" "flakes" ];` (details can be found at the [NixOS Wiki](https://nixos.wiki/wiki/Flakes)). 
+
+> **Side note:** In this article we will try to keep our use of the [Nix jargon](https://nixionary.org/) to the bare minimum required, but some concepts might still be needed to fully appreciate the last part of this article (about `hix`), here's a [little refresher in case of need](https://www.youtube.com/watch?v=gUjvnZ9ZwMs).
+
+### Philosophy and Usage
+
+The `DevX` shell approach is a one shell fits most[^1]. This means it isn't the most minimalist method for building a Haskell project with Nix, but it aims to be the most robust and generic approach.
+
+[^1]: The `DevX` shell, built on top of `haskell.nix`, heavily utilizes IFD (Import From Derivation). Consequently, it can't be used as a build framework for distributing projects on `nixpkgs`. In such cases, you might prefer using [`haskellPackages.mkDerivation`](https://nixos.org/manual/nixpkgs/stable/#haskell-mkderivation).
+
+To start using the IOG DevX shell in your Haskell project, simply run `nix develop github:input-output-hk/devx#ghc96` within your Haskell project's folder! This will spawn a developer shell, within which you can proceed with your usual Haskell tools and development workflow; for instance, use `cabal build` to build your project. For instance:
+```
+yvan@X230 ~ % cabal unpack hello
+
+yvan@X230 ~ % cd hello-1.0.0.2
+
+yvan@X230 ~/hello-1.0.0.2 % nix develop "github:input-output-hk/devx#ghc96"
+
+ _____ _____ _____    _____         _       _ _    _____ _       _ _ 
+|     |     |   __|  |  |  |___ ___| |_ ___| | |  |   __| |_ ___| | |
+|-   -|  |  |  |  |  |     | .'|_ -| '_| -_| | |  |__   |   | -_| | |
+|_____|_____|_____|  |__|__|__,|___|_,_|___|_|_|  |_____|_|_|___|_|_|
+
+Revision (input-output-hk/devx): 2a8b2cbf7ddbe83fedd4cd37130fc3c57c3f447e.
+CABAL_DIR set to /home/yvan/.cabal-devx
+
+[~/hello-1.0.0.2]$ which cabal
+/nix/store/6c46a2x0cdcnj072xf7pqsjnsn6d5p9b-cabal/bin/cabal
+
+[~/hello-1.0.0.2]$ ghc --version
+The Glorious Glasgow Haskell Compilation System, version 9.6.3
+
+[~/hello-1.0.0.2]$ cabal run hello
+Resolving dependencies...
+Build profile: -w ghc-9.6.3 -O1
+In order, the following will be built (use -v for more details):
+ - hello-1.0.0.2 (exe:hello) (first run)
+Warning: hello.cabal:36:32: version operators used. To use version operators
+the package needs to specify at least 'cabal-version: >= 1.8'.
+Configuring hello-1.0.0.2...
+Preprocessing executable 'hello' for hello-1.0.0.2..
+Building executable 'hello' for hello-1.0.0.2..
+[1 of 1] Compiling Main             ( src/hello.hs, /home/yvan/hello-1.0.0.2/dist-newstyle/build/x86_64-linux/ghc-9.6.3/hello-1.0.0.2/build/hello/hello-tmp/Main.o )
+[2 of 2] Linking /home/yvan/hello-1.0.0.2/dist-newstyle/build/x86_64-linux/ghc-9.6.3/hello-1.0.0.2/build/hello/hello
+Hello, World!
+```
+
+However, before discussing various use cases of `DevX` shell (with `direnv`, GitHub Action and Codespaces), let's take a quick detour to highlight that the developer shell's closure (all the Nix derivations it depends on) is quite large. To address this, some strategies have been developed to ensure it's quick to bootstrap the `DevX` shell.
+
+### Benchmarking
+
+We developed a strategy to speed up the overall download and evaluation time of our big Nix closure, we store it as an archive on GitHub package index, and then import all the derivations at once rather than letting `nix-daemon` evaluating and dowloading each store path separetly.
+
+To showcase the efficiency and speed that our download closure approach brings compared to a call to `nix develop`, we'll present a short benchmarking comparison that highlight a roughtly x2 speed up improvement!
+
+What isn't measured is the time for evaluating the closure, compressing it, and uploading it to `ghcr.io`, as it's done by our CI and represents the caching part of the process. We measure the speed of downloading, decompressing, and performing a `nix import` on the closure against the speed of `nix develop`, so we essentially save the time of the evaluation of the Nix expression:
+
+| `DevX` shell flavor       | `nix develop` (unmodified) | `fetch-closure.sh` (custom) |
+|---------------------------|----------------------------|-----------------------------|
+| `#ghc8107-iog`            | `11m28s`                   | `8m28s`                     |
+| `#ghc8107-static-minimal` | `5m22s`                    | `2m35s`                     |
+| `#ghc962-iog`             | `11m9s`                    | `7m26s`                     |
+| `#ghc962-static-minimal`  | `4m48s`                    | `1m35s`                     |
+
+> **Side note:** those benchmarks were run on @yvan-sraka's legacy ThinkPad X230 on a French countryside internet connection, to display how it changes loading times in contexts where internet connection and computing power are precious resources, like in a heavy CI setting. You can learn more about the methodology used [here](https://github.com/input-output-hk/devx/issues/22#issuecomment-1661841652).
+
+### Direnv
+
+`direnv` is an quite handy tool that allows you to load/unload the development environment automatically based on your current directory. It's a good way to avoid polluting your `.profile` or shell rc files.
+
+You can learn how to install `direnv` [here](https://direnv.net).
+You may then need to hook up direnv in your shell; you can consult the dedicated [`direnv` and `DevX`](./docs/direnv.md) guide for detailed instructions on integration.
+
+Then just add an `.envrc` file with the following content in the root of your project:
+```shell
+# https://github.com/nix-community/nix-direnv A fast, persistent use_nix/use_flake implementation for direnv:
+if ! has nix_direnv_version || ! nix_direnv_version 2.3.0; then
+  source_url "https://raw.githubusercontent.com/nix-community/nix-direnv/2.3.0/direnvrc" "sha256-Dmd+j63L84wuzgyjITIfSxSD57Tx7v51DMxVZOsiUD8="
+fi
+# https://github.com/input-output-hk/devx Slightly opinionated shared GitHub Action for Cardano-Haskell projects
+use flake "github:input-output-hk/devx#ghc8107"
+```
+
+`DevX` supports a variety of compiler versions, typically the latest for each series from 8.10 to latest. Along with the core GHC version, [it also offers various flavours, denoted as suffixes to the compiler names](https://github.com/input-output-hk/devx#compilers-and-flavours).
+
+The `nix-direnv` prefix is not necessary, but it's a good way to prevent Nix from garbage collecting your shell while the directory exists!
+
+### GitHub Action
+
+We also provide a `DevX` GitHub Action, that use the `fetch-closure.sh` method presented in Benchmark section. Here's how you might utilize the it in your workflow:
+```yaml
+- name: Build
+  uses: input-output-hk/actions/devx@latest
+  with:
+    platform: 'x86_64-linux'
+    target-platform: '-windows'
+    compiler-nix-name: 'ghc8107'
+    minimal: false
+    iog: false
+- name: Build
+  shell: devx {0}
+  run: |
+    cabal update
+    cabal build
+```
+
+Here we picked GHC version 8.10.7 and a Windows cross-compiler on Linux. This means that the 2 commands (`cabal update && cabal build`) runned in this GitHub Action will produce a windows binary.
+
+### VSCode DevContainer and GitHub CodeSpace
+
+Finally, we also offer support for VSCode DevContainers and GitHub CodeSpaces. This is a nice feature that let you, in one-click, load within your IDE or web browser a fully-featured developer environment (e.g., with Haskell Language Server setup). Which is particularly nice for onboarding new users and let them quickly start to contribute to a project.
+
+To make the DevX developer shell available in a VSCode DevContainer or GitHub CodeSpace, simply add a file named `.devcontainer/devcontainer.json` with the following content:
+```json
+{
+   "image":"ghcr.io/input-output-hk/devx-devcontainer:ghc8107",
+   "customizations":{
+      "vscode":{
+         "extensions":[
+            "haskell.haskell"
+         ],
+         "settings":{
+            "haskell.manageHLS":"PATH"
+         }
+      }
+   }
+}
+```
+You can follow the [Microsoft tutorial](https://code.visualstudio.com/docs/devcontainers/tutorial) to set-up your VSCode local DevContainer or you can give it a try by [opening a GitHub Codespace](https://codespaces.new/input-output-hk/cardano-base?quickstart=1) on [`cardano-base`](https://github.com/input-output-hk/cardano-base) repository!
+
+## IOGx: Flake Templates for Projects at IOG
+
+IOGx is a Nix library of functions and templates for structuring your Nix code and comes with a number of common DevX facilities to help develop your project. Its vision is to provide a JSON-like, declarative interface to Nix, enabling developers unfamiliar with the Nix language to maintain and enhance the Nix sources independently with minimal effort. This is motivated by the current fact that the learning curve for Nix is steep and the documentation is not yet as beginner friendly than it could be.
+
+### Getting Started with IOGx
+
+Kick-start your project with IOGx by running the following command:
+```
+yvan@X230 ~/iogx-demo % nix flake init --template github:input-output-hk/iogx#vanilla
+wrote: /Users/yvan/iogx-demo/nix/outputs.nix
+wrote: /Users/yvan/iogx-demo/nix/shell.nix
+wrote: /Users/yvan/iogx-demo/nix
+wrote: /Users/yvan/iogx-demo/flake.nix
+Flake Template for Vanilla Projects
+    Open flake.nix to get started.
+```
+This command generates a `flake.nix` and a `nix` folder containing various file templates.
+
+The next steps involve populating the templates in the `nix` folder and `flake.nix` itself.
+
+### IOGx Features
+
+IOGx comes packed with numerous [features](https://github.com/input-output-hk/iogx#2-features) that enhance the Haskell development process.
+
+To delve deeper into IOGx and its API, refer to the comprehensive [API Reference](https://github.com/input-output-hk/iogx/blob/main/doc/api.md). IOGX aims to facilitate a pleasant and efficient development process for Haskell projects at IOG, minimizing the need for extensive Nix language knowledge.
+
+## Hix
+
+Hix is a command-line tool designed to facilitate the addition of `haskell.nix` support to existing Haskell projects. Maintaining a large Nix expression, that uses `haskell.nix`, require quite a lot of Nix knowledge within a project team. Hix helps to reduce this expression to the bare minimum.
+
+### Quickstart with Hix
+
+The `hix init` (or `nix run "github:input-output-hk/haskell.nix#hix" -- init` if `hix` isn't in your `PATH`) command adds a `flake.nix` and `nix/hix.nix` file to the project root.
+
+Once this is done, you can utilize regular Nix tools. For instance:
+```
+yvan@X230 ~ % cabal unpack hello  
+Unpacking to hello-1.0.0.2/
+
+yvan@X230 ~ % cd hello-1.0.0.2
+
+yvan@X230 ~/hello-1.0.0.2 % nix run "github:input-output-hk/haskell.nix#hix" -- init
+`flake.nix` file created.
+`nix/hix.nix` project configuation:
+───────┬───────────────────────────────────────────────────────────────────────
+       │ File: nix/hix.nix
+───────┼───────────────────────────────────────────────────────────────────────
+   1   │ {pkgs, ...}: {
+   2   │   # name = "project-name";
+   3   │   compiler-nix-name = "ghc92"; # Version of GHC to use
+   4   │ 
+   5   │   # Cross compilation support:
+   6   │   # crossPlatforms = p: pkgs.lib.optionals pkgs.stdenv.hostPlatform.isx86_64 ([
+   7   │   #   p.mingwW64
+   8   │   #   p.ghcjs
+   9   │   # ] ++ pkgs.lib.optionals pkgs.stdenv.hostPlatform.isLinux [
+  10   │   #   p.musl64
+  11   │   # ]);
+  12   │ 
+  13   │   # Tools to include in the development shell
+  14   │   shell.tools.cabal = "latest";
+  15   │   # shell.tools.hlint = "latest";
+  16   │   # shell.tools.haskell-language-server = "latest";
+  17   │ }
+───────┴───────────────────────────────────────────────────────────────────────
+
+yvan@X230 ~/hello-1.0.0.2 % nix develop
+warning: creating lock file '/home/yvan/hello-1.0.0.2/flake.lock'
+
+[yvan@X230:~/hello-1.0.0.2]$ cabal build
+Resolving dependencies...
+Build profile: -w ghc-9.2.8 -O1
+In order, the following will be built (use -v for more details):
+ - hello-1.0.0.2 (exe:hello) (first run)
+Warning: hello.cabal:36:32: version operators used. To use version operators
+the package needs to specify at least 'cabal-version: >= 1.8'.
+Configuring hello-1.0.0.2...
+Preprocessing executable 'hello' for hello-1.0.0.2..
+Building executable 'hello' for hello-1.0.0.2..
+[1 of 1] Compiling Main             ( src/hello.hs, /home/yvan/hello-1.0.0.2/dist-newstyle/build/x86_64-linux/ghc-9.2.8/hello-1.0.0.2/build/hello/hello-tmp/Main.o )
+Linking /home/yvan/hello-1.0.0.2/dist-newstyle/build/x86_64-linux/ghc-9.2.8/hello-1.0.0.2/build/hello/hello ...
+```
+
+In this example, we (again) use `cabal` to pull the `hello` package from Hackage, then we enter the project directory and run `hix init` that generate a `flake.nix`, then to enter a development shell with `nix develop`. Now that we are in the development shell we can use `cabal` as one normally would.
+
+You can try this at home and play with `cabal list --installed` to show the difference between environments before `nix develop` and after, or view the contents of the flake using `nix flake show`!
+
+Finally, to build a specific component with Nix you can, e.g., use `nix build .#hello:exe:hello`, and to build and run a component, `nix run .#hello:exe:hello` would work as expected.
+
+### Installing Hix
+
+Hix can be installed in your `PATH` (by adding it to `~/.nix-profile/bin`) with the following command:
+```
+nix-env -iA hix -f https://github.com/input-output-hk/haskell.nix/tarball/master
+```
+
+To update Hix to the latest version, simply run `hix update`.
+
+### Using Hix Commands
+
+Commands like `hix develop`, `hix flake`, `hix build`, and `hix run` work similarly to their Nix counterparts. However, instead of utilizing the `flake.nix`, a boilerplate Haskell.nix `flake.nix` file is added to `.hix-flake/flake.nix`.
+
+Once this step is done, you can execute commands without the need for `hix init`:
+```
+hix develop
+hix flake show
+hix build .#hello:exe:hello
+hix run .#hello:exe:hello
+```
+
+### Using `hix-shell` and `hix-build`
+
+The `hix-shell` and `hix-build` commands emulate the behaviour of `nix-build` and `hix-shell` if a boilerplate `default.nix` and `shell.nix` were present. These commands are executed as follows:
+```
+hix-shell --run 'cabal build all'
+hix-build -A hsPkgs.hello.components.exes.hello
+```
+
+## Final Thoughts
+
+As we've seen the suite of tools we've developed around `nix`, and `haskell.nix` at IOG provide a fully featured set of tools from running reproducible adhoc (DevX) build environments locally, to running them in CI either via GHA or `nix`-based CI systems (via `haskell.nix` and IOGx) as appropriate, and extending this to in-browser development environment with VSCode DevContainers / GitHub CodeSpaces.