Improve stack support for HLS

[fendor]

Hi!

Setting the stage, Haskell Language Server's support for stack projects is currently lacking.
There are many reasons why using HLS with stack is a subpar experience.

I am trying to list the most important issues here, in the hope of better collaboration, so we can improve the developer experience with both HLS and stack.

Finding the compilation options of a filepath

Essentially, HLS is a compiler. As the compiler, it needs to know how to compile a component / file. Stack knows how to compile a file part of a stack project, and we want to extract that information.

The status quo is that we call a variation of stack repl with a special GHC shim that intercepts the ghc options when stack repl tries to invoke ghc --interactive. Then we use these options to compile the project ourselves.
This works reasonably well, but has some shortcomings. Most notably, the issues:

• "stack repl <test suite>" fails if "stack test --no-run-tests" is not run in advance for test suite containing errors #5380
• Internal library component not available for REPL #4564
• False positive: Main.hs (executable) does not recognise function defined in Lib.hs (library) haskell/haskell-language-server#1822
• Allow setting the working-dir to absolute directories #6135

Some of these are full-blown blockers and stack users have suffered in the past.

You can work around this using implicit-hie, which is its own can of worms, and not a proper solution for the needs of HLS.

What does HLS need

It is not a requirement for us to keep using this ugly hack of stack repl.

What we fundamentally need is this:

Given a filepath, we need to be able to find the exact compilation options for it (preferably for the whole component it belongs to) in the quickest way possible, with as little work as possible. Additionally, the compilation options must be enough to actually compile the file. Potential configure hooks must have been run before HLS tries to compile the file.
In the light of multiple home units coming to cabal and ghc, we would like even a bit more, but I can't specify right now what exactly.

Going forward

To get this stuff sorted out, there are multiple ways. In Cabal, we've added a couple of months/years ago the --enable-build-info flag with writes out build-info.json files containing the build information required to compile a component.

If someone wants to collaborate on this, I'd be happy to give more information, maybe even in a sync call.

[fendor]

@mpilgrem if possible, I would like to know what you think :)

[mpilgrem]

@fendor, I support the objective.

Stack has a command stack ide with subcommands that are provided with the aim of helping IDEs; and a command stack query that aims to output 'general build information' in JSON format. If something can be added to Stack (perhaps a new subcommand of stack ide or an enhancement of stack query?) that helps HLS support Stack users, that is all to the good.

In terms of helping with the implementation of the idea, I am conscious of my own limitations but I would do what I can to help.

To help me understand what is required from Stack, it would help to make the abstract more concrete, with a simple example. For example, if you had a plain vanilla stack new foo project, does a 'filepath' mean something like app/Main.hs or src/Lib.hs?

In this context, what is meant by 'all the exact compilation options'? If you command stack --verbose build --cabal-verbose, you can see the detail of how Stack is commanding Cabal (the library) (with the configure and build commands) and how (in turn) Cabal is commanding GHC. In this simple example, the configure step looks something like (on Windows):

--verbose=2 
--builddir=.stack-work\dist\8a54c84f 
configure 
--with-ghc=C:\Users\mikep\AppData\Local\Programs\stack\x86_64-windows\ghc-9.2.8\bin\ghc-9.2.8.exe 
--with-ghc-pkg=C:\Users\mikep\AppData\Local\Programs\stack\x86_64-windows\ghc-9.2.8\bin\ghc-pkg-9.2.8.exe 
--user 
--package-db=clear 
--package-db=global 
--package-db=C:\sr\snapshots\ee96b439\pkgdb 
--package-db=C:\Users\mikep\Documents\Code\Haskell\foo\.stack-work\install\0216c472\pkgdb 
--libdir=C:\Users\mikep\Documents\Code\Haskell\foo\.stack-work\install\0216c472\lib 
--bindir=C:\Users\mikep\Documents\Code\Haskell\foo\.stack-work\install\0216c472\bin 
--datadir=C:\Users\mikep\Documents\Code\Haskell\foo\.stack-work\install\0216c472\share 
--libexecdir=C:\Users\mikep\Documents\Code\Haskell\foo\.stack-work\install\0216c472\libexec 
--sysconfdir=C:\Users\mikep\Documents\Code\Haskell\foo\.stack-work\install\0216c472\etc 
--docdir=C:\Users\mikep\Documents\Code\Haskell\foo\.stack-work\install\0216c472\doc\foo-0.1.0.0 
--htmldir=C:\Users\mikep\Documents\Code\Haskell\foo\.stack-work\install\0216c472\doc\foo-0.1.0.0 
--haddockdir=C:\Users\mikep\Documents\Code\Haskell\foo\.stack-work\install\0216c472\doc\foo-0.1.0.0 
--dependency=base=base-4.16.4.0 
--extra-include-dirs=C:\Users\mikep\AppData\Local\Programs\stack\x86_64-windows\msys2-20210604\mingw64\include 
--extra-lib-dirs=C:\Users\mikep\AppData\Local\Programs\stack\x86_64-windows\msys2-20210604\mingw64\lib 
--extra-lib-dirs=C:\Users\mikep\AppData\Local\Programs\stack\x86_64-windows\msys2-20210604\mingw64\bin 
--exact-configuration 
--ghc-option=-fhide-source-paths

and the build step looks something like:

--verbose=2
--builddir=.stack-work\dist\8a54c84f 
build 
lib:foo 
exe:foo-exe 
--ghc-options " -fdiagnostics-color=always"

You may already know this, but Stack uses the version of Cabal (the library) that ships with the version of GHC in question, which may limit the availability to Stack of information provided by Cabal's --enable-build-info flag (it seems to me that the flag was first documented in the guide for Cabal 3.8).

[hasufell]

I'm somewhat familiar with stack codebase and don't mind contributing in this effort.

[mpilgrem]

Thinking out loud:

• other than at the command line, Stack is configured by a 'global' file common to all Stack projects (config.yaml - perhaps there is more than one on Unix-like operating systems: see https://docs.haskellstack.org/en/stable/yaml_configuration/#yaml-configuration), a unique project-level file (stack.yaml) and, for local packages within a Stack project, their individual package.yaml (Hpack) or Cabal files describing the package. Package description can be conditional, depending on, in part, Cabal flags and Stack's own configuration can include the setting or unsetting of Cabal flags;
• so, given a relative file path to a *.hs file, you need a function that identifies the project-level Stack configuration file. If the file is assumed to be relative to the working directory, you can follow Stack's own approach to identifying the relevant project-level configuration (which will be the project-level configuration for the 'global' project, as a last resort) - https://docs.haskellstack.org/en/stable/yaml_configuration/#yaml-configuration;
• once you have the project-level configuration (and, hence, know all the local packages in the Stack project) you can work out which local package(s) the *.hs is a 'member' of, by reference to the package descriptions - which might be none, one, or more than one. If more than one package (unlikely, but possible), I suppose you need a basis for picking just one - otherwise the 'compilation options' will not be unique for the file path.
• once you have the unique project (and any Cabal flags set in the Stack configuration) and the (unique) package (with its conditions satisfied), you should know what 'options' Stack will pass on to Cabal (the library) in respect of the file and, I suppose, what Cabal will pass on to GHC.

[fendor]

Thank you for your extensive comments!

Let me add some clarifications:

For example, if you had a plain vanilla stack new foo project, does a 'filepath' mean something like app/Main.hs or src/Lib.hs?

Yes, that is exactly correct. When talking about filepaths, I mean haskell source files that are part of a stack project.

In this context, what is meant by 'all the exact compilation options'?

Yes, what you correctly inferred. The problem, for example, with stack build is: it does too much work. It compiles the component/project, even though it doesn't need to, since we are going to rebuild all of that. For big projects, the start-up time would likely explode.

(it seems to me that the flag was first documented in the guide for Cabal 3.8).

That is true, but let's not delay improvements just because some people cannot benefit from it immediately.

---

I think stack ide and stack query are both fine commands, if we can tweak them to our needs, that'd be great :)

[mpilgrem]

@fendor, does the following assist?

As noted above, recent versions of Cabal (the library) offer the --enable-build-info flag. The Cabal documentation says that the flag causes Cabal to add a file build-info.json to the root of Cabal's 'build' directory.

stack Setup.hs configure --help suggests that the flag relates to the Cabal configure command. So, a Stack user can currently specify it for, say, all targets by adding to Stack's configuration file:

configure-options:
  $targets:
  - --enable-build-info

After a stack build, Cabal's 'build' directory is found at .stack-work/dist/<hash>. The location of that directory (relative to the Stack project-level confiuration file) is provided by stack path --dist-dir. With the above Stack configuration option set, it will include the build-info.json file.

For example, with a 'plain vanilla' stack new foo project (which currently uses Stackage LTS Haskell LTS 21.1, GHC 9.4.6 and Cabal-3.8.1.0), the build-info.json file contains (formatted):

{
  "cabal-lib-version": "3.8.1.0",
  "compiler": {
    "flavour": "ghc",
    "compiler-id": "ghc-9.4.6",
    "path": "C:\\Users\\mikep\\AppData\\Local\\Programs\\stack\\x86_64-windows\\ghc-9.4.6\\bin\\ghc-9.4.6.exe"
  },
  "components": [
    {
      "type": "lib",
      "name": "lib",
      "unit-id": "foo-0.1.0.0-RDnI1UeKkS9SDv3Gmnx7A",
      "compiler-args": [
        "-fbuilding-cabal-package",
        "-O",
        "-outputdir",
        ".stack-work\\dist\\51f21a8f\\build",
        "-odir",
        ".stack-work\\dist\\51f21a8f\\build",
        "-hidir",
        ".stack-work\\dist\\51f21a8f\\build",
        "-stubdir",
        ".stack-work\\dist\\51f21a8f\\build",
        "-i",
        "-i.stack-work\\dist\\51f21a8f\\build",
        "-isrc",
        "-i.stack-work\\dist\\51f21a8f\\build\\autogen",
        "-i.stack-work\\dist\\51f21a8f\\build\\global-autogen",
        "-I.stack-work\\dist\\51f21a8f\\build\\autogen",
        "-I.stack-work\\dist\\51f21a8f\\build\\global-autogen",
        "-I.stack-work\\dist\\51f21a8f\\build",
        "-IC:\\Users\\mikep\\AppData\\Local\\Programs\\stack\\x86_64-windows\\msys2-20230526\\mingw64\\include",
        "-optP-include",
        "-optP.stack-work\\dist\\51f21a8f\\build\\autogen\\cabal_macros.h",
        "-this-unit-id",
        "foo-0.1.0.0-RDnI1UeKkS9SDv3Gmnx7A",
        "-hide-all-packages",
        "-Wmissing-home-modules",
        "-no-user-package-db",
        "-package-db",
        "C:\\sr\\snapshots\\3fe291b3\\pkgdb",
        "-package-db",
        "C:\\Users\\mikep\\Documents\\Code\\Haskell\\foo\\.stack-work\\install\\9bf47987\\pkgdb",
        "-package-id",
        "base-4.17.2.0",
        "-XHaskell2010",
        "-Wall",
        "-Wcompat",
        "-Widentities",
        "-Wincomplete-record-updates",
        "-Wincomplete-uni-patterns",
        "-Wmissing-export-lists",
        "-Wmissing-home-modules",
        "-Wpartial-fields",
        "-Wredundant-constraints"
      ],
      "modules": [
        "Lib",
        "Paths_foo"
      ],
      "src-files": [],
      "hs-src-dirs": [
        "src"
      ],
      "src-dir": "C:\\Users\\mikep\\Documents\\Code\\Haskell\\foo\\",
      "cabal-file": ".\\foo.cabal"
    },
    {
      "type": "exe",
      "name": "exe:foo-exe",
      "unit-id": "foo-0.1.0.0-8kxBowjWnorJiiEogpBLXn-foo-exe",
      "compiler-args": [
        "-fbuilding-cabal-package",
        "-O",
        "-outputdir",
        ".stack-work\\dist\\51f21a8f\\build",
        "-odir",
        ".stack-work\\dist\\51f21a8f\\build",
        "-hidir",
        ".stack-work\\dist\\51f21a8f\\build",
        "-stubdir",
        ".stack-work\\dist\\51f21a8f\\build",
        "-i",
        "-i.stack-work\\dist\\51f21a8f\\build",
        "-iapp",
        "-i.stack-work\\dist\\51f21a8f\\build\\foo-exe\\autogen",
        "-i.stack-work\\dist\\51f21a8f\\build\\global-autogen",
        "-I.stack-work\\dist\\51f21a8f\\build\\foo-exe\\autogen",
        "-I.stack-work\\dist\\51f21a8f\\build\\global-autogen",
        "-I.stack-work\\dist\\51f21a8f\\build",
        "-IC:\\Users\\mikep\\AppData\\Local\\Programs\\stack\\x86_64-windows\\msys2-20230526\\mingw64\\include",
        "-optP-include",
        "-optP.stack-work\\dist\\51f21a8f\\build\\foo-exe\\autogen\\cabal_macros.h",
        "-hide-all-packages",
        "-Wmissing-home-modules",
        "-no-user-package-db",
        "-package-db",
        "C:\\sr\\snapshots\\3fe291b3\\pkgdb",
        "-package-db",
        "C:\\Users\\mikep\\Documents\\Code\\Haskell\\foo\\.stack-work\\install\\9bf47987\\pkgdb",
        "-package-id",
        "base-4.17.2.0",
        "-package-id",
        "foo-0.1.0.0-RDnI1UeKkS9SDv3Gmnx7A",
        "-XHaskell2010",
        "-Wall",
        "-Wcompat",
        "-Widentities",
        "-Wincomplete-record-updates",
        "-Wincomplete-uni-patterns",
        "-Wmissing-export-lists",
        "-Wmissing-home-modules",
        "-Wpartial-fields",
        "-Wredundant-constraints",
        "-threaded",
        "-rtsopts",
        "-with-rtsopts=-N"
      ],
      "modules": [
        "Paths_foo"
      ],
      "src-files": [
        "Main.hs"
      ],
      "hs-src-dirs": [
        "app"
      ],
      "src-dir": "C:\\Users\\mikep\\Documents\\Code\\Haskell\\foo\\",
      "cabal-file": ".\\foo.cabal"
    }
  ]
}

There is, currently, no Stack command-line equivalent of the configuration option above. However, it would be (I think) trivial to add a stack build --enable-build-info flag that was equivalent to setting the configuration file.

[fendor]

I think that works in general...

The only disadvantage is that we can't load a component if one of its dependencies doesn't build due to type errors, I believe. It would be much nicer, if we had something like haskell/cabal#8726 in stack as well.

However, I will play around with it, time to polish https://github.com/fendor/cabal-build-info/

[mpilgrem]

@fendor, I did not follow what was the relevant part of haskell/cabal#8726. The initial post refers to a goal of starting GHCi with more than one package loaded. stack ghci already has a --package option that allows you to load GHCi with an additional package: https://docs.haskellstack.org/en/stable/ghci/#specifying-extra-packages-to-build-or-depend-on

[fendor]

In the process of that PR, the author implemented the concept of a promised-dependency. Ie a dependency that is promised to exist at compile time but not at configure time. Currently, if you configure a package and one of the dependencies is not already installed in the package-db, the build crashes.

The promised-dependency features allows us to configure all components and then load them into ghci immediately, without having to build anything from stack's perspective, speeding up the startup time of HLS immensely. As a nice side effect, it allows us to load a project, even if not all local dependencies typecheck at the moment, since the build cannot fail in that case.

[mpilgrem]

@fendor, thanks. Adding here a link to the blog post on that topic: https://well-typed.com/blog/2023/03/cabal-multi-unit/.

[mpilgrem]

@fendor, I am returning to this issue. However, I realise that I am still not clear on what HLS needs from Stack.

For example, if HLS currently (assuming nothing has moved on since July 2023) just harvests GHC options (using a shim) from stack ghci <filename> when the latter calls ghc --interactive, is it just a matter of having (say) a stack ide ghc-options <filename> command that sends the same information to the standard output channel? I am conscious that stack ghci <filename> does various things before it invokes ghc --interactive.

[fendor]

Yes, exactly, that would work for us!

[mpilgrem]

@fendor, I have a draft stack ide ghc-options <filename> command at:

• Re #6154 Add Stack's ide ghc-options command #6735

and in the re6154 branch version of Stack (if not using GHCup to manage versions of Stack, it can be obtained by: stack upgrade --source-only --git --git-branch re6154).

No doubt it will need to be further refined, including as I understand better what HLS needs/wants.

Implementing it raised the following questions:

1. stack ghci <filename> sets -odir and -hidir, as:

   -odir=<absolute path to project root directory>\.stack-work\odir
   -hidir=<absolute path to project root directory>\.stack-work\odir
   

   Is HLS interested in those options, or does it choose its own location(s)? In my draft
   stack ide ghc-options <filename>, I currently assume HLS is not interested.

2. stack ghci <filename> sets -optP-include and
   -optP<absolute path to cabal_macros.h>, where cabal_macros.h is a Stack-generated file that
   combines all the CPP macros that Cabal (the library) automatically generates; it is put in
   a random directory. In my draft stack ide ghc-options <filename>, I assume that HLS is
   interested in the content of that file but will decide for itself what to do with the content
   (see below). Is that correct? Or perhaps HLS is not interested in the content at all - because
   something has to be built before Cabal (the library) will automatically generate the CPP
   macros (see the example output below)?

3. stack ghci <filename> prunes the configured GHC options that ghc --interactive will
   not accept (-O*, -debug, -threaded, -ticky, -static, -Werror). I did not know if HLS
   was interested in them or not. Is HLS interested in them? In my draft
   stack ide ghc-options <filename>, I output them separately (see below).

The output of my draft stack ide ghc-options <filename> to the standard output channel has the following format:

---
<GHC options passed to GHCi, other than the ones excluded as mentioned above, one per line>
---
<The content of the `cabal_macros.h` file>
---
<GHC options not passed to GHCi, one per line>
---

I implemented stack ide ghc-options <filename> by (a) duplicating stack ghci <filename> and (b) deleting everything that was not relevant to simply outputting the GHC options passed to --ghc --interactive and the content of the cabal_macros.h file. There may be scope for further simplification, but - initially - I prioritised reusing existing code.

Some other 'logging'-type information is still output to the standard error channel.

[mpilgrem]

So, for example:

> stack new test6154
> cd test6154
> stack ide ghc-options src/Lib.hs

generates, to standard error:

Using configuration for test6154:lib to load D:\Users\mike\Code\Haskell\test6154\src\Lib.hs

Warning: Didn't find expected autogen file:
         D:\Users\mike\Code\Haskell\test6154\.stack-work\dist\f24b2e15\build\autogen\cabal_macros.h

and, then, to standard output:

---
-i
-hide-all-packages
-XHaskell2010
-iD:\Users\mike\Code\Haskell\test6154\.stack-work\dist\f24b2e15\build
-iD:\Users\mike\Code\Haskell\test6154\src
-iD:\Users\mike\Code\Haskell\test6154\.stack-work\dist\f24b2e15\build\autogen
-iD:\Users\mike\Code\Haskell\test6154\.stack-work\dist\f24b2e15\build\global-autogen
-stubdir=D:\Users\mike\Code\Haskell\test6154\.stack-work\dist\f24b2e15\build
-ID:\sr\programs\x86_64-windows\msys2-20240727\mingw64\include
-LD:\sr\programs\x86_64-windows\msys2-20240727\mingw64\lib
-LD:\sr\programs\x86_64-windows\msys2-20240727\mingw64\bin
-package-id=base-4.19.2.0-943b
-Wall
-Wcompat
-Widentities
-Wincomplete-record-updates
-Wincomplete-uni-patterns
-Wmissing-export-lists
-Wmissing-home-modules
-Wpartial-fields
-Wredundant-constraints
-haddock
---
---
---

(In the Stack environment, GHC_PACKAGE_PATH is set to refer to the local package database, the snapshot package database and the global package database for the specified version of GHC.)

There is no dist\<hash>\build\autogen\cabal_macros.h file because, unlike stack ghci <filename>, stack ide ghc-options <filename> does not, itself, build anything.

After a stack build, the file exists and the content of the Stack-generated file is also output (extract only):

/* DO NOT EDIT: This file is automatically generated by Cabal */

/* package test6154-0.1.0.0 */
#ifndef VERSION_test6154
#define VERSION_test6154 "0.1.0.0"
#endif /* VERSION_test6154 */
#ifndef MIN_VERSION_test6154
#define MIN_VERSION_test6154(major1,major2,minor) (\
  (major1) <  0 || \
  (major1) == 0 && (major2) <  1 || \
  (major1) == 0 && (major2) == 1 && (minor) <= 0)
#endif /* MIN_VERSION_test6154 */
...

#ifndef CURRENT_PACKAGE_KEY
#define CURRENT_PACKAGE_KEY "test6154-0.1.0.0-DrhkEJjqpSaLuLOb4wVqHg"
#endif /* CURRENT_packageKey */
#ifndef CURRENT_COMPONENT_ID
#define CURRENT_COMPONENT_ID "test6154-0.1.0.0-DrhkEJjqpSaLuLOb4wVqHg"
#endif /* CURRENT_COMPONENT_ID */
#ifndef CURRENT_PACKAGE_VERSION
#define CURRENT_PACKAGE_VERSION "0.1.0.0"
#endif /* CURRENT_PACKAGE_VERSION */

#undef CURRENT_PACKAGE_KEY
#undef CURRENT_COMPONENT_ID

[mpilgrem]

Similarly, if a dependency is added to acme-missiles-0.3 (as an extra-dep, as that package is not in a Stackage snapshot) and Lib.hs imports Acme.Missiles, before building the output of my draft stack ide ghc-options src/Lib.hs includes (extract):

-package=acme-missiles-0.3
-package-id=base-4.19.2.0-943b

and after building (when acme-missiles-0.3 has been added to the snapshot package database referenced by GHC_PACKAGE_PATH in the Stack environment) the output includes (extract):

-package-id=acme-missiles-0.3-E9mKyDsckvc8jcWGFxGNEv
-package-id=base-4.19.2.0-943b

[fendor]

Ok, so, working through this:

(1)

Is HLS interested in those options, or does it choose its own location(s)?

We do override these options, so we are not interested, but I think there is still value to keep them in the output.

(2) Yeah, HLS is definitely interested in the contents of this file. However, I am not quite sure if I understand the rest of the question correctly, HLS is, at the moment, not generating this CPP macros based on the package database, or anything like that.

(3) We want to mirror stack repl behaviour as closely as possible, so if stack repl filters them out (which is fair and good), then it is fine for HLS to also get these filtered flags.

---

Since stack ide ghc-options doesn't make sure the file/component or its dependencies are already compiled, what would be the proposed workflow? How would HLS have to invoke stack, to make sure the compilation options retrieved via stack ide ghc-options src/Lib.hs are sufficient to compile all the modules of the component the file target src/Lib.hs belongs to?
IIRC, stack build src/Lib.hs is not accepted.

Another thing we need is a list of modules that can be compiled with these flags. Even if stack build was executed before stack ide ghc-options src/Lib.hs, what are the module targets that can be compiled now with these options?
Calling stack ide ghc-options for all Haskell files individually in a project would be too slow, imo.

The ghc-options for library also should have a -this-unit-id flag.
When we load the test suite and the library with different flags, then the test suite will contain this dependency: -package=hello-world-0.1.0.0. However, HLS needs to know that hello-world-0.1.0.0 is a home unit! Otherwise, we might get confused and try to load functions from the package database. The bug haskell/haskell-language-server#1822 is a manifestation of this issue.

One more thing, in multiple home unit sessions, we need to uphold the closure property. Since stack supports internal libraries, this is a thing we need to consider as well.