Support for alternative GHC backends: extra-compilation-artifacts directory installation (#8662)

* Cbits archive for c, cxx, cmm, asm sources

Build and install extra archive file for non-Haskell code.
It allows to build alternative backends and compilation pipelines for GHC.
GHC plugins can observe and process Haskell related code,
while the 'cbits' archive supply the foreign code parts.

* Install extra module related (modpak) files when available

Support '.modpak' file installation. Modpak is an optional file,
it is generated by GHC plugins (or other tools) for Haskell modules.
The modpak file format is not set, it could be a zip file or other format.
In modpak files GHC plugins can store additional data for Haskell modules.
With modpak files Cabal can support alternative GHC backends out of the box.

* add changelog entry for cbits and modpak feature

* Support for extra compilation artifacts

GHC plugins can store custom data in the 'extra-compilation-artifacts' directory
which gets installed with the package.

This is a generalization of the previous design. See the discussion in PR #8662.

* documentation for extra-compilation-artifacts installation

Author: csabahruska

Cabal/src/Distribution/Simple/GHC.hs

@@ -118,7 +118,7 @@ import System.Directory
          ( doesFileExist, doesDirectoryExist
          , getAppUserDataDirectory, createDirectoryIfMissing
          , canonicalizePath, removeFile, renameFile, getDirectoryContents
-         , makeRelativeToCurrentDirectory )
+         , makeRelativeToCurrentDirectory, doesDirectoryExist )
 import System.FilePath          ( (</>), (<.>), takeExtension
                                 , takeDirectory, replaceExtension
                                 ,isRelative )
@@ -2005,6 +2005,9 @@ installLib verbosity lbi targetDir dynlibTargetDir _builtDir pkg lib clbi = do
   whenProf    $ copyModuleFiles "p_hi"
   whenShared  $ copyModuleFiles "dyn_hi"
 
+  -- copy extra compilation artifacts that ghc plugins may produce
+  copyDirectoryIfExists "extra-compilation-artifacts"
+
   -- copy the built library files over:
   whenHasCode $ do
     whenVanilla $ do
@@ -2079,6 +2082,12 @@ installLib verbosity lbi targetDir dynlibTargetDir _builtDir pkg lib clbi = do
       findModuleFilesEx verbosity [builtDir] [ext] (allLibModules lib clbi)
       >>= installOrdinaryFiles verbosity targetDir
 
+    copyDirectoryIfExists dirName = do
+      let src = builtDir  </> dirName
+          dst = targetDir </> dirName
+      dirExists <- doesDirectoryExist src
+      when dirExists $ copyDirectoryRecursive verbosity src dst
+
     compiler_id = compilerId (compiler lbi)
     platform = hostPlatform lbi
     uid = componentUnitId clbi

cabal-testsuite/PackageTests/ExtraCompilationArtifacts/Lib.hs

@@ -0,0 +1,3 @@
+module Lib where
+
+bar = ()

cabal-testsuite/PackageTests/ExtraCompilationArtifacts/lib.cabal

@@ -0,0 +1,7 @@
+cabal-version:      2.4
+name:               lib
+version:            0.1.0.0
+library
+    exposed-modules:  Lib
+    build-depends:    base
+    default-language: Haskell2010

cabal-testsuite/PackageTests/ExtraCompilationArtifacts/test.hs

@@ -0,0 +1,32 @@
+import Distribution.Simple.LocalBuildInfo
+import Test.Cabal.Prelude
+import System.Directory
+import System.FilePath
+
+-- Test if extra-compilation-artifacts are installed
+main = setupAndCabalTest . recordMode DoNotRecord $ do
+  withPackageDb $ do
+    setup "configure" []
+    setup "build" []
+    generateExtraCompArtifactsToBuildDir
+    setup "copy" []
+
+    lbi <- getLocalBuildInfoM
+    let installedLibPath = libdir $ absoluteInstallDirs (localPkgDescr lbi) lbi NoCopyDest
+
+    shouldExist $ installedLibPath </> "extra-compilation-artifacts" </> "ghc-plugin-X" </> "data-dir" </> "content-A.txt"
+    shouldExist $ installedLibPath </> "extra-compilation-artifacts" </> "ghc-plugin-X" </> "data-dir" </> "content-B.txt"
+    shouldExist $ installedLibPath </> "extra-compilation-artifacts" </> "ghc-plugin-Y" </> "content-Y.txt"
+
+generateExtraCompArtifactsToBuildDir :: TestM ()
+generateExtraCompArtifactsToBuildDir = do
+  -- extra compilation artifacts can be generated optionally by ghc plugins
+  dist_dir <- fmap testDistDir getTestEnv
+  let genArtifact fname = liftIO $ do
+        let dst = dist_dir </> "build" </> "extra-compilation-artifacts" </> fname
+        createDirectoryIfMissing True (takeDirectory dst)
+        writeFile dst ""
+  genArtifact $ "ghc-plugin-X" </> "data-dir" </> "content-A.txt"
+  genArtifact $ "ghc-plugin-X" </> "data-dir" </> "content-B.txt"
+  genArtifact $ "ghc-plugin-Y" </> "content-Y.txt"
+

changelog.d/pr-8662

@@ -0,0 +1,9 @@
+synopsis: Installation of extra-compilation-artifacts directory
+packages: Cabal
+prs: #8662
+issues:
+description: {
+
+- GHC plugins now can store custom data in the 'extra-compilation-artifacts' directory which gets installed with the package.
+
+}

doc/setup-commands.rst

@@ -9,7 +9,9 @@ GHC provides the commands ``runhaskell`` and ``runghc`` (they are equivalent)
 to allow you to run Haskell programs without first having to compile them
 (scripts). The low-level Cabal interface is implemented using ``Setup.hs``
 scripts. You should prefer using higher level interface provided by nix-style
-builds.
+builds. However, the documentation of the low level interface below may be helpful
+to high level interface users as well, because it delves into internal details
+common to both and omitted elsewhere.
 
 ::
 
@@ -1104,6 +1106,10 @@ Copy the files into the install locations and (for library packages)
 register the package with the compiler, i.e. make the modules it
 contains available to programs.
 
+Additionally for GHC the ``extra-compilation-artifacts`` directory is copied if present.
+GHC plugins can store extra data in subfolders.
+(e.g. *extra-compilation-artifacts/PLUGIN_NAME/HS_MODULE.txt*)
+
 The `install locations <#installation-paths>`__ are determined by
 options to `runhaskell Setup.hs configure`_.