Help needed: Updating GHC to 8.4.3

[Ricardo Wurmus]

Hi Guix,

We need to update to GHC 8.4.3 because that’s the current version that
LTS Haskell depends on.  (Some packages I’m working on already depend on
packages that only come with GHC 8.4.3.)

Attached is a WIP patch to update GHC to 8.4.3.  (The patch renames the
existing “ghc-8” package to “ghc-8.0” and sets the default GHC at
“ghc-8.0” so that we can switch the default once it has been fully built
by the build farm.)

The problem is: it does not work, although the build phase succeeds.
Something with linking with other libraries fails.  Here’s an excerpt
from the end of the build phase:

--8<---------------cut here---------------start------------->8---
…
Warning: Distribution.Simple.Program: could not find link destinations for:
    findExecutable
Warning: Distribution.Simple.Compiler: could not find link destinations for:
    extensionToFlag
Warning: Distribution.Simple.Command: could not find link destinations for:
    FieldDescr
Warning: Distribution.Simple.Setup: could not find link destinations for:
    programDbPaths dispModSubstEntry
Warning: Distribution.PackageDescription.Configuration: could not find link 
destinations for:
    PDTagged PDNull
Warning: Distribution.PackageDescription.Parsec: could not find link 
destinations for:
    sectionizeFields
Warning: Distribution.Types.InstalledPackageInfo.FieldGrammar: could not find 
link destinations for:
    Basic
Warning: Distribution.InstalledPackageInfo: could not find link destinations 
for:
    LineNo
Warning: Distribution.Simple.PackageIndex: could not find link destinations for:
    DepUniqueKey
Warning: Distribution.Simple.BuildTarget: could not find link destinations for:
    matchInexactly findMatch None Unambiguous Ambiguous
Warning: Distribution.Simple.Build.Macros: could not find link destinations for:
    generateToolVersionMacros
Warning: Distribution.Simple.GHC: could not find link destinations for:
    gbuild GBuildMode
Warning: Distribution.Backpack.Configure: could not find link destinations for:
    PreExistingComponent
…
Warning: HscMain: could not find link destinations for:
    RenamedStuff
Warning: TcSplice: could not find link destinations for:
    annotThType
Warning: InteractiveEval: could not find link destinations for:
    RttiType
Warning: GhcMake: could not find link destinations for:
    NodeMap BuildModule LogQueue upsweep CompilationGraph
Warning: GHC: could not find link destinations for:
    OnOff LogOutput SseVersion BmiVersion modSummary tm_internals DesugaredMod 
WhetherHasFamInst holeUnitId
Warning: DriverBkp: could not find link destinations for:
    BkpEnv BkpM backpackProgressMsg
cd libraries && sh gen_contents_index --intree
phase `build' succeeded after 2940.0 seconds
--8<---------------cut here---------------end--------------->8---

Here’s how the check phase fails:

--8<---------------cut here---------------start------------->8---
starting phase `check'
make -C testsuite/tests CLEANUP=1 SUMMARY_FILE=../../testsuite_summary.txt
make[1]: Entering directory 
'/tmp/guix-build-ghc-8.4.3.drv-0/ghc-8.4.3/testsuite/tests'
"/tmp/guix-build-ghc-8.4.3.drv-0/ghc-8.4.3/inplace/bin/ghc-stage2" --make -o 
../mk/ghc-config ../mk/ghc-config.hs
[1 of 1] Compiling Main             ( ../mk/ghc-config.hs, ../mk/ghc-config.o )
Linking ../mk/ghc-config ...
../mk/ghc-config 
"/tmp/guix-build-ghc-8.4.3.drv-0/ghc-8.4.3/inplace/bin/ghc-stage2" 
>"../mk/ghcconfig_tmp_guix-build-ghc-8.4.3.drv-0_ghc-8.4.3_inplace_bin_ghc-stage2.mk";
 if [ $? != 0 ]; then rm -f 
"../mk/ghcconfig_tmp_guix-build-ghc-8.4.3.drv-0_ghc-8.4.3_inplace_bin_ghc-stage2.mk";
 exit 1; fi
../mk/ghc-config: error while loading shared libraries: libgmp.so.10: cannot 
open shared object file: No such file or directory
Looks like you don't have timeout, building it first...
make -C ../timeout all
make[2]: Entering directory 
'/tmp/guix-build-ghc-8.4.3.drv-0/ghc-8.4.3/testsuite/timeout'
../mk/ghc-config 
"/tmp/guix-build-ghc-8.4.3.drv-0/ghc-8.4.3/inplace/bin/ghc-stage2" 
>"../mk/ghcconfig_tmp_guix-build-ghc-8.4.3.drv-0_ghc-8.4.3_inplace_bin_ghc-stage2.mk";
 if [ $? != 0 ]; then rm -f 
"../mk/ghcconfig_tmp_guix-build-ghc-8.4.3.drv-0_ghc-8.4.3_inplace_bin_ghc-stage2.mk";
 exit 1; fi
../mk/ghc-config: error while loading shared libraries: libgmp.so.10: cannot 
open shared object file: No such file or directory
rm -f -f TimeMe.o TimeMe.hi TimeMe TimeMe.exe
python3 calibrate 
'/tmp/guix-build-ghc-8.4.3.drv-0/ghc-8.4.3/inplace/bin/ghc-stage1' > 
calibrate.out
TimeMe: error while loading shared libraries: libgmp.so.10: cannot open shared 
object file: No such file or directory
rm -rf install-inplace
mkdir install-inplace
mkdir install-inplace/bin
cp timeout.py install-inplace/bin/timeout.py
echo '#!/gnu/store/rbrandv7anzjxqkr40d7fkanzssslk4b-bash-minimal-4.4.19/bin/sh' 
> install-inplace/bin/timeout
echo 'exec "python3" $0.py "$@"' >> install-inplace/bin/timeout
chmod +x install-inplace/bin/timeout
make[2]: Leaving directory 
'/tmp/guix-build-ghc-8.4.3.drv-0/ghc-8.4.3/testsuite/timeout'
PYTHON="python3" "python3" ../driver/runtests.py  -e 
"ghc_compiler_always_flags='-dcore-lint -dcmm-lint -no-user- -rtsopts  
-dno-debug-output'" -e config.compiler_debugged= -e ghc_with_native_codegen=0 
-e config.have_vanilla=True -e config.have_dynamic=True -e 
config.have_profiling=True -e ghc_with_threaded_rts=0 -e ghc_with_dynamic_rts=0 
-e config.have_interp=True -e config.unregisterised=False -e 
config.ghc_dynamic_by_default=False -e config.ghc_dynamic=False -e 
ghc_with_smp=0 -e ghc_with_llvm=0 -e windows=False -e darwin=False -e 
config.in_tree_compiler=True -e config.cleanup=True -e config.local=True 
--rootdir=. --config-file=../config/ghc -e 'config.confdir="../config"' -e 
'config.platform=""' -e 'config.os=""' -e 'config.arch=""' -e 
'config.wordsize=""' -e 'config.timeout=int() or config.timeout' -e 
'config.exeext=""' -e 
'config.top="/tmp/guix-build-ghc-8.4.3.drv-0/ghc-8.4.3/testsuite"' --config 
'compiler="/tmp/guix-build-ghc-8.4.3.drv-0/ghc-8.4.3/inplace/bin/ghc-stage2"' 
--config 
'ghc_pkg="/tmp/guix-build-ghc-8.4.3.drv-0/ghc-8.4.3/inplace/bin/ghc-pkg"' 
--config 
'haddock="/tmp/guix-build-ghc-8.4.3.drv-0/ghc-8.4.3/inplace/bin/haddock"' 
--config 'hp2ps="/tmp/guix-build-ghc-8.4.3.drv-0/ghc-8.4.3/inplace/bin/hp2ps"' 
--config 'hpc="/tmp/guix-build-ghc-8.4.3.drv-0/ghc-8.4.3/inplace/bin/hpc"' 
--config 'gs="gs"' --config 
'timeout_prog="../timeout/install-inplace/bin/timeout"' -e "config.stage=" 
--summary-file "../../testsuite_summary.txt"   
--rootdir=../../libraries/array/tests  --rootdir=../../libraries/base/tests  
--rootdir=../../libraries/binary/tests  
--rootdir=../../libraries/bytestring/tests  
--rootdir=../../libraries/containers/tests  
--rootdir=../../libraries/deepseq/tests  
--rootdir=../../libraries/directory/tests  
--rootdir=../../libraries/filepath/tests  
--rootdir=../../libraries/ghc-compact/tests  
--rootdir=../../libraries/ghc-prim/tests  
--rootdir=../../libraries/haskeline/tests  --rootdir=../../libraries/hpc/tests  
--rootdir=../../libraries/pretty/tests  --rootdir=../../libraries/process/tests 
 --rootdir=../../libraries/stm/tests  
--rootdir=../../libraries/template-haskell/tests  
--rootdir=../../libraries/text/tests  --rootdir=../../libraries/unix/tests \
         \
         \
         \
         \
         \
         \

Traceback (most recent call last):
  File "../driver/runtests.py", line 65, in <module>
    exec(e)
  File "<string>", line 1
    config.compiler_debugged=
                            ^
SyntaxError: invalid syntax
make[1]: *** [../mk/test.mk:303: test] Error 1
make[1]: Leaving directory 
'/tmp/guix-build-ghc-8.4.3.drv-0/ghc-8.4.3/testsuite/tests'
make: *** [Makefile:223: test] Error 2
--8<---------------cut here---------------end--------------->8---

It says “invalid syntax” because the value for “$(GhcDebugged)” is
unavailable, because the ghc-config stuff above failed.

GHC 8.0 had been patched with
"ghc-dont-pass-linker-flags-via-response-files.patch" to avoid using
response files with the linker, because our ld-wrapper doesn’t seem to
behave right in some edge case that GHC depends on.  I tried porting the
patch to GHC 8.4.3 by applying this snippet:

--8<---------------cut here---------------start------------->8---
      ;; FIXME: Our ld-wrapper does not seem to fully support response files,
      ;; thus breaking the build.  We use "runSomethingFiltered" instead of
      ;; "runSomethingResponseFile" to work around this problem.  Note the
      ;; slightly different type.
      (snippet
       '(begin
          (substitute* "compiler/main/SysTools/Tasks.hs"
            (("runSomethingResponseFile dflags cc_filter \"C Compiler\" p args2 
mb_env")
             "runSomethingFiltered dflags cc_filter \"C Compiler\" p args2 
Nothing mb_env")
            (("runSomethingResponseFile dflags ld_filter \"Linker\" p args2 
mb_env")
             "runSomethingFiltered dflags ld_filter \"Linker\" p args2 Nothing 
mb_env"))
          #t))
--8<---------------cut here---------------end--------------->8---

We are using “runSomethingFiltered” instead of
“runSomethingResponseFile”, but I suppose something about the flags
still isn’t quite right.

Any ideas?

Unfortunately, Nixpkgs wasn’t of much help in this case.  AFAIU they
modified their ld wrapper so that no patch of GHC is required.

--
Ricardo

>From 7485bad6a8b8d826c8c8e8f818027c6a51433971 Mon Sep 17 00:00:00 2001
From: Ricardo Wurmus <address@hidden>
Date: Fri, 17 Aug 2018 16:55:43 +0200
Subject: [PATCH] WIP Add ghc 8.4.3

---
 gnu/packages/haskell.scm | 113 ++++++++++++++++++++++++++++++++++++++-
 1 file changed, 111 insertions(+), 2 deletions(-)

diff --git a/gnu/packages/haskell.scm b/gnu/packages/haskell.scm
index 7347c8753..d9e65ee4a 100644
--- a/gnu/packages/haskell.scm
+++ b/gnu/packages/haskell.scm
@@ -320,7 +320,7 @@ top of CLISP.")
 interactive environment for the functional language Haskell.")
     (license license:bsd-3)))
 
-(define-public ghc-8
+(define-public ghc-8.0
   (package
     (name "ghc")
     (version "8.0.2")
@@ -432,7 +432,116 @@ interactive environment for the functional language 
Haskell.")
 interactive environment for the functional language Haskell.")
     (license license:bsd-3)))
 
-(define-public ghc ghc-8)
+(define-public ghc-8
+  (package (inherit ghc-8.0)
+    (name "ghc")
+    (version "8.4.3")
+    (source
+     (origin
+      (method url-fetch)
+      (uri (string-append "https://www.haskell.org/ghc/dist/";
+                          version "/" name "-" version "-src.tar.xz"))
+      (sha256
+       (base32 "1mk046vb561j75saz05rghhbkps46ym5aci4264dwc2qk3dayixf"))
+      (modules '((guix build utils)))
+      ;; FIXME: Our ld-wrapper does not seem to fully support response files,
+      ;; thus breaking the build.  We use "runSomethingFiltered" instead of
+      ;; "runSomethingResponseFile" to work around this problem.  Note the
+      ;; slightly different type.
+      (snippet
+       '(begin
+          (substitute* "compiler/main/SysTools/Tasks.hs"
+            (("runSomethingResponseFile dflags cc_filter \"C Compiler\" p 
args2 mb_env")
+             "runSomethingFiltered dflags cc_filter \"C Compiler\" p args2 
Nothing mb_env")
+            (("runSomethingResponseFile dflags ld_filter \"Linker\" p args2 
mb_env")
+             "runSomethingFiltered dflags ld_filter \"Linker\" p args2 Nothing 
mb_env"))
+          #t))))
+    (inputs
+     `(("gmp" ,gmp)
+       ("ncurses" ,ncurses)
+       ("libffi" ,libffi)))
+    (native-inputs
+     `(("perl" ,perl)
+       ("python" ,python)               ; for tests
+       ("ghostscript" ,ghostscript)     ; for tests
+       ;; GHC 8.4.3 is built with GHC 8.
+       ("ghc-bootstrap" ,ghc-8.0)
+       ("ghc-testsuite"
+        ,(origin
+           (method url-fetch)
+           (uri (string-append
+                 "https://www.haskell.org/ghc/dist/";
+                 version "/" name "-" version "-testsuite.tar.xz"))
+           (sha256
+            (base32 
"1z55b1z0m3plqd2d1ks6w5wvx7igm7zsk3i4v7cms003z0as0hzz"))))))
+    (arguments
+     `(#:test-target "test"
+       ;; We get a smaller number of test failures by disabling parallel test
+       ;; execution.
+       #:parallel-tests? #f
+
+       ;; The DSOs use $ORIGIN to refer to each other, but (guix build
+       ;; gremlin) doesn't support it yet, so skip this phase.
+       #:validate-runpath? #f
+
+       ;; Don't pass --build=<triplet>, because the configure script
+       ;; auto-detects slightly different triplets for --host and --target and
+       ;; then complains that they don't match.
+       #:build #f
+
+       #:configure-flags
+       (list
+        (string-append "--with-gmp-libraries="
+                       (assoc-ref %build-inputs "gmp") "/lib")
+        (string-append "--with-gmp-includes="
+                       (assoc-ref %build-inputs "gmp") "/include")
+        "--with-system-libffi"
+        (string-append "--with-ffi-libraries="
+                       (assoc-ref %build-inputs "libffi") "/lib")
+        (string-append "--with-ffi-includes="
+                       (assoc-ref %build-inputs "libffi") "/include")
+        (string-append "--with-curses-libraries="
+                       (assoc-ref %build-inputs "ncurses") "/lib")
+        (string-append "--with-curses-includes="
+                       (assoc-ref %build-inputs "ncurses") "/include"))
+       #:phases
+       (modify-phases %standard-phases
+         (add-after 'unpack 'unpack-testsuite
+           (lambda* (#:key inputs #:allow-other-keys)
+             (invoke "tar" "xvf"
+                     (assoc-ref inputs "ghc-testsuite")
+                     "--strip-components=1")
+             #t))
+         (add-after 'unpack-testsuite 'fix-shell-wrappers
+           (lambda _
+             (substitute* '("driver/ghci/ghc.mk"
+                            "utils/mkdirhier/ghc.mk"
+                            "rules/shell-wrapper.mk")
+               (("echo '#!/bin/sh'")
+                (format #f "echo '#!~a'" (which "sh"))))
+             #t))
+         (add-before 'build 'fix-references
+           (lambda _
+             (substitute* '("testsuite/timeout/Makefile"
+                            "testsuite/timeout/timeout.py"
+                            "testsuite/timeout/timeout.hs"
+                            
"testsuite/tests/programs/life_space_leak/life.test"
+                            ;; libraries
+                            "libraries/process/System/Process/Posix.hs"
+                            "libraries/process/tests/process001.hs"
+                            "libraries/process/tests/process002.hs"
+                            "libraries/unix/cbits/execvpe.c")
+               (("/bin/sh") (which "sh"))
+               (("/bin/ls") (which "ls"))
+               (("/bin/rm") "rm"))
+             #t))
+         (add-before 'build 'fix-environment
+           (lambda _
+             (unsetenv "GHC_PACKAGE_PATH")
+             (setenv "CONFIG_SHELL" (which "bash"))
+             #t)))))))
+
+(define-public ghc ghc-8.0)
 
 (define-public ghc-hostname
   (package
-- 
2.18.0