flycheck-haskell breaks with library-vanilla:false and shared:true, executable-dynamic:true under cabal sandbox

[flycheck]

flycheck-haskell breaks with library-vanilla:false and shared:true, executable-dynamic:true under cabal sandbox

[swsnr]

@da-x I'm sorry, but your description is a little imprecise. Which imports fail? Imports of dependencies, or of modules from your package? Also, what does “fail” mean?

Also, I'm not sure about the last paragraph. I might be mistaken, but as far as I know the default of library-profiling is False, not True, so I'm not sure what you actually changed.

Please try to be more precise. At best, give me a recipe to reproduce the issue from scratch, ideally with a vanilla .cabal/config. Also, please show me the exact invocation of the GHC syntax checker, by typing C-c ! C-c in an affected buffer, selecting haskell-ghc, and copy and paste the output in the compilation buffer that pops up.

[da-x]

Oops, sorry - I meant to write library-vanilla, and instead copy-pasted the wrong variable. Edited. I'm happy to provide you with more details.

The imports that fail (ghc's "Could not find module") are dependencies of the local package, i.e. stuff that comes from the sandbox.

I managed to reproduce it with yaml package. But I guess it would break on any package that has dependencies that come from the cabal sandbox.

1. Modify ~/.cabal/config like above

cabal unpack yaml-0.8.10
cd yaml-0.8.10
cabal sandbox init
cabal install

So far so good. But then try to open Text/Libyaml.hs, followed your suggestion with C-c ! C-c. It would first break on CPP (an unrelated issue) but if you edit the #ifdefs away it would break on import, whereas it would have not if the same procedure has been executed with library-vanilla:true.

ghc -Wall -fno-code -no-user-package-db -package-db /home/dan/test/yaml-0.8.10/.cabal-sandbox/x86_64-linux-ghc-7.8.3-packages.conf.d -i/home/dan/test/yaml-0.8.10/dist/build -i/home/dan/test/yaml-0.8.10/dist/build/autogen -i/home/dan/test/yaml-0.8.10/dist/build/json2yaml/json2yaml-tmp -i/home/dan/test/yaml-0.8.10/dist/build/yaml2json/yaml2json-tmp -i/home/dan/test/yaml-0.8.10/ -i/home/dan/test/yaml-0.8.10/ -x hs /home/dan/test/yaml-0.8.10/Text/Libyaml.hs

/home/dan/test/yaml-0.8.10/Text/Libyaml.hs:50:8:
    Could not find module ‘Control.Monad.Trans.Resource’
    There are files missing in the ‘resourcet-1.1.3’ package,
    try running 'ghc-pkg check'.
    Use -v to see a list of the files searched for.

/home/dan/test/yaml-0.8.10/Text/Libyaml.hs:51:8:
    Could not find module ‘Data.Conduit’
    There are files missing in the ‘conduit-1.2.3’ package,
    try running 'ghc-pkg check'.
    Use -v to see a list of the files searched for.

Pasted 'cabal sandbox hc-pkg list' below just to see the dependencies are really there as far as the sandbox is concerned:

...
...
/home/dan/test/yaml-0.8.10/.cabal-sandbox/x86_64-linux-ghc-7.8.3-packages.conf.d
   aeson-0.6.2.1
   blaze-builder-0.3.3.4
   conduit-1.2.3
   dlist-0.7.1
   exceptions-0.6.1
   lifted-base-0.2.3.0
   mmorph-1.0.4
   monad-control-0.3.3.0
   nats-0.2
   resourcet-1.1.3
   scientific-0.3.3.2
   semigroups-0.15.4
   transformers-base-0.4.3
   void-0.6.1
   yaml-0.8.10

[swsnr]

@da-x Well, there are no dependencies anymore. By disabling vanilla libraries, you've just opted out from building the dependencies. You only have the profiling libraries left, but GHC doesn't look for these, because Flycheck doesn't pass -prof to GHC.

I'd argue that Flycheck is right here: It's a configuration issue on your side.

Is there a reason why you disable vanilla libraries? As far as I know, vanilla and profiling libraries can co-exist side by side, so there's no need to disable the former in order to get the latter. You just need to enable profiling for your own package with cabal configure -p --enable-executable-profiling.

[da-x]

(edited for typos/formatting)

With "library-vanilla: True" and "shared:True", ghc links both the static and the dynamic binaries of the libraries, e.g:

-rw-r--r-- 1 dan dan 777792 Dec 11 10:33 ./lib/x86_64-linux-ghc-7.8.3/scientific-0.3.3.2/libHSscientific-0.3.3.2.a
-rwxr-xr-x 1 dan dan 582122 Dec 11 10:33 ./lib/x86_64-linux-ghc-7.8.3/scientific-0.3.3.2/libHSscientific-0.3.3.2-ghc7.8.3.so

This increases the storage footprint of the cabal sandbox by a factor 2. However with executable-dynamic:true the static libraries (*.so files) are not used, and therefore we don't need to link the static archive (*.a files), and therefore we can use library-vanilla:false. From aspects unrelated to flycheck, this seems to be a valid ghc/cabal state (with 7.8, I am not sure about earlier versions).

I also have profiling disabled. The purpose of this configuration is to avoid linking static libraries completely, so that there are no static archive files under the cabal sandbox, and in addition, the executable under the bin directory are dynamically linked to their dependencies in the cabal sandbox which are all arriving as shared libraries. Dynamically linked binaries are much smaller. For example when building the hakyll package the result binary is a few KBs instead of 80 MB.

I am not sure that flycheck is right here, as 'cabal repl' is able to find the dependencies inside the sandbox even though they are no static libraries (for the yaml package, I needed to add -fPIC to cc-options, will push this fix upstream). I think this information can be transferred to flycheck via the get-cabal-configuration.hs method.

$ cabal repl
Preprocessing library yaml-0.8.10...
GHCi, version 7.8.3: http://www.haskell.org/ghc/  :? for help
Loading package ghc-prim ... linking ... done.
Loading package integer-gmp ... linking ... done.
Loading package base ... linking ... done.
Loading package array-0.5.0.0 ... linking ... done.
Loading package deepseq-1.3.0.2 ... linking ... done.
Loading package bytestring-0.10.4.0 ... linking ... done.
Loading package text-1.1.0.0 ... linking ... done.
Loading package hashable-1.2.2.0 ... linking ... done.
Loading package scientific-0.3.3.2 ... linking ... done.
Loading package filepath-1.3.0.2 ... linking ... done.
Loading package old-locale-1.0.0.6 ... linking ... done.
Loading package time-1.4.2 ... linking ... done.
Loading package unix-2.7.0.1 ... linking ... done.
Loading package directory-1.2.1.0 ... linking ... done.
Loading package containers-0.5.5.1 ... linking ... done.
Loading package transformers-0.3.0.0 ... linking ... done.
Loading package mtl-2.1.3.1 ... linking ... done.
Loading package exceptions-0.6.1 ... linking ... done.
Loading package stm-2.4.2 ... linking ... done.
Loading package transformers-base-0.4.3 ... linking ... done.
Loading package monad-control-0.3.3.0 ... linking ... done.
Loading package lifted-base-0.2.3.0 ... linking ... done.
Loading package mmorph-1.0.4 ... linking ... done.
Loading package resourcet-1.1.3 ... linking ... done.
Loading package nats-0.2 ... linking ... done.
Loading package unordered-containers-0.2.4.0 ... linking ... done.
Loading package semigroups-0.15.4 ... linking ... done.
Loading package void-0.6.1 ... linking ... done.
Loading package conduit-1.2.3 ... linking ... done.
Loading package attoparsec-0.10.4.0 ... linking ... done.
Loading package blaze-builder-0.3.3.4 ... linking ... done.
Loading package dlist-0.7.1 ... linking ... done.
Loading package syb-0.4.1 ... linking ... done.
Loading package pretty-1.1.1.1 ... linking ... done.
Loading package template-haskell ... linking ... done.
Loading package primitive-0.5.2.1 ... linking ... done.
Loading package vector-0.10.9.1 ... linking ... done.
Loading package aeson-0.6.2.1 ... linking ... done.
Loading object (static) dist/build/c/helper.o ... done
Loading object (static) dist/build/libyaml/api.o ... done
Loading object (static) dist/build/libyaml/dumper.o ... done
Loading object (static) dist/build/libyaml/emitter.o ... done
Loading object (static) dist/build/libyaml/loader.o ... done
Loading object (static) dist/build/libyaml/parser.o ... done
Loading object (static) dist/build/libyaml/reader.o ... done
Loading object (static) dist/build/libyaml/scanner.o ... done
Loading object (static) dist/build/libyaml/writer.o ... done
final link ... done
[1 of 6] Compiling Text.Libyaml     ( Text/Libyaml.hs, interpreted )
[2 of 6] Compiling Data.Yaml        ( Data/Yaml.hs, interpreted )
[3 of 6] Compiling Data.Yaml.Aeson  ( Data/Yaml/Aeson.hs, interpreted )
[4 of 6] Compiling Data.Yaml.Builder ( Data/Yaml/Builder.hs, interpreted )
[5 of 6] Compiling Data.Yaml.Parser ( Data/Yaml/Parser.hs, interpreted )
[6 of 6] Compiling Data.Yaml.Include ( Data/Yaml/Include.hs, interpreted )
Ok, modules loaded: Text.Libyaml, Data.Yaml, Data.Yaml.Aeson, Data.Yaml.Builder, Data.Yaml.Parser, Data.Yaml.Include.
*Text.Libyaml> 

/proc/<pid>/maps shows the loaded DSOs from the cabal sandbox:

...
7f34c08fb000-7f34c0905000 r-xp 00000000 08:04 12587335                   /home/dan/.cabal/test/yaml-0.8.10/.cabal-sandbox/lib/x86_64-linux-ghc-7.8.3/void-0.6.1/libHSvoid-0.6.1-ghc7.8.3.so
7f34c0905000-7f34c0b04000 ---p 0000a000 08:04 12587335                   /home/dan/.cabal/test/yaml-0.8.10/.cabal-sandbox/lib/x86_64-linux-ghc-7.8.3/void-0.6.1/libHSvoid-0.6.1-ghc7.8.3.so
7f34c0b04000-7f34c0b05000 r--p 00009000 08:04 12587335                   /home/dan/.cabal/test/yaml-0.8.10/.cabal-sandbox/lib/x86_64-linux-ghc-7.8.3/void-0.6.1/libHSvoid-0.6.1-ghc7.8.3.so
7f34c0b05000-7f34c0b06000 rw-p 0000a000 08:04 12587335                   /home/dan/.cabal/test/yaml-0.8.10/.cabal-sandbox/lib/x86_64-linux-ghc-7.8.3/void-0.6.1/libHSvoid-0.6.1-ghc7.8.3.so
7f34c0b06000-7f34c0ba2000 r-xp 00000000 08:04 12587325                   /home/dan/.cabal/test/yaml-0.8.10/.cabal-sandbox/lib/x86_64-linux-ghc-7.8.3/semigroups-0.15.4/libHSsemigroups-0.15.4-ghc7.8.3.so
7f34c0ba2000-7f34c0da2000 ---p 0009c000 08:04 12587325                   /home/dan/.cabal/test/yaml-0.8.10/.cabal-sandbox/lib/x86_64-linux-ghc-7.8.3/semigroups-0.15.4/libHSsemigroups-0.15.4-ghc7.8.3.so
7f34c0da2000-7f34c0da3000 r--p 0009c000 08:04 12587325                   /home/dan/.cabal/test/yaml-0.8.10/.cabal-sandbox/lib/x86_64-linux-ghc-7.8.3/semigroups-0.15.4/libHSsemigroups-0.15.4-ghc7.8.3.so
7f34c0da3000-7f34c0da9000 rw-p 0009d000 08:04 12587325                   /home/dan/.cabal/test/yaml-0.8.10/.cabal-sandbox/lib/x86_64-linux-ghc-7.8.3/semigroups-0.15.4/libHSsemigroups-0.15.4-ghc7.8.3.so
...

This is how the ghci is executed via 'cabal repr'. Not sure how the information about the DSOs gets there.

/opt/ghc-7.8.3/lib/ghc-7.8.3/bin/ghc -B/opt/ghc-7.8.3/lib/ghc-7.8.3 --interactive -fbuilding-cabal-package -O0 -outputdir dist/build -odir dist/build -hidir dist/build -stubdir dist/build -i -idist/build -i. -idist/build/autogen -Idist/build/autogen -Idist/build -Ic -Ilibyaml -optP-include -optPdist/build/autogen/cabal_macros.h -package-name yaml-0.8.10 -hide-all-packages -no-user-package-db -package-db /home/dan/.cabal/test/yaml-0.8.10/.cabal-sandbox/x86_64-linux-ghc-7.8.3-packages.conf.d -package-db dist/package.conf.inplace -package-id aeson-0.6.2.1-51da61d52752149eabb23fda6e899080 -package-id attoparsec-0.10.4.0-7ee6fb69f8d373108d8583068f8120f7 -package-id base-4.7.0.1-e4b74d27ad8c8987c63abc42a80e7335 -package-id bytestring-0.10.4.0-73edd0ec5827e07aeacc42c99897b171 -package-id conduit-1.2.3-03a25b103caf9f8dfb81d0b8b7701c4e -package-id containers-0.5.5.1-23e2a2b94d6e452c773209f31d8672c5 -package-id directory-1.2.1.0-d2e4ec65a1afb7a1a6cc06930dc01ea8 -package-id filepath-1.3.0.2-1580a61d3226e4be45fe2130dc2881e3 -package-id resourcet-1.1.3-d91528b040b56ff8f0d7fc56ddb346d9 -package-id scientific-0.3.3.2-884380915060af766df7df17ddeaca91 -package-id text-1.1.0.0-67a836d0620144282e62b9f73f92f738 -package-id transformers-0.3.0.0-16a97696ae672940f1523b262e566ba5 -package-id unordered-containers-0.2.4.0-f6bbc0197292a7807d3274e6b47bc6af -package-id vector-0.10.9.1-b7220db8da564c8c2b77e5d55b53a7d4 -XHaskell98 Text.Libyaml Data.Yaml Data.Yaml.Aeson Data.Yaml.Builder Data.Yaml.Parser Data.Yaml.Include dist/build/c/helper.o dist/build/libyaml/api.o dist/build/libyaml/dumper.o dist/build/libyaml/emitter.o dist/build/libyaml/loader.o dist/build/libyaml/parser.o dist/build/libyaml/reader.o dist/build/libyaml/scanner.o dist/build/libyaml/writer.o -Wall

[swsnr]

@da-x I wasn't aware that library-vanilla only refers to static archives, but other than that you are not telling me anything new, nor does what you have said contradict my previous comment: Flycheck doesn't pass -shared either, so for Flycheck there are still no dependencies available, even if the shared libraries are present.

cabal repl finds your dependencies because GHCi 7.8 uses dynamic linking by default—which is precisely the reason why you end up with two libraries by default, and why you can't opt out from dynamic libraries in cabal install. Internally, Cabal passes -dynamic-too to GHC while building in order to build static and dynamic libraries for all packages. In GHC 7.6, cabal repl would fail with static libraries absent, because it used static linking.

Besides, for this reason Shared: True doesn't affect your dependencies at all. It only affects your own package, and only for cabal build.

Flycheck however has no chance to figure out whether your sandbox is build with or without static libraries. All it could do is to pass -shared by default, and hope that dynamic libraries are present. That would work well for 7.8, where these are always there anyway, but not so well for 7.6, which doesn't build dynamic libraries by default.

OTOH, all that you gain by enable-vanilla: False is storage space. Sure, sandboxes can get (very) large, but does it really matter if it's 1GB, or just 500MB?

[da-x]

Beside space, it also affects module build compilation time, though I don't have actual measurements.

test.sh:

#!/bin/bash

cabal unpack yaml-0.8.10
cd yaml-0.8.10
cabal sandbox init
cabal install

shared-only, library-vanilla: False

time ../test.sh  93.37s user 7.63s system 224% cpu 44.960 total

shared+static, library-vanilla: True

time ../test.sh  108.30s user 8.15s system 225% cpu 51.753 total

Space:

du -s *
13528   shared
26504   static

How about detecting ghc's version, then passing -shared if it's 7.8 and above and not pass it if otherwise? If you post a branch I'll test it.

[swsnr]

@da-x I appreciate your offer, but you'll understand that I have a hard time to see why I should write the code that saves you some 15 seconds of build time and a dozen of megabytes disk space. I find that the balance between effort and gain isn't quite right here.

If that's so important to you, copy the definition of haskell-ghc to your init.el, add -shared to it, and call it a day.

[swsnr]

@da-x I'll add a catch-all option for arbitrary arguments to haskell-ghc, so as soon as https://github.com/flycheck/flycheck/issues/542 is implemented, you'll be able to use (setq flycheck-ghc-args '("-shared")).

Closing as wontfix, since I don't think that this case warrants some special work around in Flycheck.