Comments (12)
To be quite honest, I don't think Cabal's test framework is very good, and I far prefer working with GHC's test suite which is fast (no need to compile/link the test binaries), easy to write tests for (Cabal tests need more boilerplate, as opposed to the expected stdout/stderr which is very convenient), parallel by default, more flexible (I can write an arbitrary shell script for the test if need be) and generally more convenient to work with. So I'd be more keen on B.
from directory.
Is there an easy way to get a minimal GHC test runner that is not tied specifically to a given GHC repository? Being able to run it on Travis without downloading and building GHC would be nice.
from directory.
Well, the test runner is all in Python and can be configured to point to your system GHC, so while the fact that the runner lives in the GHC repo proper might be a pain, you certainly don't have to build GHC.
from directory.
I just pulled the GHC git repo (from github, since I couldn't get the canonical link to respect --depth
) and positioned the testsuite/
in a suitable location for directory/test/Makefile
to find it.
$ git clone --depth 1 [email protected]:ghc/ghc.git
$ mv ghc/testsuite/ .
$ cd src/directory/tests/
$ make
Some time later,
Unexpected results from:
TEST="createDirectoryIfMissing001"
OVERALL SUMMARY for test run started at Sat Feb 21 18:40:51 2015 EST
0:01:31 spent to go through
13 total tests, which gave rise to
130 test cases, of which
2 were skipped
0 had missing libraries
118 expected passes
0 expected failures
0 caused framework failures
0 unexpected passes
10 unexpected failures
0 unexpected stat failures
Unexpected failures:
. createDirectoryIfMissing001 [bad stdout] (normal,hpc,optasm,profasm,threaded1,threaded2,dyn,profthreaded,optllvm)
. createDirectoryIfMissing001 [bad stdout or stderr] (ghci)
from directory.
Might be related to this. I'm fairly certain a large number of the tests in directory are not safe to parallelize.
from directory.
Well, that's well worth fixing!
from directory.
Truthfully, I had only intended the above as a demonstration of not needing to build GHC to run the testsuite. The issue it raises does seem to be worth looking into, however.
from directory.
Failures in the above were due to accidentally linking against directory-1.2.1.0, so it appears that the technique is fundamentally sound (and the code isn't broken). Assuming directory
to be in a sandbox, the fix for linking properly is to either run make from a shell spawned with cabal exec ${SHELL}
or to cabal sandbox init --sandbox="../.cabal-sandbox"
from tests/
.
Using the second method,
$ cabal exec make
<cut>
OVERALL SUMMARY for test run started at Sun Feb 22 13:33:53 2015 EST
0:01:41 spent to go through
13 total tests, which gave rise to
130 test cases, of which
2 were skipped
0 had missing libraries
128 expected passes
0 expected failures
0 caused framework failures
0 unexpected passes
0 unexpected failures
0 unexpected stat failures
Suffice it to say, I also push for option B.
from directory.
As another data point, I've been unable to duplicate this success on Windows. Still looking into why that is (likely PEBCAK), but I can say it isn't as straight forward as running on Unix.
from directory.
I went with Option B since that seems to be the consensus. The changes can be seen in 23b416 to eaf723. Travis CI can now run the entire test suite automatically. It should also work on Windows with MSYS+Git+Python installed.
In case anyone wants to use them in other packages like unix or win32: it can be done with minimal effort, especially if the relative file structure remains the same. The crucial files are:
- tools/run-tests, which downloads the GHC repository and runs the tests;
- tools/ghc.patch, which fixes
timeout
so it compiles on Windows with older versions of GHC; - tools/dispatch-tests.hs, which allows Cabal to invoke the tests.
from directory.
Reopening this because, after having used it many times now, I really don't think it's worth the trouble anymore:
- It interacts poorly with sandboxes and there's no way to fix that because Cabal does not give us the necessary environment.
- It forces us to use a Custom
.cabal
to reconcile with Nix's strict dependency management, which - by itself was already a bad sign: it means we are bringing in a lot of needless dependencies (Python and
which
, but also many POSIX utilities likemake
). - It's very slow on Windows because every test is a separate shell invocation.
Just today I ran into another problem where it fails on Windows Command Prompt, because the GHC test suite changes the codepage to 65001 (UTF-8), which Python 2.7 fails to recognize and immediately bails. It works fine on MinTTY, and also fine on Python 3+ (probably because this bug got fixed). It's odd that I never discovered this until now, probably because I rarely use Command Prompt.
In summary, it's a very fragile solution: the pieces just don't fit well in Cabal build system and I had to invent several dirty hacks to make it work.
Future (long-term) plan: Rewrite a small pure-Haskell test library that mimics the essential features and then port all the tests over.
Update: it seems that the codepage change was a red herring: although that was disruptive for interactive mode, it did not affect the actual scripts. The problem seems to manifest on both Command Prompt and MinTTY. The real reason was this change: the variable config.compiler
is now double-quoted, which prevents MinGW from automatically converting the path from MinGW-style (e.g. /c/Windows
) to Windows-style (e.g. C:\Windows
), and as a result Python fails to locate GHC because Python I have is not built for MinGW but for generic Windows.
As a side-note: installing a GnuWin32 which
added some additional confusion to the problem, as it also breaks the test suite because it produces Windows-style paths that, when eval
'ed by Python, becomes interpreted incorrectly as escape characters.
TL;DR. This change broke testing for out-of-tree compilers on MinGW. Also, do not use GnuWin32 which
with the test suite.
P.S. And do not use the make
bundled with Git either, because it will crash/hang. Instead, make sure the Git path is put after everything else in PATH
.
from directory.
Fixed: 6323523...3df3097
from directory.
Related Issues (20)
- Updating the CI and the cabal file HOT 2
- makeAbsolute ~ gives wrong path HOT 2
- Release for GHC 9.6.1 HOT 3
- TMPDIR environment variable not honoured on windows HOT 2
- Feature request: createLink HOT 1
- FR: a variant of getModificationTime that works on symlinks HOT 1
- Cabal file doesn't list install-includes HOT 3
- [Feature request] A way to copy a directory with metadata (and/or recursively) HOT 5
- directory bundled with GHC-9.6 is not safe (inferred) anymore. HOT 5
- Support filepath >= 1.5.0.0 and os-string
- Support Win32 2.14 HOT 5
- canonicalizePath regressed on Windows between directory-1.3.7.1 and directory-1.3.8.0 HOT 2
- Please revise `time` dependency to `< 1.15` HOT 1
- Support filepath 1.4.300.1 HOT 4
- Builds with allow-newer unnecessarily fail HOT 1
- on Windows, copyFile makes empty source and destination files when the source file is absent HOT 8
- FR: findExecutable "" should return Nothing, like on linux HOT 4
- `directory-1.3.8.3`: relax upper bound of dependency on `base` HOT 2
- Resolve TODO regarding AFPP support for openBinaryTempFile HOT 2
- Refactor remaining code to use *at APIs
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from directory.