Comments (17)
in a similar vein, see also the new ticket scala/scala-dev#295, which proposes removing a lot of tests from partest's clutches entirely
(not to mention the recently merged scala/scala#5661, which removed our ScalaCheck-based tests from partest's clutches)
from scala-partest.
example failure causes to watch out for:
- SI-5941 makes printing of type names sensitive to classloader details
- use of security manager calls
- calls to
System.exit
(Dale is working on it) altering of system properties(Dale added code that saves & restores them)- @retronym says using Scala reflection may alter the state of the VM in ways that may influence later tests
- ...?
from scala-partest.
very rough numbers: in our local tests so far this saves about 2/3 of a second per run test. if we can enable this for 1500 (out of 1800) tests, we'll save 1000 seconds = 16 minutes. PR validation currently takes 71 minutes total.
from scala-partest.
Some progress towards fixing SI-5941: https://github.com/retronym/scala/tree/ticket/5941
from scala-partest.
so it isn't a trivial matter of just running any failed test the old way
I think Dale convinced me that wouldn't be so bad as a starting point. When test A does something that later causes test B to fail, it's hard to say who's at fault. Is it test A for changing something, or test B for depending on something that could change? So yes, mass quarantine of all the test Bs will catch some innocent victims, but only some.
from scala-partest.
I've not forgotten this. It's on my todo list (of out-of-hours todos).
from scala-partest.
It's still very much on my mind as well.
from scala-partest.
To rekindle this, I've created a branch of scala/scala that uses partest-as-a-library from a JUnit suite, overriding the way that run tests are executed as discussed here.
I wrote this from scratch as I can't remember what we did with the code we wrote in Banff: system property setup, capturing stdout/stderr, trapping sys.exit.
https://github.com/retronym/scala/tree/topic/jptest
18 minutes later:
[info] Test run finished: 37 failed, 0 ignored, 1872 total, 1051.299s
[error] Failed: Total 1872, Failed 37, Errors 0, Passed 1835
[error] Failed tests:
[error] partest.RunTests
[error] (junit/test:testOnly) sbt.TestsFailedException: Tests unsuccessful
[error] Total time: 1057 s, completed 14/02/2018 6:36:51 PM
Running all 1877 partests under the existing sbt 'partest --run'
is glacially slow -- 71 mins -- on the same machine:
[error] Failed: Total 1877, Failed 3, Errors 0, Passed 1874
[error] Failed tests:
[error] partest
[error] (test/it:testOnly) sbt.TestsFailedException: Tests unsuccessful
[error] Total time: 4255 s, completed 14/02/2018 7:50:01 PM
... and that's with use of all four cores!
Maybe something is off with my measurements, but they suggest an even greater improvement than the 2x we got from the back-of-the-envelope calcs above.
from scala-partest.
One advantage of using JUnit as the harness is that it we get some tooling for free. For example, running this suite in IntelliJ, I can sort by test duration.
I'm not sure why the IntelliJ runner took 29min in total, rather than the 17min I measured through SBT. But in that context, the slowest 10 tests account for > 30% of the runtime.
Some of these tests are doing black box testing for absence of memory leaks in ways that stress GC (t6969, reflection-mem-tests*). Others are performing stress tests on collections (t6853, ctries-new, ctries-old, t3989).
Test of reify
tend to be slow-ish, and there are lots of them (135).
Tests that themselves perform another layer of programmatic compilation (through Toolbox.eval
or through ReplTest
) are also sluggish.
At the other end of the scale, the fastest executing tests still take around 130ms. This is likely limited by the time taken to do a hello world compile with a new Global
, even with a warmed up JVM.
from scala-partest.
breadcrumb to self:
[info] Test partest.RunTests.t7046-2 started
32 tests failed with partest.RunTests, rerun with:
junit/testOnly partest.RunTests -- --tests=\Qlazy$minusconcurrent\E|\QbcodeInlinerMixed\E|\Qt7805$minusrepl$minusi\E|\Qdynamic$minusupdateDynamic\E|\Qt2318\E|\Qt4332\E|\Qt4788\E|\Qt5938\E|\Qt6669\E|\Qt7106\E|\Qt7852\E|\Qt7974\E|\Qt8046\E|\Qt8433\E|\Qt9359\E|\Qt9403\E|\Qt7817$minustree$minusgen\E|\Qicode$minusreader$minusdead$minuscode\E|\QjunitForwarders\E|\Qt8266$minusoctal$minusinterp\E|\Qreflection$minusvalueclasses$minusmagic\E|\Qt6240$minusuniverse$minuscode$minusgen\E|\Qshutdownhooks\E|\QscalapInvokedynamic\E|\Qt4788$minusseparate$minuscompilation\E|\QWeakHashSetTest\E|\Qt8601$minusclosure$minuselim\E|\Qdynamic$minusselectDynamic\E|\Qreify_copypaste1\E|\Qdynamic$minusapplyDynamic\E|\Qreflection$minusmagicsymbols$minusinvoke\E|\Qt7046$minus1\E
[info] Test run finished: 33 failed, 0 ignored, 1872 total, 1441.14s
[error] Failed: Total 1872, Failed 33, Errors 0, Passed 1839
[error] Failed tests:
[error] partest.RunTests
[error] (junit/test:testOnly) sbt.TestsFailedException: Tests unsuccessful
[error] Total time: 1547 s, completed 15/02/2018 1:46:14 PM
from scala-partest.
At the other end of the scale, the fastest executing tests still take around 130ms. This is likely limited by the time taken to do a hello world compile with a new Global, even with a warmed up JVM.
It occurs to me that reusing a Global
instance for this use case is probably quite achievable. After compilation, we can walk the Symbol table and unlink symbols that originate from the just-compiled set of source files. We can leave the rest of the symbols in place (e.g. standard library) without needing to worry about invalidating them if the corresponding JAR changes.
@lrytz mentioned that the Junit tests that perform programatic compilation already to Global
instance re-use. They don't actually bother unlinking symbols from the previous run at all, and just make sure the tests use distinct class names or that overlaps don't trigger buggy behaviour.
from scala-partest.
Another means to speedup might be reducing IO by using virtual directories for the target, which could be directly used as the classpath of the executing project.
from scala-partest.
make sure the tests use distinct class names
The tests reuse the same names often, actually; there were only few bugs, I fixed some, for example scala/scala@59d6dbc
from scala-partest.
Ah, there are still open issues; scala/scala-dev#214
from scala-partest.
I'm currently working a branch of partest to support JVM reuse: https://github.com/retronym/scala-partest/tree/ticket/75
I think it makes sense to get this change through before changing to using JUnit as the front end.
from scala-partest.
Okay, I've got a working prototype of Global reuse in https://github.com/retronym/scala-partest/tree/ticket/75. The complementary config changes to scala/scala are in topic/jptest
That gets the single threaded partest --run
down to 14m on my laptop.
from scala-partest.
JVM reuse is now merged in scala/scala#6413
PR validation down to ~23 mins, down from 55 mins.
I'm closing this ticket as fixed, as the cost/benefit of the Global reuse idea doesn't seem worth it right now.
from scala-partest.
Related Issues (20)
- possibly dead branches HOT 1
- Drop scala-xml dependency? HOT 1
- update branches and readmes now that re-in-sourcing is complete for 2.13 HOT 5
- relicense under Apache 2
- Checkfiles are not properly preprocessed when diffing HOT 2
- Consolidate Scala.js fork HOT 3
- switch to sbt-git HOT 2
- Partest shouldn't pick up just any old `java` command HOT 4
- Partest mark test that timed out as skipped instead of failed HOT 2
- MemoryTest could be more robust, precise about failure HOT 1
- Mistyped test name shouldn't test the world HOT 5
- release 1.0.17 against 2.12.0-M5 but not 2.11 HOT 2
- Partest detects wrong java exectuable on Windows HOT 3
- feature: report which tests are slow to compile, slow to run HOT 1
- Flags parsing doesn't do "foo bar" HOT 6
- reverse sense of diffs (with `--show-diff` and `--verbose`) HOT 1
- Release v1.1.1 and switch to it in 2.12.x/2.13.x? HOT 1
- --dryrun HOT 6
- Current dir is on classpath during `run` test HOT 3
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from scala-partest.