juliaci/pkgeval.jl

Keeping tabs on the julia ecosystem

pkgeval.jl's Issues

New name and home for this package

Should this be in the JuliaCI org?

Issue with stopping a container

From JuliaLang/julia@a5758aa#commitcomment-48923479:

Mar 31 01:35:17 amdci8 bash[29029]:       From worker 2:        Error response from daemon: No such container: Percival-WDVmlJXs
Mar 31 01:35:17 amdci8 bash[29029]:       From worker 2:
Mar 31 01:35:19 amdci8 bash[29029]:       From worker 2:        Error response from daemon: No such container: NumericalAlgorithms-FsFmcjW6
Mar 31 01:35:22 amdci8 bash[29029]: [ Info: [Node 2 | 2021-03-31T01:35:22.895]: reporting results for PkgEvalJob JuliaLang/julia@a5758aa [daily]
Mar 31 01:35:22 amdci8 bash[29029]: [ Info: [Node 2 | 2021-03-31T01:35:22.897]: ...generating report...
Mar 31 01:35:22 amdci8 bash[29029]: [ Info: [Node 2 | 2021-03-31T01:35:22.968]: ...tarring data...
Mar 31 01:35:23 amdci8 bash[29029]: [ Info: [Node 2 | 2021-03-31T01:35:23.013]: ...moving /storage/pkgeval/Nanosoldier/workdir/tmpresults to /storage/pkgeval/Nanosoldier/workdir/NanosoldierReports/pkgeval/by_date/2021-03/31...
Mar 31 01:35:23 amdci8 bash[29029]: [ Info: [Node 2 | 2021-03-31T01:35:23.015]: ...pushing /storage/pkgeval/Nanosoldier/workdir/NanosoldierReports/pkgeval/by_date/2021-03/31 to GitHub...
Mar 31 01:35:23 amdci8 bash[29029]:       From worker 2:        [master 779b9ae97c] upload report for PkgEvalJob JuliaLang/julia@a5758aa [daily]
Mar 31 01:35:23 amdci8 bash[29029]:       From worker 2:         3 files changed, 53 insertions(+), 1 deletion(-)
Mar 31 01:35:23 amdci8 bash[29029]:       From worker 2:         create mode 100644 pkgeval/by_date/2021-03/31/data.tar.xz
Mar 31 01:35:23 amdci8 bash[29029]:       From worker 2:         create mode 100644 pkgeval/by_date/2021-03/31/report.md
Mar 31 01:35:23 amdci8 bash[29029]:       From worker 2:        Already up to date.
Mar 31 01:35:28 amdci8 bash[29029]:       From worker 2:        To https://github.com/JuliaCI/NanosoldierReports.git
Mar 31 01:35:28 amdci8 bash[29029]:       From worker 2:           297a357dc0..779b9ae97c  master -> master
Mar 31 01:35:28 amdci8 bash[29029]: ┌ Error: An exception occurred during job execution
Mar 31 01:35:28 amdci8 bash[29029]: │   exception =
Mar 31 01:35:28 amdci8 bash[29029]: │    NanosoldierError: failed to run tests: TaskFailedException:
Mar 31 01:35:28 amdci8 bash[29029]: │    failed process: Process(`docker stop Percival-WDVmlJXs`, ProcessExited(1)) [1]
Mar 31 01:35:28 amdci8 bash[29029]: │
Mar 31 01:35:28 amdci8 bash[29029]: │    Stacktrace:
Mar 31 01:35:28 amdci8 bash[29029]: │     [1] pipeline_error at ./process.jl:525 [inlined]
Mar 31 01:35:28 amdci8 bash[29029]: │     [2] run(::Base.CmdRedirect; wait::Bool) at ./process.jl:440
Mar 31 01:35:28 amdci8 bash[29029]: │     [3] run at ./process.jl:438 [inlined]
Mar 31 01:35:28 amdci8 bash[29029]: │     [4] kill_container at /storage/pkgeval/dev/PkgEval/src/run.jl:364 [inlined]
Mar 31 01:35:28 amdci8 bash[29029]: │     [5] stop at /storage/pkgeval/dev/PkgEval/src/run.jl:167 [inlined]
Mar 31 01:35:28 amdci8 bash[29029]: │     [6] #36 at /storage/pkgeval/dev/PkgEval/src/run.jl:234 [inlined]
Mar 31 01:35:28 amdci8 bash[29029]: │     [7] lock(::PkgEval.var"#36#44"{String,PkgEval.var"#stop#38"{String},Base.Process}, ::ReentrantLock) at ./lock.jl:161
Mar 31 01:35:28 amdci8 bash[29029]: │     [8] macro expansion at /storage/pkgeval/dev/PkgEval/src/run.jl:231 [inlined]
Mar 31 01:35:28 amdci8 bash[29029]: │     [9] (::PkgEval.var"#35#43"{Int64,String,Pipe,PkgEval.var"#stop#38"{String},ReentrantLock,Base.Process})() at ./task.jl:356
Mar 31 01:35:28 amdci8 bash[29029]: │    Stacktrace:
Mar 31 01:35:28 amdci8 bash[29029]: │     [1] wait at ./task.jl:267 [inlined]
Mar 31 01:35:28 amdci8 bash[29029]: │     [2] fetch at ./task.jl:282 [inlined]
Mar 31 01:35:28 amdci8 bash[29029]: │     [3] run_sandboxed_test(::String, ::NamedTuple{(:name, :uuid, :path, :registry),Tuple{String,Base.UUID,String,String}}; log_limit::Int64, time_limit::Int64, do_depwarns::Bool, kwargs::Base.Iterator
Mar 31 01:35:28 amdci8 bash[29029]: │     [4] macro expansion at /storage/pkgeval/dev/PkgEval/src/run.jl:584 [inlined]
Mar 31 01:35:28 amdci8 bash[29029]: │     [5] (::PkgEval.var"#50#57"{Int64,Base.Iterators.Pairs{Union{},Union{},Tuple{},NamedTuple{(),Tuple{}}},String,Array{NamedTuple{(:julia, :install, :cache, :pkg),Tuple{VersionNumber,String,String,Nam
Mar 31 01:35:28 amdci8 bash[29029]: │    Stacktrace:
Mar 31 01:35:28 amdci8 bash[29029]: │     [1] sync_end(::Channel{Any}) at ./task.jl:314
Mar 31 01:35:28 amdci8 bash[29029]: │     [2] macro expansion at ./task.jl:333 [inlined]
Mar 31 01:35:28 amdci8 bash[29029]: │     [3] run(::Array{VersionNumber,1}, ::Array{Any,1}; ninstances::Int64, retries::Int64, kwargs::Base.Iterators.Pairs{Union{},Union{},Tuple{},NamedTuple{(),Tuple{}}}) at /storage/pkgeval/dev/PkgEval/s
Mar 31 01:35:28 amdci8 bash[29029]: │     [4] (::Nanosoldier.var"#30#36"{Nanosoldier.PkgEvalJob,Dict{String,VersionNumber},Array{Any,1}})() at /storage/pkgeval/Nanosoldier/src/jobs/PkgEvalJob.jl:231
Mar 31 01:35:28 amdci8 bash[29029]: │     [5] withenv(::Nanosoldier.var"#30#36"{Nanosoldier.PkgEvalJob,Dict{String,VersionNumber},Array{Any,1}}, ::Pair{String,Bool}) at ./env.jl:161
Mar 31 01:35:28 amdci8 bash[29029]: │     [6] execute_tests!(::Nanosoldier.PkgEvalJob, ::Dict{String,Nanosoldier.BuildRef}, ::Dict{String,Array{String,1}}, ::Dict{Any,Any}) at /storage/pkgeval/Nanosoldier/src/jobs/PkgEvalJob.jl:229
Mar 31 01:35:28 amdci8 bash[29029]: │     [7] run(::Nanosoldier.PkgEvalJob) at /storage/pkgeval/Nanosoldier/src/jobs/PkgEvalJob.jl:355
Mar 31 01:35:28 amdci8 bash[29029]: │     [8] (::Distributed.var"#106#108"{Distributed.CallMsg{:call_fetch}})() at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.5/Distributed/src/process_messages.jl:294
Mar 31 01:35:28 amdci8 bash[29029]: │     [9] run_work_thunk(::Distributed.var"#106#108"{Distributed.CallMsg{:call_fetch}}, ::Bool) at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.5/Distributed/src/process_messages.
Mar 31 01:35:28 amdci8 bash[29029]: │     [10] macro expansion at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.5/Distributed/src/process_messages.jl:294 [inlined]
Mar 31 01:35:28 amdci8 bash[29029]: │     [11] (::Distributed.var"#105#107"{Distributed.CallMsg{:call_fetch},Distributed.MsgHeader,Sockets.TCPSocket})() at ./task.jl:356
Mar 31 01:35:28 amdci8 bash[29029]: └ @ Nanosoldier /storage/pkgeval/Nanosoldier/src/jobs/PkgEvalJob.jl:425

Maybe PkgEval decided to stop the container when it was already exiting?

Add a keyword argument for not skipping failed packages

Oft requested and @JeffBezanson volunteered to send a PR ;).

Show 1.0.x compatibility? And not 1.2?

Maybe:

https://maleadt.github.io/NewPkgEval.jl/

should show for the LTS version of Julia (and if bad for taking too much space, drop unsupported 1.2)?

Use a PkgServer

#68 (comment)

Easiest is probably to expect the user to set-up his own server, specify it using the environment variable, and have NewPkgEval forward that to test containers (taking care of connectivity, I guess we'd need to run with --net=host to be able to connect to a local server).

FIFO purge

Purge currently removes all files, for Nanosoldier it would be interesting if it only removes builds and downloads that haven't been used for a while.

Storing things only based on branch name + version can be problematic when the branch is updated.

Let's say you have run the backports-1.3.0 branch and now have updated the branch and want to rerun PkgEval. Trying to give the same branch name will run with the cached built version of Julia.

Add a license

There doesn't appear to be any licensing information in here. It'd be good to have in general, but also because I'd like to reuse some of this code for nefarious other projects. 🙂

Register NewPkgEval in the General registry?

I noticed that NewPkgEval is not registered in the General registry. Perhaps we could register it?

Automated retry flag

From discussion with @maleadt:

It would be nice if there was a flag that told PkgEval to automatically do a second run with the tests that failed (like you usually do manually after it finishes anyway). Just thinking it’d be nice to invoke that before heading to bed and then wake up to a report with some of the stochastic failures already filtered out

Add Gtk.jl to NewPkgEval CI

To test the X11 functionality,

/tmp fills up with Julia temporary directories

JuliaLang/julia@83d7eba

Share precompilation cache

Instead of precompiling stuff over and over, try share precompilation files between sandboxes.

Right now, this would run into race condition galore so that would need to be fixed.

Support for testing a PackageSpec

This could be useful for e.g. running NewPkgEval on private unregistered packages behind a firewall.

Provide tooling to build system image as a github action

It would be nice to be able to use PkgEval as a github action. The motivation was for package authors to be able to ensure that their packages do not have issues when they are included into a system image.

Java support

Just a reminder issue to eventually add Java support to PkgEval.

Occasional Docker crash

      From worker 2:    1876 combinations to test (3609 succeeded, 1875 failed, 223 killed, 835 skipped, 10 hours)
      From worker 2:    docker: Error response from daemon: driver failed programming external connectivity on endpoint ConstantTime-mCwMbrL2 (0dc45dea13ad2edccc4ab5830c8a24671f139524ac9af9553f74a19001e6d403): failed to update bridge endpoint 0dc45de to store: failed to update bridge store for object type *bridge.bridgeEndpoint: timeout.
      From worker 2:
[ Info: [Node 2 | 2020-09-07T01:00:18.193]: reporting results for PkgEvalJob JuliaLang/julia@bf886b5 [daily]
[ Info: [Node 2 | 2020-09-07T01:00:18.214]: ...generating report...
      From worker 2:    docker: Error response from daemon: invalid mount config for type "bind": bind source path does not exist: /tmp/jl_x1Qkht/julia-b36515781c.
      From worker 2:    BloomFilters-KpwTyeQb
[ Info: [Node 2 | 2020-09-07T01:00:20]: ...tarring data...
[ Info: [Node 2 | 2020-09-07T01:00:20.162]: ...moving /home/maleadt/Nanosoldier/workdir/tmpresults to /home/maleadt/Nanosoldier/workdir/NanosoldierReports/pkgeval/by_date/2020-09/06...
[ Info: [Node 2 | 2020-09-07T01:00:20.186]: ...pushing /home/maleadt/Nanosoldier/workdir/NanosoldierReports/pkgeval/by_date/2020-09/06 to GitHub...
      From worker 2:    docker: Error response from daemon: OCI runtime create failed: container_linux.go:349: starting container process caused "process_linux.go:449: container init caused \"rootfs_linux.go:58: mounting \\\"/tmp/jl_x1Q
kht/julia-b36515781c\\\" to rootfs \\\"/var/lib/docker/btrfs/subvolumes/66d8d2fecab80ff07a903dce2982baa7414ff0fb0fec7673b5c3345a42e0c667\\\" at \\\"/opt/julia\\\" caused \\\"stat /tmp/jl_x1Qkht/julia-b36515781c: no such file or director
y\\\"\"": unknown.
      From worker 2:    [master f1fd70a9da7] upload report for PkgEvalJob JuliaLang/julia@bf886b5 [daily]
      From worker 2:     3 files changed, 72 insertions(+), 1 deletion(-)
      From worker 2:     create mode 100644 pkgeval/by_date/2020-09/06/data.tar.gz
      From worker 2:     create mode 100644 pkgeval/by_date/2020-09/06/report.md
      From worker 2:    Already up to date.
      From worker 2:    0 combinations to test (4639 succeeded, 2447 failed, 268 killed, 1060 skipped, 1 day)
      From worker 2:    0 combinations to test (4830 succeeded, 2392 failed, 114 killed, 1048 skipped, 5 days)
      From worker 2:    0 combinations to test (4616 succeeded, 2455 failed, 275 killed, 1056 skipped, 3 days)
      From worker 2:    0 combinations to test (4639 succeeded, 2447 failed, 268 killed, 1060 skipped, 1 day)
      From worker 2:    0 combinations to test (4830 succeeded, 2392 failed, 114 killed, 1048 skipped, 5 days)
      From worker 2:    0 combinations to test (4616 succeeded, 2455 failed, 275 killed, 1056 skipped, 3 days)
      From worker 2:    0 combinations to test (4639 succeeded, 2447 failed, 268 killed, 1060 skipped, 1 day)
      From worker 2:    0 combinations to test (4830 succeeded, 2392 failed, 114 killed, 1048 skipped, 5 days)
      From worker 2:    0 combinations to test (4616 succeeded, 2455 failed, 275 killed, 1056 skipped, 3 days)
      From worker 2:    To https://github.com/JuliaCI/NanosoldierReports.git
      From worker 2:       3d697efc2fd..f1fd70a9da7  master -> master
┌ Error: An exception occurred during job execution
│   exception =
│    NanosoldierError: failed to run tests: TaskFailedException:
│    failed process: Process(`docker run --detach --gpus all -e NVIDIA_VISIBLE_DEVICES=all -e NVIDIA_DRIVER_CAPABILITIES=all --mount type=bind,source=/tmp/jl_x1Qkht/julia-b36515781c,target=/opt/julia,readonly --mount type=bind,source=/home/maleadt/.julia/registries,target=/usr/local/share/julia/registries,readonly --mount type=bind,source=/home/maleadt/.cache/NewPkgEval/artifacts,target=/var/cache/julia/artifacts --env JULIA_DEPOT_PATH=::/usr/local/share/julia --tmpfs /home/pkgeval:exec,uid=1000,gid=1000 --cpus=2 --env JULIA_NUM_THREADS=2 --name ConstantTime-mCwMbrL2 --rm newpkgeval xvfb-run /opt/julia/bin/julia -e '    using InteractiveUtils
│        versioninfo()
│        println()
│
│        mkpath(".julia")
│        symlink("/var/cache/julia/artifacts", ".julia/artifacts")
│
│        using Pkg
│        Pkg.UPDATED_REGISTRY_THIS_SESSION[] = true
│
│        ENV["CI"] = true
│        ENV["PKGEVAL"] = true
│        ENV["JULIA_PKGEVAL"] = true
│
│        ENV["PYTHON"] = ""
│        ENV["R_HOME"] = "*"
│
│        Pkg.add(ARGS...)
│        Pkg.test(ARGS...)
│    ' ConstantTime`, ProcessExited(125)) [125]
│
│    Stacktrace:
│     [1] pipeline_error at ./process.jl:525 [inlined]
│     [2] read(::Cmd) at ./process.jl:412
│     [3] read(::Cmd, ::Type{String}) at ./process.jl:421
│     [4] #spawn_sandboxed_julia#25(::Bool, ::String, ::Int64, ::Bool, ::typeof(NewPkgEval.spawn_sandboxed_julia), ::String, ::Cmd) at /home/maleadt/.julia/dev/NewPkgEval/src/run.jl:82
│     [5] #spawn_sandboxed_julia at ./none:0 [inlined]
│     [6] #run_sandboxed_julia#24 at /home/maleadt/.julia/dev/NewPkgEval/src/run.jl:37 [inlined]
│     [7] (::NewPkgEval.var"#kw##run_sandboxed_julia")(::NamedTuple{(:stdout, :stderr, :stdin, :interactive, :wait, :name),Tuple{IOStream,IOStream,Base.DevNull,Bool,Bool,String}}, ::typeof(NewPkgEval.run_sandboxed_julia), ::String, ::Cmd) at ./none:0
│     [8] (::NewPkgEval.var"#27#30"{Int64,Int64,Base.Iterators.Pairs{Union{},Union{},Tuple{},NamedTuple{(),Tuple{}}},String,NamedTuple{(:name, :uuid, :path, :registry),Tuple{String,Base.UUID,String,String}},String})(::String, ::IOStream) at /home/maleadt/.julia/dev/NewPkgEval/src/run.jl:130
│     [9] mktemp(::NewPkgEval.var"#27#30"{Int64,Int64,Base.Iterators.Pairs{Union{},Union{},Tuple{},NamedTuple{(),Tuple{}}},String,NamedTuple{(:name, :uuid, :path, :registry),Tuple{String,Base.UUID,String,String}},String}, ::String) at ./file.jl:611
│     [10] mktemp at ./file.jl:609 [inlined]
│     [11] #run_sandboxed_test#26(::Int64, ::Int64, ::Bool, ::Base.Iterators.Pairs{Union{},Union{},Tuple{},NamedTuple{(),Tuple{}}}, ::typeof(NewPkgEval.run_sandboxed_test), ::String, ::NamedTuple{(:name, :uuid, :path, :registry),Tuple{String,Base.UUID,String,String}}) at /home/maleadt/.julia/dev/NewPkgEval/src/run.jl:129
│     [12] run_sandboxed_test at /home/maleadt/.julia/dev/NewPkgEval/src/run.jl:104 [inlined]
│     [13] macro expansion at /home/maleadt/.julia/dev/NewPkgEval/src/run.jl:399 [inlined]
│     [14] (::NewPkgEval.var"#37#44"{Int64,Base.Iterators.Pairs{Union{},Union{},Tuple{},NamedTuple{(),Tuple{}}},Array{NamedTuple{(:julia, :install, :pkg),Tuple{VersionNumber,String,NamedTuple{(:name, :uuid, :path, :registry),Tuple{String,Base.UUID,String,String}}}},1},Array{Union{Nothing, NamedTuple{(:julia, :install, :pkg),Tuple{VersionNumber,String,NamedTuple{(:name, :uuid, :path, :registry),Tuple{String,Base.UUID,String,String}}}}},1},Array{Dates.DateTime,1},NewPkgEval.var"#stop_work#40"{Array{Union{Nothing, NamedTuple{(:julia, :install, :pkg),Tuple{VersionNumber,String,NamedTuple{(:name, :uuid, :path, :registry),Tuple{String,Base.UUID,String,String}}}}},1},Array{Task,1}},Int64})() at ./task.jl:333
│    Stacktrace:
│     [1] sync_end(::Array{Any,1}) at ./task.jl:300
│     [2] macro expansion at ./task.jl:319 [inlined]
│     [3] #run#33(::Int64, ::Int64, ::Base.Iterators.Pairs{Union{},Union{},Tuple{},NamedTuple{(),Tuple{}}}, ::typeof(NewPkgEval.run), ::Array{VersionNumber,1}, ::Array{Any,1}) at /home/maleadt/.julia/dev/NewPkgEval/src/run.jl:334
│     [4] (::NewPkgEval.var"#kw##run")(::NamedTuple{(:ninstances,),Tuple{Int64}}, ::typeof(NewPkgEval.run), ::Array{VersionNumber,1}, ::Array{Any,1}) at ./none:0
│     [5] (::Nanosoldier.var"#30#36"{Nanosoldier.PkgEvalJob,Dict{String,VersionNumber},Array{Any,1}})() at /home/maleadt/Nanosoldier/src/jobs/PkgEvalJob.jl:199
│     [6] withenv(::Nanosoldier.var"#30#36"{Nanosoldier.PkgEvalJob,Dict{String,VersionNumber},Array{Any,1}}, ::Pair{String,Bool}) at ./env.jl:161
│     [7] execute_tests!(::Nanosoldier.PkgEvalJob, ::Dict{String,Nanosoldier.BuildRef}, ::Dict{Any,Any}) at /home/maleadt/Nanosoldier/src/jobs/PkgEvalJob.jl:197
│     [8] run(::Nanosoldier.PkgEvalJob) at /home/maleadt/Nanosoldier/src/jobs/PkgEvalJob.jl:307
│     [9] (::Distributed.var"#108#110"{Distributed.CallMsg{:call_fetch}})() at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.3/Distributed/src/process_messages.jl:294
│     [10] run_work_thunk(::Distributed.var"#108#110"{Distributed.CallMsg{:call_fetch}}, ::Bool) at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.3/Distributed/src/process_messages.jl:79
│     [11] macro expansion at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.3/Distributed/src/process_messages.jl:294 [inlined]
│     [12] (::Distributed.var"#107#109"{Distributed.CallMsg{:call_fetch},Distributed.MsgHeader,Sockets.TCPSocket})() at ./task.jl:333
└ @ Nanosoldier ~/Nanosoldier/src/jobs/PkgEvalJob.jl:376
┌ Error: [Node 2 | 2020-09-07T01:01:56.82]: Encountered error: encountered job loop error: MethodError(Base.show_backtrace, (IOBuffer(data=UInt8[...], readable=true, writable=true, seekable=true, append=false, size=1099, maxsize=Inf, ptr=1100, mark=-1), nothing), 0x00000000000068c0)
│   exception =
│    MethodError: no method matching show_backtrace(::Base.GenericIOBuffer{Array{UInt8,1}}, ::Nothing)
│    Closest candidates are:
│      show_backtrace(::IO, !Matched::Array{Any,1}) at errorshow.jl:595
│      show_backtrace(::IO, !Matched::Array{T,1} where T) at errorshow.jl:567
│    Stacktrace:
│     [1] #showerror#656(::Bool, ::typeof(showerror), ::Base.GenericIOBuffer{Array{UInt8,1}}, ::ProcessFailedException, ::Nothing) at ./errorshow.jl:79
│     [2] showerror(::Base.GenericIOBuffer{Array{UInt8,1}}, ::ProcessFailedException, ::Nothing) at ./errorshow.jl:74
│     [3] showerror(::Base.GenericIOBuffer{Array{UInt8,1}}, ::TaskFailedException) at ./task.jl:76
│     [4] showerror(::Base.GenericIOBuffer{Array{UInt8,1}}, ::CompositeException) at ./task.jl:49
│     [5] show at /home/maleadt/Nanosoldier/src/Nanosoldier.jl:44 [inlined]
│     [6] print(::Base.GenericIOBuffer{Array{UInt8,1}}, ::Nanosoldier.NanosoldierError{CompositeException}) at ./strings/io.jl:35
│     [7] print_to_string(::Nanosoldier.NanosoldierError{CompositeException}) at ./strings/io.jl:138
│     [8] string(::Nanosoldier.NanosoldierError{CompositeException}) at ./strings/io.jl:177
│     [9] delegate_job(::Nanosoldier.Server, ::Nanosoldier.PkgEvalJob, ::Int64) at /home/maleadt/Nanosoldier/src/server.jl:108
│     [10] (::Nanosoldier.var"#57#58"{Nanosoldier.Server,DataType,Int64,Array{Int64,1}})() at /home/maleadt/Nanosoldier/src/server.jl:68
└ @ Nanosoldier ~/Nanosoldier/src/config.jl:51

So I guess the driver failed programming external kills the runner task, after which the extracted Julia is removed (causing lots of jobs, somehow subsequently launched, to fail). For some reason, the printer task then remains:

      From worker 2:    0 combinations to test (4639 succeeded, 2447 failed, 268 killed, 1060 skipped, 1 day)
      From worker 2:    0 combinations to test (4830 succeeded, 2392 failed, 114 killed, 1048 skipped, 5 days)
      From worker 2:    0 combinations to test (4616 succeeded, 2455 failed, 275 killed, 1056 skipped, 3 days)
      From worker 2:    0 combinations to test (4639 succeeded, 2447 failed, 268 killed, 1060 skipped, 1 day)
      From worker 2:    0 combinations to test (4830 succeeded, 2392 failed, 114 killed, 1048 skipped, 5 days)
      From worker 2:    0 combinations to test (4616 succeeded, 2455 failed, 275 killed, 1056 skipped, 3 days)
      From worker 2:    0 combinations to test (4639 succeeded, 2447 failed, 268 killed, 1060 skipped, 1 day)
      From worker 2:    0 combinations to test (4830 succeeded, 2392 failed, 114 killed, 1048 skipped, 5 days)
      From worker 2:    0 combinations to test (4616 succeeded, 2455 failed, 275 killed, 1056 skipped, 3 days)
      From worker 2:    0 combinations to test (4639 succeeded, 2447 failed, 268 killed, 1060 skipped, 1 day)
      From worker 2:    0 combinations to test (4830 succeeded, 2392 failed, 114 killed, 1048 skipped, 5 days)
      From worker 2:    0 combinations to test (4616 succeeded, 2455 failed, 275 killed, 1056 skipped, 3 days)

Collect deprecations separately

To inform femtocleaner which deprecations are most common

Julia build not always CI tested

Depending on the availability, it might download. Also looks broken: https://travis-ci.com/JuliaComputing/NewPkgEval.jl/jobs/271476640

Feature request: run package tests in an environment that supports terminal colors

As discussed in JuliaLang/julia#41885 (comment) it seems that when nanosoldier is run it does not properly produce terminal colors when requesting it in sprint. The example shared there is:

Expression: sprint(show, df, context = :color => true) == "\e[1m2×2 DataFrame\e[0m\n\e[1m Row \e[0m│\e[1m Fish   \e[0m\e[1m Mass      \e[0m\n\e[1m     \e[0m│\e[90m String \e[0m\e[90m Float64?  \e[0m\n─────┼───────────────────\n   1 │ Suzy          1.5\n   2 │ Amir   \e[90m missing   \e[0m"
   Evaluated: "2×2 DataFrame\n Row │ Fish    Mass\n     │ String  Float64?\n─────┼───────────────────\n   1 │ Suzy          1.5\n   2 │ Amir    missing" == "\e[1m2×2 DataFrame\e[0m\n\e[1m Row \e[0m│\e[1m Fish   \e[0m\e[1m Mass      \e[0m\n\e[1m     \e[0m│\e[90m String \e[0m\e[90m Float64?  \e[0m\n─────┼───────────────────\n   1 │ Suzy          1.5\n   2 │ Amir   \e[90m missing   \e[0m"

and we can see that sprint(show, df, context = :color => true) did not produce output with color escape sequences. In Github Actions tests set up in DataFrames.jl GitHub CI these tests pass correctly.

Make a package error if it tries to read from stdin during testing

Otherwise it hangs forever trying to read something.

Classify errors

E.g.:

• Parser failure
• Binary dependency failure
• Test failure

Output is not printed in chronological order

Test Summary: | Pass  Total
Parser        |   86     86
Test Summary: | Pass  Total
Serializer    |   48     48
Test Summary: | Pass  Total
Integration   |  108    108
Test Summary: | Pass  Total
Regression    |    8      8
 Resolving package versions...
 Installed JSON ──── v0.21.0
 Installed Parsers ─ v0.3.7
  Updating `~/.julia/environments/v1.3/Project.toml`
  [682c06a0] + JSON v0.21.0
  Updating `~/.julia/environments/v1.3/Manifest.toml`
  [682c06a0] + JSON v0.21.0
  [69de0a69] + Parsers v0.3.7
  [2a0f44e3] + Base64 
  [ade2ca70] + Dates 
  [8ba89e20] + Distributed 
  [b77e0a4c] + InteractiveUtils 
  [56ddb016] + Logging 
  [d6f4376e] + Markdown 
  [a63ad114] + Mmap 
  [de0858da] + Printf 
  [9a3f8284] + Random 
  [9e88b42a] + Serialization 
  [6462fe0b] + Sockets 
  [8dfed614] + Test 
  [4ec0a83e] + Unicode 
   Testing JSON
 Resolving package versions...
 Installed FixedPointNumbers ── v0.6.1
 Installed OrderedCollections ─ v1.1.0
 Installed OffsetArrays ─────── v0.11.1
 Installed DataStructures ───── v0.17.1
    Status `/tmp/jl_u0rzd6/Manifest.toml`
  [864edb3b] DataStructures v0.17.1
  [53c48c17] FixedPointNumbers v0.6.1
  [682c06a0] JSON v0.21.0
  [6fe1bfb0] OffsetArrays v0.11.1
  [bac558e1] OrderedCollections v1.1.0
  [69de0a69] Parsers v0.3.7
  [2a0f44e3] Base64  [`@stdlib/Base64`]
  [ade2ca70] Dates  [`@stdlib/Dates`]
  [8ba89e20] Distributed  [`@stdlib/Distributed`]
  [b77e0a4c] InteractiveUtils  [`@stdlib/InteractiveUtils`]
  [56ddb016] Logging  [`@stdlib/Logging`]
  [d6f4376e] Markdown  [`@stdlib/Markdown`]
  [a63ad114] Mmap  [`@stdlib/Mmap`]
  [de0858da] Printf  [`@stdlib/Printf`]
  [9a3f8284] Random  [`@stdlib/Random`]
  [9e88b42a] Serialization  [`@stdlib/Serialization`]
  [6462fe0b] Sockets  [`@stdlib/Sockets`]
  [8dfed614] Test  [`@stdlib/Test`]
  [4ec0a83e] Unicode  [`@stdlib/Unicode`]
   Testing JSON tests passed 

Different buffering between streams?

Prevent cloning the registry for each test

There are some commented pieces in the code to this end but just commenting them out does not seem to work.

Install LaTeX in the Docker image?

PGFPlotsX.jl fails because it needs LaTeX available (example log).

What do you think about adding LaTeX to the image? Doesn’t need to be a full TeX Live install. Just the minimum required to get PGFPlotsX.jl to pass.

cc: @maleadt

Also cc-ing @KristofferC since PGFPlotsX.jl is his package.

Quit packages that produce too much output

I just had a case where I was generating multiple 250GB+ files (until system filled up) during a PkgEval run (oxinabox/DataDeps.jl#104). We should detect bad behaving packages and quit when they have spammed too much output.

Simplify the scheduler

The current scheduler was written when upgrading packages to 1.0 and one wanted to know "if I fixed this package, how many other packages would that unblock". Packages are tested in priority order and a package that is depending on another only starts being tested after it's dependency has passed. This was nice in the "upgrading packages to 1.0"-sprint but right now it doesn't serve much of a purpose.

If we just run through all given packages "linearly" I think that could simplify quite a lot of the code.

Feature Request: Add a pass to test PackageCompiler on each package

We want to get to a point where PackageCompiler reliably produces system images and those system images are relocatable.

It would be nice to have an optional pass for testing this, which:

1. Compiles a package into a system image
2. Runs the tests as a different user in a different path, and compares the test results with those obtained without compiling the package into a system image.

@aviks @JeffBezanson @StefanKarpinski

Programmatically disable colors

We now rely on the master process being launched with --color=no.

CI recipe

Once we have #61, it would be interesting to have a CI recipe to test a package in the NewPkgEval environment. This should be possible with Travis + Docker.

Catch network errors

PkgEval failed: failed to clone from https://github.com/JuliaDiff/ChainRulesCore.jl.git, error: GitError(Code:ERROR, Class:Net, failed to resolve address for github.com: Temporary failure in name resolution)

ERROR: LoadError: InitError: HTTP/2 502 while requesting https://github.com/CalebBell/chemicals/raw/master/chemicals/Identifiers/chemical%20identifiers%20pubchem%20small.tsv

[04:07:35] curl: (6) Could not resolve host: github.com

ERROR: LoadError: Could not resolve host: math.nist.gov while requesting math.nist.gov/MatrixMarket/matrices.html

ERROR: LoadError: HTTP/2 301 (Resolving timed out after 30000 milliseconds) while requesting https://repo.continuum.io/miniconda/Miniconda3-latest-Linux-x86_64.sh

ERROR: LoadError: Could not download https://github.com/JuliaPackaging/Yggdrasil/releases/download/OpenSpecFun-v0.5.3+0/OpenSpecFun.v0.5.3.x86_64-linux-gnu-gcc8.tar.gz to /home/pkgeval/.julia/packages/SpecialFunctions/ne2iw/deps/usr/downloads/OpenSpecFun.v0.5.3.x86_64-linux-gnu-gcc8.tar.gz:

GitHub: Rate limiting

Fix optional GitHub authenitcation

Allow status "not applicable" for packages with external dependencies

My package

https://juliahub.com/ui/Packages/BitemporalPostgres/ZmypI/1.0.0

depends on POSTGRES. I would prefer status "not applicable" or similar instead of "failed".

allow packages to request more time

I noticed that ChainRules tests were timing out a while ago, because running the tests takes a while. Since ChainRules has become quite an important package in the ecosystem, it would be a shame if regressions were to go unnoticed and we don't really want to make the test suite less thorough. Just increasing the timeout for every package might not be a good idea, since there are often hangs when testing certain packages, but perhaps there could be a way for packages to tell PkgEval that their tests might take a little longer.

busywait duration test usually fails on newer commit but passes on older

I've noticed that StressTest.jl usually fails on the newer of the commits being tested, and passes on the older. For instance https://s3.amazonaws.com/julialang-reports/nanosoldier/pkgeval/by_date/2022-08/06/report.html
https://s3.amazonaws.com/julialang-reports/nanosoldier/pkgeval/by_hash/2415a14_vs_843b322/report.html

StressTest.jl currently just exports a busywait.

function busywait(sec::Real; yield_loop = false)
    sec ≥ 0 || throw(ArgumentError("cannot busywait for $sec seconds"))
    ns = sec * 1e9
    t = Base.time_ns()
    while Base.time_ns() - t < ns
        yield_loop && yield()
    end
    nothing
end

And the test is

@testset "busywait(seconds) duration within 1% of target duration" begin
    rng = Random.seed!(1234)
    for duration in (rand(5) .* 2)
        e = @elapsed busywait(duration)
        @test isapprox(e,duration,rtol=0.01)
    end
end

Given CI is, as a rule of thumb, hard to rely on that precisely I should probably just make the test more tolerant..

But I thought I'd report it, just in case it's highlighting that the runs aren't on a level playing field.
Could the system load during the evaluation the newer commit be worse than the older?

Support multiple architectures like `aarch64` and `powerpc64le`

Right now it looks like we hardcode a bunch of things for x86_64. Would be great if we could use PkgEval for acceptance testing on PPC and AArch64.

Deprecated packages

Should we maintain a list with "deprecated" packages (with, e.g., PaddedViews, ImagineFormat, NRRD, and MetaImageFormat. UnalignedVectors)? Or should the registry contain bounds such that these packages don't get installed?

PkgEval badges?

Now that we are running NewPkgEval every day, perhaps we can bring back the PkgEval badges that package authors can put in their package READMEs? The badge would say “passing” or “failing” to indicate the status of the package’s latest registered version on the most recent daily NewPkgEval run.

Retry flaky tests

Don't hard-code versions

Use https://julialang-s3.julialang.org/bin/versions.json.

Conditionally use Xvfb

Start from the list in https://github.com/JuliaCI/PackageEvaluator.jl/blob/d1cc2a42fa3640086b73c274903fec46ab4d1861/src/constants.jl

Remove package from blacklist

I just noticed that AbstractAlgebra is still blacklisted, because it "hangs forever" (observed over a year ago). I doubt this is still the case. I would really like to see our package removed from this list. Maybe it would be a good idea to rerun the blacklisted packages?

Add package testing time limits?

It might be worthwhile as the number of registered packages grows to set a maximum time allotted for each package. I had a package hang and I had to manually kill it as it was still going after 4 hours.

Write outside of package directory

Building Julia and writing version information probably shouldn't happen in the package directory (or at least not write to files that are part of the package). Maybe use ~/.cache/NewPkgEval.jl? That would make it easier to purge, too (currently it's possible that some files are left over since we only remove based on the contents of the toml files).

Use Sandbox.jl

https://github.com/staticfloat/Sandbox.jl

Prune breaks the cache

Probably caused by #68; after a couple of days PkgEval breaks:

┌ Error: An exception occurred during job execution
│   exception =
│    NanosoldierError: failed to run tests: TaskFailedException:
│    failed process: Process(`docker run --detach --gpus all -e NVIDIA_VISIBLE_DEVICES=all -e NVIDIA_DRIVER_CAPABILITIES=all --mount type=bind,source=/home/maleadt/.cache/NewPkgEval/usr/julia-1.6.0-DEV-8f512f3f6d/julia-8f512f3f6d,target=/opt/julia,readonly --mount type=bind,source=/home/maleadt/.julia/registries,target=/usr/local/share/julia/registries,readonly --mount type=bind,source=/home/maleadt/.cache/NewPkgEval/artifacts,target=/var/cache/julia/artifacts --env JULIA_DEPOT_PATH=::/usr/local/share/julia --tmpfs /home/pkgeval:exec,uid=1000,gid=1000 --cpus=2 --env JULIA_NUM_THREADS=2 --name Julia_v1.6.0-DEV-8f512f3f6d-DevIL-md8m3MeY --rm newpkgeval xvfb-run /opt/julia/bin/julia -e '    using InteractiveUtils
│        versioninfo()
│        println()
│
│        mkpath(".julia")
│        symlink("/var/cache/julia/artifacts", ".julia/artifacts")
│
│        using Pkg
│        Pkg.UPDATED_REGISTRY_THIS_SESSION[] = true
│
│        ENV["CI"] = true
│        ENV["PKGEVAL"] = true
│        ENV["JULIA_PKGEVAL"] = true
│
│        Pkg.add(ARGS...)
│        Pkg.test(ARGS...)
│    ' DevIL`, ProcessExited(125)) [125]
│
│    Stacktrace:
│     [1] pipeline_error at ./process.jl:525 [inlined]
│     [2] read(::Cmd) at ./process.jl:412
│     [3] read(::Cmd, ::Type{String}) at ./process.jl:421
│     [4] #spawn_sandboxed_julia#23(::Bool, ::String, ::Int64, ::Bool, ::typeof(NewPkgEval.spawn_sandboxed_julia), ::VersionNumber, ::Cmd) at /home/maleadt/.julia/dev/NewPkgEval/src/run.jl:83
│     [5] #spawn_sandboxed_julia at ./none:0 [inlined]
│     [6] #run_sandboxed_julia#22 at /home/maleadt/.julia/dev/NewPkgEval/src/run.jl:37 [inlined]
│     [7] (::NewPkgEval.var"#kw##run_sandboxed_julia")(::NamedTuple{(:stdout, :stderr, :stdin, :interactive, :wait, :name),Tuple{IOStream,IOStream,Base.DevNull,Bool,Bool,String}}, ::typeof(NewPkgEval.run_sandboxed_julia), ::VersionNumber, ::Cmd) at ./none:0
│     [8] (::NewPkgEval.var"#25#28"{Int64,Int64,Base.Iterators.Pairs{Union{},Union{},Tuple{},NamedTuple{(),Tuple{}}},VersionNumber,NamedTuple{(:name, :uuid, :path, :registry),Tuple{String,Base.UUID,String,String}},String})(::String, ::IOStream) at /home/maleadt/.julia/dev/NewPkgEval/src/run.jl:160
│     [9] mktemp(::NewPkgEval.var"#25#28"{Int64,Int64,Base.Iterators.Pairs{Union{},Union{},Tuple{},NamedTuple{(),Tuple{}}},VersionNumber,NamedTuple{(:name, :uuid, :path, :registry),Tuple{String,Base.UUID,String,String}},String}, ::String) at ./file.jl:611
│     [10] mktemp at ./file.jl:609 [inlined]
│     [11] #run_sandboxed_test#24(::Int64, ::Int64, ::Bool, ::Base.Iterators.Pairs{Union{},Union{},Tuple{},NamedTuple{(),Tuple{}}}, ::typeof(NewPkgEval.run_sandboxed_test), ::VersionNumber, ::NamedTuple{(:name, :uuid, :path, :registry),Tuple{String,Base.UUID,String,String}}) at /home/maleadt/.julia/dev/NewPkgEval/src/run.jl:159
│     [12] run_sandboxed_test at /home/maleadt/.julia/dev/NewPkgEval/src/run.jl:106 [inlined]
│     [13] macro expansion at /home/maleadt/.julia/dev/NewPkgEval/src/run.jl:363 [inlined]
│     [14] (::NewPkgEval.var"#35#42"{Int64,Base.Iterators.Pairs{Union{},Union{},Tuple{},NamedTuple{(),Tuple{}}},Array{NamedTuple{(:julia, :pkg),Tuple{VersionNumber,NamedTuple{(:name, :uuid, :path, :registry),Tuple{String,Base.UUID,String,String}}}},1},Array{Union{Nothing, NamedTuple{(:julia, :pkg),Tuple{VersionNumber,NamedTuple{(:name, :uuid, :path, :registry),Tuple{String,Base.UUID,String,String}}}}},1},Array{Dates.DateTime,1},NewPkgEval.var"#stop_work#38"{Array{Union{Nothing, NamedTuple{(:julia, :pkg),Tuple{VersionNumber,NamedTuple{(:name, :uuid, :path, :registry),Tuple{String,Base.UUID,String,String}}}}},1},Array{Task,1}},Int64})() at ./task.jl:333
│    Stacktrace:
│     [1] sync_end(::Array{Any,1}) at ./task.jl:300
│     [2] macro expansion at ./task.jl:319 [inlined]
│     [3] #run#31(::Int64, ::Int64, ::Base.Iterators.Pairs{Union{},Union{},Tuple{},NamedTuple{(),Tuple{}}}, ::typeof(NewPkgEval.run), ::Array{VersionNumber,1}, ::Array{Any,1}) at /home/maleadt/.julia/dev/NewPkgEval/src/run.jl:355
│     [4] (::NewPkgEval.var"#kw##run")(::NamedTuple{(:ninstances,),Tuple{Int64}}, ::typeof(NewPkgEval.run), ::Array{VersionNumber,1}, ::Array{Any,1}) at ./none:0
│     [5] (::Nanosoldier.var"#29#34"{Nanosoldier.PkgEvalJob,Dict{String,VersionNumber},Array{Any,1}})() at /home/maleadt/Nanosoldier/src/jobs/PkgEvalJob.jl:197
│     [6] withenv(::Nanosoldier.var"#29#34"{Nanosoldier.PkgEvalJob,Dict{String,VersionNumber},Array{Any,1}}, ::Pair{String,Bool}) at ./env.jl:161
│     [7] execute_tests!(::Nanosoldier.PkgEvalJob, ::Dict{String,Nanosoldier.BuildRef}, ::Dict{Any,Any}) at /home/maleadt/Nanosoldier/src/jobs/PkgEvalJob.jl:195
│     [8] run(::Nanosoldier.PkgEvalJob) at /home/maleadt/Nanosoldier/src/jobs/PkgEvalJob.jl:306
│     [9] (::Distributed.var"#108#110"{Distributed.CallMsg{:call_fetch}})() at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.3/Distributed/src/process_messages.jl:294
│     [10] run_work_thunk(::Distributed.var"#108#110"{Distributed.CallMsg{:call_fetch}}, ::Bool) at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.3/Distributed/src/process_messages.jl:79
│     [11] macro expansion at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.3/Distributed/src/process_messages.jl:294 [inlined]
│     [12] (::Distributed.var"#107#109"{Distributed.CallMsg{:call_fetch},Distributed.MsgHeader,Sockets.TCPSocket})() at ./task.jl:333
└ @ Nanosoldier ~/Nanosoldier/src/jobs/PkgEvalJob.jl:375

From worker 2:    docker: Error response from daemon: OCI runtime create failed: container_linux.go:349: starting container process caused "process_linux.go:449: container init caused \"rootfs_linux.go:58: mounting \\\"/home/maleadt/.cache/NewPkgEval/usr/julia-1.6.0-DEV-b828bbc426/julia-b828bbc426\\\" to rootfs \\\"/var/lib/docker/btrfs/subvolumes/afb4d8fb6cba36826d6f6279faace0775deb0dd09ca1543c92a1808d46442313\\\" at \\\"/opt/julia\\\" caused \\\"stat /home/maleadt/.cache/NewPkgEval/usr/julia-1.6.0-DEV-b828bbc426/julia-b828bbc426: no such file or directory\\\"\"": unknown.

That build, b828bbc426, had already been successfully executed before:

[ Info: [Node 2 | 2020-05-10T14:00:30.796]: running tests for PkgEvalJob JuliaLang/julia@8f512f3 [daily]
[ Info: [Node 2 | 2020-05-10T14:00:41.453]: Resolved primary build to Julia 1.6.0-DEV-8f512f3f6d (commit 8f512f3f6d5d69403be4b8179bd80ba64229e761 at JuliaLang/julia)
[ Info: [Node 2 | 2020-05-10T14:00:53.63]: Resolved against build to Julia 1.6.0-DEV-b828bbc426 (commit b828bbc426e85467665091f97cc16eac4587bb76 at JuliaLang/julia)

Versions.toml:

["1.6.0-DEV-4c3f57fb55"]
file = "julia.v1.6.0-DEV-4c3f57fb55.x86_64-linux-gnu.tar.gz"
sha = "6f816969fcce0abd87370a3cd44374be5411d1ad37adbbe63363fb7e3de20fb2"

["1.5.0-DEV-f7de6d242f"]
file = "julia-f7de6d242f-linux64.tar.gz"
sha = "0f2696d0f9097f760ea3a7d8de7767541b6baedb182abce7a234a6fe4cda1827"

["1.6.0-DEV-8f512f3f6d"]
file = "julia-8f512f3f6d-linux64.tar.gz"
sha = "42feb59454ef35713d0b245398ab280778014b30b014e809058cd4d5a80826af"

["1.6.0-DEV-b828bbc426"]
file = "julia-b828bbc426-linux64.tar.gz"
sha = "cf51f1cdb44684ce4b03ad9a009a5ae1789a7570249d50396da684c68271e36f"

["1.6.0-DEV-ddf7ce9a59"]
file = "julia-ddf7ce9a59-linux64.tar.gz"
sha = "3dcc89b708b04b7594bd683a39a78a81197d5ef2cf107c37920d797649963e42"

.cache/downloads:

downloads:
julia-1.4.0.tar.gz               julia-8f512f3f6d-linux64.tar.gz.sha256  julia-ddf7ce9a59-linux64.tar.gz         julia-f7de6d242f-linux64.tar.gz.sha256               julia.v1.6.0-DEV-4c3f57fb55.x86_64-linux-gnu.tar.gz.sha256
julia-1.4.0.tar.gz.sha256        julia-b828bbc426-linux64.tar.gz         julia-ddf7ce9a59-linux64.tar.gz.sha256  JuliaLang
julia-8f512f3f6d-linux64.tar.gz  julia-b828bbc426-linux64.tar.gz.sha256  julia-f7de6d242f-linux64.tar.gz         julia.v1.6.0-DEV-4c3f57fb55.x86_64-linux-gnu.tar.gz

Standard libraries are not tested properly

We currently do Pkg.add and Pkg.test, which results in the vendored stdlib getting tested, and not the upstream one.

use of `cd` is unsafe in parallel

This package depends on changing the global working directory, making it potentially unpredictable and unsuitable for use in applications that utilize Tasks. I recommend using joinpath(dir, base) and setenv(cmd::Cmd; dir) instead

Store timeouts as a separate result

So we can distinguish timed out packages vs failed packages.

Nanosoldier's suggested invocation to re-run failed test does not contain extra options like build flags.

E.g. https://s3.amazonaws.com/julialang-reports/nanosoldier/pkgeval/by_hash/4983135_vs_843b322/report.html

suggests

@nanosoldier `runtests(["Balloons", "COPT", "CellListMap", "CitableImage", "CountdownNumbers", ...], vs = ":release-1.8")`

even though this was launched with buildflags=["LLVM_ASSERTIONS=1", "FORCE_ASSERTIONS=1"], vs_buildflags=["LLVM_ASSERTIONS=1", "FORCE_ASSERTIONS=1".