ohwgiles / laminar Goto Github PK
View Code? Open in Web Editor NEWFast and lightweight Continuous Integration
Home Page: https://laminar.ohwg.net
License: GNU General Public License v3.0
Fast and lightweight Continuous Integration
Home Page: https://laminar.ohwg.net
License: GNU General Public License v3.0
running laminar behind a proxy and the custom stylesheet isn't taken into account.
would it be possible to sort the runs by result (success,failure), date started, duration, reason ....?
what are you thoughts on this, please.
when watching progress in specific tag tab the jobs don't refresh with new queued jobs.
Hi I am trying to run laminar in a container (LXD gentoo with OpenRC). And this is what I am getting as output.
The container is named laminar-ci it has internet access,
`
terminate called after throwing an instance of 'kj::ExceptionImpl'
what(): kj/async-io-unix.c++:748: failed: DNS lookup failed.; params.host = unix-abstract; params.service = laminar; gai_strerror(status) = Servname not supported for ai_socktype
stack: 0x7f1abdbe3063 0x7f1abde311c1 0x7f1abde28e2d 0x7f1abde2963f 0x7f1abde11bb2 0x7f1abde23a10 0x42af70 0x42e99e 0x7f1abde18286 0x7f1abde117cb 0x7f1abde14e75 0x431d5f 0x42b4e2 0x4582eb 0x4297f8 0x7f1abc455f10 0x4299e9
Aborted
`
So there is some problem with dns? Can you please help me? Thank you
I don't know if this will pass your KISS principle, but would it be possible to start a job from the command interface?
Thanks
I am trying to execute a script over ssh, but get different results when compared to starting a local script.
This is the pkg-builder-remote.run file on laminar server
#!/bin/bash
echo Running on robor ...
ssh [email protected] /bin/bash /root/test.sh
echo Done
And this is the test.sh file on the remote server:
#!/bin/bash
C_NAME=pkg-builder-1
echo $C_NAME
lxc launch -e -p prf-funtoo -p default fun-generic "$C_NAME"
sleep 3
lxc exec "$C_NAME" ip a add 10.214.101.2/24 dev eth0
lxc exec "$C_NAME" -- ping -c3 10.214.101.1
echo "Waiting 3 seconds"
sleep 3
lxc exec "$C_NAME" -- poweroff
Results from the run as displayed on laminar:
pkg-builder-remote #1
Console output
[laminar] Executing /var/lib/laminar/cfg/jobs/pkg-builder-remote.run
Running on robor ...
pkg-builder-1
Creating pkg-builder-1
Starting pkg-builder-1
PING 10.214.101.1 (10.214.101.1) 56(84) bytes of data.
64 bytes from 10.214.101.1: icmp_seq=1 ttl=64 time=0.152 ms
64 bytes from 10.214.101.1: icmp_seq=2 ttl=64 time=0.056 ms
64 bytes from 10.214.101.1: icmp_seq=3 ttl=64 time=0.054 ms
--- 10.214.101.1 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2118ms
rtt min/avg/max/mdev = 0.054/0.087/0.152/0.046 ms
Waiting 3 seconds
Done
This is the pkg-builder.run file on laminar server:
#!/bin/bash
C_NAME=pkg-builder-$RUN
echo Connecting to remote host and starting container $C_NAME
ssh [email protected] "env C_NAME=$C_NAME /bin/bash -xe"<<"EOF"
echo $C_NAME
lxc launch -e -p prf-funtoo -p default fun-generic "$C_NAME"
sleep 3
lxc exec "$C_NAME" ip a add 10.214.101.2/24 dev eth0
lxc exec "$C_NAME" -- ping -c3 10.214.101.1
echo "Waiting 3 seconds"
sleep 3
lxc exec "$C_NAME" -- poweroff
EOF
This is the output from laminar:
Console output
[laminar] Executing /var/lib/laminar/cfg/jobs/pkg-builder.run
We are going to start container:
Connecting to remote host and starting container pkg-builder-37
+ lxc launch -e -p prf-funtoo -p default fun-generic pkg-builder-37
Creating pkg-builder-37
Starting pkg-builder-37
+ sleep 3
+ lxc exec pkg-builder-37 ip a add 10.214.101.2/24 dev eth0
So the ping command and sleep and poweroff commands never get executed.
Am I doing something wrong?
Thanks
When I set LAMINAR_ARCHVE_URL
, my artefact suddenly got a strange URL: http://192.168.1.33:8080/jobs/hello/https://ci.<mydomain.tld>/archive/hello/1/hello-20170922
.
I'm accessing Laminar via HTTP because I cannot proxy websockets with my lighttpd server - it lacks mod_websocket to do it, and Laminar is trying to access unsecured socket anyway - but the reason is not important as different URL schemes probably are.
for "Builds per job in the last 24 hours" and "Average build time per job this week" graphs if a lot of jobs are running the graph is unusable.
When the laminar web UI is behind a reverse proxy (nginx), the website does not work. It does not show any UI elements:
The means of connecting to the website does not seem to have any impact. I tried connecting via a unix socket and via a network socket. The nginx rule used to forward requests to laminar is simple:
location / {
#proxy_pass http://unix:/var/lib/laminar/laminard.sock;
proxy_pass http://example.com:8081;
}
When I connect to http://example.com:8081 directly, the web UI is displayed correctly.
boost::filesystem
and websocketpp
can be replaced with new functionality in capnproto
capnproto
http features for serving files (see #37)SIGCHLD
handling with the new functionality from capnproto
capnp
code as used in Server.cpp
as compared to other files. It already leaked into Run
in a few recent commits.client.cpp
and RpcImpl
The current behaviour of LAMINAR_KEEP_RUNDIR
is that if the env variable is set, all job runs will be preserved. This leads to jobs piling up on the disk, which is why this setting is disabled in my setup. However, if I need to inspect the environment of a failed job, I have to change the configuration, restart laminar, and re-run the failing job.
I suggest that LAMINAR_KEEP_RUNDIR
supports integer values. The integer represents the number of run directories that will be preserved.
For example: LAMINAR_KEEP_RUNDIR=3
will preserve the three most recent run directories for every job.
This would allow me to enable the setting permanently. It would be easier to inspect and debug the environment in which a job failed without having to re-configure and restart laminar.
I am bringing this idea up, because it seems somewhat related to #18 .
Hi Oliver,
it is me again.
I am getting this on my box, again:
terminate called after throwing an instance of 'kj::ExceptionImpl'
what(): kj/async-io-unix.c++:297: disconnected: ::read(fd, buffer, maxBytes): Connection reset by peer
stack: 0x7f7ca9133236 0x7f7ca91093f4 0x4456c0 0x42fb0c 0x7f7ca911076d 0x7f7ca9108ea9 0x7f7ca910ceb4 0x4338ba 0x42c08c 0x458274 0x42a157 0x7f7ca79e672c 0x42a368
What info do you need to debug?
Is this expected behaviour?
If I run the command with ENV=env laminarc trigger job the ENV doesn't get passed to the job, but if I do laminarc trigger jobs ENV=env - env gets passed to the job.
The current Bootstrap 3 layout is starting to look a bit dated
When I am trying to download a file from laminar's archive I am getting this:
This is even directly downloading
wget http://laminar.lxd:8080/archive/fun-prime/89/dev-db/mariadb-10.2.9.tbz2
--2018-04-12 08:29:42-- http://laminar.lxd:8080/archive/fun-prime/89/dev-db/mariadb-10.2.9.tbz2
Resolving laminar.lxd... 127.0.0.1
Connecting to laminar.lxd|127.0.0.1|:8080... connected.
HTTP request sent, awaiting response... 200 OK
Length: 43209261 (41M)
Saving to: ‘mariadb-10.2.9.tbz2’
mariadb-10.2.9.tbz2 8%[========> ] 3.56M --.-KB/s in 0.01s
2018-04-12 08:29:42 (320 MB/s) - Connection closed at byte 3732424. Retrying.
--2018-04-12 08:29:43-- (try: 2) http://laminar.lxd:8080/archive/fun-prime/89/dev-db/mariadb-10.2.9.tbz2
Connecting to laminar.lxd|127.0.0.1|:8080... connected.
HTTP request sent, awaiting response... 200 OK
Length: 43209261 (41M)
Saving to: ‘mariadb-10.2.9.tbz2’
mariadb-10.2.9.tbz2 20%[====================> ] 8.37M --.-KB/s in 0.02s
2018-04-12 08:29:43 (517 MB/s) - Connection closed at byte 8774615. Retrying.
^C
I am running without proxy, so :8080 is directly laminar listening for requests.
But running with proxy doesn't make it better.
2018-04-12 16:39:16 (602 KB/s) - Connection closed at byte 3404021. Retrying.
How can I help to debug it for you some more?
are you planning to extend the tool with some notifications? not only browser notifications, but support for external stuff?
Could the running jobs show the current execution time of the job?
If a job run is aborted due to a timeout, the .after
script should still run, because that's probably where a notification would be sent
So currently if you stay on the jobs page and let jobs run, you are able to see all jobs that started after you have opened the jobs page. If you follow a link to details about single job and come back you only see approx 30 jobs.
It would be nice if you scroll to the bottom of the page that the page would dynamically load more finished jobs (it can be 30 at a time).
Do you know what I am trying to say?
So I have 30 executors running they are shown as busy and no idle. After refresh of the page it shows. 30 busy and 30 idle, which is not correct.
It should be possible to display each tab group on the jobs page in a high-visibility mode for displaying on large screens in the office
Hi,
it would be nice if job/node (at least job) config was parsed before job run (trigger event) - in more complex scenarios with nodes, there is problem with jobs queued forever.
Example: Create job hello2.run
and it's config hello2.conf
, then execute laminarc trigger hello2
- job si queued forever. (no hello2.conf
parsed until daemon restart).
By studying sources, I didn't found any simple workaround - should I create patch/pullrequest? I'd appraise you to handle this simple fix by yourself - I won't like to mess your clean code by my hand :-D
Currently the executor seems to be set to 8 - at least on my machine. Can you tweak this setting?
I have:
nodes/default.conf
EXECUTORS=2
nodes/emerge.conf
EXECUTORS=1
TAGS=emerge
but laminar spins up 3 jobs for a job tagged emerge using jobs/job.conf
Am I using it wrong?
Thank you
So I am creating jobs every 10 seconds like this:
for i in `cat $(wc -l pkgcat/*|sort|awk '{ print $2 }'|grep -v total |head -10)` ;do LAMINAR_REASON="$i" laminarc trigger fun-prime PKG=$i;sleep 10;done
The input of i$ is something like this:
cat $(wc -l pkgcat/*|sort|awk '{ print $2 }'|grep -v total |head -10)
app-antivirus/clamav
app-antivirus/clamav-unofficial-sigs
app-antivirus/skyldav
x11-base/xorg-drivers
x11-base/xorg-server
x11-base/xorg-x11
....
The scipt fun-prime is starting a container and then building the package inside the LXD container.
Can supply those as well if you need them.
So there is a new run added to the job for a different package.
This is inside the laminard.log:
cat /var/log/laminard
terminate called after throwing an instance of 'kj::ExceptionImpl'
what(): laminar.cpp:376: failed: expected siginfo->ssi_signo == SIGCHLD
stack: 0x42be9e 0x42dde7 0x7fa345d8a544 0x42ebec 0x7fa345d918bd 0x7fa345d89ff9 0x7fa345d8e004 0x43284a 0x42b6d9 0x4574b1 0x429877 0x7fa3446866fb 0x429a98
When asked to shutdown, laminar should terminate all child processes and mark their result as "aborted"
When starting a job, amount of time passed starts on negative value. Looks very strange.
Also, how is estimate progress calculated? Maybe this is connected.
When there are jobs, being added to queue and you are on the jobs page the queued jobs increment and get shown correctly. Once you refresh the page or navigate away and come back the queued jobs are not shown. If there are even more jobs being added to the queue then the queue starts counting from 1 again, which is not correct.
So this happens only with run number 1 of a job all subsequent runs behave correctly.
I wanted to do a screencast, but failing on some ffmpeg (x11grab deprecated) so will add this later if needed.
When packaging Laminar, I got this warning from rpmlint:
laminar.x86_64: W: executable-stack /usr/sbin/laminard
The binary declares the stack as executable. Executable stack is usually an
error as it is only needed if the code contains GCC trampolines or similar
constructs which uses code on the stack. One common source for needlessly
executable stack cases are object files built from assembler files which don't
define a proper .note.GNU-stack section.
That’s a CVE waiting to happen, so it should be fixed.
A job's current status could be served as a badge, e.g. laminar:8080/jobs/JOB/badge.svg
. Remember to set a short cache period.
how should I pass variable with space? eg. USE="this that"
because every time I use laminarc set USE=$USE
it gets truncated to USE='"this'
am I doing something wrong?
If the connection from the browser to laminard is interrupted (e.g. server dies, or reverse proxy times out the websocket link), the browser should attempt to reconnect, and display a notification if this fails
Screenshot
https://postimg.org/image/4l12ra9tgr/
Tried to compile on Gentoo on my Raspberry Pi. Got this instead:
/home/kisik21/laminar/src/laminar.cpp: In member function ‘void Laminar::runFinished(Run*)’:
/home/kisik21/laminar/src/laminar.cpp:689:48: error: invalid conversion from ‘size_t* {aka unsigned int*}’ to ‘uLongf* {aka long unsigned int*}’ [-fpermissive]
(unsigned char*)&r->log[0], logsize) == Z_OK) {
^
additionally, I had to install rapidjson for build to process, it happens to be included somewhere.
When I configure the environment variable LAMINAR_KEEP_WORKDIR, the directories of the individual job runs are not preserved.
Steps to reproduce
Expected results
The directory run/my-job/1/ is visible after the job crashed or finished.
Actual results
The directory run/my-job/1/ is temporarily visible. As soon as the job execution stops, the directory is removed.
BTW: I do like the project idea very much! Despite being a very young project, I chose to work with Laminar, because everything looks so tidy and polished :) Thanks for your efforts.
Replace the existing trigger
(which queues a job) and start
(which runs a job to completion) with queue
(queues a job), start
(blocks until job starts) and run
(blocks until completion)
Hi, I have tried to rebuild last version (commit 9c25681) from git with script from this repo and it failed.
I think, it is due to some old lib or something similar, but I'm not familiar with it enough to find, where issue is. Any kick possible?
Thanks, Valicek1
Error (from command make -j4
in debian docker script, no errors in steps before).
root@95e46937385e:/build/laminar# make -j4
[ 8%] Generating laminar.capnp.c++, laminar.capnp.h
[ 8%] Generating css/bootstrap.min.css.z
[ 11%] Generating index.html.z
[ 8%] Generating laminar.capnp.c++, laminar.capnp.h
[ 13%] Generating js/app.js.z
[ 16%] Generating favicon.ico.z
[ 19%] Generating favicon-152.png.z
[ 25%] Generating icon.png.z
[ 25%] Generating js/vue-router.min.js.z
Scanning dependencies of target laminarc
[ 27%] Generating js/vue.min.js.z
[ 33%] Generating js/ansi_up.js.z
[ 33%] Generating js/Chart.min.js.z
[ 36%] Generating js/Chart.HorizontalBar.js.z
[ 41%] Generating css/bootstrap.min.css.o
[ 41%] Building CXX object CMakeFiles/laminarc.dir/src/client.cpp.o
[ 44%] Building CXX object CMakeFiles/laminarc.dir/laminar.capnp.c++.o
[ 47%] Generating index.html.o
[ 50%] Generating js/app.js.o
[ 52%] Generating favicon.ico.o
[ 55%] Generating favicon-152.png.o
[ 58%] Generating icon.png.o
[ 61%] Generating js/vue-router.min.js.o
[ 63%] Generating js/vue.min.js.o
[ 66%] Generating js/ansi_up.js.o
[ 69%] Generating js/Chart.min.js.o
[ 72%] Generating js/Chart.HorizontalBar.js.o
Scanning dependencies of target laminard
[ 77%] Building CXX object CMakeFiles/laminard.dir/src/database.cpp.o
[ 77%] Building CXX object CMakeFiles/laminard.dir/src/main.cpp.o
[ 80%] Building CXX object CMakeFiles/laminard.dir/src/server.cpp.o
[ 83%] Building CXX object CMakeFiles/laminard.dir/src/laminar.cpp.o
[ 86%] Linking CXX executable laminarc
[ 86%] Built target laminarc
[ 88%] Building CXX object CMakeFiles/laminard.dir/src/conf.cpp.o
[ 91%] Building CXX object CMakeFiles/laminard.dir/src/resources.cpp.o
[ 94%] Building CXX object CMakeFiles/laminard.dir/src/run.cpp.o
[ 97%] Building CXX object CMakeFiles/laminard.dir/laminar.capnp.c++.o
/build/laminar/src/server.cpp: In member function 'void Server::start()':
/build/laminar/src/server.cpp:438:16: error: 'class kj::TaskSet' has no member named 'onEmpty'
childTasks.onEmpty().wait(ioContext.waitScope);
^~~~~~~
/build/laminar/src/server.cpp:440:25: error: 'class kj::WaitScope' has no member named 'poll'
ioContext.waitScope.poll();
^~~~
CMakeFiles/laminard.dir/build.make:205: recipe for target 'CMakeFiles/laminard.dir/src/server.cpp.o' failed
make[2]: *** [CMakeFiles/laminard.dir/src/server.cpp.o] Error 1
make[2]: *** Waiting for unfinished jobs....
CMakeFiles/Makefile2:67: recipe for target 'CMakeFiles/laminard.dir/all' failed
make[1]: *** [CMakeFiles/laminard.dir/all] Error 2
Makefile:127: recipe for target 'all' failed
make: *** [all] Error 2
When triggering child jobs, as described by [1], the only returned information about child job's result is exit code ([2]). Having lots of parent and child jobs executed it's easy (at least for me) to lose track which child job "belongs" to the parent one. Thus I think it would be cool for start
result to contain also any "pointer" to the child job.
Examples of what I have in mind:
$ laminarc start ajob
75
$ echo $?
0
laminard
to become self-conscious about its URL, and somehow it would need to became configurable - not sure whether this is cool approach?$ laminarc start ajob
http://laminar.example/jobs/ajob/75
$ echo $?
0
[1] http://laminar.ohwgiles.documentup.com/#job-chains
[2] https://github.com/ohwgiles/laminar/blob/master/src/laminar.capnp#L26
it would be nice to have tooltips for most of the graphs. showing the number of runs, or execution time ...
Steps to reproduce:
Thanks for looking into it
PS: This doesn't happen if you input new url into the address bar, only if you navigate within the application.
Would be nice if an open Laminar tab in browser could ping you with a browser notification when something finishes cooking up.
I hope it doesn't break your minimalistic approach (which is cool, because Laminar is blazing fast on a low-power Raspberry Pi!), but would be probably useful for me, as my job notifier doesn't work for some reason...
Trying to grasp laminar's power, but something went wrong... I queued a "hello" job, tried to run it 5 times, and received no log output, when I looked into Web UI!
Tried to run it 6th time, received some log output while the job is still running, but after reloading, the job is shown like it never was - everything is NaN or undefined, and no logs are saved.
Maybe it's a bug, maybe it's intentional, but how to make laminar at least save the job logs and stats? By the way, the "Last success" and "Last build" stats on job screen also suffer from this.
It should be possible to set a maximum execution time for any job, after which laminar should abort it. Depends on #29
I have noticed following:
I have couple of jobs queued and there are already let's say 4 jobs running. If I edit cfg/nodes/TAG.conf while jobs running, after I save laminar will launch another 4 jobs.
It should check if the limit of jobs is not already running and launch jobs only if the current running jobs < jobs defined in .conf
I have a couple of failed builds that are covering the passed ones in the Total builds graph. Could these be made semi-transparent?
Thank you
Hi Oliver,
so I have laminar crashing upon me under load. And if I restart it there are stale run directories, which don't allow to start new runs or jobs.
It would be good to solve the crashing problem of course, but also if a crash happens enable laminar to recover from it:
Let me know what you think.
Pavol
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.