Unable to perform Incremental build

[openshift]

Unable to perform Incremental build

[soltysh]

@jhadvig can you provide more details around how did you achieve that, because by default openshift/ruby-20-centos7 does not provide save-artifacts which is required to perform incremental build. If you built your own, can you please share? The error:

Error reading next tar header: io: read/write on closed pipe
Error saving previous build artifacts: timeout waiting for tar stream

indicates your save-artifacts script did not streamed tar contents into stdout.
Could you also describe all the steps you did to get to this point?

[Maxim-Filimonov]

I'm having similar issue. But looks like it actually works for me at least.
Our save-artifacts script
Sti log
Looks like incremental build does work but it throws some weird errors for some reason. At least when I run it without incremental it does bundle installation from the beginning while in this case it used previous bundle.
Is there a way to easy debug what happens during save-artifacts stage? It seems like it's impossible to insert any extra logs there as output is piped directly into tar.

[bparees]

if your save-artifacts script outputs to stderr that ought to get displayed and not piped into the tar, so you could try that.

[soltysh]

@Maxim-Filimonov thx for your comment, this line from the log worries me:

E0514 21:56:35.199285 12608 tar.go:192] Error creating file /var/folders/tb/0b6sc_rs2m112jvm89ft3hl80000gn/T/sti721578848/upload/artifacts/bundle/ruby/2.1.0/gems/builder-3.2.2/doc/releases/builder-2.1.1.rdoc: open /var/folders/tb/0b6sc_rs2m112jvm89ft3hl80000gn/T/sti721578848/upload/artifacts/bundle/ruby/2.1.0/gems/builder-3.2.2/doc/releases/builder-2.1.1.rdoc: too many open files

You're hitting too many open files limit. Can I ask you to verify 2 things:

1. Your open files limit ulimit -n
2. The number of files that save-artifacts puts into tar

Based on that answer I'll be able to diagnose the problem better.

[Maxim-Filimonov]

@soltysh Not sure which one applies i'll give you all 3

# inside container
ulimit -n
524288
# On Mac OS X
ulimit -n
2560
# On host vagrant machine (docker host)
ulimit -n
1024

find bundle -type f | wc -l
2552

Not saving assets yet only bundle. Need to fix this error first.

[soltysh]

@Maxim-Filimonov thanks, you've perfectly shown me what I was suspecting, we're relying on Go's defer in here instead of manually closing those files. Will fix that.

[bparees]

@soltysh looks like i was wrong about stderr, we should be capturing stderr and echoing it out (along with capturing stdout into the tar file) so there's a way to debug a save artifacts script.

[Maxim-Filimonov]

@soltysh yeah I have just realised that it is causing problems downline as gems are not properly untarred and app cannot run anymore.
Can you think of a possible workaround?

[soltysh]

@Maxim-Filimonov I'm currently working on a fix for that, I'll have a PR ready in some time.

[bparees]

@Maxim-Filimonov can you try with the latest STI? we'll get a mac release out momentarily.

[bparees]

@Maxim-Filimonov osx+windows releases are available now as well (See the v0.5.1 release)

[Maxim-Filimonov]

@bparees getting new error now - cp: error writing '/root/src/data/dump/local.1': No space left on device

[Maxim-Filimonov]

@bparees actually ditch this i was running it from wrong folder. it works now 👍

[bparees]

great! closing.

[babelop]

I'm still seeing this on latest version of STI. Though more often it fails with a random filename rather than just the next tar header. Ex:

Error extracting file bundle/ruby/gems/nokogiri-1.6.6.2/lib/nokogiri/xml/attr.rb: io: read/write on closed pipe

If I catch it, I can stop the build and run a new one and it will usually work the next time. I have increased both the ulimit and fs hard limits, which didn't seem to make a noticeable difference. Disk space and RAM are also plentiful.

OS is deployed on CentOS 7 official cloud image (via OpenStack). Single node.

Any suggestions?

[bparees]

Can you please open this as a new issue and give full recreate steps? are you performing an incremental build? full logs with --loglevel=5 would also be useful.