sti build hangs

[openshift]

sti build hangs

[bparees]

Can you give us steps to recreate? what STI image, what STI scripts, what commit level of STI, etc?

(also what docker version)

[iocanel]

I've been using the https://github.com/fabric8io/base-sti base image. The sti scripts are part of the image and they are referenced using their github url.

My docker is at 1.4.1 and I'll try to upgrade and see if it makes any difference.

Here's an example of how I use the command:

sti build https://github.com/fabric8io/example-camel-cdi.git fabric8/base-sti fabric8/camel-example --loglevel=3

[bparees]

yup, I was able to reproduce this just running your sti command. and I do see the container has exited after running assemble successfully.

Given that the sti output hangs partway through the assemble script output, i'm guessing we're screwing up/overfilling a buffer somewhere. @soltysh can you take a look?

[iocanel]

The behaviour that I see makes me feel that the issue is related with the readers of the containers output and error. It seems like on of them stops reading and fails to close preventing the method to return? (limited understanding of golang here).

[bparees]

in the meantime, could you edit your assemble script to redirect the maven output to /dev/null? that might let you make progress...

[iocanel]

It does seem to make a difference.

[soltysh]

Looking...

[iocanel]

Using the latest master, I still have the same issue.

[soltysh]

@iocanel can you please check this once more. I just did a fresh build of STI (this revision) and the command:

sti build https://github.com/fabric8io/example-camel-cdi.git fabric8/base-sti fabric8/camel-example --loglevel=3

worked without any problem resulting in fabric8/camel-example image. The only problem I hit was running that result image ended in:

$ docker run fabric8/camel-example
/tmp/scripts/run: line 3: deploy-and-start: command not found

But that is not a STI problem from what I see. If you're still hitting the problem, please include the entire output of the build here (with --loglevel=5), I'll try to have a closer look at it.

[iocanel]

Will check and get back to you.

[goldmann]

I'm seeing this too. It feels like the build is stuck on downloading some Maven artifacts in my case, but in fact, the build container already exited with status 0 and new image was created as it should. My example command is:

$ sti build --loglevel=5 --forcePull=false https://github.com/goldmann/openshift-examples --contextDir=spring-eap6-quickstart 8fffd6ce363a test-jee-app

Where the 8fffd6ce363a image is a custom EAP image, but probably the wildfly8-centos image could be used too.

One note: this does not happen every time. If I re-run the command, there is a high chance that I'll get the full output.

[goldmann]

Any news on this? Since we added STI to our CI this is now failing our builds.

[mfojtik]

#160

@goldmann this might fix you (--assebleOutput=false)

[goldmann]

@mfojtik Well, this is more like a workaround :) Is there a way to redirect the output to somewhere? We would still need the logs for further investigation.

[soltysh]

@goldmann sorry but I can't still reproduce it and I'm still investigating the the source of the problem.

[goldmann]

@soltysh Fair enough, update this issues if you have something. Thanks!

[bparees]

@goldmann still an issue?

[goldmann]

Haven't seen it in a while.

[bparees]

closing, feel free to reopen if it recurs.

[goldmann]

After we upgraded to ac0b251, there is no output at all on our CI and the STI process is waiting (hanging) forever for the container, which exited cleanly.

Last lines of the container log:

Downloading: https://maven.repository.redhat.com/techpreview/all/org/apache/apache/7/apache-7.pom
Downloading: https://repository.jboss.org/nexus/content/groups/public/org/apache/apache/7/apache-7.pom
[WARNING] Checksum validation failed, no checksums available from the repository for https://repository.jboss.org/nexus/content/groups/public/org/apache/apache/7/apache-7.pom
Downloaded: https://repository.jboss.org/nexus/content/groups/public/org/apache/apache/7/apache-7.pom (15 KB at 7.4 KB/sec)
Downloading: https://maven.repository.redhat.com/techpreview/all/org/apache/maven/plugins/maven-clean-plugin/2.4.1/maven-clean-plugin-2.4.1.jar
Downloading: https://repository.jboss.org/nexus/content/groups/public/org/apache/maven/plugins/maven-clean-plugin/2.4.1/maven-clean-plugin-2.4.1.jar
[WARNING] Checksum validation failed, no checksums available from the repository for https://repository.jboss.org/nexus/content/groups/public/org/apache/maven/plugins/maven-clean-plugin/2.4.1/maven-clean-plugin-2.4.1.jar
Downloaded: https://repository.jboss.org/nexus/content/groups/public/org/apache/maven/plugins/maven-clean-plugin/2.4.1/maven-clean-plugin-2.4.1.jar (23 KB at 13.5 KB/sec)
[INFO] 
[INFO] --- maven-clean-plugin:2.4.1:clean (default-clean) @ spring-eap6-quickstart ---
[INFO] Deleting /home/jboss/source/target
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 10.249s
[INFO] Finished at: Mon May 18 14:02:34 EDT 2015
[INFO] Final Memory: 17M/903M
[INFO] ------------------------------------------------------------------------
Copying all WAR and EAR artifacts from /home/jboss/source/deployments directory into /opt/eap/standalone/deployments for later deployment...
'/home/jboss/source/deployments/ROOT.war' -> '/opt/eap/standalone/deployments/ROOT.war'

[mfojtik]

@goldmann is #149 (comment) still a reproducer for this?

[goldmann]

Just to let all know - we talked with @mfojtik and the issue is probably with the goroutines and wait when reading the container output.