Can not get docker builds passing travis-ci check

[civctp2]

Can not get docker builds passing travis-ci check

[DEVoytas]

@ptitSeb, @LynxAbraxas could you have a look at this? I hope you know autotools better than I do.

[ptitSeb]

I'll try to have a look at this, but not sure I can before this WeekEnd.

[DEVoytas]

Thank you @ptitSeb ! No rush.
BTW, do you still sustain your consent to be the acting maintainer, until we find someone else to step in?

[LynxAbraxas]

Did you have a look at my comment #16 (comment) on your initial PR? I think this is due to a not clean docker context after the travis build.
The docker build should have a clean environment and ideally its own travis job. That's what I tried to achieve in f188765 that makes #13 and its squashed version #23 succeed on travis.

[ptitSeb]

I just looked, but I'm not fond of a solution that use a submodule. (also, why is docker build important?)

[LynxAbraxas]

See #23, it does not use submodules, (#13 does not use any either, it only originates from a history, where submodules were used).
Docker builds lead to a docker images that "contain" and "preserve" the OS config and therefore make it possible to run the app under Linux (even Win or MacOS) even if the host OS does not provide the necessary libraries any more, and that will be possible as long as docker is compatible to the generated image, so likely for decades. So if you have a working docker image of ctp2 you will be able to run and play the game for years without the need to recompile;-)

[ptitSeb]

I understand what Docker is, what I don't get is the purpose of building ctp2 in a docker.

Isn't the point of maintaining the source is to have the software compiling and running on current OS, and on a as large as possible OS target.

[LynxAbraxas]

Well, my reason to build ctp2 in docker is to get a docker image of a playable ctp2 such that I always have a running version as long as my last functional docker image is still supported by docker (lost many days over the years just for getting ctp2 to compile again on new/upgraded systems due to new compiler, libs, etc). For a playable image, I need to include some of the non-open game files and thats the reason why I initially started with a separate repos (https://github.com/LynxAbraxas/ctp2DF) with ctp2 and ctp2CD as submodules. In order to not conflict with the ctp2 code release conditions, I have to keep ctp2CD private (it only contains the unpacked civlang.ctp and civmain.ctp from the original CD, with file names adjusted to fit to those that are in ctp2, as described in the docs). Since the resulting docker image is fully playable, I also have to keep that private. That's why I chose to use GitLab (GL), which offers private repos and docker registry for free. Since @DEVoytas asked for the Dockerfile (DF), I created PR #13 with the full history of the DF creation and its reasoning, and a compressed form in case the history is not wanted (#23).
When this GL feature request is realized, it would be possible to mirror the official civctp2/civctp2 repos on GL, keeping a ctp2CD repos private as well as the civctp2 registry, but have the rest of the repos public, including test results and the automatically taken screen shots.

An advantage to the DF approach is, that a playable image can be used for automated smoke tests (which I couldn't resist to implement...). At the current state, GL executes 3 test after the build: One test if the start screen shows up, one if a new game can be loaded and one if a saved game can be loaded. An Xvfb instance is started for the headless rendering and SikuliX for automated mouse interaction based on pattern images. If all tests pass, a release image is pushed to the registry. This now works for the master branch and the linux branch.
That way it was easy to find out that this change causes a segfault after the start screen is up but before a new or saved game is loaded, see #25.
Or e.g. that the linux branch compiles for i386 and x86_64 systems but segfaults at start up on x86_64 (and runs fine on i386), as the tests for linux pass while those for linux-64 all fail.

I guess travis is also running the jobs in a docker environment, but the compilation (and test/run) environment is only clearly defined when providing the Dockerfile in the repos because then every layer up to the source image is well known.
In addition, even if not used for building and testing, having the DF in the repos will be helpful for those interested in playing the game from a docker image (like me). That's why I left the run.sh script in #13 (or use x11docker instead).
In order to ensure, that the DF at least is in a state that ctp2 compiles, it makes sens to have a build job in travis for testing that.

[DEVoytas]

Hi @LynxAbraxas an @ptitSeb. Sorry I was not active recently. This is why I think we really need maintainer for this project! I would like to pass credential to any of you, although, I remember @ptitSeb agreed to become acting maintainer temporary, until someone else agrees to take a role. @ptitSeb are you still OK with that?

Regarding the Docker... Maintainer will decide whatever he thinks, but let my give my view on it.
In my opinion it's OK to have docker build in the main game repo as long as it does not introduce additional dependencies. So the source code in this repo is what is required to generate main game binary, and if we have docker file, its only role should be to do exactly that but in docker container.
For Docker that have some external dependencies (requires game data, for playing for example), there should be separate project, that requires (among others) binary from this repo, game data from original CD, and possibly some other stuff. This is how I think about it, feel free to disagree.

Also apart from whatever I wrote above... @LynxAbraxas I always wondered why game data has to be in git repository (I get the private part, just asking about git)? To my understanding this data does not change (or does it?), so there is no need to track changes in it. So why not use regular local directory for that, instead of git repo?

[LynxAbraxas]

The game data does not have to be in a git repos (though it is quite convenient for switching between e.g. the English and the German version), but I found it the easiest way to provide the data to GL when building the docker image. As a git repos, it can be pushed to GL, kept private and if you stick to the relative path for the subrepos, you don't even need to provide extra credentials, GL will just see it as a relative repos of your account. Just give it a try! Clone ctp2DF, create your own ctp2DF/../ctp2CD with the game data (appropriately renamed to fit that of civctp2), add that commit sha as the status of ctp2DF/ctp2CD/ (git submodule sync; git add ctp2CD) commit and push both to your GL account as private repos. Then wait for about 30 minutes and when the docker image is ready in your GL registry, just do a ./run.sh registry.gitlab.com/yourAccount/ctp2df/master ./ctp2 and start playing;-) You will also have the screenshots of the last tests shown in the README.md on GL.

[ptitSeb]

Hi @LynxAbraxas an @ptitSeb. Sorry I was not active recently. This is why I think we really need maintainer for this project! I would like to pass credential to any of you, although, I remember @ptitSeb agreed to become acting maintainer temporary, until someone else agrees to take a role. @ptitSeb are you still OK with that?

I don't have much time lately for civctp2 (too many other project for now). So I'm not the best choice right now.
Also, while I find all this docker/gitlab stuff neet, I don't se the point of having that on the "main" repo. I would prefer all this to be set as a separate repo and leave the main repo Source only. For me, the point of this repo is to have the sources of civctp2, compilable on as much system as possible (so I'm in line with what @DEVoytas already expressed).

[LynxAbraxas]

For me, the point of this repo is to have the sources of civctp2, compilable on as much system as possible

That's fine, but the aim should not only be that the source compiles but actually leads to a usable executable, and ideally that changes do not introduce bugs or regressions. If such testing is in a separate repos, there won't be the comfort of having GH test each pushed commit automatically.
For example some changes (just for the sake of compilation) caused a segfault when loading a game (see #25) and some even remove any kind of successful diplomacy (#35). Creating test cases to avoid the former are much easier than for the latter. I can offer some for GL, if they should be for travis, they have to be contributed from someone else.

Not sure if we are still discussing the actual issue here. Both build types are passing with either #13 or #23, and the actual reason for the issue is also adressed, so I think this issue can be closed. The discussion wether to include a Dockerfile in the source repo should be done in a separate issue.

[DEVoytas]

If such testing is in a separate repos, there won't be the comfort of having GH test each pushed commit automatically.

But if those are not automatic tests that could run on travis, and need to be run manually, there seem to be not much difference if they are in the same repo or not. I don’t think it is worth the cost of introducing extra dependency, especially dependency on binary files that need to be private.
On the other hand I would be very much in favour of some automatic tests that could be run by travis, but I agree with you that discussion on that topic probably dos not belong to this issue.

Both build types are passing with either #13 or #23, and the actual reason for the issue is also adressed, so I think this issue can be closed

You right @LynxAbraxas , it should probably be closed, but before I do that I would really like to understand why for my PR I got failure from travis and your one is passing. Could you help me with that? Is it something different in Docker file, travis file, or something else ?

[LynxAbraxas]

But if those are not automatic tests that could run on travis, and need to be run manually, there seem to be not much difference if they are in the same repo or not.

But they are automatic (thanks to sikuli)! They just depend on a private repo with the CD files. I chose GL to realize that because I love GL but I guess you could realize something similar with travis and some camouflaged login to a private GH repos.
However, GL offers syncing with a GH repos, and I think pushing to the GH repos then even triggers the GL pipelines (I haven't tested this though).

... I would really like to understand why for my PR I got failure from travis and your one is passing. Could you help me with that? Is it something different in Docker file, travis file, or something else ?

The only difference between your approach and mine is that I let the docker build run in its own travis job, which ensures that no files from the direct build environment (as configure by @ptitSeb) get into the docker context. In your approach that happens (because run after the direct build) and since the build tools in the direct and in the docker environment do not have the same version, you run into the observed problem with libtool, see my answer here: #16 (comment).
Another way to avoid that problem would be to use git clone in the DF instead of COPY but that then easily leads to stale caches (though this is not enabled in the travis job so far, I'm not familiar with travis defaults on sharing caches between builds).

[DEVoytas]

Thanks to @LynxAbraxas docker builds now work fine on master: https://travis-ci.com/civctp2/civctp2
so let's close this issue.