Load variables from properties file

[adriaanm]

Load variables from properties file

[cunei]

The section in the end was named 'vars' rather than 'globals' in the end, according to Iulian's original request in #40. That was also because configuration files may end up being nested into one another, in the future, therefore 'globals' would not be an accurate description in that case. The fix for #40 does preserve the overall schema checking in the configuration file, whereas the initial hack was just disabling schema checking altogether. The fix works by specifying a custom deserializer that simply skips over the 'vars' section, while the rest is processed as normal by the jacks/jackson libraries (mostly using reflection).

It is possible to "smuggle" inside properties in a number of ways. Normally, when you define them as bash vars for instance, they are expanded in the config file directly when the Typesafe config library loads the configuration file, and the same happens for java properties. That all happens in the line:

readValue[DBuildConfiguration](configFile)

in SbtBuildMain.scala; the implementation of readValue is:

def readValue[T](f: File)(implicit m: Manifest[T]) = readValueT[T](parseFile(f))

where the parseFile is the call to the config library that reads the file and expands all the vars, converting all to a single string. Then, the subsequent readValueT performs the actual JSON parsing.

There are various ways in which loading a properties file can be accomplished. The first would be to add some code before the readValue() that loads the properties file and manually adds properties to the System properties beforehand. This is a bit undesirable, as it would pollute the system properties unnecessarily.

Another way could be to load the properties file(s), and then "merge" them together with the original loaded configuration by using the typesafe config's API, either using the properties as-is, or by lifting them and adding the 'vars' prefix.

The properties file(s) could be specified by either using a command line option, or as an additional option in the top-level 'options' section of the config file. I am trying to store the entire configuration in the config file, therefore the second approach would be preferable.

In the first case, the option would have to be introduced via an option introduced by a flag; however, there is no command line option parsing code in dbuild, at this time.

The second case is better, but it involves a bit of a catch-22: the readValueT can only be performed once the config is complete, but in order to extract the properties file path from the config file it is usually necessary to run readValueT on it first. We could be 'cheating' by hardcoding the relevant option path, and loading it manually by using the typesafe config library (rather than using the JSON parsing code), right after the config file is loaded. I would have to think whether a cleaner approach is possible.

It can be done, it just involves a bit of work; I'll be happy to work on it. If you need this soon, I can work on it now for inclusion in the next upcoming release of dbuild (due soon).

[adriaanm]

It would be very handy to have something like this, as we're already seeing the pain of keeping all the version numbers in synch, as well as parsing the properties file in bash, exporting to environment variables, and importing those again in the builds. If we could standardize on property names across builds, it would be a lot nicer. It's not a must-have, though -- I can work around it in the jenkins scripts.

[adriaanm]

Also, thanks for the thorough explanation! One thing that occurred to me is that repeatable builds are important, so we should only read property files from git repos, so that the sha of the commit determines the content of the property file.

[cunei]

I already have most of the code in place, a pull request should follow soon. The repeatable build data is only generated after all the expansions have already taken place, both concerning the environment variables as well as the conversion of branches/tags to fixed commits. Therefore the place where the properties are coming from does not affect repeatability.
Conversely, the vars expansion takes place at the very beginning in the code: we need to have everything expanded early. By the time we get to looking into the git repositories, the configuration file has already been processed and expanded, so it is too late. The properties files need to be accessible when dbuild starts.
However, to ease the pain, I can offer access to properties files via an arbitrary URL (which can also be built using environment variables). So you only need to set a tag/branch env var from bash, and you can then use that to denote within the config file the URL on github where the properties file is located, as well as the URL of the corresponding git repo. Would that work?

[cunei]

Pull request #73 contains the relevant patch. Please let me know if any additional changes are needed.

[adriaanm]

Perfect! The properties file is already available, so no need for getting it from git. Your explanation re: repeatable builds makes sense. I'll try it out tomorrow! BTW, are there nightly builds of dbuild? I don't mind building my own -- just wondering :-)

[cunei]

I'll publish a test version later today; stay tuned. :)