Question: How can we dynamically generate different templates from a consul K/V JSON object ?

[criteo]

Question: How can we dynamically generate different templates from a consul K/V JSON object ?

[pierresouchay]

I don't think it is easily doable as there is no API to dynamically create templates in consul-templaterb. So, taking the exact same approach would not work as it is.

Personally, I don't like much the approach of generating multiple files from the template because cleanup is then very hard:

when 1 of your vhost disappear, what should happen: does the files needs to be removed? What would happen to another file being in this repo. We might think it is not a problem : the program keep track of the files being generated, but what if the program generating is being killed, and during that time one of the entries gets removed? -> will the my_old_vhost.conf needs to be cleaned up? -> this is actually a hard problem to solve.

So, my advice would be to keep a number of predictable files. If splitting is an requirement, it would even be possible to shard vhosts such as a_to_m_vhosts.conf (a rendered file with domains starting with a up to m domain names) and m_to_z_vhosts.conf (domains from n to z...).

Still, if you want to keep the existing rendering with 1 file per domain, there are still several possibilities (as many as you wish since erb is a full programming language).

The easiest IMHO would likely be to generate multiple files with a single template:

consul-templaterb is actually a full featured ruby interpreter, so it is easily possible to write files in ruby. This is actually a bit tricky, because it would require you to take care about the state to void constantly writing files, furthermore, it implies that you handle cleanup: when a value is removed, you then will have to do the cleanup. consul-templaterb can render itself a configurable template (see https://github.com/criteo/consul-templaterb/blob/master/samples/render_template_from_kv.erb for instance), so: Render the template in a string, if is different from content of file on disk -> write to disk the content.
Once again cleanup is complicated (but it would work). You can also keep a state in your template (see https://github.com/criteo/consul-templaterb/blob/master/samples/consul-ui/timeline.json.erb#L10 on how doing it)

[kamaradclimber]

To elaborate on @pierresouchay answer, a common pattern amongst consul-templaterb is to use the feature that allows to execute a binary when template is edited.

What a regular person would do: --template host.conf.erb:host.conf:nginx -c host.conf to generate the host.conf file used by a nginx process.

What can be done instead: --template hostconf_generator.erb:hostconf_generator.rb:ruby hostconf_generator.rb. The template here would generate a ruby file (it can be anything) that will be executed.

For instance, you might want to write a template that will produce the following output:

File.write("/tmp/file1", <<~EOH)
server {
  listen 80;
  server_name fqdn1.yourorg.com
  location / { ...}
}
EOH

File.write("/tmp/file2", <<~EOH)
server {
  listen 80;
  server_name fqdn2.yourorg.com
  location / { ...}
}
EOH

Here consul-templaterb generates a generator instead of directly the config file.

[abarre]

Thank you for all this details. I'm closing the issue. I will reopen it if I need more details.