support for nginx

[certbot]

support for nginx

[jdkasten]

Hi Martin!

Welcome to the project. I have seen quite a few requests for nginx support so yesterday I made a ton of refactoring changes I had been planning on a long time in order to make this happen.

I have been discussing the changes in #24. Last night I submitted a stub nginx_configurator.py to the configurator_refactor branch. This is where you would like to edit, and I would absolutely be willing to help you get setup!

Let me finish testing out the branch though... and I will let you know when I pull the changes into master.

[golodhrim]

hi james!

great to hear... :) thx for the quick reply, sure, I would like to see that packages make its way into http://funtoo.org and support us that way even more with a testing in a small system but mostly full of hackers, also one of my instructors at the Seminar I'm appending this week and does a lot in security seemed totally interessted...

Greetings

[jdkasten]

Great! I just committed the patch, and it should enable nginx to be worked on alongside Apache.
I created a starter nginx_configurator.py class that includes method stubs you will need and some commented out code that may be useful.

I made the stub nginx class, a child of the AugeasConfigurator class as that is how I would proceed.
Augeas is an opensource configuration editing framework that supports many different configuration files. (It is also what I used to enable the Apache configuration changes... and you will get to use the existing save system - which I am about to write a wiki article about)

I will spend the rest of the day pulling in user fixes, answering developer questions, and writing detailed documentation on the current state of the code and how to get started. Let me know if you have any questions!

Best,
James

[jdkasten]

I have added additional information in the Wiki to aid in development. Let me know if you have questions.

[jdkasten]

This is a fairly large project and requires quite a bit of knowledge about nginx configurations. That being said, the project can be done in a piece wise fashion. We can create a branch for development purposes... which would allow you to commit code that others might find useful as you experiment with editing the configuration.

In order to avoid this topic from sinking into oblivion, I will add the "Good Volunteer Work" tag, as it may be awhile before the rest of the codebase is in good enough shape for me to look at it.

[craigloftus]

@jdkasten I had a quick look at the work required. There is still some refactoring work required to allow switching between configurators, in client/client.py and scripts/main.py.

# TODO: Can probably figure out which configurator to use
#       without special packaging based on system info Command
#       line arg or client function to discover

I wasn't entirely sure how to 'parse' that comment. A command line arg would be fine as a starting point, with a preference for something more magical later on?

[jdkasten]

@craigloftus Yes, some refactoring work will have to go into fully supporting the nginx_configurator.
That comment was just a placeholder acknowledging that choosing the correct configurator will have to eventually be performed. I wasn't fully satisfied with any interface I had come up with for selecting the proper configurator. If you have a good idea on how to do this.... I would be happy to add it with a pull request.

For now, I was thinking a separate branch could be created for the necessary changes/additions to support nginx. This branch would eventually be merged into master once the functionality is complete.

Thoughts?

[craigloftus]

An nginx branch sounds good.

For anyone else following the issue, as a starting point, I have put up a gist of an nginx SSL config.

[craigloftus]

The Augeas Nginx lens is very poor; it will fail to parse any Nginx server config files (hercules-team/augeas#179). I have tried to fix the lens, and got recursion working (craigloftus/augeas@04cb46d), but I was not able to fix the parsing of the location blocks.

The Augeas developers are reasonably responsive, so I think it is worth waiting to see if we can get the lens fixed, as there is currently a lot of work being done by the AugeasConfigurator.

[aychedee]

I would love to volunteer for this issue. I have a good knowledge of Nginx SSL config so once the branch is created I can start contributing.

[kuba]

Note that this is github and anyone can create branch in their local repo - there is no need for this project to create such in advance. Technically speaking, branch in git is in fact a pointer at a commit and since there is no commit yet... Once you have some commits, just create a pull request :).

[schoen]

Hi @aychedee, I agree with @kuba that in git in general anyone can create a branch, no need for the project to do so, so you can make your own branch in your own local copy if you like, or in your GitHub account, or wherever you like!

The current master tree does contain letsencrypt/client/nginx_configurator.py, which is the code to attempt to configure Nginx (it's just mostly not implemented yet). As folks earlier in this thread were discussing, the client uses a library called Augeas to interact with Apache configurations (both for reading and writing them, and both for satisfying challenges from the CA and for deploying the cert once it's issued). We're hoping to use Augeas for the Nginx configuration too, because it's a general-purpose, not Apache-specific, configuration file reading and writing tool. But it sounds like it still needs some work to get good enough Nginx integration, because Augeas needs to be taught about the format of each kind of configuration file it will read or write. Maybe you could help experiment with that aspect.

[schoen]

The Augeas parsers for config files are called "lenses" and are basically grammars for the config files in question.

http://augeas.net/docs/lenses.html
http://augeas.net/docs/references/lenses/files/nginx-aug.html

According to @craigloftus earlier, it's going to take some work to make the Augeas lens for Nginx more complete.

[aychedee]

Okay, I'll have a look at it over the holidays. I do understand git's branching abilities ;) I was just not sure if the mostly not implemented code was actually in the repo at all. Thanks for all the extra context. Very much appreciated.

[ThomasWaldmann]

was there any progress on this?

[aychedee]

I spent some time in December looking through the Augeas lenses but ended up but gave up before I managed to get a working ML environment.

[ThomasWaldmann]

see #200 - that might become a solution until there is specific support code for nginx.

[diracdeltas]

Heya, apparently I'm picking up this ticket. What I've learned so far is:

• The augeas lens for nginx is wildly insufficient right now. It only checks /etc/nginx/nginx.conf and /usr/portage/www-servers/nginx/files/nginx.conf, and it will fail to parse files that contain blocks nested with depth > 1.
• Writing your own lens seems somewhat complicated [1], and I haven't figured out how to tell the Python augeas wrapper to use a custom lens.
• There is a nice, 100-line MIT-licensed python module to parse/write nginx configs [2] that has some bugs but is way better than the Augeas lens. IMO the best approach is to fork this and either include it in letsencrypt or a separate repo.

[1] https://github.com/hercules-team/augeas/wiki/Creating-a-lens-step-by-step
[2] https://github.com/fatiherikli/nginxparser

[pde]

A 100 line script sounds too small to be worth suffering the pain of git submodule. So perhaps whould just put it in our tree, and then politely send any changes we make back upstream...

[jdkasten]

Great! You can still probably use the Reverter module in order to save/revert config changes, if it helps you out.

https://letsencrypt.readthedocs.org/en/latest/api/client/reverter.html
https://github.com/letsencrypt/lets-encrypt-preview/blob/master/letsencrypt/client/reverter.py

Before you make changes to a configuration file... you can add the files to a checkpoint.
add_to_checkpoint(save_files, save_notes)
or
add_to_temp_checkpoint(save_files, save_notes) <- This allows you to save challenge configuration changes separately and provides some guards to make sure permanent and temporary config changes don't collide with each other.

Before you create any new files... call
register_file_creation(temporary, *files)

When you are done with a significant amount of work... (issued a cert, performed an enhancement), call
finalize_checkpoint(title)

To revert temporary changes (cleanup from a challenge)
revert_temporary_config()

In case of a program crash, Control-C etc... you can restore the last finalized_checkpoint with...
recovery_routine()

Note: This class also provides many of the installer methods...
view_checkpoints()
rollback_checkpoints(N)

Example of Basic Usage...
https://github.com/letsencrypt/lets-encrypt-preview/blob/master/letsencrypt/client/augeas_configurator.py#L99

[lutter]

I've just updated the Augeas lens for Nginx (hercules-team/augeas#225) and would love to hear feedback on whether that is sufficient for lets-encrypt's purposes and/or how it fails.

To try out the lens from the PR easily, download https://raw.githubusercontent.com/lutter/augeas/lns/nginx/lenses/nginx.aug and store it in /usr/share/augeas/lenses and run augtool. If you just want to make sure that your particular nginx.conf works, you can do something like

augtool -A -t 'Nginx incl /path/to/nginx.conf' 'print /augeas//error'

If that prints nothing, all is well.

[diracdeltas]

@lutter thanks, i'll take a quick look.

I have basically rolled our own nginx parser at this point, but Augeas is nice because of the path traversal methods and the save functionality. WIP branch at https://github.com/diracdeltas/lets-encrypt-preview/tree/feature/nginx

[diracdeltas]

This is basically functional now that #351 and #387 are merged! 💃

[kuba]

Can we close it for now, @golodhrim?