Giter Site home page Giter Site logo

python-github-webhooks's Introduction

Python GitHub Webhooks

Simple Python WSGI application to handle GitHub webhooks.

Install

git clone https://github.com/carlos-jenkins/python-github-webhooks.git
cd python-github-webhooks

Dependencies

sudo pip install -r requirements.txt

Setup

You can configure what the application does by copying the sample config file config.json.sample to config.json and adapting it to your needs:

{
    "github_ips_only": true,
    "enforce_secret": "",
    "return_scripts_info": true,
    "hooks_path": "/.../hooks/"
}
github_ips_only:Restrict application to be called only by GitHub IPs. IPs whitelist is obtained from GitHub Meta (endpoint). Default: true.
enforce_secret:Enforce body signature with HTTP header X-Hub-Signature. See secret at GitHub WebHooks Documentation. Default: '' (do not enforce).
return_scripts_info:Return a JSON with the stdout, stderr and exit code for each executed hook using the hook name as key. If this option is set you will be able to see the result of your hooks from within your GitHub hooks configuration page (see "Recent Deliveries"). Default: true.
hooks_path:Configures a path to import the hooks. If not set, it'll import the hooks from the default location (/.../python-github-webhooks/hooks)

Adding Hooks

This application will execute scripts in the hooks directory using the following order:

hooks/{event}-{name}-{branch}
hooks/{event}-{name}
hooks/{event}
hooks/all

The application will pass to the hooks the path to a JSON file holding the payload for the request as first argument. The event type will be passed as second argument. For example:

hooks/push-myrepo-master /tmp/sXFHji push

Hooks can be written in any scripting language as long as the file is executable and has a shebang. A simple example in Python could be:

#!/usr/bin/env python
# Python Example for Python GitHub Webhooks
# File: push-myrepo-master

import sys
import json

with open(sys.argv[1], 'r') as jsf:
  payload = json.loads(jsf.read())

### Do something with the payload
name = payload['repository']['name']
outfile = '/tmp/hook-{}.log'.format(name)

with open(outfile, 'w') as f:
    f.write(json.dumps(payload))

Not all events have an associated branch, so a branch-specific hook cannot fire for such events. For events that contain a pull_request object, the base branch (target for the pull request) is used, not the head branch.

The payload structure depends on the event type. Please review:

https://developer.github.com/v3/activity/events/types/

Deploy

Apache

To deploy in Apache, just add a WSGIScriptAlias directive to your VirtualHost file:

<VirtualHost *:80>
    ServerAdmin [email protected]
    ServerName  my.site.com
    DocumentRoot /var/www/site.com/my/htdocs/

    # Handle Github webhook
    <Directory "/var/www/site.com/my/python-github-webhooks">
        Order deny,allow
        Allow from all
    </Directory>
    WSGIScriptAlias /webhooks /var/www/site.com/my/python-github-webhooks/webhooks.py

</VirtualHost>

You can now register the hook in your Github repository settings:

https://github.com/youruser/myrepo/settings/hooks

To register the webhook select Content type: application/json and set the URL to the URL of your WSGI script:

http://my.site.com/webhooks

Docker

To deploy in a Docker container you have to expose the port 5000, for example with the following command:

git clone http://github.com/carlos-jenkins/python-github-webhooks.git
docker build -t carlos-jenkins/python-github-webhooks python-github-webhooks
docker run -d --name webhooks -p 5000:5000 carlos-jenkins/python-github-webhooks

You can also mount volume to setup the hooks/ directory, and the file config.json:

docker run -d --name webhooks \
  -v /path/to/my/hooks:/src/hooks \
  -v /path/to/my/config.json:/src/config.json \
  -p 5000:5000 python-github-webhooks

Test your deployment

To test your hook you may use the GitHub REST API with curl:

https://developer.github.com/v3/
curl --user "<youruser>" https://api.github.com/repos/<youruser>/<myrepo>/hooks

Take note of the test_url.

curl --user "<youruser>" -i -X POST <test_url>

You should be able to see any log error in your webapp.

Debug

When running in Apache, the stderr of the hooks that return non-zero will be logged in Apache's error logs. For example:

sudo tail -f /var/log/apache2/error.log

Will log errors in your scripts if printed to stderr.

You can also launch the Flask web server in debug mode at port 5000.

python webhooks.py

This can help debug problem with the WSGI application itself.

License

Copyright (C) 2014-2015 Carlos Jenkins <[email protected]>

Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing,
software distributed under the License is distributed on an
"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
KIND, either express or implied.  See the License for the
specific language governing permissions and limitations
under the License.

Credits

This project is just the reinterpretation and merge of two approaches:

Thanks.

python-github-webhooks's People

Contributors

carlos-jenkins avatar codebam avatar framawiki avatar fvanderbiest avatar kpriceyahoo avatar lrineau avatar mathias-baumann-sociomantic avatar mfin avatar monperrus avatar nmaupu avatar orkohunter avatar sidheshenator avatar tbarn avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

python-github-webhooks's Issues

not work with secret

File "/app/webhooks.py", line 82, in index
    mac = hmac.new(str(secret), msg=request.data, digestmod='sha1')
  File "/usr/local/lib/python2.7/hmac.py", line 136, in new
    return HMAC(key, msg, digestmod)
  File "/usr/local/lib/python2.7/hmac.py", line 52, in __init__
    self.outer = self.digest_cons()
  File "/usr/local/lib/python2.7/hmac.py", line 50, in <lambda>
    self.digest_cons = lambda d='': digestmod.new(d)
AttributeError: 'str' object has no attribute 'new'

More logs is needed

At this time, when there is an issue with GitHub integration (like test returning 301 code), there is absolutely no logs or whatever.

We need both access and error logs to get a chance to understand where the issue can be.

500 Server Error

Hi,

I have my webhook server up and running (apache calling python-github-webhooks via WSGI), however, I see the following error in my logs whenever GitHub tries to POST something to this endpoint:

[Wed Jan 25 12:31:26.499245 2017] [:info] [pid 62282] [client 10.240.0.187:40078] mod_wsgi (pid=62282, process='', application='blahblah.c9users.io|/webhooks'): Loading WSGI script '/home/ubuntu/python-github-webhooks/webhooks.py'.
[Wed Jan 25 12:31:26.661105 2017] [:error] [pid 62282] [client 10.240.0.187:40078] [2017-01-25 12:31:26,659] ERROR in app: Exception on / [POST]
[Wed Jan 25 12:31:26.661239 2017] [:error] [pid 62282] [client 10.240.0.187:40078] Traceback (most recent call last):
[Wed Jan 25 12:31:26.661279 2017] [:error] [pid 62282] [client 10.240.0.187:40078]   File "/usr/local/lib/python2.7/dist-packages/flask/app.py", line 1982, in wsgi_app
[Wed Jan 25 12:31:26.661286 2017] [:error] [pid 62282] [client 10.240.0.187:40078]     response = self.full_dispatch_request()
[Wed Jan 25 12:31:26.661291 2017] [:error] [pid 62282] [client 10.240.0.187:40078]   File "/usr/local/lib/python2.7/dist-packages/flask/app.py", line 1614, in full_dispatch_request
[Wed Jan 25 12:31:26.661295 2017] [:error] [pid 62282] [client 10.240.0.187:40078]     rv = self.handle_user_exception(e)
[Wed Jan 25 12:31:26.661300 2017] [:error] [pid 62282] [client 10.240.0.187:40078]   File "/usr/local/lib/python2.7/dist-packages/flask/app.py", line 1517, in handle_user_exception
[Wed Jan 25 12:31:26.661305 2017] [:error] [pid 62282] [client 10.240.0.187:40078]     reraise(exc_type, exc_value, tb)
[Wed Jan 25 12:31:26.661309 2017] [:error] [pid 62282] [client 10.240.0.187:40078]   File "/usr/local/lib/python2.7/dist-packages/flask/app.py", line 1612, in full_dispatch_request
[Wed Jan 25 12:31:26.661314 2017] [:error] [pid 62282] [client 10.240.0.187:40078]     rv = self.dispatch_request()
[Wed Jan 25 12:31:26.661318 2017] [:error] [pid 62282] [client 10.240.0.187:40078]   File "/usr/local/lib/python2.7/dist-packages/flask/app.py", line 1598, in dispatch_request
[Wed Jan 25 12:31:26.661323 2017] [:error] [pid 62282] [client 10.240.0.187:40078]     return self.view_functions[rule.endpoint](**req.view_args)
[Wed Jan 25 12:31:26.661326 2017] [:error] [pid 62282] [client 10.240.0.187:40078]   File "/home/ubuntu/python-github-webhooks/webhooks.py", line 52, in index
[Wed Jan 25 12:31:26.661329 2017] [:error] [pid 62282] [client 10.240.0.187:40078]     config = loads(cfg.read())
[Wed Jan 25 12:31:26.661332 2017] [:error] [pid 62282] [client 10.240.0.187:40078]   File "/usr/lib/python2.7/json/__init__.py", line 338, in loads
[Wed Jan 25 12:31:26.661335 2017] [:error] [pid 62282] [client 10.240.0.187:40078]     return _default_decoder.decode(s)
[Wed Jan 25 12:31:26.661338 2017] [:error] [pid 62282] [client 10.240.0.187:40078]   File "/usr/lib/python2.7/json/decoder.py", line 366, in decode
[Wed Jan 25 12:31:26.661341 2017] [:error] [pid 62282] [client 10.240.0.187:40078]     obj, end = self.raw_decode(s, idx=_w(s, 0).end())
[Wed Jan 25 12:31:26.661344 2017] [:error] [pid 62282] [client 10.240.0.187:40078]   File "/usr/lib/python2.7/json/decoder.py", line 382, in raw_decode
[Wed Jan 25 12:31:26.661346 2017] [:error] [pid 62282] [client 10.240.0.187:40078]     obj, end = self.scan_once(s, idx)
[Wed Jan 25 12:31:26.661355 2017] [:error] [pid 62282] [client 10.240.0.187:40078] ValueError: Expecting , delimiter: line 5 column 5 (char 129)

I tried configuring the webhooks on the GitHub side, as having the Content Type header application/json and also application/x-www-form-urlencoded but for both, I'm getting the same error, what could be wrong?

Many thanks in advance,
VG

Major issue: not closing temporary file

I finally noticed that you missed close() after an uptime of one week:

ERROR:uwsgi_file__***:Exception on / [POST]  
Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/flask/app.py", line 1817, in wsgi_app  
    response = self.full_dispatch_request()
  File "/usr/lib/python2.7/dist-packages/flask/app.py", line 1477, in full_dispatch_request  
    rv = self.handle_user_exception(e)
  File "/usr/lib/python2.7/dist-packages/flask/app.py", line 1381, in handle_user_exception  
    reraise(exc_type, exc_value, tb)
  File "/usr/lib/python2.7/dist-packages/flask/app.py", line 1475, in full_dispatch_request  
    rv = self.dispatch_request()
  File "/usr/lib/python2.7/dist-packages/flask/app.py", line 1461, in dispatch_request  
    return self.view_functions[rule.endpoint](**req.view_args)
  File "/var/www/cgi-bin/***/webhooks/github.py", line 52, in index  
    with open(join(path, 'github/config.json'), 'r') as cfg:

Ouch.

This was determined by this:

# sh -c 'readlink /proc/30756/fd/*'
/tmp/tmpjCxknv (deleted)
/tmp/tmp5p0JBV (deleted)
/tmp/tmp7W680K (deleted)
/tmp/tmpLp_FMI (deleted)

Test the web app before deploying?

Is there a recommended way to test the web app defined in the script webhooks.py locally? Sorry if this is obvious, but I'm new to webhooks and web apps in general.

Docker build

I need to extend your image and so i want to base my Dockerfile on yours. I get this error:

Sending build context to Docker daemon   2.56kB
Step 1/5 : FROM carlos-jenkins/python-github-webhooks
pull access denied for carlos-jenkins/python-github-webhooks, repository does not exist or may require 'docker login'

Can you please make a docker hub integration that will pull the master releases and build images? It's very easy, just follow these instructions: https://docs.docker.com/docker-hub/builds/

hmac.compare_digest does not exist in Python < 2.7.7

One issue I ran into with this webhooks script is that hmac.compare_digest is new in 2.7.7, and for legacy reasons I'm stuck on 2.6.

I have a few different ways to solve this problem in a general way, and I wanted to get your feedback before I submit a pull request.

Option 1: fall back to a simple string comparison if hmac.compare_digest is unavailable. This is functionally equivalent, but simply fails to provide protection against a timing attack. (It's not clear to me that a timing attack is even a concern, given the inherit network latency involved, so this may be perfectly safe.)

Option 2: have an option, disabled by default, to allow the fallback. This is what I have implemented locally. Code snippet below.

Option 3: have a fallback to a python-based version of compare_digest. This would be slower than the C implementation in the standard library, but would provide the same protection. It would also require adding a python-fallback to the repo, as I don't think there's one available in a pip package anywhere.

The code for option 2 would look something like:

        # HMAC requires the key to be bytes, but data is string
        mac = hmac.new(str(secret), msg=request.data, digestmod=sha1)

        # Python prior to 2.7.7 does not have hmac.compare_digest
        try:
            if not hmac.compare_digest(str(mac.hexdigest()), str(signature)):
                abort(403)
        except AttributeError as e:
            # What compare_digest provides is protection against timing attacks, but if
            # the user is okay with not having this protection, let's let them allow a
            # trivial fallback
            if not config.get('allow_insecure_compare_digest', False):
                logging.error('Could not call hmac.compare_digest; are you running python < 2.7.7?', exc_info=True)
                abort(500)

            if not str(mac.hexdigest()) == str(signature):
                abort(403)

Let me know your thoughts, and I can prepare a pull request.

hooks on docker

In my hooks, I've had to use git (among other tools) as a specific user (whose uid is 999).

Since git was not provided by the carlos-jenkins/python-github-webhooks docker image (and I did not want to hack into it) I derived a customized image based on yours, with this Dockerfile.extended:

FROM carlos-jenkins/python-github-webhooks

MAINTAINER "François Van Der Biest" <[email protected]>

# add packages required to run your hooks, eg:
RUN apk update && apk add bash git openssh-client

# create user which will run hooks (group ping has gid=999 in base image)
RUN adduser -S -G ping -s /bin/bash -u 999 sftp

# required here to populate root's known_hosts so that git pull command 
# does not interactively ask to check RSA key fingerprint:
RUN mkdir -p /root/.ssh && \
    chmod 700 /root/.ssh && \
    ssh-keyscan github.com >> /root/.ssh/known_hosts

then: docker build -t fvanderbiest/python-github-webhooks -f Dockerfile.extended .

Finally, I set the setuid bit on my hook, and gave it to user with uid 999:
chmod u+s push-myrepo-mybranch

It works great !
This is not really an issue, but I thought it might be useful to others...

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.