Giter Site home page Giter Site logo

django-package-monitor's Introduction

https://travis-ci.org/yunojuno/django-package-monitor.svg?branch=master

Django Package Monitor

This package is now Python 3.6 and Django 2.1 and above. For previous versions please refer to the Python2 branch.

A Django app for keeping track of dependency updates.

Background

At YunoJuno we have a Django project that includes almost 100 external packages. In order to manage updates to these we have a rolling development task that comes around in the first week of each month, and includes the following:

  1. Using pip list --outdated list out all available updates
  2. Group updates (using semver) into Major, Minor, Patch, Other
  3. Apply patch updates in a single update / commit
  4. Apply minor updates as a group, see what breaks, remove, rinse, repeat
  5. Take a view on major updates

This task is a PITA, and so we decided to make it simpler.

Implementation

This project contains a Django app that can be used to monitor your packages.

It consists of a single model, PackageVersion, an admin list view that you can use to view current package versions, and load latest versions from PyPI, and a single management command that can be used to load local requirements and update remote versions from the shell - which you could run overnight if you felt the need.

It is important to note that this app does not update your requirements for you - it simply displays the requirements that you have, and the latest that is available on PyPI.

In order to illustrate how it works, the app itself contains a Django project that can be used to demonstrate the feature.

Installation

Download / install the app using pip:

pip install django-package-monitor

Add the app package_monitor to your INSTALLED_APPS Django setting:

# settings.py
INSTALLED_APPS = (
    'django.contrib.admin',
    'django.contrib.auth',
    'django.contrib.contenttypes',
    'django.contrib.sessions',
    'django.contrib.messages',
    'django.contrib.staticfiles',
    'package_monitor',
    ...
)

Set the PACKAGE_MONITOR_REQUIREMENTS_FILE setting to point to your project requirements file:

# settings.py
PACKAGE_MONITOR_REQUIREMENTS_FILE = path.join(PROJECT_ROOT, 'requirements.txt')

Add the app URLs to your project - NB it must have the namespace set:

# urls.py
urlpatterns = patterns(
    '',
    url(r'^admin/', include(admin.site.urls)),
    url(r'^package_monitor/', include('package_monitor.urls', namespace='package_monitor')),
)

At this point you should have a working implementation. You can test this by running the management command to load your local requirements:

# load up the local requirements file
$ python manage.py refresh_packages --local

This will load all of the requirements it finds in the requirements file specified into the database. If you then want to check PyPI for updated version, run the command with the --remote option. You can run both of these command together:

# load up the local requirements file, and check PyPI
$ python manage.py refresh_packages --local --remote

If you want to clean out the existing PackageVersion table before loading the local file, use the --clean option:

# clear out database, load up the local requirements file, and check PyPI
$ python manage.py refresh_packages --clean --local --remote

Tests

There is a test suite that can be run using tox:

$ pip install -r requirements
$ tox

In addition to the unit tests, the source distribution also includes a fully-functioning Django project, that can be run from the repo root, and used to demonstrate how it works:

$ git clone [email protected]:yunojuno/django-package-monitor.git
$ cd django-package-monitor
$ pip install -r requirements.txt
# you will need to create a superuser in order to access the admin site
$ python manage.py createsuperuser
$ python manage.py runserver

If you then log in to the app (http://localhost:8000/admin by default), you can then see the admin list page:

Screenshot of admin list view (empty)

If you click on the "Reload local requirements" button in the top-right, it will load up the contents of the requirements file that you used earlier:

Screenshot of admin list view populated with local requirements

NB If any requirements cannot be parsed by the semantic_version.Version.coerce method, then the is_parseable property is set to False, and the package is in effect unmanaged.

At this point it has parsed the requirements file, and stored the current working version of each package (as current_version). In order to see what the latest versions are, select all the packages, and choose "Update selected packages from PyPI" form the actions list:

Screenshot of admin list view with all requirements selected

This may take some time, as it will call the PyPI API for each package (excluding those that are marked as editable), and download the latest version info for each. At the end of this, you should see the page updated with the new version information (as latest_version) - as well as the licence information that is stored in the PyPI metadata:

Screenshot of admin list view with requirement info updated from PyPI

If you drill down to the detail on an individual package, you can see all of the available versions:

Screenshot of Django package details

Contributing

This is by no means complete - it can't cope with requirements that are anything other than '==', and it doesn't (yet) help with updating the requirements file itself. However, it's good enough to be of value, hence releasing it. If you would like to contribute to the project, usual Github rules apply:

  1. Fork the repo to your own account
  2. Submit a pull request
  3. Add tests for any new code
  4. Follow coding style of existing project

Licence

This project is MIT licensed - see the LICENCE file for details.

django-package-monitor's People

Contributors

ebertti avatar eriktelepovsky avatar hugorodgerbrown avatar miphreal avatar srtab avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

django-package-monitor's Issues

Update failure if PyPI json failure

If an installed package doesnt exist on PyPI, the refresh process fails.

I have a patch for this, and possibly a few other PyPI errors I encountered.

Strip comments from requirement lines

Pip supports inline comments in the requirements file:

social-auth-core==1.3.0   # via social-auth-app-django
sqlparse==0.2.3           # via django-request-token

As we parse the file into individual requirements, we should strip out any such comments. We can't split purely on # as this is used for appending package names to pinned commits, e.g.:

-e git+git://github.com/jsocol/django-waffle.git@a2c41676e4f9be54dab2522b10dec09189245e74#egg=django_waffle

Proposal is to split on ' #'.

KeyError: 'HTTP_REFERER'

/package_monitor/views.py in reload:

Traceback (most recent call last):
  File "/Users/erik/env/cargo/lib/python2.7/site-packages/django/core/handlers/base.py", line 132, in get_response
    response = wrapped_callback(request, *callback_args, **callback_kwargs)
  File "/Users/erik/env/cargo/lib/python2.7/site-packages/django/contrib/auth/decorators.py", line 22, in _wrapped_view
    return view_func(request, *args, **kwargs)
  File "/Users/erik/env/cargo/lib/python2.7/site-packages/package_monitor/views.py", line 14, in reload
    url = request.META['HTTP_REFERER']
KeyError: 'HTTP_REFERER'

Handle semantic_version.parse errors

If one package in the list of requirements cannot be parsed (see #10), then the local requirements cannot be loaded.

Steps to reproduce:

  1. Add Unideocode==0.04.7 to requirements.txt
  2. Open the admin site / package_updates
  3. Navigate to Package_Monitor > Package versions
  4. Click on "Reload local requirements"

-or-

  1. Add Unideocode==0.04.7 to requirements.txt
  2. Run ./manage.py refresh_packages --local

Expected result:

All local packages are loaded into the database.

Actual result:

An exception is raised by the semantic_version parse function:

ValueError: Invalid leading zero in minor: '0.04.17'

Suggestion: configurable warnings about unwanted licences

(Low priority)

In the spirit of https://github.com/onur/cargo-license/, it would be interesting to be able to mark out certain licences as 'unwanted' in an application stack - eg GPL

The barest implementation could be just to make the admin changelist sortable by licence type.

Next level up might be to support a list of licence names (strings, I guess) which are used to highlight relevant packages in the admin and/or allow filtering by 'unwanted' licences in the changelist sidebar.

A management command could also be handy.

IndexError in models.py

I added django-package-monitor to my requirements.txt and it fails on python manage.py refresh_packages --clean --remote --local command now:

File "/Users/erik/env/cargo/lib/python2.7/site-packages/package_monitor/management/commands/refresh_packages.py", line 81, in handle
    local()
  File "/Users/erik/env/cargo/lib/python2.7/site-packages/package_monitor/management/commands/refresh_packages.py", line 34, in local
    create_package_version(r)
  File "/Users/erik/env/cargo/lib/python2.7/site-packages/package_monitor/management/commands/refresh_packages.py", line 21, in create_package_version
    PackageVersion(requirement=requirement).save()
  File "/Users/erik/env/cargo/lib/python2.7/site-packages/package_monitor/models.py", line 95, in __init__
    self.current_version = Version.coerce(requirement.specs[0][1])
IndexError: list index out of range

Without django-package-monitor in the requirements.txt works fine.

Tested on version 0.2.3

Apps aren't loaded yet

I try to run Django project with your app, but every time I run server with package_monitor in INSTALLED_APPS this exception raises:

  File "manage.py", line 10, in <module>
    execute_from_command_line(sys.argv)
  File "/Users/erik/env/cargo/lib/python2.7/site-packages/django/core/management/__init__.py", line 351, in execute_from_command_line
    utility.execute()
  File "/Users/erik/env/cargo/lib/python2.7/site-packages/django/core/management/__init__.py", line 343, in execute
    self.fetch_command(subcommand).run_from_argv(self.argv)
  File "/Users/erik/env/cargo/lib/python2.7/site-packages/django/core/management/__init__.py", line 177, in fetch_command
    commands = get_commands()
  File "/Users/erik/env/cargo/lib/python2.7/site-packages/django/utils/lru_cache.py", line 101, in wrapper
    result = user_function(*args, **kwds)
  File "/Users/erik/env/cargo/lib/python2.7/site-packages/django/core/management/__init__.py", line 72, in get_commands
    for app_config in reversed(list(apps.get_app_configs())):
  File "/Users/erik/env/cargo/lib/python2.7/site-packages/django/apps/registry.py", line 137, in get_app_configs
    self.check_apps_ready()
  File "/Users/erik/env/cargo/lib/python2.7/site-packages/django/apps/registry.py", line 124, in check_apps_ready
    raise AppRegistryNotReady("Apps aren't loaded yet.")
django.core.exceptions.AppRegistryNotReady: Apps aren't loaded yet.

Tested on Django==1.8.5.

TypeError in models.py

Result of python manage.py refresh_packages --clean --local --remote:

Traceback (most recent call last):
  File "manage.py", line 10, in <module>
    execute_from_command_line(sys.argv)
  File "/Users/erik/env/cargo/lib/python2.7/site-packages/django/core/management/__init__.py", line 351, in execute_from_command_line
    utility.execute()
  File "/Users/erik/env/cargo/lib/python2.7/site-packages/django/core/management/__init__.py", line 343, in execute
    self.fetch_command(subcommand).run_from_argv(self.argv)
  File "/Users/erik/env/cargo/lib/python2.7/site-packages/django/core/management/base.py", line 394, in run_from_argv
    self.execute(*args, **cmd_options)
  File "/Users/erik/env/cargo/lib/python2.7/site-packages/django/core/management/base.py", line 445, in execute
    output = self.handle(*args, **options)
  File "/Users/erik/env/cargo/lib/python2.7/site-packages/package_monitor/management/commands/refresh_packages.py", line 83, in handle
    remote()
  File "/Users/erik/env/cargo/lib/python2.7/site-packages/package_monitor/management/commands/refresh_packages.py", line 40, in remote
    pv.update_from_pypi()
  File "/Users/erik/env/cargo/lib/python2.7/site-packages/package_monitor/models.py", line 117, in update_from_pypi
    self.licence = package_licence(info)[:100]
TypeError: 'NoneType' object has no attribute '__getitem__'

DB field length overflow problem

We just found a DataError: value too long for type character varying(200) when trying to view the package-monitor results in the Admin. We don't have the context/stack available but it might be related to a pinned commit (git -e ...) as a dependency

017-06-02T10:25:28.949518+00:00 app[web.1]: ERROR Internal Server Error: /package_monitor/reload/
2017-06-02T10:25:28.949531+00:00 app[web.1]: Traceback (most recent call last):
2017-06-02T10:25:28.949533+00:00 app[web.1]:   File "/app/.heroku/python/lib/python2.7/site-packages/django/core/handlers/exception.py", line 42, in inner
2017-06-02T10:25:28.949533+00:00 app[web.1]:     response = get_response(request)
2017-06-02T10:25:28.949534+00:00 app[web.1]:   File "/app/.heroku/python/lib/python2.7/site-packages/django/core/handlers/base.py", line 187, in _get_response
2017-06-02T10:25:28.949535+00:00 app[web.1]:     response = self.process_exception_by_middleware(e, request)
2017-06-02T10:25:28.949535+00:00 app[web.1]:   File "/app/.heroku/python/lib/python2.7/site-packages/django/core/handlers/base.py", line 185, in _get_response
2017-06-02T10:25:28.949536+00:00 app[web.1]:     response = wrapped_callback(request, *callback_args, **callback_kwargs)
2017-06-02T10:25:28.949537+00:00 app[web.1]:   File "/app/.heroku/python/lib/python2.7/site-packages/django/contrib/auth/decorators.py", line 23, in _wrapped_view
2017-06-02T10:25:28.949537+00:00 app[web.1]:     return view_func(request, *args, **kwargs)
2017-06-02T10:25:28.949538+00:00 app[web.1]:   File "/app/.heroku/python/lib/python2.7/site-packages/package_monitor/views.py", line 12, in reload
2017-06-02T10:25:28.949539+00:00 app[web.1]:     refresh_packages.local()
2017-06-02T10:25:28.949540+00:00 app[web.1]:   File "/app/.heroku/python/lib/python2.7/site-packages/package_monitor/management/commands/refresh_packages.py", line 36, in local
2017-06-02T10:25:28.949540+00:00 app[web.1]:     create_package_version(r)
2017-06-02T10:25:28.949542+00:00 app[web.1]:   File "/app/.heroku/python/lib/python2.7/site-packages/package_monitor/management/commands/refresh_packages.py", line 23, in create_package_version
2017-06-02T10:25:28.949542+00:00 app[web.1]:     PackageVersion(requirement=requirement).save()
2017-06-02T10:25:28.949543+00:00 app[web.1]:   File "/app/.heroku/python/lib/python2.7/site-packages/django/db/models/base.py", line 796, in save
2017-06-02T10:25:28.949543+00:00 app[web.1]:     force_update=force_update, update_fields=update_fields)
2017-06-02T10:25:28.949544+00:00 app[web.1]:   File "/app/.heroku/python/lib/python2.7/site-packages/django/db/models/base.py", line 824, in save_base
2017-06-02T10:25:28.949545+00:00 app[web.1]:     updated = self._save_table(raw, cls, force_insert, force_update, using, update_fields)
2017-06-02T10:25:28.949546+00:00 app[web.1]:   File "/app/.heroku/python/lib/python2.7/site-packages/django/db/models/base.py", line 908, in _save_table
2017-06-02T10:25:28.949546+00:00 app[web.1]:     result = self._do_insert(cls._base_manager, using, fields, update_pk, raw)
2017-06-02T10:25:28.949547+00:00 app[web.1]:   File "/app/.heroku/python/lib/python2.7/site-packages/django/db/models/base.py", line 947, in _do_insert
2017-06-02T10:25:28.949548+00:00 app[web.1]:     using=using, raw=raw)
2017-06-02T10:25:28.949548+00:00 app[web.1]:   File "/app/.heroku/python/lib/python2.7/site-packages/django/db/models/manager.py", line 85, in manager_method
2017-06-02T10:25:28.949552+00:00 app[web.1]:     return getattr(self.get_queryset(), name)(*args, **kwargs)
2017-06-02T10:25:28.949553+00:00 app[web.1]:   File "/app/.heroku/python/lib/python2.7/site-packages/django/db/models/query.py", line 1045, in _insert
2017-06-02T10:25:28.949553+00:00 app[web.1]:     return query.get_compiler(using=using).execute_sql(return_id)
2017-06-02T10:25:28.949554+00:00 app[web.1]:   File "/app/.heroku/python/lib/python2.7/site-packages/django/db/models/sql/compiler.py", line 1054, in execute_sql
2017-06-02T10:25:28.949554+00:00 app[web.1]:     cursor.execute(sql, params)
2017-06-02T10:25:28.949555+00:00 app[web.1]:   File "/app/.heroku/python/lib/python2.7/site-packages/django/db/backends/utils.py", line 79, in execute
2017-06-02T10:25:28.949556+00:00 app[web.1]:     return super(CursorDebugWrapper, self).execute(sql, params)
2017-06-02T10:25:28.949556+00:00 app[web.1]:   File "/app/.heroku/python/lib/python2.7/site-packages/django/db/backends/utils.py", line 64, in execute
2017-06-02T10:25:28.949557+00:00 app[web.1]:     return self.cursor.execute(sql, params)
2017-06-02T10:25:28.949557+00:00 app[web.1]:   File "/app/.heroku/python/lib/python2.7/site-packages/django/db/utils.py", line 94, in __exit__
2017-06-02T10:25:28.949558+00:00 app[web.1]:     six.reraise(dj_exc_type, dj_exc_value, traceback)
2017-06-02T10:25:28.949559+00:00 app[web.1]:   File "/app/.heroku/python/lib/python2.7/site-packages/django/db/backends/utils.py", line 64, in execute
2017-06-02T10:25:28.949559+00:00 app[web.1]:     return self.cursor.execute(sql, params)
2017-06-02T10:25:28.949560+00:00 app[web.1]:   File "/app/.heroku/python/lib/python2.7/site-packages/opbeat/instrumentation/packages/dbapi2.py", line 169, in execute
2017-06-02T10:25:28.949561+00:00 app[web.1]:     return self._trace_sql(self.__wrapped__.execute, sql, params)
2017-06-02T10:25:28.949561+00:00 app[web.1]:   File "/app/.heroku/python/lib/python2.7/site-packages/opbeat/instrumentation/packages/dbapi2.py", line 182, in _trace_sql
2017-06-02T10:25:28.949562+00:00 app[web.1]:     return method(sql, params)
2017-06-02T10:25:28.949562+00:00 app[web.1]: DataError: value too long for type character varying(200)

django-package-monitor

First of all, good evening.
I tried your app to get the feeling of the package itself and I have:

package_monitor_packageversion.licence may not be NULL

when trying to update all the packages.

Feel free to ask me for more details.

Kind regards,

Andre Carpinteiro

Invalid leading zero

First of all, thank you very much for the previous fix ๐Ÿ‘

We I tried with my own project packages there was another error that I would like to share:

ValueError at /package_monitor/reload/
Invalid leading zero in minor: '0.04.17'

The version is related to Unicode==0.04.17

Is it something related to semantic? If so, warn me if I am on the right repo.

Kind regards,

Andre Carpinteiro

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.