pyupio / changelogs Goto Github PK

View Code? Open in Web Editor NEW

60.0 11.0 24.0 25.81 MB

A changelog finder and parser for packages available on pypi, npm and rubygems.

Home Page: https://pyup.io

License: MIT License

Makefile 1.30% Python 98.70%

changelogs's Introduction

A changelog finder and parser with command line interface for packages available on pypi, npm, rubygems and launchpad.net. Originally developed for pyup.io.

Installation

To install changelogs, run this command in your terminal:

$ pip install changelogs

Usage

To use changelogs in a Python project:

import changelogs

logs = changelogs.get("flask")
logs = changelogs.get("babel", vendor="npm")
logs = changelogs.get("bundler", vendor="npm")

Or, from the command line:

changelogs flask
changelogs babel npm
changelogs bundler gem

If you are on macOS, you can chain the open command:

changelogs babel npm >> babel.log && open babel.log

About

When trying to get a changelog for a given package, there are a bunch of problems:

There is no central place to store a changelog. If a project has a changelog, it's most likely somewhere in the git repo at all kinds of different places. This makes it hard to find.
The package index meta data often has no direct link to the git repo. This makes the repo hard to find.
There is no changelog standard. Everyone uses a different approach. This makes it hard to parse.

This project is trying to solve this by:

first querying the package vendor for package meta data like the homepage or docs URL.
if the meta data doesn't contain a valid URL to a repo, visit all available URLs and scrape them to find one.
if there is a valid repo URL, visit the repo and look for possible changelogs like Changes.txt, NEWS.md or history.rst.
fetch the content and somewhat try to parse it.

changelogs's People

Contributors

Stargazers

Watchers

changelogs's Issues

Changelogs incorrect for mysqlclient

The changelog at https://pyup.io/changelogs/mysqlclient/ says:

5.0.10

Revamped the build system. Edit site.cfg if necessary (probably not
in most cases)
...

However this is from the bottom of the history file:
https://github.com/PyMySQL/mysqlclient-python/blame/c56146d137b2c9d55f9d15a5e3b502dd5a8457a1/HISTORY.md#L286

...and the version numbers are wrong. (It's interpreting (MySQL > 5.0.10) as a version number).

requests.exceptions.MissingSchema: Invalid URL

Package: como

Traceback (most recent call last):
  File "/usr/local/lib/python3.6/code.py", line 91, in runcode
    exec(code, self.locals)
  File "<console>", line 3, in <module>
  File "/usr/local/lib/python3.6/site-packages/celery/local.py", line 191, in __call__
    return self._get_current_object()(*a, **kw)
  File "/usr/local/lib/python3.6/site-packages/celery_once/tasks.py", line 134, in __call__
    return super(QueueOnce, self).__call__(*args, **kwargs)
  File "/usr/local/lib/python3.6/site-packages/celery/app/task.py", line 379, in __call__
    return self.run(*args, **kwargs)
  File "/app/pyupio/packages/tasks.py", line 295, in sync_changelog
    raise self.retry(exc=e, countdown=60)
  File "/usr/local/lib/python3.6/site-packages/celery/app/task.py", line 652, in retry
    raise_with_context(exc or Retry('Task can be retried', None))
  File "/app/pyupio/packages/tasks.py", line 277, in sync_changelog
    d['parsed_changelog'] = changelogs.get(name=name, functions=fns)
  File "/usr/local/lib/python3.6/site-packages/changelogs/changelogs.py", line 168, in get
    find_changelogs_fn=fns["find_changelogs"]
  File "/app/pyupio/packages/tasks.py", line 265, in patched_get_urls
    find_changelogs_fn=find_changelogs_fn, **kwargs
  File "/usr/local/lib/python3.6/site-packages/changelogs/pypi.py", line 63, in get_urls
    candidates=candidates)
  File "/usr/local/lib/python3.6/site-packages/changelogs/finder.py", line 208, in find_changelogs
    repos = set(find_repo_urls(session, name, candidates))
  File "/usr/local/lib/python3.6/site-packages/changelogs/finder.py", line 71, in find_repo_urls
    resp = session.get(_url)
  File "/usr/local/lib/python3.6/site-packages/requests/sessions.py", line 501, in get
    return self.request('GET', url, **kwargs)
  File "/usr/local/lib/python3.6/site-packages/opbeat/instrumentation/packages/base.py", line 63, in __call__
    args, kwargs)
  File "/usr/local/lib/python3.6/site-packages/opbeat/instrumentation/packages/base.py", line 220, in call_if_sampling
    return wrapped(*args, **kwargs)
  File "/usr/local/lib/python3.6/site-packages/requests/sessions.py", line 474, in request
    prep = self.prepare_request(req)
  File "/usr/local/lib/python3.6/site-packages/requests/sessions.py", line 407, in prepare_request
    hooks=merge_hooks(request.hooks, self.hooks),
  File "/usr/local/lib/python3.6/site-packages/requests/models.py", line 302, in prepare
    self.prepare_url(url, params)
  File "/usr/local/lib/python3.6/site-packages/requests/models.py", line 382, in prepare_url
    raise MissingSchema(error)
requests.exceptions.MissingSchema: Invalid URL 'como.cwoebker.com': No schema supplied. Perhaps you meant http://como.cwoebker.com?

pyzmq-static resolves to pyzmq changelog

cheroot changelog

https://pypi.python.org/pypi/Cheroot has its changelog at https://github.com/cherrypy/cheroot/blob/master/CHANGES.rst (but no repo link in its PyPI description...)

AttributeError: 'NoneType' object has no attribute 'xpath'

Package: django-selenium-test-runner

AttributeError: 'NoneType' object has no attribute 'xpath'
  File "celery/app/trace.py", line 367, in trace_task
    R = retval = fun(*args, **kwargs)
  File "celery_once/tasks.py", line 134, in __call__
    return super(QueueOnce, self).__call__(*args, **kwargs)
  File "celery/app/trace.py", line 622, in __protected_call__
    return self.run(*args, **kwargs)
  File "pyupio/packages/tasks.py", line 270, in sync_changelog
    d['parsed_changelog'] = changelogs.get(name=name, functions=fns)
  File "changelogs/changelogs.py", line 168, in get
    find_changelogs_fn=fns["find_changelogs"]
  File "pyupio/packages/tasks.py", line 257, in patched_get_urls
    d['urls'], d['repos'] = get_urls(session, name, data, find_changelogs_fn, **kwargs)
  File "changelogs/pypi.py", line 78, in get_urls
    candidates=new_candidates)
  File "changelogs/finder.py", line 208, in find_changelogs
    repos = set(find_repo_urls(session, name, candidates))
  File "changelogs/finder.py", line 74, in find_repo_urls
    for link in frozenset([str(l) for l in tree.xpath("//a/@href")]):

vcrpy

https://raw.githubusercontent.com/kevin1024/vcrpy/master/docs/changelog.rst

Issue with django-fernet-fields

The changelog of django-fernet-fields isn't parsed correctly.

cc @eranrund

flake8 Changelog

Located here:

https://gitlab.com/pycqa/flake8/tree/master/docs/source/release-notes

Fall back to raw commit logs as a last resort

Changelogs out of date for graphene

The changelogs page (https://pyup.io/changelogs/graphene/) lists the latest version as 1.0.5, whereas the latest is 1.3:
https://pypi.python.org/pypi/graphene
https://github.com/graphql-python/graphene/releases

XMLSyntaxError: line 3: b'htmlParseStartTag: invalid element name'

Package: gcloud-storage-api

XMLSyntaxError: line 3: b'htmlParseStartTag: invalid element name'
  File "celery/app/trace.py", line 367, in trace_task
    R = retval = fun(*args, **kwargs)
  File "celery_once/tasks.py", line 134, in __call__
    return super(QueueOnce, self).__call__(*args, **kwargs)
  File "celery/app/trace.py", line 622, in __protected_call__
    return self.run(*args, **kwargs)
  File "pyupio/packages/tasks.py", line 270, in sync_changelog
    d['parsed_changelog'] = changelogs.get(name=name, functions=fns)
  File "changelogs/changelogs.py", line 168, in get
    find_changelogs_fn=fns["find_changelogs"]
  File "pyupio/packages/tasks.py", line 257, in patched_get_urls
    d['urls'], d['repos'] = get_urls(session, name, data, find_changelogs_fn, **kwargs)
  File "changelogs/pypi.py", line 78, in get_urls
    candidates=new_candidates)
  File "changelogs/finder.py", line 208, in find_changelogs
    repos = set(find_repo_urls(session, name, candidates))
  File "changelogs/finder.py", line 73, in find_repo_urls
    tree = etree.HTML(resp.content)
  File "src/lxml/lxml.etree.pyx", line 3161, in lxml.etree.HTML (src/lxml/lxml.etree.c:77178)
  File "src/lxml/parser.pxi", line 1830, in lxml.etree._parseMemoryDocument (src/lxml/lxml.etree.c:116674)
  File "src/lxml/parser.pxi", line 1718, in lxml.etree._parseDoc (src/lxml/lxml.etree.c:115324)
  File "src/lxml/parser.pxi", line 1090, in lxml.etree._BaseParser._parseDoc (src/lxml/lxml.etree.c:109723)
  File "src/lxml/parser.pxi", line 584, in lxml.etree._ParserContext._handleParseResultDoc (src/lxml/lxml.etree.c:103584)
  File "src/lxml/parser.pxi", line 694, in lxml.etree._handleParseResult (src/lxml/lxml.etree.c:105238)
  File "src/lxml/parser.pxi", line 633, in lxml.etree._raiseParseError (src/lxml/lxml.etree.c:104323)

django-statici18n changelogs for release 1.3.0

https://django-statici18n.readthedocs.io/en/latest/changelog.html

h2o

Probably one of the worst changelogs: https://raw.githubusercontent.com/h2oai/h2o-3/master/Changes.md

Support changelogs in GitHub release descriptions

A number of packages use the GitHub "releases" feature to describe the changes in that release (and don't have a separate changelog file at all).

For example:
https://github.com/syrusakbary/promise/releases
https://github.com/graphql-python/graphql-core/releases
https://github.com/graphql-python/graphene-django/releases
https://github.com/google/brotli/releases

Support changelogs for kombu

https://github.com/celery/kombu/blob/master/Changelog

matrix-angular-sdk resolves to synapse/matrix-js-sdk/matrix-react-sdk

Changelog for Django-Jinja

It's just an .adoc instead of .md 😄

https://github.com/niwinz/django-jinja/blob/master/CHANGES.adoc

BeautifulSoup4 release notes

https://bazaar.launchpad.net/~leonardr/beautifulsoup/bs4/view/head:/NEWS.txt

Initial Update

Hi 👊

This is my first visit to this fine repo, but it seems you have been working hard to keep all dependencies updated so far.

Once you have closed this issue, I'll create seperate pull requests for every update as soon as I find one.

That's it for now!

Happy merging! 🤖

sqlalchemy changelog

http://docs.sqlalchemy.org/en/latest/changelog/changelog_11.html#change-1.1.6

PyAudio changelog

It is possible to find a changelog for PyAudio at http://people.csail.mit.edu/hubert/pyaudio/.

django-countries changelog not updated

See PR for current state: dimagi/commcare-hq#14932

https://github.com/SmileyChris/django-countries/blob/master/CHANGES.rst has correct info

pbr

History is here: https://docs.openstack.org/developer/pbr/history.html

Add support for packages on Gitlab

Selenium Changelog

Looks like you already know it's there:smile:

Use PyPI API to keep version lists up to date

In a few cases, the changelogs page for a package showed an out of date version as the "latest", presumably since that was the last version for which a changelog could be found.

In cases where no new changelog could be found, I think it would be less confusing if empty placeholders were added for the newer releases, using PyPI as the source of truth for which versions exist.

For example for https://pyup.io/changelogs/graphene/ :

v2.0:
unable to load changelog! (file issue link)

v1.0.5

Improved behavior of NdbConnectionField when ...

Linking to github compare view

For github-hosted projects, there are usually tagged version numbers. For certain projects we like to review the code difference based off of these tagged versions, so having a direct link to the github compare view for version updates would be useful (especially when there's no changelog)

For example, here is a link that compares Django 1.10.1 to 1.10.2's release.

Changelog missing latest version for django-cors-headers

https://pyup.io/changelogs/django-cors-headers/ says the newest version is 2.0.1.

However https://github.com/ottoyiu/django-cors-headers/blob/259181ca7a554183b15efd786850e09f6901da20/HISTORY.rst lists the latest PyPI version of 2.0.2.

Python Selenium changelog location

https://github.com/SeleniumHQ/selenium/blob/master/py/CHANGES

pyup not finding changelog

changelogger version: 0.5.0
Python version: 3.5.2
Operating System: macOS Sierra

Description

I'm maintain a couple of PyPI packages (webassets-elm, for example) but everytime pyup opens a PR (in repositories that use my package) it says it can't find any changelog.

What I Did

I installed changelogs and tested it, but always had an empty stdout. My changelog is the CHANGELOG file in the root of the repo, but I tried renaming and moving it all around (no success):

$ ls -laGh | grep CHANGELOG
-rw-r--r--@  1 cuducos  staff   362B Jan 27 13:17 CHANGELOG
$ changelogs webassets_elm
$ git mv CHANGELOG changelog.md
$ changelogs webassets_elm
$ gt mv changelog.md changelog
$ changelogs webassets_elm
$ git mv changelog.md webassets_elm/CHANGELOG
$ changelogs webassets_elm
$

Am I missing anything or is this a bug?

TypeError: string indices must be integers

Package: monasca-events-api

TypeError: string indices must be integers
  File "celery/app/trace.py", line 367, in trace_task
    R = retval = fun(*args, **kwargs)
  File "celery_once/tasks.py", line 134, in __call__
    return super(QueueOnce, self).__call__(*args, **kwargs)
  File "celery/app/trace.py", line 622, in __protected_call__
    return self.run(*args, **kwargs)
  File "pyupio/packages/tasks.py", line 270, in sync_changelog
    d['parsed_changelog'] = changelogs.get(name=name, functions=fns)
  File "changelogs/changelogs.py", line 189, in get
    _depth=_depth+1)
  File "changelogs/changelogs.py", line 177, in get
    get_head_fn=fns["get_head"]
  File "changelogs/launchpad.py", line 79, in parse
    return {e["version"]: e["changelog"] for e in content["entries"]

pinax-stripe

https://github.com/pinax/pinax-stripe/blob/master/docs/about/release-notes.md

PyPI: Changelogs for newrelic

https://pypi.python.org/pypi/newrelic

The release notes overview is at:
https://docs.newrelic.com/docs/release-notes/agent-release-notes/python-release-notes

The per-release pages have URLs of form:
https://docs.newrelic.com/docs/release-notes/agent-release-notes/python-release-notes/python-agent-282163

...where '282163' is the full version string (eg '2.82.1.63') with periods removed.

pandas release notes

This seems like a good start: http://pandas-docs.github.io/pandas-docs-travis/whatsnew.html

TypeError: expected string or bytes-like object

Package: ots

TypeError: expected string or bytes-like object
  File "celery/app/trace.py", line 367, in trace_task
    R = retval = fun(*args, **kwargs)
  File "celery_once/tasks.py", line 134, in __call__
    return super(QueueOnce, self).__call__(*args, **kwargs)
  File "celery/app/trace.py", line 622, in __protected_call__
    return self.run(*args, **kwargs)
  File "pyupio/packages/tasks.py", line 270, in sync_changelog
    d['parsed_changelog'] = changelogs.get(name=name, functions=fns)
  File "changelogs/changelogs.py", line 168, in get
    find_changelogs_fn=fns["find_changelogs"]
  File "pyupio/packages/tasks.py", line 257, in patched_get_urls
    d['urls'], d['repos'] = get_urls(session, name, data, find_changelogs_fn, **kwargs)
  File "changelogs/pypi.py", line 76, in get_urls
    new_candidates = changelogs.url_re.findall(data["info"]["description"])

pyparsing changelog

https://pypi.python.org/pypi/pyparsing has its changelog at https://sourceforge.net/p/pyparsing/code/HEAD/tree/trunk/src/CHANGES

Issues with coverage.py changelog

https://pyup.io/changelogs/coverage/ only has 4.4b1 available, while the comment at qutebrowser/qutebrowser#2609 has the 4.4 release too
Looks like < and > are escaped inside code blocks while they shouldn't be, leading to <source>

It's in the readme.

XMLSyntaxError: line 738: b'Tag path invalid'

Package: 'gomobile.xhtmlmp'

XMLSyntaxError: line 738: b'Tag path invalid'
  File "celery/app/trace.py", line 367, in trace_task
    R = retval = fun(*args, **kwargs)
  File "celery_once/tasks.py", line 134, in __call__
    return super(QueueOnce, self).__call__(*args, **kwargs)
  File "celery/app/trace.py", line 622, in __protected_call__
    return self.run(*args, **kwargs)
  File "pyupio/packages/tasks.py", line 270, in sync_changelog
    d['parsed_changelog'] = changelogs.get(name=name, functions=fns)
  File "changelogs/changelogs.py", line 168, in get
    find_changelogs_fn=fns["find_changelogs"]
  File "pyupio/packages/tasks.py", line 257, in patched_get_urls
    d['urls'], d['repos'] = get_urls(session, name, data, find_changelogs_fn, **kwargs)
  File "changelogs/pypi.py", line 63, in get_urls
    candidates=candidates)
  File "changelogs/finder.py", line 208, in find_changelogs
    repos = set(find_repo_urls(session, name, candidates))
  File "changelogs/finder.py", line 73, in find_repo_urls
    tree = etree.HTML(resp.content)
  File "src/lxml/lxml.etree.pyx", line 3161, in lxml.etree.HTML (src/lxml/lxml.etree.c:77178)
  File "src/lxml/parser.pxi", line 1830, in lxml.etree._parseMemoryDocument (src/lxml/lxml.etree.c:116674)
  File "src/lxml/parser.pxi", line 1718, in lxml.etree._parseDoc (src/lxml/lxml.etree.c:115324)
  File "src/lxml/parser.pxi", line 1090, in lxml.etree._BaseParser._parseDoc (src/lxml/lxml.etree.c:109723)
  File "src/lxml/parser.pxi", line 584, in lxml.etree._ParserContext._handleParseResultDoc (src/lxml/lxml.etree.c:103584)
  File "src/lxml/parser.pxi", line 694, in lxml.etree._handleParseResult (src/lxml/lxml.etree.c:105238)
  File "src/lxml/parser.pxi", line 633, in lxml.etree._raiseParseError (src/lxml/lxml.etree.c:104323)