lilydjwg/nvchecker

New version checker for software releases

nvchecker (short for new version checker) is for checking if a new version of some software has been released.

This is the version 2.0 branch. For the old version 1.x, please switch to the v1.x branch.

Dependency

• Python 3.8+
• Python library: structlog, platformdirs, tomli (on Python < 3.11)
• One of these Python library combinations (ordered by preference):
  • tornado + pycurl
  • aiohttp
  • httpx with http2 support (experimental; only latest version is supported)
  • tornado
• All commands used in your software version configuration files

Install and Run

To install:

pip3 install nvchecker

To use the latest code, you can also clone this repository and run:

pip install .

To see available options:

nvchecker --help

Run with one or more software version files:

nvchecker -c config_file

You normally will like to specify some "version record files"; see below.

Documentation

For detailed documentation, see https://nvchecker.readthedocs.io/en/latest/.

nvchecker's Issues

list options not working with BitBucket source

ignored and exclude_pattern do not seem to be working when using a BitBucket source with use_max_tag=true.

Deprecated ignored_tags does work though.

I'm using nvchecker 1.4.4.

Add module __main__

> python3 -m nvchecker --help
/usr/bin/python3: No module named nvchecker.__main__; 'nvchecker' is a package and cannot be directly executed

Email notification?

Would it be possible to build in email notification of new versions or should i just run it via cron and have it email on output?

TIA,

machine-readable errors

A JSON output with versions and errors?

Use structlog and configure it to output json?

remove usage of environment variables for tokens

Use keyfile instead. Also remove the token option from gitlab source.

Add support for more distro package managers

I suspect it might be easier to support some existing cross/meta package-manager tool, even if the results are not as good.

I've been compiling a list at
https://gitlab.com/coala/package_manager/issues/237

Version 1.5 is not available via pip

$ pip3 install nvhecker
Collecting nvhecker
Exception:
Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/pip/basecommand.py", line 215, in main
    status = self.run(options, args)
  File "/usr/lib/python3/dist-packages/pip/commands/install.py", line 353, in run
    wb.build(autobuilding=True)
  File "/usr/lib/python3/dist-packages/pip/wheel.py", line 749, in build
    self.requirement_set.prepare_files(self.finder)
  File "/usr/lib/python3/dist-packages/pip/req/req_set.py", line 380, in prepare_files
    ignore_dependencies=self.ignore_dependencies))
  File "/usr/lib/python3/dist-packages/pip/req/req_set.py", line 554, in _prepare_file
    require_hashes
  File "/usr/lib/python3/dist-packages/pip/req/req_install.py", line 278, in populate_link
    self.link = finder.find_requirement(self, upgrade)
  File "/usr/lib/python3/dist-packages/pip/index.py", line 465, in find_requirement
    all_candidates = self.find_all_candidates(req.name)
  File "/usr/lib/python3/dist-packages/pip/index.py", line 423, in find_all_candidates
    for page in self._get_pages(url_locations, project_name):
  File "/usr/lib/python3/dist-packages/pip/index.py", line 568, in _get_pages
    page = self._get_page(location)
  File "/usr/lib/python3/dist-packages/pip/index.py", line 683, in _get_page
    return HTMLPage.get_page(link, session=self.session)
  File "/usr/lib/python3/dist-packages/pip/index.py", line 795, in get_page
    resp.raise_for_status()
  File "/usr/share/python-wheels/requests-2.18.4-py2.py3-none-any.whl/requests/models.py", line 935, in raise_for_status
    raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 404 Client Error: Not Found for url: https://pypi.org/simple/nvhecker/

include_regex does not work with pagination

for repositories with so many tags that it needs multiple pages include_regex does not work while the deprecated include_tags_pattern does work

For example following repository returns no result with the new setting but does return results with the old setting:

[puppetserver]
github = puppetlabs/puppetserver
use_max_tag = true
max_page = 5
include_regex = ^[0-9.]+

with instead

include_tags_pattern = ^[0-9.]+

it currently correctly returns 6.2.1

it appears apply_list_options only gets the first page multiple times, not each page:

>/.local/lib/python3.6/site-packages/nvchecker-1.4.2-py3.6.egg/nvchecker/get_version.py(46)apply_list_options()
-> if pattern:
(Pdb) p len(versions), len(set(versions))
(150, 30)

Add more complete tests

Support github apikey without requiring environment variables

I'd much prefer to use another config option, e.g. apikey = ~/.config/nvchecker/apikey, where this file contains the OAuth token.

More generally this could probably collect the various gitlab tokens, and I think this is everything which uses tokens.

版本号能不能把-后面的小版本去掉

比如aur和arch官方的包，-后面的数字基本没啥用，但是仍然会参与版本号比较

Feature-Request: use proxy-configuration from environment variables

As far as I can see you can only configure a proxy for nvchecker via the config-file.

It would be cool, if nvchecker would consider the environment variable if nothing is provided in the configuration-file.

Can you release new ver?

Can you push a new version to pypi at some point?

Thanks for the great script!

TypeError: on_no_result() missing 1 required positional argument: 'oldver'

An exception is raised when nvchecker can't find the version for a package

$ cat bug.ini
[__config__]

[android-ndk-beta]
url = https://developer.android.com/ndk/downloads/index.html
regex = Latest Beta Version \((r\d+ Beta \d)\)
from_pattern = r(\d+) Beta (\d)
to_pattern = r\1_beta\2

$ nvchecker bug.ini
[E 05-10 18:43:36.244 _base:136] android-ndk-beta: version string not found.
[W 05-10 18:43:36.245 _base:136] android-ndk-beta: no-result
Traceback (most recent call last):
  File "/usr/bin/nvchecker", line 11, in <module>
    load_entry_point('nvchecker==1.1.dev0', 'console_scripts', 'nvchecker')()
  File "/usr/lib/python3.6/site-packages/nvchecker/main.py", line 42, in main
    ioloop.run_until_complete(s.check())
  File "/usr/lib/python3.6/asyncio/base_events.py", line 468, in run_until_complete
    return future.result()
  File "/usr/lib/python3.6/site-packages/nvchecker/core.py", line 181, in check
    self.on_no_result(name)
TypeError: on_no_result() missing 1 required positional argument: 'oldver'

nvchecker version: archlinuxcn/nvchecker-git 1:1.1dev.r61-1

Redirection not handled

Hello,

I'd like to track this repo : http://hg.nginx.org/nginx/branches with:

url = https://hg.nginx.org/nginx/branches/
regex = stable-\d+\.\d+

but it does not work. There is a redirection, I guess this is why it fails ?

AsyncHTTPClient get 599 error when runnning in cron

here is the cron command:
source $HOME/.bash_profile && /usr/local/bin/nvchecker $HOME/nvchecker/nvchecker.ini > $HOME/nvchecker/update.log 2>&1
only in cron it runs with the following error:

[E 09-14 10:30:21.776 ioloop:612] Exception in callback functools.partial(<function wrap.<locals>.wrapped at 0x7f173141dae8>, HTTPResponse(_body=None,buffer=None,code=599,effective_url='https://www.archlinux.org/packages/search/json/?name=warsow',error=CurlError('HTTP 599: Operation timed out after 0 milliseconds with 0 out of 0 bytes received',),headers={},reason='Unknown',request=<tornado.httpclient.HTTPRequest object at 0x7f173141e390>,request_time=20.00719165802002,time_info={}))
    Traceback (most recent call last):
      File "/usr/local/lib/python3.4/dist-packages/tornado/ioloop.py", line 592, in _run_callback
        ret = callback()
      File "/usr/local/lib/python3.4/dist-packages/tornado/stack_context.py", line 343, in wrapped
        raise_exc_info(exc)
      File "<string>", line 3, in raise_exc_info
      File "/usr/local/lib/python3.4/dist-packages/tornado/stack_context.py", line 355, in _handle_exception
        if tail.exit(*exc):
      File "/usr/local/lib/python3.4/dist-packages/tornado/stack_context.py", line 186, in exit
        return self.exception_handler(type, value, traceback)
      File "/usr/local/lib/python3.4/dist-packages/nvchecker-0.4dev-py3.4.egg/nvchecker/core.py", line 98, in _handle_exception
        raise value.with_traceback(traceback)
      File "/usr/local/lib/python3.4/dist-packages/tornado/stack_context.py", line 314, in wrapped
        ret = fn(*args, **kwargs)
      File "/usr/local/lib/python3.4/dist-packages/nvchecker-0.4dev-py3.4.egg/nvchecker/source/archpkg.py", line 17, in _pkg_done
        raise res.error
    tornado.curl_httpclient.CurlError: HTTP 599: Operation timed out after 0 milliseconds with 0 out of 0 bytes received

Feature Request: Support Docker Hub images

It would be really nice if this would also support Docker images (particularly https://hub.docker.com). Their API documentation is https://docs.docker.com/registry/spec/api.

Example for tags: https://registry.hub.docker.com/repositories/library/alpine/tags/

Thank you in advance.

HTTP 404 for okular and linux

Like the title said, I get 404 when using nvchecker to check for newer version of linux and okular.

Here is my sources.ini

[__config__]
oldver=old_ver
newver=new_ver

[linux]
github=torvalds/linux
use_latest_release=true

[okular]
github=KDE/okular
use_latest_release=true

My old_ver (I know the version I wrote in this file is wrong, but other packages like conky doesn't get 404, and version compare should be done after retrieving newest version so I think this is not the root of the problem):

linux 5.4.0
okular 19.08.3

Here is the exact error when I run nvchecker sources.ini

[E 12-09 10:24:19.410 core:185] okular: unexpected error happened error=HTTP 404: Not Found
    Traceback (most recent call last):
      File "/usr/lib/python3.8/site-packages/nvchecker/core.py", line 158, in worker
        ret = await get_version(
      File "/usr/lib/python3.8/site-packages/nvchecker/get_version.py", line 90, in get_version
        version = await func(name, conf, **kwargs)
      File "/usr/lib/python3.8/site-packages/nvchecker/source/github.py", line 25, in get_version
        check_ratelimit(e, name)
      File "/usr/lib/python3.8/site-packages/nvchecker/source/github.py", line 23, in get_version
        return await get_version_real(name, conf, **kwargs)
      File "/usr/lib/python3.8/site-packages/nvchecker/source/github.py", line 68, in get_version_real
        async with session.get(url, headers=headers, **kwargs) as res:
      File "/usr/lib/python3.8/site-packages/nvchecker/source/tornado_httpclient.py", line 65, in __aenter__
        return await to_asyncio_future(client.fetch(self.req))
    tornado.httpclient.HTTPClientError: HTTP 404: Not Found
[E 12-09 10:24:19.413 core:185] linux: unexpected error happened error=HTTP 404: Not Found
    Traceback (most recent call last):
      File "/usr/lib/python3.8/site-packages/nvchecker/core.py", line 158, in worker
        ret = await get_version(
      File "/usr/lib/python3.8/site-packages/nvchecker/get_version.py", line 90, in get_version
        version = await func(name, conf, **kwargs)
      File "/usr/lib/python3.8/site-packages/nvchecker/source/github.py", line 25, in get_version
        check_ratelimit(e, name)
      File "/usr/lib/python3.8/site-packages/nvchecker/source/github.py", line 23, in get_version
        return await get_version_real(name, conf, **kwargs)
      File "/usr/lib/python3.8/site-packages/nvchecker/source/github.py", line 68, in get_version_real
        async with session.get(url, headers=headers, **kwargs) as res:
      File "/usr/lib/python3.8/site-packages/nvchecker/source/tornado_httpclient.py", line 65, in __aenter__
        return await to_asyncio_future(client.fetch(self.req))
    tornado.httpclient.HTTPClientError: HTTP 404: Not Found

I install nvchecker from Arch official repo.

Proxy and pycurl

Hello,

When i add the pycurl dependency, i have this error :

[E 11-22 09:16:33.365 core:129] unexpected error happened with influxdb
    Traceback (most recent call last):
      File "/usr/local/lib/python3.5/dist-packages/nvchecker/core.py", line 125, in future_done
        version = fu.result()
      File "/usr/lib/python3.5/asyncio/futures.py", line 274, in result
        raise self._exception
      File "/usr/lib/python3.5/asyncio/tasks.py", line 239, in _step
        result = coro.send(None)
      File "/usr/local/lib/python3.5/dist-packages/nvchecker/get_version.py", line 19, in get_version
        return await func(name, conf)
      File "/usr/local/lib/python3.5/dist-packages/nvchecker/source/github.py", line 36, in get_version
        async with session.get(url, headers=headers, **kwargs) as res:
      File "/usr/local/lib/python3.5/dist-packages/nvchecker/source/tornado_httpclient.py", line 54, in __aenter__
        return await to_asyncio_future(client.fetch(self.req))
      File "/usr/lib/python3.5/asyncio/futures.py", line 363, in __iter__
        return self.result()  # May raise too.
      File "/usr/lib/python3.5/asyncio/futures.py", line 274, in result
        raise self._exception
      File "/usr/local/lib/python3.5/dist-packages/tornado/curl_httpclient.py", line 214, in _process_queue
        curl.info["headers"])
      File "/usr/local/lib/python3.5/dist-packages/tornado/curl_httpclient.py", line 474, in _curl_setup_request
        request.prepare_curl_callback(curl)
      File "/usr/local/lib/python3.5/dist-packages/tornado/stack_context.py", line 277, in null_wrapper
        return fn(*args, **kwargs)
      File "/usr/local/lib/python3.5/dist-packages/nvchecker/source/tornado_httpclient.py", line 24, in try_use_http2
        curl.setopt(pycurl.HTTP_VERSION, 4)
    pycurl.error: (1, '')

If i disable the line curl.setopt(pycurl.HTTP_VERSION, 4) it works

pytest-asyncio 0.14 + pycurl = test suite hang

$ pytest -v
======================================== test session starts ========================================
platform linux -- Python 3.8.3, pytest-5.4.3, py-1.9.0, pluggy-0.13.1 -- /usr/bin/python
cachedir: .pytest_cache
rootdir: /home/yen/var/local/Computer/archlinux/nvchecker, inifile: setup.cfg
plugins: asyncio-0.14.0, flaky-3.6.1, httpbin-1.0.0
collected 75 items

tests/test_android.py::test_android_addon PASSED                                               [  1%]
tests/test_android.py::test_android_package PASSED                                             [  2%]
tests/test_anitya.py::test_anitya

And then it hangs. If I downgrade python-pytest-asyncio to 0.12, or comment out the following line

nvchecker/nvchecker/source/tornado_httpclient.py

Line 14 in e176078

AsyncHTTPClient.configure("tornado.curl_httpclient.CurlAsyncHTTPClient", max_clients=20)

The test suite is fine.

I run git bisect on pytest-asyncio. The first "bad" commit is pytest-dev/pytest-asyncio@51d986c

$ git bisect log
git bisect start
# good: [b8e2a45e152a196cef7fdb6ddcf3e2d67a0f01ca] 0.12.0
git bisect good b8e2a45e152a196cef7fdb6ddcf3e2d67a0f01ca
# bad: [c0c8838e975ea63c7208599e0ea67461ea87f1e1] Prepare for release
git bisect bad c0c8838e975ea63c7208599e0ea67461ea87f1e1
# bad: [51d986cec83fdbc14fa08015424c79397afc7ad9] To solve test cases that fail:
git bisect bad 51d986cec83fdbc14fa08015424c79397afc7ad9
# good: [c1131f8b5313189508dc81d7ef1937ccb136658b] 1) A new test case that fails with 0.12.0, and pass with this commit.    test_async_fixtures_with_finalizer_scope.py: 2) Main problem is due to side effects of asyncio.get_event_loop(). See:    https://github.com/python/cpython/blob/3.8/Lib/asyncio/events.py#L636    This method could either return an existing loop or create a new one.
git bisect good c1131f8b5313189508dc81d7ef1937ccb136658b
# good: [f97e900f1fcb51a572a1b861c95ac49e69bbfdf9] 1) Test case (test_async_fixtures_with_finalizer) refactoring to pass on python 3.6 & 3.5 2) An additional test case (test_module_with_get_event_loop_finalizer).    Refactoring previous test case I got to another potential issue: Finalizers using get_event_loop() instead of event_loop [on a module scope fixture]    To be able to pass this new test case I needed some additional changes on next event_loop fixture block(  https://github.com/pytest-dev/pytest-asyncio/blob/v0.12.0/pytest_asyncio/plugin.py#L54-L66):    - This block needs to apply on all event_loop fixture scopes (so I remove 'asyncio' in request.keywords, that only apply to function scope)    - The get_event_loop() method inside this block has following side effects (See https://github.com/python/cpython/blob/3.8/Lib/asyncio/events.py#L636):      - As no loop is still set, then L636 is executed, and so a new event_loop (let's call it L2) is first created and then set.         - And then finalizer will restore L2 at the end of test. What has no sense, to restore L2 that wasn't before the test. And additionally this L2 is never closed.     So it seeems that get_event_loop() should be removed, and so I see no reasons for any finalizer.
git bisect good f97e900f1fcb51a572a1b861c95ac49e69bbfdf9
# first bad commit: [51d986cec83fdbc14fa08015424c79397afc7ad9] To solve test cases that fail:

That commit destorys event loop after all tests each module are processed. Based on that fact, I managed to work-around the issue with

diff --git a/tests/conftest.py b/tests/conftest.py
index bddf44a..daae169 100644
--- a/tests/conftest.py
+++ b/tests/conftest.py
@@ -54,7 +54,7 @@ async def get_version():
 
   return __call__
 
[e-mail](scope="module")
[e-mail](scope="session")
 def event_loop(request):
   """Override pytest-asyncio's event_loop fixture,
      Don't create an instance of the default event loop for each test case.

Above all, I guess the pycurl backend in tornado gets confused with multiple event loops.

pull in configuration from $XDG_CONFIG_HOME automatically

(On Linux at least) It would be pretty awesome, if a call to nvchecker leads to an automatic resolve of the application using a configuration below ~/.config/nvchecker/nvchecker.ini (i.e. $XDG_CONFIG_HOME/nvchecker/nvchecker.ini), if available.
This is standard behavior and a good default.

If you include appdirs, you can easily make this cross-platform.

Allow nvtake to 'take' all updates

Would it be possible to have a flag to nvtake to 'sync' all versions ? My use case is I have it set up to email me daily via cron and I just want to know about each new version once.

Any thoughts?

TIA,

feature request: add a --retry <ntimes> option

Thanks for this very useful piece of software. We're using it to watch for software updates for applications we centrally install for users on our cluster, but we get errors sometimes due to transient issues with the remote servers. It'd be nice to be able to automatically retry checking a particular entry that failed.

Feature request: Support Yum/CentOS/Fedora Repos?

It would be nice if you would support Yum/RH-Based repos!

TIA,

Add repology source

http://repology.org could be used as a 'fallback' source of new version data if a better source cant be found.

How to setup NVCHECKER_GITHUB_TOKEN?

Hi,
I tried to setup NVCHECKER_GITHUB_TOKEN variable, but it doesn't work.
[config]
oldver = old_ver.txt
newver = new_ver.txt
NVCHECKER_GITHUB_TOKEN = my_token

Please help me.
Thanks!

Intended usage

I've just found this tool and thought "how great". But I don't really get how to use it.

Here is my config:

[__config__]
oldver = old_ver.txt
newver = new_ver.txt

[holoviews]
github = pyviz/holoviews

old_ver contains
holoviews 20190721.234313

and new_ver
holoviews 20190717.131740

But the newest version in reality is 1.12.3 from the 20th of May.

Why is nvchecker finding daily snapshots here?

Missing dependence python-setuptools

Missing dependence python-setuptools for pkg_resources module.

Github release tag

hi,

Your tools is awesome, thanks for it.

Funny, but for nvchecker release check does not work (only commit can be checked):

Where it works properly this endpoint returns with valid information:
https://api.github.com/repos/go-gitea/gitea/releases/latest

For your repo it does not work:

https://api.github.com/repos/lilydjwg/nvchecker/releases/latest

However, both Github projects have releases on the project page. So I simply don't understand, why is that? Why is that endpoint is missing for nvchecker?

Cache results for reuse

Some entries may be duplicate (e.g. different packages from the same source).

gitcafe已经不复存在了

貌似合并到别的站点去了
我也没空看新的API
删了吧

change cli argument processing

Use -c to specify a config file, default to $XDG_CONFIG_DIR/nvchecker/nvchecker.ini. (superseding #96).

Feature Request: vcs检查更新时也可以有proxy选项

如题，这样没准freedesktop.org就有办法解决了
仅支持http/https proxy就可以了。。

ignored_tags not powerful enough to filter github releases not available with the releases API

See for example https://github.com/EFForg/https-everywhere/

No Github releases exist, so I need use_max_tag, but they've also got lots of legacy tags, so I'd need to use this too, but unfortunately that is 1) insane, 2) "IndexError: list index out of range":

ignored_tags = trac-5275-1 testtag deterministic-0.1 chrome-0.0.8.1 chrome-0.0.8.1 chrome-0.0.8.4 chrome-0.0.8.4 chrome-0.0.8.5 chrome-0.0.8.5 chrome-0.0.8.6 chrome-0.0.8.6 chrome-0.0.8.7 chrome-0.0.8.7 chrome-0.0.8.8 chrome-0.0.8.8 chrome-2012.1.26 chrome-2012.10.18 chrome-2012.10.18 chrome-2012.10.31 chrome-2012.10.31 chrome-2012.10.9 chrome-2012.10.9 chrome-2012.12.17 chrome-2012.12.17 chrome-2012.2.27 chrome-2012.2.27 chrome-2012.2.6 chrome-2012.2.6 chrome-2012.2.6.1 chrome-2012.2.6.1 chrome-2012.2.9 chrome-2012.2.9 chrome-2012.3.14 chrome-2012.3.14 chrome-2012.3.9 chrome-2012.3.9 chrome-2012.5.1 chrome-2012.5.1 chrome-2012.6.18 chrome-2012.6.18 chrome-2012.6.21 chrome-2012.6.21 chrome-2012.8.15 chrome-2012.8.15 chrome-2012.8.16 chrome-2012.8.16 chrome-2012.9.10 chrome-2012.9.10 chrome-2012.9.21 chrome-2012.9.21 chrome-2013.1.18 chrome-2013.1.18 chrome-2013.10.16 chrome-2013.10.16 chrome-2013.3.7 chrome-2013.3.7 chrome-2013.4.30 chrome-2013.4.30 chrome-2013.6.4 chrome-2013.6.4 chrome-2013.7.10 chrome-2013.7.10 chrome-2013.8.16 chrome-2013.8.16 chrome-2013.8.17 chrome-2013.8.17 chrome-2014.1.3 chrome-2014.1.3 chrome-2014.11.25 chrome-2014.11.25 chrome-2014.4.14 chrome-2014.4.14 chrome-2014.4.14.1 chrome-2014.4.14.1 chrome-2014.4.15 chrome-2014.4.15 chrome-2014.4.16 chrome-2014.4.16 chrome-2014.4.25 chrome-2014.4.25 chrome-2014.6.25 chrome-2014.6.25 chrome-2014.6.26 chrome-2014.6.26 chrome-2014.8.22 chrome-2014.8.22 chrome-2015.1.22 chrome-2015.1.22 chrome-2015.12.16 chrome-2015.12.16 chrome-2015.3.23 chrome-2015.3.23 chrome-2015.3.31 chrome-2015.3.31 chrome-2015.4.23 chrome-2015.4.23 chrome-2015.4.7 chrome-2015.4.7 chrome-2015.5.12 chrome-2015.5.12 chrome-2015.5.28 chrome-2015.5.28 chrome-2015.7.13 chrome-2015.7.13 chrome-2015.7.17 chrome-2015.7.17 chrome-2015.8.13 chrome-2015.8.13 chrome-2016.10.20 chrome-2016.10.20 chrome-2016.11.30 chrome-2016.11.30 chrome-2016.11.8 chrome-2016.11.8 chrome-2016.12.19 chrome-2016.12.19 chrome-2016.2.18 chrome-2016.2.18 chrome-2016.2.23 chrome-2016.2.23 chrome-2016.3.23 chrome-2016.3.23 chrome-2016.4.4 chrome-2016.4.4 chrome-2016.5.10 chrome-2016.5.10 chrome-2016.6.9 chrome-2016.6.9 chrome-2016.7.18 chrome-2016.7.18 chrome-2016.7.19 chrome-2016.7.19 chrome-2016.8.24 chrome-2016.8.24 chrome-2016.9.1 chrome-2016.9.1 chrome-2016.9.21 chrome-2016.9.21 chrome-2017.1.25 chrome-2017.1.25 chrome-2017.2.13 chrome-2017.2.13 chrome-2017.3.17 chrome-2017.3.17 chrome-2017.3.9 chrome-2017.3.9 chrome-2017.4.19 chrome-2017.4.19 chrome-2017.4.5 chrome-2017.4.5 chrome-2017.5.22 chrome-2017.5.22 chrome-2017.5.8 chrome-2017.5.8 chrome-2017.6.20 chrome-2017.6.20 chrome-2017.6.5 chrome-2017.6.5 chrome-2017.7.18 chrome-2017.7.18 chrome-2017.7.5 chrome-2017.7.5

Needless to say, being able eliminate the 79 tags beginning with "chrome-" from being considered would be very useful here.

problems checking tags from bitbucket

checking tags on bitbucket recently stopped working and it can be reproduced via the test suite:

____________________________ test_bitbucket_max_tag ____________________________

get_version = <function get_version.<locals>.__call__ at 0x7f5fb0befb70>

    async def test_bitbucket_max_tag(get_version):
>       assert await get_version("example", {"bitbucket": "prawee/git-tag", "use_max_tag": 1}) == "1.7.0"

tests/test_bitbucket.py:11: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
tests/conftest.py:50: in __call__
    return await _get_version(name, config)
/usr/lib/python3/dist-packages/nvchecker/get_version.py:90: in get_version
    version = await func(name, conf, **kwargs)
/usr/lib/python3/dist-packages/nvchecker/source/bitbucket.py:22: in get_version
    async with session.get(url) as res:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = <nvchecker.source.tornado_httpclient.ResponseManager object at 0x7f5fb0ba0ba8>

    async def __aenter__(self):
>     return await to_asyncio_future(client.fetch(self.req))
E     tornado.httpclient.HTTPClientError: HTTP 410: Gone

/usr/lib/python3/dist-packages/nvchecker/source/tornado_httpclient.py:62: HTTPClientError
___________________ test_bitbucket_max_tag_with_ignored_tags ___________________

get_version = <function get_version.<locals>.__call__ at 0x7f5fb0befb70>

    async def test_bitbucket_max_tag_with_ignored_tags(get_version):
>       assert await get_version("example", {"bitbucket": "prawee/git-tag", "use_max_tag": 1, "ignored_tags": "1.6.0 1.7.0"}) == "v1.5"

tests/test_bitbucket.py:14: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
tests/conftest.py:50: in __call__
    return await _get_version(name, config)
/usr/lib/python3/dist-packages/nvchecker/get_version.py:90: in get_version
    version = await func(name, conf, **kwargs)
/usr/lib/python3/dist-packages/nvchecker/source/bitbucket.py:22: in get_version
    async with session.get(url) as res:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = <nvchecker.source.tornado_httpclient.ResponseManager object at 0x7f5fb09f49b0>

    async def __aenter__(self):
>     return await to_asyncio_future(client.fetch(self.req))
E     tornado.httpclient.HTTPClientError: HTTP 410: Gone

/usr/lib/python3/dist-packages/nvchecker/source/tornado_httpclient.py:62: HTTPClientError

There are other failures in the test suite, too, but the bitbucket issue is the one that's affecting me currently.

Missing Dependency: "mercurial"

Please add "mercurial" as a dependency because without it checking some packages will generate the following error:

/usr/lib/python3.4/site-packages/nvchecker/source/vcs.sh: line 71: hg: command not found

`RuntimeWarning` with the aiohttp backend

For example, in this Travis CI test: https://travis-ci.org/lilydjwg/nvchecker/jobs/395356024

==================== 50 passed, 6 skipped in 15.02 seconds =====================
sys:1: RuntimeWarning: coroutine 'ClientSession.close' was never awaited
The command "pytest" exited with 0.

Running the nvchecker command results in the same warning

$ nvchecker
sys:1: RuntimeWarning: coroutine 'ClientSession.close' was never awaited
[E 06-26 22:38:31.527 base_events:1266] Unclosed client session
    client_session: <nvchecker.source.aiohttp_httpclient.BetterClientSession object at 0x10a637e10>

BTW, why are there multiple HTTP backends? It's confusing and bring issues to packaging. For example, Arch Linux's pacman does not support the OR operator (|) like dpkg of Debian, so aiohttp, tornado and pycurl are listed as optional dependencies. User intervention are required to make nvchecker actually works. If the plan is to replace Tornado+PyCURL with aiohttp as described in #53, I vote for dropping old backends, so that packagers can just list aiohttp as a normal dependency.

Use a common module for those JSON API sources

"vcs" source type doesn't return errors

I got this:

svn: warning: W170000: URL 'https://cvs.khronos.org/svn/repos/registry/trunk/public/cl/api/2.0' non-existent in revision 32529
svn: E200009: Could not display info for all targets because some targets don't exist

But this error didn't get to nvchecker.

@yuyichao

"vcs =" returns incorrect commit hashes

With the following nvchecker.ini: https://fars.ee/9Ajz (generated by grep -B 1 '^vcs =\s\+$' ~lilydjwg/.lilac/nvchecker.ini | grep -v '\--' on build.archlinuxcn.org), commit hashes appear messed up - the same commit appears in multiple packages.

An example run:

[I 12-15 20:23:13.251 core:190] lximage-qt-git: updated to 1.1.a977db28e5a50fd10f0ebe627e8b24c531b9574c
[I 12-15 20:23:13.251 core:190] lxqt-notificationd-git: updated to 1.1.a977db28e5a50fd10f0ebe627e8b24c531b9574c
[I 12-15 20:23:13.252 core:190] musescore-git: updated to 1.1.a977db28e5a50fd10f0ebe627e8b24c531b9574c
[I 12-15 20:23:13.252 core:190] openspecfun-git: updated to 1.1.a977db28e5a50fd10f0ebe627e8b24c531b9574c
[I 12-15 20:23:13.255 core:190] oce-git: updated to 1.1.a977db28e5a50fd10f0ebe627e8b24c531b9574c
[I 12-15 20:23:13.255 core:190] scribus-svn: updated to 1.1.a977db28e5a50fd10f0ebe627e8b24c531b9574c
[I 12-15 20:23:13.255 core:190] virtualbox-svn: updated to 1.1.a977db28e5a50fd10f0ebe627e8b24c531b9574c
[I 12-15 20:23:13.257 core:190] depot-tools-git: updated to 1.1.a977db28e5a50fd10f0ebe627e8b24c531b9574c
[I 12-15 20:23:13.258 core:190] rr-git: updated to 1.1.a977db28e5a50fd10f0ebe627e8b24c531b9574c
[I 12-15 20:23:13.258 core:190] emacs-llvm-mode-git: updated to 1.1.a977db28e5a50fd10f0ebe627e8b24c531b9574c
[I 12-15 20:23:13.261 core:190] beignet-git: updated to 1.1.a977db28e5a50fd10f0ebe627e8b24c531b9574c
[I 12-15 20:23:13.261 core:190] openlibm-git: updated to 1.1.a977db28e5a50fd10f0ebe627e8b24c531b9574c
[I 12-15 20:23:13.262 core:190] libunwind-git: updated to 1.1.a977db28e5a50fd10f0ebe627e8b24c531b9574c
[I 12-15 20:23:13.264 core:190] emacs-julia-mode-git: updated to 1.1.a977db28e5a50fd10f0ebe627e8b24c531b9574c
[I 12-15 20:23:13.264 core:190] julia-pkg-scripts: updated to 1.1.a977db28e5a50fd10f0ebe627e8b24c531b9574c
[I 12-15 20:23:13.265 core:190] openblas-lapack-git: updated to 1.1.a977db28e5a50fd10f0ebe627e8b24c531b9574c
[I 12-15 20:23:13.265 core:190] libutf8proc-git: updated to 1.1.a977db28e5a50fd10f0ebe627e8b24c531b9574c
[I 12-15 20:23:13.265 core:190] emacs-markdown-mode-git: updated to 1.1.a977db28e5a50fd10f0ebe627e8b24c531b9574c
[I 12-15 20:23:13.266 core:190] devtools-cn-git: updated to 1.1.a977db28e5a50fd10f0ebe627e8b24c531b9574c
[I 12-15 20:23:13.266 core:190] spice-vdagent-git: updated to 1.1.a977db28e5a50fd10f0ebe627e8b24c531b9574c
[I 12-15 20:23:13.320 core:190] lxqt-globalkeys-git: updated to 1.1.7ac6ecbeda5c274c2e0baed0682d49f0449b373d
[I 12-15 20:23:13.323 core:190] liblxqt-git: updated to 1.1.0a6ae1d8d0ac3cbb9b0e5c253a690202abf66c34
[I 12-15 20:23:13.330 core:190] lxqt-runner-git: updated to 1.1.07c93bd4c4cc62cef9ffae85421cbed91d1de554
[I 12-15 20:23:13.356 core:190] libfm-qt-git: updated to 1.1.ed1f91db0d92bd0a813d795bbec79aab6f0753b2
[I 12-15 20:23:13.404 core:190] pcmanfm-qt-git: updated to 1.1.0dc3bb3cd35571918a01cdded5caaf20ccb4a404
[I 12-15 20:23:13.483 core:190] lxqt-sudo-git: updated to 1.1.8de11dcb2dbd53abf44e949f556ca104ea3b9198
[I 12-15 20:23:13.497 core:190] lxqt-session-git: updated to 1.1.7895162bdc30fc648d1406e7a2ef591a26f0a524
[I 12-15 20:23:13.548 core:190] julia-git: updated to 1.1.01fe7c26986723a10cb9c97ef903b7d14f832fe6
[I 12-15 20:23:13.598 core:190] lxqt-panel-git: updated to 1.1.c6f910fda61cd78109271d99edc56ac11b53b335
[I 12-15 20:23:13.835 core:190] libqtxdg-git: updated to 1.1.1a52e5cb5203a51c1a450bb7fd7268e22fc7c03b
[I 12-15 20:23:13.961 core:190] spim-svn: updated to 1.1.720
[I 12-15 20:23:13.969 core:190] lxqt-archiver-git: updated to 1.1.ae87dac376bc4c16fc7ca1ffd2881bf3175068db
[I 12-15 20:23:14.169 core:190] obconf-qt-git: updated to 1.1.eb5851386b334dbc70a31ae875c9a40ee98c01a5
[I 12-15 20:23:14.217 core:190] lxqt-qtplugin-git: updated to 1.1.bbd3d1d7834a9c8a5b23333e2ff204b41e2c0e2e
[I 12-15 20:23:14.363 core:190] lxqt-config-git: updated to 1.1.cb5aa9da01b49bc6cb79cf8c80fb50c7b098cc72
[I 12-15 20:23:14.376 core:190] lxqt-about-git: updated to 1.1.ebcce6cc121936843097c7d024592d2e41a36be2
[I 12-15 20:23:14.436 core:190] lxqt-powermanagement-git: updated to 1.1.db4c079acc0914543e7dfd9e2e81abe3a8f7ab94
[I 12-15 20:23:14.449 core:190] lxqt-policykit-git: updated to 1.1.cb89036cd777e2bd1298d20794ea328b073c43e1
[I 12-15 20:23:15.041 core:190] version-control-tools-hg: updated to 1.1.4285740c40de @

Commits for the first 20 packages are all a977db28e5a50fd10f0ebe627e8b24c531b9574c. This is the correct commit for lximage-qt-git - lxqt/lximage-qt@a977db2.

support include_tags_pattern for gitlab

hi
gitlab does not have a include_tags_pattern configuration like github does. The configuration is useful to ignore e.g alpha/beta/rc versions.
Can this configuration also be added to gitlab (and probably all other git based check backends)?

version comparison gives bad results

Maybe we should remove the logic and use equation to judge if it's updated? cc @felixonmars @farseerfc

See also archlinuxcn/repo#351 and #40.

This logic was implemented three years ago by a5aad83.

pip3 install nvchecker error on Linux

main.ConfigurationError: Could not run curl-config: [Errno 2] No such file or directory: 'curl-config': 'curl-config'

remove deprecated `include_tags_pattern` and `ignored_tags` options

Use list options instead. We need to remove them from github, gitlab and bitbucket sources.

Hide traceback for network issues

Please replace the traceback with something more compact.

$ nvchecker --tries 9 ~/source.ini
[I 04-28 11:38:31.002 core:201] nextcloud: updated from 15.0.7-1 to 16.0.0-1
[I 04-28 11:38:31.204 core:201] nginx-mainline-mod-dav-ext: updated from 3.0.0-4 to 3.0.0-5
[W 04-28 11:38:49.079 core:163] nginx-mainline: failed, retrying
    Traceback (most recent call last):
      File "/usr/lib/python3.7/site-packages/nvchecker/core.py", line 158, in worker
        name, conf, keyman=self.keymanager)
      File "/usr/lib/python3.7/site-packages/nvchecker/get_version.py", line 90, in get_version
        version = await func(name, conf, **kwargs)
      File "/usr/lib/python3.7/site-packages/nvchecker/source/archpkg.py", line 17, in get_version
        async with session.get(URL, params={"name": pkg}) as res:
      File "/usr/lib/python3.7/site-packages/nvchecker/source/tornado_httpclient.py", line 62, in __aenter__
        return await to_asyncio_future(client.fetch(self.req))
    tornado.curl_httpclient.CurlError: HTTP 599: Connection timed out after 20000 milliseconds
[W 04-28 11:38:49.100 core:163] php: failed, retrying
    Traceback (most recent call last):
      File "/usr/lib/python3.7/site-packages/nvchecker/core.py", line 158, in worker
        name, conf, keyman=self.keymanager)
      File "/usr/lib/python3.7/site-packages/nvchecker/get_version.py", line 90, in get_version
        version = await func(name, conf, **kwargs)
      File "/usr/lib/python3.7/site-packages/nvchecker/source/archpkg.py", line 17, in get_version
        async with session.get(URL, params={"name": pkg}) as res:
      File "/usr/lib/python3.7/site-packages/nvchecker/source/tornado_httpclient.py", line 62, in __aenter__
        return await to_asyncio_future(client.fetch(self.req))
    tornado.curl_httpclient.CurlError: HTTP 599: Connection timed out after 20016 milliseconds
[W 04-28 11:38:49.103 core:163] postgresql-old-upgrade: failed, retrying
    Traceback (most recent call last):
      File "/usr/lib/python3.7/site-packages/nvchecker/core.py", line 158, in worker
        name, conf, keyman=self.keymanager)
      File "/usr/lib/python3.7/site-packages/nvchecker/get_version.py", line 90, in get_version
        version = await func(name, conf, **kwargs)
      File "/usr/lib/python3.7/site-packages/nvchecker/source/archpkg.py", line 17, in get_version
        async with session.get(URL, params={"name": pkg}) as res:
      File "/usr/lib/python3.7/site-packages/nvchecker/source/tornado_httpclient.py", line 62, in __aenter__
        return await to_asyncio_future(client.fetch(self.req))
    tornado.curl_httpclient.CurlError: HTTP 599: Connection timed out after 20012 milliseconds
[I 04-28 11:38:49.486 core:201] nginx-mainline: updated from 1.15.11-1 to 1.15.12-1

Tests fail when not connected to internet

While bundling this for openSUSE, the tests are run inside an isolated VM, disconnected from the internet.

Some mechanism is needed to get the rest of the tests to pass, and these ones to be skipped.

Options include

• a pytest mark on internet tests, or non-internet tests, so pytest -k notnet or pytest -k 'not net can be used
• split the tests into two subdirectories, one for infrastructure code and the other for resolvers needing network
• detect network is missing in a pytest fixture and dynamically skip those tests
• adding mocks for the tests

A quick win for tests/test_regex.py is to use the pytest plugin for httpbin, which provides a local instance of httpbin that works in an isolated VM.

Probably other approaches exist too.

Atm I am doing

mkdir tests_ok
mv tests/test_cache.py \
   tests/test_cmd.py \
   tests/test_manual.py \
   tests/test_substitute.py \
   tests/conftest.py \
   tests_ok
pytest tests_ok/

Project at https://build.opensuse.org/package/show/home:jayvdb:py-new/python-nvchecker

Unable to run nvchecker executable

I'm having difficulty trying to install the executable by following these steps on macOS 10.13.4 using python --version Python 3.6.2. Note that I haven't used Python for a while.

• Clone the repo

git clone https://github.com/lilydjwg/nvchecker;
cd nvchecker;

* Build and install the package

python setup.py install

The above creates a build/ directory (containing bdist.macosx-10.12-x86_64 and folder lib) and dist/ directory (containing nvchecker-1.2.dev0-py3.6.egg)

* Run the package (Attempt 1)

./nvchecker --help
-bash: ./nvchecker: is a directory

* Run the package (Attempt 2)

$ python ./build/lib/nvchecker/main.py
Traceback (most recent call last):
File "./build/lib/nvchecker/main.py", line 10, in
from .lib import notify
ModuleNotFoundError: No module named 'main.lib'; 'main' is not a package

tornado版本的aur链接访问出错

如题，运行会出下述错误

[E 08-24 21:10:14.158 ioloop:633] Exception in callback functools.partial(<function wrap.<locals>.wrapped at 0x7f947bdc9f28>, HTTPResponse(_body=None,buffer=None,code=599,effective_url='https://aur.archlinux.org/rpc/?v=5&type=info&arg[]=bazel',error=OSError(97, 'Address
family not supported by protocol'),headers=<tornado.httputil.HTTPHeaders object at 0x7f9478beaa90>,reason='Unknown',request=<tornado.httpclient.HTTPRequest object at 0x7f947bddc940>,request_time=0.3073916435241699,time_info={}))
    Traceback (most recent call last):
      File "/usr/lib/python3.5/site-packages/tornado/ioloop.py", line 604, in _run_callback
        ret = callback()
      File "/usr/lib/python3.5/site-packages/tornado/stack_context.py", line 343, in wrapped
        raise_exc_info(exc)
      File "<string>", line 3, in raise_exc_info
      File "/usr/lib/python3.5/site-packages/tornado/stack_context.py", line 355, in _handle_exception
        if tail.exit(*exc):
      File "/usr/lib/python3.5/site-packages/tornado/stack_context.py", line 186, in exit
        return self.exception_handler(type, value, traceback)
      File "/usr/lib/python3.5/site-packages/nvchecker/core.py", line 100, in _handle_exception
        raise value.with_traceback(traceback)
      File "/usr/lib/python3.5/site-packages/tornado/stack_context.py", line 314, in wrapped
        ret = fn(*args, **kwargs)
      File "/usr/lib/python3.5/site-packages/nvchecker/source/aur.py", line 21, in _aur_done
        raise res.error
      File "/usr/lib/python3.5/site-packages/tornado/stack_context.py", line 314, in wrapped
        ret = fn(*args, **kwargs)
      File "/usr/lib/python3.5/site-packages/tornado/tcpclient.py", line 130, in on_timeout
        self.try_connect(iter(self.secondary_addrs))
      File "/usr/lib/python3.5/site-packages/tornado/tcpclient.py", line 96, in try_connect
        future = self.connect(af, addr)
      File "/usr/lib/python3.5/site-packages/tornado/tcpclient.py", line 180, in _create_stream
        stream = IOStream(socket.socket(af),
      File "/usr/lib/python3.5/socket.py", line 134, in __init__
        _socket.socket.__init__(self, family, type, proto, fileno)
    OSError: [Errno 97] Address family not supported by protocol

装了pycurl后一切正常，但是pycurl是optional dependency啊

Feature request: 可以追踪的上游增加manual项

当然就是看到manual就skip了。。

不过这样的话，nvchecker.ini中的每一项就与源里的文件夹一一对应了
可以在pre-commit中加上对这一点的检测

还有一点可以检测的就是 不是manual的必须要有lilac.py

Set HTTP client's pool size to `max_concurrent`