prototype-x/blacklist-rkn-tool

Python3 скрипт для работы с реестром запрещенных сайтов http://vigruzki.rkn.gov.ru/

blacklist-rkn-tool's Issues

метод cleaner вызывает эксепшн

вызов метода

        if self.check_diff():
            self.cleaner()
            return 1

приводит к
pymysql.err.NotSupportedError: (1235, "This version of MariaDB doesn't yet support 'LIMIT & IN/ALL/ANY/SOME subquery'")

"IndexError: list index out of range" in _url_dedup_sql on finish.

On console:

# /home/rkn/blacklist-rkn-tool/bl-rkn.py --dump
Traceback (most recent call last):
  File "/home/rkn/blacklist-rkn-tool/bl-rkn.py", line 653, in <module>
    main()
  File "/home/rkn/blacklist-rkn-tool/bl-rkn.py", line 649, in main
    BlrknCLI()
  File "/home/rkn/blacklist-rkn-tool/bl-rkn.py", line 581, in __init__
    self._get_dump()
  File "/home/rkn/blacklist-rkn-tool/bl-rkn.py", line 609, in _get_dump
    message = self.report.statistics_show()
  File "/home/rkn/blacklist-rkn-tool/bl-rkn.py", line 104, in statistics_show
    url_add_sql = self._url_dedup_sql(diff, 'ignore', 1)
  File "/home/rkn/blacklist-rkn-tool/bl-rkn.py", line 437, in _url_dedup_sql
    url_diff_sql = URL.select(fn.Distinct(URL.url)).where(URL.add == self.idx_list[diff])
IndexError: list index out of range

In logs:

2018-04-04 23:51:48,970  Starting script.
2018-04-04 23:51:49,160  Check database: PostgreSQL Ok
2018-04-04 23:51:51,078  Current versions: webservice: 3, dump: 2.2, doc: 4
2018-04-04 23:51:51,079  New webservice: 3.1
2018-04-04 23:51:51,083  New dumpFormatVersion: 2.3
2018-04-04 23:51:51,086  New docVersion: 4.8
2018-04-04 23:51:53,789  Send email from ... to ...
2018-04-04 23:51:53,790  Current webservice:3
New webservice: 3.1

Current dumpFormatVersion: 2.2
New dumpFormatVersion: 2.3

Current docVersion: 4
New docVersion: 4.8


2018-04-04 23:51:53,790  Check if dump.xml has updates since last sync.
2018-04-04 23:51:53,794  Current date: lastDumpDate: 2012-01-01 00:00:00, lastDumpDateUrgently: 2012-01-01 00:00:00
2018-04-04 23:51:53,794  Last date: lastDumpDate: 2018-04-04 22:56:00, lastDumpDateUrgently: 2018-04-04 22:56:00
2018-04-04 23:51:53,794  New dump is available.
2018-04-04 23:51:53,798  Sending request.
2018-04-04 23:51:55,735  Checking request status.
2018-04-04 23:51:55,735  Got code 245874f427a272a4deab875eba47fe24
2018-04-04 23:51:55,741  Save code in History
2018-04-04 23:51:55,744  Waiting for a 90 sec.
2018-04-04 23:53:25,823  Trying to get result...
2018-04-04 23:53:28,747  Got a dump ver. 2.3 for the ...
2018-04-04 23:53:28,811  Downloaded dump 6826514 bytes, MD5 hashsum: 53ab1e693f36707235bb42642e8f3941
2018-04-04 23:53:28,811  Unpacking.
2018-04-04 23:53:29,314  dump.xml already exists.
2018-04-04 23:53:30,345  Got updateTime: 1522893360.
2018-04-04 23:53:30,347  Got updateTimeUrgently: 1522893360.
2018-04-04 23:53:30,908  New Item, IP, Domain, URL id: 786432.
2018-04-04 23:53:31,020  New Item, IP, Domain, URL id: 786433.

...

2018-04-05 01:59:41,953  New Item, IP, Domain, URL id: 786430.
2018-04-05 01:59:42,059  New Item, IP, Domain, URL id: 786431.
2018-04-05 02:02:44,339  Dump cleaner run
2018-04-05 02:02:44,341  Item deleted: 0
2018-04-05 02:02:44,342  IP deleted: 0
2018-04-05 02:02:44,343  Domain deleted: 0
2018-04-05 02:02:44,345  URL deleted: 0
2018-04-05 02:02:44,347  History deleted: 1

Core.get_request ignores DumpPath= value

Core.get_request uses hardcoded "/dumps/" value instead of DumpPath option.

Ошибка на пустой базе

Похоже, что не полная инициализация базы. На пустой базе вылетает эксепшн при попытке чтения Dump.get(Dump.param == 'webServiceVersion').value:

python3 bl-rkn.py --dump
Traceback (most recent call last):
  File "/home/maxim/.pyenv/versions/3.5.2/lib/python3.5/site-packages/peewee.py", line 3097, in get
    return next(clone.execute())
  File "/home/maxim/.pyenv/versions/3.5.2/lib/python3.5/site-packages/peewee.py", line 2271, in next
    obj = self.iterate()
  File "/home/maxim/.pyenv/versions/3.5.2/lib/python3.5/site-packages/peewee.py", line 2253, in iterate
    raise StopIteration
StopIteration

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "bl-rkn.py", line 631, in <module>
    main()
  File "bl-rkn.py", line 627, in main
    BlrknCLI()
  File "bl-rkn.py", line 561, in __init__
    self._get_dump()
  File "bl-rkn.py", line 569, in _get_dump
    srv_msg = self.dump.check_service_upd()
  File "/home/maxim/Dropbox/Python/ver3/blacklist-rkn-tool/core.py", line 42, in check_service_upd
    Dump.get(Dump.param == 'webServiceVersion').value,
  File "/home/maxim/.pyenv/versions/3.5.2/lib/python3.5/site-packages/peewee.py", line 4795, in get
    return sq.get()
  File "/home/maxim/.pyenv/versions/3.5.2/lib/python3.5/site-packages/peewee.py", line 3101, in get
    % self.sql())
db.DumpDoesNotExist: Instance matching query does not exist:
SQL: SELECT `t1`.`param`, `t1`.`value` FROM `dump` AS t1 WHERE (`t1`.`param` = %s)
PARAMS: ['webServiceVersion']

zapretinfo.ZapretInfoException: No request file

Это как-то лечится или больше не работает скриптик?