release-engineering / kobo Goto Github PK
View Code? Open in Web Editor NEWA pile of python modules used by Red Hat release engineering to build their tools.
License: GNU Lesser General Public License v2.1
A pile of python modules used by Red Hat release engineering to build their tools.
License: GNU Lesser General Public License v2.1
sys.exc_type
has been deprecated a long time ago and was removed in Python 3 which causes this error:
response = xmlrpclib.dumps(
> xmlrpclib.Fault(1, "%s: %s" % (sys.exc_type.__name__, sys.exc_info()[1])),
allow_none=self.allow_none, encoding=self.encoding)
E AttributeError: module 'sys' has no attribute 'exc_type'
venv3/lib/python3.7/site-packages/kobo/django/xmlrpc/dispatcher.py:112: AttributeError
Currently, we release kobo to PyPI as a single egg containing all kobo components. This doesn't make much sense since kobo consists of at least the following components:
These are generally deployed to different environments and it doesn't make sense to have them all packaged together as one "all or nothing" egg.
It means that we're discouraged from defining any requirements in the kobo egg, since anyone using that may be forced to pull in a lot more dependencies than they need. For example, somebody using some of the kobo utility classes shouldn't have to install Django; and it should not be necessary to install Django to (for example) create a container image for a kobo worker via pip.
We should instead be shipping separate eggs, probably at least 4 (plus one for backwards-compat), and these should define their dependencies correctly.
For example, we could release eggs:
_utf8_chunk
and _tail fails
raises an error:
TypeError: memoryview: a bytes-like object is required, not 'str'
These functions also have different returns on Python 2 and 3. For py2 it returns str
for py3 it returns bytes
, this must be due to unicode errors.
run
reads output in chunks. If a unicode character happens to be split between two chunks, decoding fails and there is a crash.
This bug triggered a failure in Fedora 31 updates-testing compose: https://pagure.io/releng/failed-composes/issue/237
It would be convenient if it was possible to concatenate lists in configuration files. Particularly if one file is included in another, it would be nice to be able to define common base in one file and append something in another.
Proposed syntax:
a = [1, 2, 3]
a = a + [4]
There is already a precedent for having operators in the configuration: it supports string interpolation with %
operator.
kobo should provide a command for resetting the priority of an already-scheduled task. The use-case is to adjust the relative priorities of tasks so that desired tasks are executed first, when the queue is full.
We already have "create-task". Suggestion: this should be a new "modify-tasks" counterpart. Only certain fields of a task could be modified; initially, at least the priority.
Example:
Usage: <command> modify-tasks task_id [task_id...]
Options:
--priority=PRIORITY priority
Currently if you try to set key starting with underscore, KeyError
is raised.
https://github.com/release-engineering/kobo/blob/master/kobo/conf.py#L92-L95
This is confusing for users encountering this exception, as the documentation says:
Raised when a mapping (dictionary) key is not found in the set of existing keys.
I assume there is some reasoning why these keys are rejected, but I suggest using different exception.
Passing a Context or a RequestContext is still possible when the template is loaded by a DjangoTemplates backend but itβs deprecated and isn't supported since Django 1.10.
This code
force_list({'a': True, 'b': True}.keys())
should return ["a", "b"]
but instead it returns [dict_keys(['a', 'b'])]
(list of dict_items).
This only happens on Python 3.
The code for interrupt_tasks and timeout_tasks are very similar.
def timeout_tasks(request, task_list):
result = True
for task_id in task_list:
task = Task.objects.get_and_verify(task_id=task_id, worker=request.worker)
if task:
try:
task.timeout_task(recursive=True)
except:
raise
result = False
return result
The strange part is the result
variable, this function only returns True
, if for some reason task.timeout_task raises an error result = False
is never reached because it re-raises the error.
Some options available for fixing this:
worker log:
2016-04-19 09:46:42 [INFO ] Waking up task #24993 [...].
2016-04-19 09:46:42 [INFO ] Task #24993 exited with status 0
2016-04-19 09:46:42 [INFO ] Task has finished: 24993
2016-04-19 09:46:42 [INFO ] kill_group: Process (pgrp 9253) exited
2016-04-19 09:47:03 [WARNING ] Closing interrupted tasks: [24993]
2016-04-19 09:47:03 [ERROR ] <Fault 1: Traceback (most recent call last):
File "/usr/lib/python2.6/site-packages/kobo/django/xmlrpc/dispatcher.py", line 95, in _marshaled_dispatch
response = self._dispatch(method, params)
File "/usr/lib64/python2.6/SimpleXMLRPCServer.py", line 418, in _dispatch
return func(*params)
File "/usr/lib/python2.6/site-packages/kobo/hub/decorators.py", line 24, in _new_func
return func(request, *args, **kwargs)
File "/usr/lib/python2.6/site-packages/kobo/hub/xmlrpc/worker.py", line 111, in interrupt_tasks
task.interrupt_task(recursive=True)
File "/usr/lib/python2.6/site-packages/django/db/transaction.py", line 371, in inner
return func(*args, **kwargs)
File "/usr/lib/python2.6/site-packages/kobo/hub/models.py", line 737, in interrupt_task
task.interrupt_task(recursive=True)
File "/usr/lib/python2.6/site-packages/django/db/transaction.py", line 371, in inner
return func(*args, **kwargs)
File "/usr/lib/python2.6/site-packages/kobo/hub/models.py", line 733, in interrupt_task
raise Exception("Cannot interrupt task %d, state is %s" % (self.id, self.state))
Exception: Cannot interrupt task 24994, state is 3
At 9:42, process of 24993 finished with 0 so kobo correctly closed the task and cleaned up:
2016-04-19 09:46:42 [INFO ] Task #24993 exited with status 0
2016-04-19 09:46:42 [INFO ] Task has finished: 24993
2016-04-19 09:46:42 [INFO ] kill_group: Process (pgrp 9253) exited
I believe that here comes the race condition. In next cycle, we can see
2016-04-19 09:47:03 [WARNING ] Closing interrupted tasks: [24993]
which is incorrect since task is already closed. When checking code:
if task_info["state"] == TASK_STATES["OPEN"] and task_info["id"] not in self.pid_dict:
# an interrupted task appears to be open, but running task manager doesn't track it in it's pid list
# this happens after a power outage, for example
interrupted_list.append(task_info["id"])
finished_tasks.add(task_info["id"])
continue
this condition is evaluated as true which means that the task is still open even though it's suppose to be closed (because it doesn't have an entry in pid_dict
)
So, why it's not closed? Looking at code, this is how tasks are supposed to be closed
try:
task.run()
except ShutdownException:
...
thread.stop()
if failed:
hub.worker.fail_task(task.task_id, task.result)
else:
hub.worker.close_task(task.task_id, task.result)
Since that code contains very little logging, it's hard to figure out what could possibly go wrong.
Please update kobo release on PyPI to match upstream one.
Latest kobo release on PyPI: 0.3.8
Latest upstream release: 0.5.2
StateEnumField is broken:
ERROR: test_to_python (__main__.TestStateEnumField)
----------------------------------------------------------------------
Traceback (most recent call last):
File "test_types.py", line 206, in setUp
self.field = StateEnumField(self.state_enum, default='NEW')
File "/home/dmach/github/kobo-github/kobo/django/fields.py", line 48, in __init__
super(StateEnumField, self).__init__(*args, **kwargs)
File "/home/dmach/github/kobo-github/devel/lib/python2.7/site-packages/django/db/models/fields/__init__.py", line 162, in __init__
self.choices = choices or []
TypeError: unbound method _set_choices() must be called with StateChoiceFormField instance as first argument (got StateEnumField instance instead)
The pkgset.FileCache
object behaves like a dict, but misses __setitem__
method. This means that a user needs to access file_cache
member. This leaks abstraction details.
With Django 1.8, starting with an empty database, with INSTALLED_APPS containing 'django.contrib.auth' and 'kobo.django.auth', manage.py migrate will fail with:
ERRORS:
auth.User.groups: (fields.E304) Reverse accessor for 'User.groups' clashes with reverse accessor for 'User.groups'.
HINT: Add or change a related_name argument to the definition for 'User.groups' or 'User.groups'.
auth.User.user_permissions: (fields.E304) Reverse accessor for 'User.user_permissions' clashes with reverse accessor for 'User.user_permissions'.
HINT: Add or change a related_name argument to the definition for 'User.user_permissions' or 'User.user_permissions'.
kobo_auth.User.groups: (fields.E304) Reverse accessor for 'User.groups' clashes with reverse accessor for 'User.groups'.
HINT: Add or change a related_name argument to the definition for 'User.groups' or 'User.groups'.
kobo_auth.User.user_permissions: (fields.E304) Reverse accessor for 'User.user_permissions' clashes with reverse accessor for 'User.user_permissions'.
HINT: Add or change a related_name argument to the definition for 'User.user_permissions' or 'User.user_permissions'.
It is possible to silence this by use of SILENCED_SYSTEM_CHECKS, but then migrations will fail because both apps try to create the table for the Users model.
It's unclear how to make this work; the migration in kobo.django.auth doesn't create related tables such as Groups. So, disabling the migrations from django.contrib.auth doesn't work (the User.username field will be left at the default 30 chars, defeating the purpose of kobo_auth's User) and disabling the migrations from django.contrib.auth doesn't work (kobo's migrations will fail to create necessary tables).
The current latest version of kobo on pypi is 0.5.2 which has some incompatible code with python3, could you upload the latest version to pypi?
We can install the latest version via pip with github link, but it's a little inconvenient to maintain the setup.py to parsing the requirements in our projects.
There is no kobo-0.5.1
git tag despite such version of kobo was released in Fedora. It is also not clear to me why the last version is tagged 0.5.2
instead of kobo-0.5.2
as all the previous releases. Please keep the release tags consistent.
Hello,
I'd expect that if rpmlib.parse_nvra() would call strip() or any other tool to get rid of possible newline character on the end of string.
I expect that "\n" isn't valid arch anyway.
Execution:
lkocman@xxx> python -c 'from kobo.shortcuts import run; import kobo.rpmlib; print kobo.rpmlib.parse_nvra(run("rpm -q rhsm-tools")[1])'
Actual results:
{'src': False, 'name': 'rhsm-tools', 'epoch': '', 'version': '1.26', 'release': '1.el6eng', 'arch': 'noarch\n'}
lkocman@rcm-dev:tmp>
Expected results (execution is enforcing .strip() on rpm output)
lkocman@rcm-dev:tmp> python -c 'from kobo.shortcuts import run; import kobo.rpmlib; print kobo.rpmlib.parse_nvra(run("rpm -q rhsm-tools")[1].strip())'
{'src': False, 'name': 'rhsm-tools', 'epoch': '', 'version': '1.26', 'release': '1.el6eng', 'arch': 'noarch'}
lkocman@rcm-dev:tmp>
TaskManager has a custom implementation for get_query_set
that returns only non-achieved Tasks.
The problem is, it does not filter at all in any version after 1.6 because this function was renamed as it shows here.
This means that if we call Task.objects.all()
it does not filter achieved tasks and this can result in errors.
If we rename the funcion it could work, but it needs to be tested.
Though not documented, the intended behavior of kobo.shortcuts.run seems to be:
But on Python 3, this is broken in every text mode case other than universal_newlines. This is because the code has special cases to deal with the universal_newlines keyword argument, but in Python 3 this is one of only several arguments for enabling text mode (and it's not the preferred one).
(on Python 3)
>>> from kobo.shortcuts import run
>>> run('echo Hello')
(0, b'Hello\n')
>>> run('echo Hello', universal_newlines=True)
(0, 'Hello\n')
>>> run('echo Hello', text=True)
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/home/rmcgover/src/kobo/kobo/shortcuts.py", line 338, in run
lines = out_reader.read(buffer_size)
File "/home/rmcgover/dev/python3/lib64/python3.7/codecs.py", line 500, in read
data = self.bytebuffer + newdata
TypeError: can't concat str to bytes
>>> run('echo Hello', encoding='utf-8')
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/home/rmcgover/src/kobo/kobo/shortcuts.py", line 338, in run
lines = out_reader.read(buffer_size)
File "/home/rmcgover/dev/python3/lib64/python3.7/codecs.py", line 500, in read
data = self.bytebuffer + newdata
TypeError: can't concat str to bytes
>>> run('echo Hello', errors='strict')
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/home/rmcgover/src/kobo/kobo/shortcuts.py", line 338, in run
lines = out_reader.read(buffer_size)
File "/home/rmcgover/dev/python3/lib64/python3.7/codecs.py", line 500, in read
data = self.bytebuffer + newdata
TypeError: can't concat str to bytes
In cases 3-5 above, exceptions are raised.
In cases 3-5 above, output is returned as a str
.
It happens when a running task has a child which is already in closed state and child should be interrupted. Interrupt happens if new patch is added to scan, so it tries to interrupt the parent repeatedly. However, the parent is trying to shut down the child which has already finished (esp. in closed state) and query is looking for opened task, which throws an exception. I think the best idea is to add condition when the query finds no results, and if the state of the task is in finished state, do not throw exception (maybe write something to log).
Django 1.11 is next version with LTS and the last version with Python 2.7 support so Kobo should support it. Also, Django 1.8 EOL will be in April 2018.
https://www.djangoproject.com/download/
Related to: #2
There are many modules in this library, some of which bring lots of dependencies like koji or django. It would be nice to split the library into multiple smaller libraries so that people can install only what they need.
This is already done to some level in Fedora RPM packaging with subpackages:
kobo
kobo-admin
python3-kobo
python3-kobo-admin
python3-kobo-client
python3-kobo-django
python3-kobo-hub
python3-kobo-rpmlib
python3-kobo-worker
For example in Pungi we use only these modules: log, pkgset, rpmlib, shortcuts and threads. No Django needed, yet when installing with pip during development, it gets pulled in.
Function shutdown_worker
from kobo/hub/xmlrpc/client.py
allows passing None as the worker_name
parameter.
If my guess is correct, this means any worker can catch the "shutdown task" and be killed which is not good.
The method cancel_subtasks
has the following line:
result &= task.cancel_task()
cancel_task
does not return a boolean, actually, it does not return anything.
This causes the error:
TypeError: unsupported operand type(s) for &=: 'bool' and 'NoneType'
If the command fails, an error message is printed to stderr
. If a program using the function wants to run in quiet mode, there is no easy way to silence it.
The message could for example only be printed when can_fail
is True
. Otherwise an exception is raised with the same error message, so calling code can print it as it sees fit.
Just two questions as per title. Thank you.
From kobo hub, if you request a log file which doesn't exist and one of the following is true:
Then you'll get a successful but empty response like this:
$ curl -v 'http://pub-dev-hub/pub/task/56/log/notexist.htm'
* About to connect() to pub-dev-hub port 80 (#0)
* Trying 172.17.0.3...
* Connected to pub-dev-hub (172.17.0.3) port 80 (#0)
> GET /pub/task/56/log/notexist.htm HTTP/1.1
> User-Agent: curl/7.29.0
> Host: pub-dev-hub
> Accept: */*
>
< HTTP/1.1 200 OK
< Date: Mon, 04 Sep 2017 23:46:54 GMT
< Server: Apache/2.2.15 (Red Hat)
< Vary: Cookie
< Content-Length: 0
< Connection: close
< Content-Type: text/html; charset=UTF-8
<
* Closing connection 0
Expected behavior: should get a 404 response.
When a task is running in a kobo worker, it's the responsibility of the LoggingThread to send log messages back to the hub.
This is done by the following loop:
def run(self):
"""Send queue content to hub."""
while self._running or not self._queue.empty() or self._send_data:
if self._queue.empty():
self._event.wait(5)
self._event.clear()
while True:
try:
self._send_data += self._queue.get_nowait()
except six.moves.queue.Empty:
break
if not self._send_data:
continue
now = int(time.time())
if self._running and len(self._send_data) < 1200 and now - self._send_time < 5:
continue
if isinstance(self._send_data, six.text_type):
self._send_data = self._send_data.encode('utf-8')
try:
self._hub.upload_task_log(StringIO(self._send_data), self._task_id, "stdout.log", append=True)
self._send_time = now
self._send_data = ""
except Fault:
continue
Problem: if self._hub.upload_task_log raises an error (other than an XML-RPC Fault), then the logging thread simply stops. However, the main thread of the task being executed doesn't necessarily stop.
The end result of this is that if the kobo hub has a temporary outage while a task is in progress, that task might continue executing but all logs after the outage would be lost.
It would be better if the LoggingThread kept retrying the task log uploads, for as long as the task's main thread is alive.
Worker.timeout_task
interrupt subtasks if recursive is True and the task has subtasks.
The subtasks are set to INTERRUPTED
but they should be set to TIMEOUT
.
This is the current code of timeout_task:
def timeout_task(self, recursive=True):
"""Set the task state to timeout."""
try:
self.__lock(self.worker_id, new_state=TASK_STATES["TIMEOUT"], initial_states=(TASK_STATES["OPEN"], ))
except (MultipleObjectsReturned, ObjectDoesNotExist):
raise Exception("Cannot interrupt task %d, state is %s" % (self.id, self.get_state_display()))
if recursive:
for task in self.subtasks():
task.interrupt_task(recursive=True)
self.logs.gzip_logs()
It's a very silly mistake, looks like it was copy and paste and someone forgot to change it.
set_task_weight
from kobo/hub/xmlrpc/worker.py
fails because it uses task.setWeight
which is not implemented on Task
model.
LoggingThread
fails on Python 3 due to encoding errors.
It fails on Python 3 because string.encode returns a bytes-like object and StringIO should receive an str object.
Traceback (most recent call last):
File "/usr/lib64/python3.6/threading.py", line 916, in _bootstrap_inner
self.run()
File "/kobo/kobo/worker/logger.py", line 54, in run
self._hub.upload_task_log(StringIO(self._send_data), self._task_id, "stdout.log", append=True)
TypeError: initial_value must be str or None, not bytes
kobo depends on python-krbV, which is python2-only and dead. We need to use something else to be able to use dependent functionality on Python 3.
Function main_loop
has this code:
try:
log_file = conf.get("LOG_FILE", None)
logger = logging.Logger("TaskManager")
logger.setLevel(logging.DEBUG)
if log_file:
log_level = logging._levelNames.get(conf.get("LOG_LEVEL", "DEBUG").upper())
kobo.log.add_rotating_file_logger(logger, log_file, log_level=log_level)
tm = TaskManager(conf=conf, logger=logger)
except Exception as ex:
raise
sys.stderr.write("Error initializing TaskManager: %s\n" % ex)
sys.exit(1)
If some exception occurs it's re-raised instead of executing the code above the raise
line. This code is unreachable and it's difficult to know what should be the behavior.
Getting the following error when I do python manage.py migrate
Traceback (most recent call last):
File "manage.py", line 13, in <module>
execute_from_command_line(sys.argv)
File "/home/akhilsp/.local/share/virtualenvs/Nitrate-9dTMyaLq/lib/python3.6/site-packages/django/core/management/__init__.py", line 381, in execute_from_command_line
utility.execute()
File "/home/akhilsp/.local/share/virtualenvs/Nitrate-9dTMyaLq/lib/python3.6/site-packages/django/core/management/__init__.py", line 375, in execute
self.fetch_command(subcommand).run_from_argv(self.argv)
File "/home/akhilsp/.local/share/virtualenvs/Nitrate-9dTMyaLq/lib/python3.6/site-packages/django/core/management/base.py", line 316, in run_from_argv
self.execute(*args, **cmd_options)
File "/home/akhilsp/.local/share/virtualenvs/Nitrate-9dTMyaLq/lib/python3.6/site-packages/django/core/management/base.py", line 353, in execute
output = self.handle(*args, **options)
File "/home/akhilsp/.local/share/virtualenvs/Nitrate-9dTMyaLq/lib/python3.6/site-packages/django/core/management/base.py", line 83, in wrapped
res = handle_func(*args, **kwargs)
File "/home/akhilsp/.local/share/virtualenvs/Nitrate-9dTMyaLq/lib/python3.6/site-packages/django/core/management/commands/migrate.py", line 82, in handle
executor = MigrationExecutor(connection, self.migration_progress_callback)
File "/home/akhilsp/.local/share/virtualenvs/Nitrate-9dTMyaLq/lib/python3.6/site-packages/django/db/migrations/executor.py", line 18, in __init__
self.loader = MigrationLoader(self.connection)
File "/home/akhilsp/.local/share/virtualenvs/Nitrate-9dTMyaLq/lib/python3.6/site-packages/django/db/migrations/loader.py", line 49, in __init__
self.build_graph()
File "/home/akhilsp/.local/share/virtualenvs/Nitrate-9dTMyaLq/lib/python3.6/site-packages/django/db/migrations/loader.py", line 206, in build_graph
self.load_disk()
File "/home/akhilsp/.local/share/virtualenvs/Nitrate-9dTMyaLq/lib/python3.6/site-packages/django/db/migrations/loader.py", line 108, in load_disk
migration_module = import_module(migration_path)
File "/home/akhilsp/.local/share/virtualenvs/Nitrate-9dTMyaLq/lib/python3.6/importlib/__init__.py", line 126, in import_module
return _bootstrap._gcd_import(name[level:], package, level)
File "<frozen importlib._bootstrap>", line 994, in _gcd_import
File "<frozen importlib._bootstrap>", line 971, in _find_and_load
File "<frozen importlib._bootstrap>", line 955, in _find_and_load_unlocked
File "<frozen importlib._bootstrap>", line 665, in _load_unlocked
File "<frozen importlib._bootstrap_external>", line 678, in exec_module
File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
File "/home/akhilsp/.local/share/virtualenvs/Nitrate-9dTMyaLq/lib/python3.6/site-packages/kobo/django/xmlrpc/migrations/0001_initial.py", line 8, in <module>
class Migration(migrations.Migration):
File "/home/akhilsp/.local/share/virtualenvs/Nitrate-9dTMyaLq/lib/python3.6/site-packages/kobo/django/xmlrpc/migrations/0001_initial.py", line 22, in Migration
('user', models.ForeignKey(blank=True, to=settings.AUTH_USER_MODEL, null=True)),
TypeError: __init__() missing 1 required positional argument: 'on_delete'
I was able to apply the migrations after editing /kobo/django/xmlrpc/migrations/0001_initial.py
by adding on_delete=models.CASCADE
to line number 22.
Is there any work around without editing this?
In TaskManager class, here's the body of fork_task currently:
def fork_task(self, task_info):
self.log_debug("Forking task %s" % self._task_str(task_info))
pid = os.fork()
if pid:
self.log_info("Task forked %s: pid=%s" % (self._task_str(task_info), pid))
return pid
# in no circumstance should we return after the fork
# nor should any exceptions propagate past here
try:
# set process group
os.setpgrp()
# set a do-nothing handler for sigusr2
# do not use signal.signal(signal.SIGUSR2, signal.SIG_IGN) - it completely masks interrups !!!
signal.signal(signal.SIGUSR2, lambda *args: None)
# set a default handler for SIGTERM
signal.signal(signal.SIGTERM, signal.SIG_DFL)
# run the task
self.run_task(task_info)
finally:
# die
os._exit(os.EX_OK)
If run_task raises any exception, it's never logged, and the process exits with 0 exit code giving no indication of an error.
It seems that any exceptions here should be logged, so that problems from run_task (e.g. failed XML-RPC calls to hub) leave some evidence behind.
This traceback was thrown when running pungi-4.1.21, which switched to python3:
Traceback (most recent call last):
File "/usr/bin/pungi-koji", line 468, in <module>
main()
File "/usr/bin/pungi-koji", line 254, in main
run_compose(compose, create_latest_link=create_latest_link, latest_link_status=latest_link_status)
File "/usr/bin/pungi-koji", line 362, in run_compose
init_phase.start()
File "/usr/lib/python3.6/site-packages/pungi/phases/base.py", line 64, in start
self.run()
File "/usr/lib/python3.6/site-packages/pungi/phases/init.py", line 41, in run
write_global_comps(self.compose)
File "/usr/lib/python3.6/site-packages/pungi/phases/init.py", line 84, in write_global_comps
get_file_from_scm(scm_dict, tmp_dir, logger=compose._logger)
File "/usr/lib/python3.6/site-packages/pungi/wrappers/scm.py", line 249, in get_file_from_scm
scm.export_file(scm_repo, i, scm_branch=scm_branch, target_dir=tmp_dir)
File "/usr/lib/python3.6/site-packages/pungi/wrappers/scm.py", line 147, in export_file
self.run_process_command(tmp_dir)
File "/usr/lib/python3.6/site-packages/pungi/wrappers/scm.py", line 48, in run_process_command
universal_newlines=True)
File "h", line 326, in run
lines = proc.stdout.read(buffer_size)
File "/usr/lib64/python3.6/encodings/ascii.py", line 26, in decode
return codecs.ascii_decode(input, self.errors)[0]
UnicodeDecodeError: 'ascii' codec can't decode byte 0xe2 in position 0: ordinal not in range(128)
Frame decode in /usr/lib64/python3.6/encodings/ascii.py at line 26
<CODE>
Kobo fails on Python 3 due missing smart_unicode
function.
from django.conf import settings
from django.core.exceptions import ImproperlyConfigured
from django.core.urlresolvers import reverse
from django.utils.safestring import mark_safe
> from django.utils.encoding import smart_unicode
E ImportError: cannot import name 'smart_unicode' from 'django.utils.encoding'
venv3/lib/python3.7/site-packages/kobo/django/menu/__init__.py:95: ImportError
Worker.update_worker
call save but it does not set any variable, which is nonsense.
It should set worker attributes based on function parameters.
https://github.com/release-engineering/kobo/blob/master/kobo/hub/models.py#L253
Code should be:
def update_worker(self, enabled, ready, task_count):
"""Update worker attributes. Return worker_info.
Update only if data provided from a worker differs.
"""
if (self.enabled, self.ready, self.task_count) != (enabled, ready, task_count):
self.enabled = enabled
self.ready = ready
self.task_count = task_count
self.save()
return self.export()
If kobo.shortcuts.run is called with a buffer_size other than -1, it can generate useless additional whitespace.
Can be trivially reproduced like:
$ python
Python 2.7.5 (default, Aug 2 2016, 04:20:16)
[GCC 4.8.5 20150623 (Red Hat 4.8.5-4)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> from kobo.shortcuts import run
>>> x = run("echo Hello world", stdout=True, buffer_size=2)
He ll o wo rl d
The reason is that run internally uses the print method, which is documented as:
A space is written before each object is (converted and) written, unless the output system believes it is positioned at the beginning of a line.
In the above example, there's an extra space introduced every 2 characters since run will loop, reading a buffer of size 2, then printing it.
Since the intent is to pass through the stdout from the process unmodified, this should be changed to use write rather than print.
File kobo/worker/main/py
makes use of a private variable which is not available on Python 3.
log_level = logging._levelNames.get(conf.get("LOG_LEVEL", "DEBUG").upper())
This causes the error bellow
AttributeError: 'module' object has no attribute '_levelNames'
We should add some Django 1.11 configuration to the Travis CI. This would be needed to ensure the changes for issue #44 are working.
In scope:
Not in scope:
If get_next_task
is called and TaskManager is currently locked and has awaited tasks it will fail to run the awaited tasks if they are not exclusive because of missing extra fields.
The reason is that take_task
crash if the task is not exclusive because it requires extra fields from arch and channel if flat=True
on get_awaited_tasks
these extra fields are not provided.
Hi.
Could you please make a new release? There is a lot of changes (Python 3 compatibility for example) and it would be nice to release a new version of RPM packages based on a new version of Kobo.
Thanks!
Class SafeCookieTransport() which is in xmlrpc submodule uses old (2.7) interface of xmlrpclib library.
Following code snippet shows the bug:
import xmlrpc.client
from kobo.xmlrpc import SafeCookieTransport
url = 'https://errata.devel.redhat.com/errata/xmlrpc.cgi'
xmlrpc = xmlrpc.client.ServerProxy(url, transport=SafeCookieTransport())
print(xmlrpc.get_advisory_cdn_file_list('RHSA-2018:2942-03'))
Here is the output:
Traceback (most recent call last):
File "xml_rpc_test.py", line 9, in <module>
print(xmlrpc.get_advisory_cdn_file_list('RHSA-2018:2942-03'))
File "/usr/lib64/python3.6/xmlrpc/client.py", line 1112, in __call__
return self.__send(self.__name, args)
File "/usr/lib64/python3.6/xmlrpc/client.py", line 1452, in __request
verbose=self.__verbose
File "/usr/lib64/python3.6/xmlrpc/client.py", line 1154, in request
return self.single_request(host, handler, request_body, verbose)
File "/home/araszka/.local/lib/python3.6/site-packages/kobo/xmlrpc.py", line 450, in _single_request
self.send_request(h, handler, request_body)
File "/home/araszka/.local/lib/python3.6/site-packages/kobo/xmlrpc.py", line 557, in send_request
return xmlrpclib.SafeTransport.send_request(self, connection, handler, request_body)
TypeError: send_request() missing 1 required positional argument: 'debug'
New version of xmlrpc library for python 3 and its interface have been updated and 'debug' argument has been added to send_request() function.
https://github.com/python/cpython/blob/3.7/Lib/xmlrpc/client.py#L1266
When can the python-3-dev
branch merge to master and land in a point release?
Have a look at this error handling logic in kobo/django/xmlrpc/dispatcher.py, _marshaled_dispatch method:
try:
if dispatch_method is not None:
response = dispatch_method(method, params)
else:
response = self._dispatch(method, params)
# wrap response in a singleton tuple
response = (response,)
response = xmlrpclib.dumps(response, methodresponse=1, allow_none=self.allow_none, encoding=self.encoding)
except xmlrpclib.Fault as fault:
response = xmlrpclib.dumps(fault, allow_none=self.allow_none, encoding=self.encoding)
except:
# report exception back to server
if settings.DEBUG:
from kobo.tback import Traceback
response = xmlrpclib.dumps(
xmlrpclib.Fault(1, u"%s" % Traceback().get_traceback()),
allow_none=self.allow_none, encoding=self.encoding)
else:
response = xmlrpclib.dumps(
xmlrpclib.Fault(1, "%s: %s" % (sys.exc_type.__name__, sys.exc_info()[1])),
allow_none=self.allow_none, encoding=self.encoding)
Consider what happens if the XML-RPC method raises any unexpected exception. Except for DEBUG mode, the exception details (traceback) will be discarded. This can make it virtually impossible to identify which code raised an exception. Even in DEBUG mode, the exception info is only passed to the client, so the root cause of an error on the server requires looking into logs on the client.
This should be improved so that unexpected exceptions during XML-RPC methods are always logged on the server.
Current Travis builds run in Ubuntu Trusty, where Koji is not available so tests which depend on RPM and Koji are skipped.
Possibilities:
Task.__lock
fails when running unit tests on Python 3.
cursor.rowcount
always returns -1
on Python 3, on Python 2 it returned the actual rows count affected. Unit tests are using SQLite, but this problem seems to happen with other drivers.
The biggest problem here is that cursor.rowcount
is used to raise or not exceptions. We need to fix it or use Django ORM instead of a cursor.
As I understand it, Kobo supports loading configuration values from a file:
Line 111 in 7500cd7
And the extent of runtime configuration for a (global) profile is generally implemented through environment variables in a downstream client:
Line 35 in 7500cd7
Users would have to write wrapper scripts to ensure the correct profile is set if using this approach.
Could configuration profiles be implemented as a CLI option? The option could look like this:
$ downstream-kobo-client --help
Usage: downstream-kobo-client <command> [args] [--help]
Options:
-h, --help show this help message and exit
--username=USERNAME specify user
--password=PASSWORD specify password
--profile=PROFILE specify profile
Which could run like this:
$ downstream-kobo-client --profile=${PROFILE}
And this in turn could look in a designated configuration folder like so:
/etc/downstream-kobo-client/${PROFILE}.conf
A declarative, efficient, and flexible JavaScript library for building user interfaces.
π Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. πππ
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google β€οΈ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.