jortel/gofer

The gofer project provides a lightweight, extensible python agent.

gofer's Issues

Log link and connection errors at WARN instead of ERROR.

Updated the reliability modules to log link an connection errors at WARN instead of ERROR. Requested by Katello bugzilla[1].

[1] https://bugzilla.redhat.com/show_bug.cgi?id=1608217

ModelError: 'NoneType' object has no attribute 'closed'

We use the Gofer as a part of the Katello installation.

Sometimes on many boxes I see the exception:

`
Oct 30 09:33:36 example.host goferd: [INFO][pulp.agent.2777c01c-ef54-4937-aaff-0aafaacd9693] gofer.messaging.adapter.connect:28 - connecting: proton+amqps://foreman-proxy.pac.nhl.com:5647
Oct 30 09:33:36 example.host goferd: [INFO][pulp.agent.2777c01c-ef54-4937-aaff-0aafaacd9693] gofer.messaging.adapter.connect:30 - connected: proton+amqps://foreman-proxy.pac.nhl.com:5647
Oct 30 09:33:36 example.host goferd: [ERROR][pulp.agent.2777c01c-ef54-4937-aaff-0aafaacd9693] gofer.messaging.adapter.model:44 - 'NoneType' object has no attribute 'closed'
Oct 30 09:33:36 example.host goferd: [ERROR][pulp.agent.2777c01c-ef54-4937-aaff-0aafaacd9693] gofer.messaging.adapter.model:44 - Traceback (most recent call last):
Oct 30 09:33:36 example.host goferd: [ERROR][pulp.agent.2777c01c-ef54-4937-aaff-0aafaacd9693] gofer.messaging.adapter.model:44 -   File "/usr/lib/python2.7/site-packages/gofer/messaging/adapter/model.py", line 40, in _fn
Oct 30 09:33:36 example.host goferd: [ERROR][pulp.agent.2777c01c-ef54-4937-aaff-0aafaacd9693] gofer.messaging.adapter.model:44 -     return fn(*args, **keywords)
Oct 30 09:33:36 example.host goferd: [ERROR][pulp.agent.2777c01c-ef54-4937-aaff-0aafaacd9693] gofer.messaging.adapter.model:44 -   File "/usr/lib/python2.7/site-packages/gofer/messaging/adapter/model.py", line 584, in open
Oct 30 09:33:36 example.host goferd: [ERROR][pulp.agent.2777c01c-ef54-4937-aaff-0aafaacd9693] gofer.messaging.adapter.model:44 -     self._impl.open()
Oct 30 09:33:36 example.host goferd: [ERROR][pulp.agent.2777c01c-ef54-4937-aaff-0aafaacd9693] gofer.messaging.adapter.model:44 -   File "/usr/lib/python2.7/site-packages/gofer/messaging/adapter/proton/reliability.py", line 45, in _fn
Oct 30 09:33:36 example.host goferd: [ERROR][pulp.agent.2777c01c-ef54-4937-aaff-0aafaacd9693] gofer.messaging.adapter.model:44 -     return fn(messenger, *args, **kwargs)
Oct 30 09:33:36 example.host goferd: [ERROR][pulp.agent.2777c01c-ef54-4937-aaff-0aafaacd9693] gofer.messaging.adapter.model:44 -   File "/usr/lib/python2.7/site-packages/gofer/messaging/adapter/proton/consumer.py", line 73, in open
Oct 30 09:33:36 example.host goferd: [ERROR][pulp.agent.2777c01c-ef54-4937-aaff-0aafaacd9693] gofer.messaging.adapter.model:44 -     self.receiver = self.connection.receiver(self.node.address)
Oct 30 09:33:36 example.host goferd: [ERROR][pulp.agent.2777c01c-ef54-4937-aaff-0aafaacd9693] gofer.messaging.adapter.model:44 -   File "/usr/lib/python2.7/site-packages/gofer/messaging/adapter/proton/connection.py", line 121, in receiver
Oct 30 09:33:36 example.host goferd: [ERROR][pulp.agent.2777c01c-ef54-4937-aaff-0aafaacd9693] gofer.messaging.adapter.model:44 -     return self._impl.create_receiver(address, name=name, dynamic=dynamic, options=options)
Oct 30 09:33:36 example.host goferd: [ERROR][pulp.agent.2777c01c-ef54-4937-aaff-0aafaacd9693] gofer.messaging.adapter.model:44 -   File "/usr/lib64/python2.7/site-packages/proton/_utils.py", line 415, in create_receiver
Oct 30 09:33:36 example.host goferd: [ERROR][pulp.agent.2777c01c-ef54-4937-aaff-0aafaacd9693] gofer.messaging.adapter.model:44 -     options=options), fetcher, credit=prefetch)
Oct 30 09:33:36 example.host goferd: [ERROR][pulp.agent.2777c01c-ef54-4937-aaff-0aafaacd9693] gofer.messaging.adapter.model:44 -   File "/usr/lib64/python2.7/site-packages/proton/_utils.py", line 202, in __init__
Oct 30 09:33:36 example.host goferd: [ERROR][pulp.agent.2777c01c-ef54-4937-aaff-0aafaacd9693] gofer.messaging.adapter.model:44 -     super(BlockingReceiver, self).__init__(connection, receiver)
Oct 30 09:33:36 example.host goferd: [ERROR][pulp.agent.2777c01c-ef54-4937-aaff-0aafaacd9693] gofer.messaging.adapter.model:44 -   File "/usr/lib64/python2.7/site-packages/proton/_utils.py", line 41, in __init__
Oct 30 09:33:36 example.host goferd: [ERROR][pulp.agent.2777c01c-ef54-4937-aaff-0aafaacd9693] gofer.messaging.adapter.model:44 -     msg="Opening link %s" % link.name)
Oct 30 09:33:36 example.host goferd: [ERROR][pulp.agent.2777c01c-ef54-4937-aaff-0aafaacd9693] gofer.messaging.adapter.model:44 -   File "/usr/lib64/python2.7/site-packages/proton/_utils.py", line 475, in wait
Oct 30 09:33:36 example.host goferd: [ERROR][pulp.agent.2777c01c-ef54-4937-aaff-0aafaacd9693] gofer.messaging.adapter.model:44 -     self.container.process()
Oct 30 09:33:36 example.host goferd: [ERROR][pulp.agent.2777c01c-ef54-4937-aaff-0aafaacd9693] gofer.messaging.adapter.model:44 -   File "/usr/lib64/python2.7/site-packages/proton/_reactor.py", line 239, in process
Oct 30 09:33:36 example.host goferd: [ERROR][pulp.agent.2777c01c-ef54-4937-aaff-0aafaacd9693] gofer.messaging.adapter.model:44 -     event.dispatch(self._global_handler)
Oct 30 09:33:36 example.host goferd: [ERROR][pulp.agent.2777c01c-ef54-4937-aaff-0aafaacd9693] gofer.messaging.adapter.model:44 -   File "/usr/lib64/python2.7/site-packages/proton/_events.py", line 162, in dispatch
Oct 30 09:33:36 example.host goferd: [ERROR][pulp.agent.2777c01c-ef54-4937-aaff-0aafaacd9693] gofer.messaging.adapter.model:44 -     _dispatch(handler, type.method, self)
Oct 30 09:33:36 example.host goferd: [ERROR][pulp.agent.2777c01c-ef54-4937-aaff-0aafaacd9693] gofer.messaging.adapter.model:44 -   File "/usr/lib64/python2.7/site-packages/proton/_events.py", line 125, in _dispatch
Oct 30 09:33:36 example.host goferd: [ERROR][pulp.agent.2777c01c-ef54-4937-aaff-0aafaacd9693] gofer.messaging.adapter.model:44 -     handler.on_unhandled(method, *args)
Oct 30 09:33:36 example.host goferd: [ERROR][pulp.agent.2777c01c-ef54-4937-aaff-0aafaacd9693] gofer.messaging.adapter.model:44 -   File "/usr/lib64/python2.7/site-packages/proton/_reactor.py", line 856, in on_unhandled
Oct 30 09:33:36 example.host goferd: [ERROR][pulp.agent.2777c01c-ef54-4937-aaff-0aafaacd9693] gofer.messaging.adapter.model:44 -     event.dispatch(self.base)
Oct 30 09:33:36 example.host goferd: [ERROR][pulp.agent.2777c01c-ef54-4937-aaff-0aafaacd9693] gofer.messaging.adapter.model:44 -   File "/usr/lib64/python2.7/site-packages/proton/_events.py", line 162, in dispatch
Oct 30 09:33:36 example.host goferd: [ERROR][pulp.agent.2777c01c-ef54-4937-aaff-0aafaacd9693] gofer.messaging.adapter.model:44 -     _dispatch(handler, type.method, self)
Oct 30 09:33:36 example.host goferd: [ERROR][pulp.agent.2777c01c-ef54-4937-aaff-0aafaacd9693] gofer.messaging.adapter.model:44 -   File "/usr/lib64/python2.7/site-packages/proton/_events.py", line 123, in _dispatch
Oct 30 09:33:36 example.host goferd: [ERROR][pulp.agent.2777c01c-ef54-4937-aaff-0aafaacd9693] gofer.messaging.adapter.model:44 -     m(*args)
Oct 30 09:33:36 example.host goferd: [ERROR][pulp.agent.2777c01c-ef54-4937-aaff-0aafaacd9693] gofer.messaging.adapter.model:44 -   File "/usr/lib64/python2.7/site-packages/proton/_handlers.py", line 1292, in on_selectable_expired
Oct 30 09:33:36 example.host goferd: [ERROR][pulp.agent.2777c01c-ef54-4937-aaff-0aafaacd9693] gofer.messaging.adapter.model:44 -     self.update(t, s, r.now)
Oct 30 09:33:36 example.host goferd: [ERROR][pulp.agent.2777c01c-ef54-4937-aaff-0aafaacd9693] gofer.messaging.adapter.model:44 -   File "/usr/lib64/python2.7/site-packages/proton/_handlers.py", line 1351, in update
Oct 30 09:33:36 example.host goferd: [ERROR][pulp.agent.2777c01c-ef54-4937-aaff-0aafaacd9693] gofer.messaging.adapter.model:44 -     if transport.closed:
Oct 30 09:33:36 example.host goferd: [ERROR][pulp.agent.2777c01c-ef54-4937-aaff-0aafaacd9693] gofer.messaging.adapter.model:44 - AttributeError: 'NoneType' object has no attribute 'closed'
Oct 30 09:33:36 example.host goferd: [ERROR][pulp.agent.2777c01c-ef54-4937-aaff-0aafaacd9693] gofer.messaging.consumer:79 - pulp.agent.2777c01c-ef54-4937-aaff-0aafaacd9693
Oct 30 09:33:36 example.host goferd: [ERROR][pulp.agent.2777c01c-ef54-4937-aaff-0aafaacd9693] gofer.messaging.consumer:79 - Traceback (most recent call last):
Oct 30 09:33:36 example.host goferd: [ERROR][pulp.agent.2777c01c-ef54-4937-aaff-0aafaacd9693] gofer.messaging.consumer:79 -   File "/usr/lib/python2.7/site-packages/gofer/messaging/consumer.py", line 72, in open
Oct 30 09:33:36 example.host goferd: [ERROR][pulp.agent.2777c01c-ef54-4937-aaff-0aafaacd9693] gofer.messaging.consumer:79 -     self.reader.open()
Oct 30 09:33:36 example.host goferd: [ERROR][pulp.agent.2777c01c-ef54-4937-aaff-0aafaacd9693] gofer.messaging.consumer:79 -   File "/usr/lib/python2.7/site-packages/gofer/messaging/adapter/model.py", line 45, in _fn
Oct 30 09:33:36 example.host goferd: [ERROR][pulp.agent.2777c01c-ef54-4937-aaff-0aafaacd9693] gofer.messaging.consumer:79 -     raise ModelError(*e.args)
Oct 30 09:33:36 example.host goferd: [ERROR][pulp.agent.2777c01c-ef54-4937-aaff-0aafaacd9693] gofer.messaging.consumer:79 - ModelError: 'NoneType' object has no attribute 'closed'

CentOS Linux release 7.8.2003 (Core)
-----
gofer.noarch                    2.12.5-3.el7               @foreman
python-gofer.noarch             2.12.5-3.el7               @foreman
python-gofer-proton.noarch      2.12.5-3.el7               @foreman
-----
python-gofer-proton.noarch      2.12.5-3.el7               @foreman
python2-qpid-proton.x86_64      0.31.0-3.el7               @epel-7
qpid-proton-c.x86_64            0.31.0-3.el7               @epel-7

goferd.service is failing, when starting without network ready

https://bugzilla.redhat.com/show_bug.cgi?id=1681182

Description of problem:
If network is not already online and in case network.service is disabled, the goferd.service will report error in logs.

How reproducible:
maybe only in slow networks, where assinging IP take some time

Steps to Reproduce:

1. install client from satellite-6
2. disable network.service on client
3. boot client

Actual results:
Feb 06 10:57:23 sample-host goferd[14030]: [WARNING][MainThread] gofer.agent.plugin:647 - plugin:demo, DISABLED
Feb 06 10:57:23 sample-host goferd[14030]: [INFO][Thread-1] gofer.rmi.store:108 - Using: /var/lib/gofer/messaging/pending/demo
Feb 06 10:57:23 sample-host goferd[14030]: [INFO][Thread-2] gofer.rmi.store:108 - Using: /var/lib/gofer/messaging/pending/katello
Feb 06 10:57:23 sample-host goferd[14030]: [INFO][MainThread] gofer.agent.plugin:692 - plugin:katello loaded using: katello.agent.goferd.plugin
Feb 06 10:57:23 sample-host goferd[14030]: [INFO][MainThread] rhsm.connection:871 - Connection built: host=sat-qs.wob.vw.vwg port=443 handler=/rhsm auth=identity_cert ca_dir=/etc/rhsm/ca/ insecure=False
Feb 06 10:57:23 sample-host goferd[14030]: [ERROR][MainThread] katello.agent.goferd.plugin:206 - [Errno -2] Name or service not known
Feb 06 10:57:23 sample-host goferd[14030]: [ERROR][MainThread] katello.agent.goferd.plugin:206 - Traceback (most recent call last):
Feb 06 10:57:23 sample-host goferd[14030]: [ERROR][MainThread] katello.agent.goferd.plugin:206 -   File "/usr/lib/python2.7/site-packages/katello/agent/goferd/plugin.py", line 197, in validate_registration
Feb 06 10:57:23 sample-host goferd[14030]: [ERROR][MainThread] katello.agent.goferd.plugin:206 -     consumer = uep.getConsumer(consumer_id)
Feb 06 10:57:23 sample-host goferd[14030]: [ERROR][MainThread] katello.agent.goferd.plugin:206 -   File "/usr/lib64/python2.7/site-packages/rhsm/connection.py", line 1105, in getConsumer
Feb 06 10:57:23 sample-host goferd[14030]: [ERROR][MainThread] katello.agent.goferd.plugin:206 -     return self.conn.request_get(method)
Feb 06 10:57:23 sample-host goferd[14030]: [ERROR][MainThread] katello.agent.goferd.plugin:206 -   File "/usr/lib64/python2.7/site-packages/rhsm/connection.py", line 693, in request_get
Feb 06 10:57:23 sample-host goferd[14030]: [ERROR][MainThread] katello.agent.goferd.plugin:206 -     return self._request("GET", method, headers=headers)
Feb 06 10:57:23 sample-host goferd[14030]: [ERROR][MainThread] katello.agent.goferd.plugin:206 -   File "/usr/lib64/python2.7/site-packages/rhsm/connection.py", line 719, in _request
Feb 06 10:57:23 sample-host goferd[14030]: [ERROR][MainThread] katello.agent.goferd.plugin:206 -     info=info, headers=headers)
Feb 06 10:57:23 sample-host goferd[14030]: [ERROR][MainThread] katello.agent.goferd.plugin:206 -   File "/usr/lib64/python2.7/site-packages/rhsm/connection.py", line 557, in _request
Feb 06 10:57:23 sample-host goferd[14030]: [ERROR][MainThread] katello.agent.goferd.plugin:206 -     conn.request(request_type, handler, body=body, headers=final_headers)
Feb 06 10:57:23 sample-host goferd[14030]: [ERROR][MainThread] katello.agent.goferd.plugin:206 -   File "/usr/lib64/python2.7/httplib.py", line 1041, in request
Feb 06 10:57:23 sample-host goferd[14030]: [ERROR][MainThread] katello.agent.goferd.plugin:206 -     self._send_request(method, url, body, headers)
Feb 06 10:57:23 sample-host goferd[14030]: [ERROR][MainThread] katello.agent.goferd.plugin:206 -   File "/usr/lib64/python2.7/httplib.py", line 1075, in _send_request
Feb 06 10:57:23 sample-host goferd[14030]: [ERROR][MainThread] katello.agent.goferd.plugin:206 -     self.endheaders(body)
Feb 06 10:57:23 sample-host goferd[14030]: [ERROR][MainThread] katello.agent.goferd.plugin:206 -   File "/usr/lib64/python2.7/httplib.py", line 1037, in endheaders
Feb 06 10:57:23 sample-host goferd[14030]: [ERROR][MainThread] katello.agent.goferd.plugin:206 -     self._send_output(message_body)
Feb 06 10:57:23 sample-host goferd[14030]: [ERROR][MainThread] katello.agent.goferd.plugin:206 -   File "/usr/lib64/python2.7/httplib.py", line 881, in _send_output
Feb 06 10:57:23 sample-host goferd[14030]: [ERROR][MainThread] katello.agent.goferd.plugin:206 -     self.send(msg)
Feb 06 10:57:23 sample-host goferd[14030]: [ERROR][MainThread] katello.agent.goferd.plugin:206 -   File "/usr/lib64/python2.7/httplib.py", line 843, in send
Feb 06 10:57:23 sample-host goferd[14030]: [ERROR][MainThread] katello.agent.goferd.plugin:206 -     self.connect()
Feb 06 10:57:23 sample-host goferd[14030]: [ERROR][MainThread] katello.agent.goferd.plugin:206 -   File "/usr/lib64/python2.7/httplib.py", line 1251, in connect
Feb 06 10:57:23 sample-host goferd[14030]: [ERROR][MainThread] katello.agent.goferd.plugin:206 -     HTTPConnection.connect(self)
Feb 06 10:57:23 sample-host goferd[14030]: [ERROR][MainThread] katello.agent.goferd.plugin:206 -   File "/usr/lib64/python2.7/httplib.py", line 824, in connect
Feb 06 10:57:23 sample-host goferd[14030]: [ERROR][MainThread] katello.agent.goferd.plugin:206 -     self.timeout, self.source_address)
Feb 06 10:57:23 sample-host goferd[14030]: [ERROR][MainThread] katello.agent.goferd.plugin:206 -   File "/usr/lib64/python2.7/socket.py", line 553, in create_connection
Feb 06 10:57:23 sample-host goferd[14030]: [ERROR][MainThread] katello.agent.goferd.plugin:206 -     for res in getaddrinfo(host, port, 0, SOCK_STREAM):
Feb 06 10:57:23 sample-host goferd[14030]: [ERROR][MainThread] katello.agent.goferd.plugin:206 - gaierror: [Errno -2] Name or service not known
Feb 06 10:57:23 sample-host goferd[14030]: [WARNING][MainThread] katello.agent.goferd.plugin:87 - [Errno -2] Name or service not known

Expected results:
service should start without any error reporting.

Additional info:
It's possible to fix this, when the service-unit is changed like.

 [Unit]
 Description=Gofer Agent
-After=network.target
+After=network-online.target
+Wants=network-online.target