The fedmsg's discuss from fedora-infra

fedmsg.schema questions

First, there are problems on py2.6

13:54:36 < abadger1999> threebean: When I print schema.keys => ['log', 'update', 'agent', [...], 'tag', 'user', 'test']
13:54:43 < abadger1999> Is that [...]  meant to be there?
14:02:41 < abadger1999> huh.. I still get [...] when I import in python shell... /me tries pip installing the actual module
14:06:59 < abadger1999> threebean: The [...]   -- Try it on a RHEL/Centos6 box
14:07:14 < abadger1999> threebean: Looks like it's not happening on python-2.7 but it is on python-2.66

14:09:14 < abadger1999> threebean: I'd also make that a frozenset

Should we just get rid of the schema all together? It may be over-engineering.

fedmsg.text internationalization

fedmsg.text has a _ function that isn't used or hooked up.

fedmsg-irc truncates super long messages

For super large messages (like those coming from bodhi), fedmsg-irc truncates its output in a way that looks like a bug or a mistake.

Badges Consumers don't shutdown gracefully

When ctrl-cing fedmsg-relay with a FedoraBadgesConsumer running, it doesn't shutdown cleanly (or at all)

It seems that no more than one consumer can be run at a time

@rossdylan found it while working on fedbadges.

Import badges and new badge data from askbot.

http://ask.fedoraproject.org/badges/ has badges. We should incorporate them at some point.

Logging system

fedmsg needs a common logging system that is independent (for console scripts) but which can work well along side other logging mechanisms (like inside other FI apps).

obey the privacy policy in fas

Don't emit any real names if they have private set.

``/etc/fedmsg-config.py`` not overwritten by puppet config correctly.

The copy of /etc/fedmsg-config.py was still the copy from the RPM when I checked. I overwrote it by hand for now.

Need a 'fuzz' analysis of fedmsg.

What happens when you open up telnet on it and just start barfing garbage on it.

What happens when you spam it? How many messages/s and bytes/s can it handle?

Split up ``/etc/fedmsg-config.py`` into ``/etc/fedmsg.d/*.py``

The config file is getting bloated. It would be nice to be able to install things piece-meal.

This should affect some of the work you're doing, @rossdylan.

daemonize long-running processes

The following commands should be able to be daemonized:

fedmsg-hub
fedmsg-relay
fedmsg-irc

There should be some sort of standardized way of daemonizing them and standard options to run them in the foreground.

file a ticket with infra to discuss keys

Need docs for how to make a badge

This should be a "from A to Z" document so that any joe-schmoe can make a new Fedora badge.

It should detail:

How to make the image (and make it compliant with a standard style we decide on). Talk to mizmo about this.
How to write the consumer that awards the badges.
How to test the consumer.
How/where to submit the badge-image+consumer to so it gets actually included in the infrastructure.
Will there be a review process? Who will be in charge?

Need to open ports up between app0{1,2}.stg for fedmsg talk.

Don't forget community01.dev!

We should probably write a fedmsg script that will output iptables commands for us, so when we change the config we can change iptables seamlessly.

Add license header to every file.

Need documentation of the plan for open badges

We have a pretty good idea, but we just need to write it down so others can check it out.

handle time.struct_time in fedmsg.encoding

/usr/lib/python2.7/site-packages/fedmsg/encoding.py

I got this traceback when developing supybot-fedmsg

Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/supybot/log.py", line 351, in m
    return f(self, *args, **kwargs)
  File "/usr/lib/python2.7/site-packages/supybot/irclib.py", line 132, in __call__
    method(irc, msg)
  File "/usr/lib/python2.7/site-packages/supybot/plugins/MeetBot/plugin.py", line 116, in doPrivmsg
    M.addline(nick, payload)
  File "/usr/lib/python2.7/site-packages/supybot/plugins/MeetBot/meeting.py", line 494, in addline
    linenum=linenum, time_=time_)
  File "/usr/lib/python2.7/site-packages/supybot/plugins/Fedmsg/plugin.py", line 83, in wrapper
    msg=dict(args=args, kw=kw),
  File "/usr/lib/python2.7/site-packages/fedmsg/__init__.py", line 60, in _wrapper
    return func(*args, **kw)
  File "/usr/lib/python2.7/site-packages/fedmsg/__init__.py", line 89, in publish
    return __local.__context.publish(topic, msg, **kw)
  File "/usr/lib/python2.7/site-packages/fedmsg/core.py", line 190, in publish
    self.publisher.send_multipart([topic, fedmsg.encoding.dumps(msg)])
  File "/usr/lib64/python2.7/json/encoder.py", line 203, in encode
    chunks = list(chunks)
  File "/usr/lib64/python2.7/json/encoder.py", line 428, in _iterencode
    for chunk in _iterencode_dict(o, _current_indent_level):
  File "/usr/lib64/python2.7/json/encoder.py", line 402, in _iterencode_dict
    for chunk in chunks:
  File "/usr/lib64/python2.7/json/encoder.py", line 402, in _iterencode_dict
    for chunk in chunks:
  File "/usr/lib64/python2.7/json/encoder.py", line 402, in _iterencode_dict
    for chunk in chunks:
  File "/usr/lib64/python2.7/json/encoder.py", line 436, in _iterencode
    o = _default(o)
  File "/usr/lib/python2.7/site-packages/fedmsg/encoding.py", line 42, in default
    return super(FedMsgEncoder, self).default(obj)
  File "/usr/lib64/python2.7/json/encoder.py", line 178, in default
    raise TypeError(repr(o) + " is not JSON serializable")
TypeError: time.struct_time(tm_year=2012, tm_mon=8, tm_mday=21, tm_hour=15, tm_min=38, tm_sec=20, tm_wday=1, tm_yday=234, tm_isdst=0) is not JSON serializable

Missing diff versions for wiki _msg2link

Important for posterity.

This will require editing extras/mediawiki/fedmsg-emit.php to get the revision information out in a reliable way.

signature-from-json is unreliable

12:05:00 < ricky> threebean: Random note on fedmsg's crypto.py: In the sign function, you sign the json encoding of the message dict, and then continue to pass the dict around

12:05:22 < ricky> However, it might not be guaranteed that the json encoding of a dict is the same on different machines

13:42:26 < threebean> ricky: oo, right. there could be a difference of the ordering of the pairs.

13:45:50 < threebean> hmm, I can trying monkeying around with an ordereddict that sorts its keys before encoding.

13:51:32 < threebean> a better idea might be to derive the signature from a different encoding. I'm thinking msgpack: http://pypi.python.org/pypi/msgpack-python/

color coded topics in fedmsg-irc

for @nirik

15:45:30 < nirik> threebean: suggestion/RFE for git fedmsg messages: if they have more than one line, include the top line (like we do now), but add a "...more" or something to indicate the changelog is longer?

Separate the bus into channels

One for normal events, and the other for more verbose statistical events like httpd logs.

http://httpd.apache.org/docs/2.2/logs.html#piped can be used for httpd logging.

mediawiki plugin allocates too many zmq sockets

There can be only one. We need to use php's memory allocation features...

http://stackoverflow.com/questions/7952209/zmq-getsocket-method

do fedmsg daemons reload config when it changes?

We want fedmsg daemons to pick up their configs with a little downtime as possible. If we can get them to do it without a formal restart, that would be awesome.

unicode safety

To start with, I saw this exception come up in /var/log/fedmsg/fedmsg-irc.log:

  File "/usr/lib64/python2.6/site-packages/twisted/python/log.py", line 84, in callWithLogger
    return callWithContext({"system": lp}, func, *args, **kw)
  File "/usr/lib64/python2.6/site-packages/twisted/python/log.py", line 69, in callWithContext
    return context.call({ILogContext: newCtx}, func, *args, **kw)
  File "/usr/lib64/python2.6/site-packages/twisted/python/context.py", line 59, in callWithContext
    return self.currentContext().callWithContext(ctx, func, *args, **kw)
  File "/usr/lib64/python2.6/site-packages/twisted/python/context.py", line 37, in callWithContext
    return func(*args,**kw)
  File "/usr/lib/python2.6/site-packages/txzmq/pubsub.py", line 60, in messageReceived
    self.gotMessage(message[1], message[0])
  File "/usr/lib/python2.6/site-packages/moksha/hub/zeromq/zeromq.py", line 143, in chain_over_moksha_callbacks
    f(_body, _topic)
  File "/usr/lib/python2.6/site-packages/moksha/hub/zeromq/zeromq.py", line 165, in intercept
    return callback(ZMQMessage(_topic, _body))
  File "/usr/lib/python2.6/site-packages/moksha/api/hub/consumer.py", line 100, in _consume_json
    self._consume(message_as_dict)
  File "/usr/lib/python2.6/site-packages/moksha/api/hub/consumer.py", line 109, in _consume
    self.consume(message)
  File "/usr/lib/python2.6/site-packages/fedmsg/consumers/ircbot.py", line 268, in consume
    terse=client.factory.terse,
  File "/usr/lib/python2.6/site-packages/fedmsg/consumers/ircbot.py", line 233, in prettify
    link=fedmsg.text._msg2link(msg, **self.hub.config),
  File "/usr/lib/python2.6/site-packages/fedmsg/consumers/ircbot.py", line 87, in ircprettify
    return fmt.format(title=title, subtitle=subtitle, link=link)
exceptions.UnicodeEncodeError: 'ascii' codec can't encode character u'\u0160' in position 5: ordinal not in range(128)

Replace puppet/modules-staging/files with puppet/modules-staging/templates

That way we don't have to maintain two separate sets of config files but just one but with variables/macros.

Need to be able to globally disable consumers/producers.

The entry-points model of moksha is getting in the way here.

I was going to test out fedmsg-irc on app02.stg, but fedoracommunity has an old moksha producer lurking there which gets picked up by the hub underlying fedmsg-irc. From the python API, the irc command needs to be able to say: "Don't load anything via extension points... only load what I tell you."

Ugly tracebacks on ctrl-C

When you hit ctrl-C during fedmsg-status you get an ugly traceback.

Traceback (most recent call last):
  File "/usr/bin/fedmsg-tail", line 9, in <module>
    load_entry_point('fedmsg==0.2.0', 'console_scripts', 'fedmsg-tail')()
  File "/usr/lib/python2.6/site-packages/fedmsg/commands/__init__.py", line 70, in wrapper
    return func(**config)
  File "/usr/lib/python2.6/site-packages/fedmsg/commands/tail.py", line 67, in tail
    for name, ep, topic, message in fedmsg.__context._tail_messages(**kw):
  File "/usr/lib/python2.6/site-packages/fedmsg/core.py", line 206, in _tail_messages
    subs[e].recv_multipart(zmq.NOBLOCK)
  File "socket.pyx", line 680, in zmq.core.socket.Socket.recv_multipart (zmq/core/socket.c:6095)
  File "socket.pyx", line 583, in zmq.core.socket.Socket.recv (zmq/core/socket.c:5725)
  File "socket.pyx", line 617, in zmq.core.socket.Socket.recv (zmq/core/socket.c:5587)
  File "socket.pyx", line 118, in zmq.core.socket._recv_copy (zmq/core/socket.c:1567)
  File "error.pyx", line 92, in zmq.core.error.ZMQError.__init__ (zmq/core/error.c:910)
KeyboardInterrupt

systemctl scripts for daemonized processes

We need systemctl scripts for the long-running processes. Take a look at how moksha does it.

Document fedmsg-config.py

There's a lot of stuff going on there, which should be documented.

Config file isn't flexible

/etc/fedmsg.ini currently only allows a single sending host on any given machine.

Instead the config file should allow a list of sending services on the machine, each with a different publish_endpoint.

Explicit hostname fails for pub+bind

The endpoint name needs to be rewritten for publishers from tcp://host.stg.fp.o:2001" totcp://*:2001``.

fedmsg-irc doesn't clean itself up correctly.

start it and stop it a few times and you'll see a traceback fly by.

fedmsg-irc sometimes gets stuck

I think we'll need to use @lmacken's pyrasite to figure it out, amirite?

Use by a multi-process WSGI app fails

Each process tries to bind its own zmq socket and they fail with ZMQError: Address already in use.

all the *_exists functions in the TahrirDatabase class fail

No idea why this is happening but when using mysql all the exists functions return false no matter what

ircbot should show a link to wiki edits

for @nirik

Set up a mailing list for the messaging sig

Make sure to provide links to it both from the Messaging_SIG wiki and from the README of this repo.

get mediawiki+fedmsg setup in app0{1,2}.stg

I wrote the mediawiki plugin back in fedmsg-0.1.0 but now we're on fedmsg-0.2.1. The config file has changed considerably, so the plugin will likely need to be updated first.

fedmsg-status doesn't work the way it was intended

The purpose it was meant to serve no longer makes sense since every endpoint isn't coupled with a fedmsg-hub. I once thought they would be 1-to-1.

fedmsg-status should either be deleted or the config convention should be changed to denote which endpoints should have heartbeats (run under fedmsg-hub) and which should not (standalone send_message).

Document messages being emitted/to-be-emitted.

There should be a master list somewhere so we don't lose track of all this.

Strip out the tahrir DB api so it's usable other places.

@decause is drooling.

relay_inbound config switch-a-roo should be hidden

The following conversation went down in #fedora-admin

08:51:30 < misc> +config['endpoints']['relay_inbound'] = config['relay_inbound'] , seems weird, why not get the
configuration right at the first time ?
08:52:03 < threebean> long story.. this is one of the few places you need to do that :/
08:52:28 < threebean> ugh.. now that you mention it though, that confusing-ness should be hidden inside the fedmsg API.

This will affect the git-hook, the mediawiki hook, and fedmsg-logger.

fedora-infra / fedmsg Goto Github PK

fedmsg's Issues

Recommend Projects

Recommend Topics

Recommend Org