rouxrc/gazouilleur

IRC bot for collaborative use and monitoring of Twitter

gazouilleur's Issues

Handle new Twitter API DM routes

https://developer.twitter.com/en/docs/direct-messages/sending-and-receiving/guides/direct-message-migration

ignore stack

Sometimes it could be desirable to ignore at least temporarily some twitter account. Here, someone is sending the same tweet, sequentially to -- apparently -- each and every French Members, including a link on a monitored domain, resulting in a large flood on the channel.

An idea could be some %ignore command with --list --add --rm (--rm last) and --global options, managing a stack (a space separated stack of @accounts should be enough) of accounts to be ignored. Or finer, but more complex maybe, regular expressions to permit filtering out not the whole account, but only the repeated message from that account in our case.

Such an %ignore stack should probably be IRC-channel specific by default (unless --global is used, which option should be restricted to bot admins ?)

tosscoin feature

add a tosscoin feature to let randomness decide when we can't ? :)

Maintain support for python2.6

twisted <= 14
pypump <= 0.6 cf tsyesika/PyPump#139

Error after update : pymongo.errors.OperationFailure: Index with name [...] already exists

After an update (via bin/update.sh), gazouilleur refused to connect to irc with the following error :

    Traceback (most recent call last):
  File "lib/python2.7/site-packages/twisted/internet/defer.py", line 434, in errback
    self._startRunCallbacks(fail)
  File "lib/python2.7/site-packages/twisted/internet/defer.py", line 501, in _startRunCallbacks
    self._runCallbacks()
  File "lib/python2.7/site-packages/twisted/internet/defer.py", line 588, in _runCallbacks
    current.result = callback(current.result, *args, **kw)
  File "lib/python2.7/site-packages/twisted/internet/defer.py", line 1184, in gotResult
    _inlineCallbacks(r, g, deferred)
--- <exception caught here> ---
  [...] 
  File "lib/python2.7/site-packages/txmongo/protocol.py", line 401, in get_last_error
    raise OperationFailure(err, code=code)
pymongo.errors.OperationFailure: Index with name: channel_1_timestamp_-1 already exists with different options

To solve the problem, I had to drop the indexes of gazouilleur's mongo collections (via mongo shell)

use mygazouilleurdb;
db.getCollectionNames().forEach(function(col_name) { var coll = db.getCollection(col_name); coll.dropIndexes(); });

Do not accept empty runlater commands

Be able to connect to a channel that require a password

Hello

It would be awesome to be able to connect to irc channel that require password.

Thanks a lot

Bot keeps getting kicked on connect (Excess flood)

Screenshots of log file:
http://i.imgur.com/NIFcylj.png
http://i.imgur.com/yYusH1W.png

Dirty hack that fixed it for now:
replace line 383 of gazouilleur/lib/microblog.py into :
err = e.response_data.get("errors", [{"code": 500, "message": "weird bug"}])[0]

Clean twitter urls with query

OpenDataZurich: Nicht verpassen und schnell anmelden, bevor es zu spät ist... ;) #IoT #SmartCity #CivicTech #LoRaWAN #OpenData #sensing « @makezurich: Challenges for #MakeZurich Vol. II are live & Registration is now open! 🚀 https://makezurich.ch https://twitter.com/makezurich/status/994750449621364738/photo/1 »?s=21 — https://twitter.com/OpenDataZurich/status/997766058231648256

Erreur avec TLSMemoryBIOProtocol

J'ai updaté mon os récement et gazouilleur ne reboot plus. Je renté une install from scratch mais sans résultat. Je sèche.

2017-03-15 21:29:31 [IRC/#gazouilleur] Left.
2017-03-15 21:29:31 [IRC] Connection lost because: [Failure instance: Traceback: <class 'OpenSSL.SSL.Error'>: [('SSL routines', 'SSL3_GET_RECORD', 'wrong version number')]
        /home/n3b/.virtualenvs/gazouilleur/local/lib/python2.7/site-packages/twisted/internet/posixbase.py:614:_doReadOrWrite
        /home/n3b/.virtualenvs/gazouilleur/local/lib/python2.7/site-packages/twisted/internet/tcp.py:209:doRead
        /home/n3b/.virtualenvs/gazouilleur/local/lib/python2.7/site-packages/twisted/internet/tcp.py:215:_dataReceived
        /home/n3b/.virtualenvs/gazouilleur/local/lib/python2.7/site-packages/twisted/protocols/tls.py:415:dataReceived
        --- <exception caught here> ---
        /home/n3b/.virtualenvs/gazouilleur/local/lib/python2.7/site-packages/twisted/protocols/tls.py:554:_write
        /home/n3b/.virtualenvs/gazouilleur/local/lib/python2.7/site-packages/OpenSSL/SSL.py:1271:send
        /home/n3b/.virtualenvs/gazouilleur/local/lib/python2.7/site-packages/OpenSSL/SSL.py:1191:_raise_ssl_error
        /home/n3b/.virtualenvs/gazouilleur/local/lib/python2.7/site-packages/OpenSSL/_util.py:48:exception_from_error_queue
        ].
2017-03-15 21:29:31 [IRCBot (TLSMemoryBIOProtocol),client] <twisted.internet.tcp.Connector instance at 0x3c73b00> will retry in 39 seconds
2017-03-15 21:29:31 [IRCBot (TLSMemoryBIOProtocol),client] Stopping factory <__builtin__.IRCBotFactory instance at 0x3c73e60>

Handle Twitter's new extra 140 characters

cf https://dev.twitter.com/overview/api/upcoming-changes-to-tweets

New tweets payload:

• handle new tweet format in rest api calls
• handle new tweet format in stream calls

New statuses.update API args:

• exclude_reply_user_ids : to remove preincluded people from answers to a conversation unuseful for gazou
• auto_populate_reply_metadata='true' when 'in_reply_to_status_id' set => don't count leading @mentions
• clean unneeded mentions
• test answer RT
• attachment_url for tweet quote only
• handle quote and img simultaneously
• handle check accouts on answerlast
• fix bug with mentions not preprocessed

Adapt !count:

• no more count imgs
• no more tweet quote
• handle quote and img simultaneously
• no more answered accounts
• adapt docs

Problem when answerlast to rt if not quoting original author

Handle Mastodon

python port : https://mastodonpy.readthedocs.io/en/latest/

• display user mentions
• display user repouets
• display user pouets ? (redondant des tweets, si différent?)
• display pouets matching search filters ? (potentiellement flood redondant des tweets)

handle text starting with numbers by adding a space after color code

Example with :

t6qt: Key to #OpenGov & #OpenData innovation? All players, sectors, jurisdictions must engage #OGT14 Ottawa Sept 16
https://www.eventbrite.ca/e/ottawa-open-government-grand-bazaar-friends-of-ogt14-talk-show-tickets-12518732867 —
https://twitter.com/6t6qt/statuses/505303902674485248

Warning on @accounts in !twitter shouln't activate on e-mails...

ERROR No module named pystache

Hello

With the last update bin/update.sh

ERROR Oups, looks like something is wrong somewhere in the code, shouldn't be committed...
ERROR No module named pystache
  File "/home/n3b/gazouilleur/gazouilleur/lib/templater.py", line 5, in <module>
    from pystache import Renderer

Configurize setpad to setanything

cf setquadrapero
https://github.com/RouxRC/gazouilleur/blob/master/gazouilleur/bot.py#L1526-L1543

Unable to start without pystache installed

Even when URL_STATS is not set in config.py, gazoulleur will currently fail to start as it is currently being imported unconditionally in the templater module:

ERROR No module named pystache
File "/opt/gazouilleur/gazouilleur/lib/templater.py", line 5, in <module>
from pystache import Renderer

I would except pystache would not be a hard requirement unless the web feature has been configured.

Thanks!

Update/Freeze dependencies

Add a !monitor command to follow changes on a webpage

Back:

• Add monitor/unmonitor commands
• Handle list pages
• Handle newsurl for pages
• Add feeder to collect html of pages
• handle page errors : warn once when 404 or 503 ? irrelevant
• Save html data to filesystem
• apply url_rewrite for relative links
• extract rawtext
• extract ordered links
• check if exists, store if not, else check diff and warn
• store screenshot via manet
• generate screenshot thumbnails
• adjust timer of feeder
• display url with version arguments for better backlog
• no warn if back and forth
• handle pdfs cf #42

Front:

• refacto templates
• build template for page on each diff
• load versions in iframe dynamically + update name in h2
• handle js diff views
• buttons to shift to fullscreen
• css full height
• switch iframe/screenshots
• screenshots by default
• visual first
• style slider and/or calendar versions
• juggle navigate or diff
• fix firefox css
• fix when only one version
• url arguments
• automatically choose left or right when clicking on a version
• add sources
• button goto next diff cf https://www.regardscitoyens.org/gazouilleur/monitor_regardscitoyens_reglement-senat.html#if&text
• make visual diff transparent or github diffimages lib? cf #43

[Bug] !ping or !pingteam generates an exception

On #sequanux chan, using !ping or !pingteam command generates an exception http://pastebin.com/ydzRAnNB

Some jpeg images not recognized as such ?

[twitter] Could not find a proper image to send at https://img.bfmtv.com/i/0/0/727/bfae5b415d841c22ecd5f62c791a3.jpeg (only jpeg, png & gif accepted).

No attribute NullHandler

I get this message when starting:

Failed to load application: 'module' object has no attribute 'NullHandler'

Add a --force option to !filter

to always filter tweets even when chan's account is quoted

Add fortune commands

Filter RTs with master account only quoted

Error with the last version (Twisted : connectionLost)

Log are here - > http://paste.debian.net/hidden/fa902b4f/

Mongod log (who seems working well)

Sat Nov 2 20:14:30.975 [initandlisten] connection accepted from 127.0.0.1:34028 #3 (1 connection now open)
Sat Nov 2 20:14:30.976 [conn3] authenticate db: Bauer { authenticate: 1, user: "Bauer", nonce: "xxxxxxxxxxxxxx", key: "xxxxxxxxxxxxxxxxxxxxxx" }

I did the update as mention on ReadMe.

I had problem to connect to irc with ssl (tried with socat and without) but with the update it seems that i have more problems.

Thanks in advance for your help.

--chan is correctly interpreted by runlater, but not removed

!runlater --chan=lqdn-rp 0 Aie

donne :

lqdn-rp : UnGarage2 : --chan=lqdn-rp Aie

indentation as a legibility feature in human-speaking channels

[WebMonitor] ImageDiff features on screenshots

cf github's image diff

!pingteam doesn't return correct users

Hello

When you use !pingteam and you have set channel users in config.py only GLOBAL_USERSare return.

Example on config.py :
GLOBAL_USERS = ['n3b', 'userG']
...

CHANNELS = {  'mychannel': {
...
    'USERS': ['n3b', 'user3', 'user4'],
},       

!pingteam hello :
botname n3b: hello n3b userG
!pingall hello :
botname n3b: hello n3b user3 user4

Show DM in different color

DM usually ask answers and a more specific treatment than regular tweets.
However, they are the same colors than the tweets and, therefore difficult to identify.

Displaying DM in another color, may help reader to better see them.

Thanks @RouxRC for the amazing jobs done

Handle quote tweets not being displayed with quote

ZEbergenyi: Please READ this   ⬇⬇⬇⬇⬇⬇ — https://twitter.com/ZEbergenyi/status/1023676227687657472
!show 1023676227687657472
ZEbergenyi (35 followers): Please READ this   ⬇⬇⬇⬇⬇⬇ « @CopernicusEU: With heat waves usually comes pollution episodes Look at Nitrogen dioxide levels in Holland, Belgium, Germany, Paris, as seen last Friday by the @tropomi instrument of our #Sentinel5P🇪🇺🛰 #SSP18 #OpenData https://twitter.com/CopernicusEU/status/1023607229705084930/photo/1 » — https://twitter.com/ZEbergenyi/status/1023676227687657472 (2018-07-29 23:06:46 - Twitter Lite - 1 ♥)

Add a new command '!favorite'

This command will allow chan users to favorite liked tweets easily.

display deleted tweets already stored with !show

Weird error in logs...

Found in logs...

2016-05-20 13:53:57 [MongoProtocol,client] Traceback (most recent call last):
2016-05-20 13:53:57 [MongoProtocol,client]   File "/usr/lib64/python2.7/logging/**init**.py", line 876, in emit
2016-05-20 13:53:58 [MongoProtocol,client]     stream.write(fs % msg.encode("UTF-8"))
2016-05-20 13:53:58 [MongoProtocol,client] UnicodeDecodeError: 'ascii' codec can't decode byte 0xe2 in position 96: ordinal not in range(128)
2016-05-20 13:53:58 [MongoProtocol,client] Logged from file filelogger.py, line 35

Add chan option to hide @users from tweet answers when more than N

option to specify tweets (not) to be printed per account and type (new, rt, answer)

The idea is an option to filter, for instance, self-tweets, so that while using %twitter or %answer, the full message (which was already printed on the IRC channel while sending the command) would NOT be printed "back", but only the tweet ID.

This would not remove any functionality, while reducing the amount of the flow on the IRC channel.

Handle new quoted status format

https://twittercommunity.com/t/updating-how-urls-are-rendered-in-the-quote-tweet-payload/105473

Fix rewrite TweetUrlImg on long replies

Add colors on twitter and news feeds

Hello

Having the ability to enable colors on feeds to be more readable.

Colors will follow the MIRC colors.

Similars works has been made on Willie irc bot, on rss.py module

Example here on irssi.

Thanks

Fix url rewrite in emails

example:
laquadrature: @Thibaultj Le plus simple est de nous envoyer un mail à contact[at]http://laquadrature.net, on vous fera un reçu/une facture selon vos besoins. A noter cependant qu'il ne peut s'agir de reçu fiscal car malheureusement les dons faits à La Quadrature ne sont pas déductibles des impôts — https://twitter.com/laquadrature/status/950827831562563584

Fix count problems

• when "img:" present , exemples:

!count Que dit http://www.lafabriquedelaloi.fr/articles.html?loi=pjl15-nouvelles_libertes_entreprises_actifs le texte du #PJLTravail adopté hier suite au #49al3? Explorez les diffs ds LaFabrique img:http://framapic.org/BE2C7r79bn3v/ehm9UJ8tRnUX.png
!count http://www.lafabriquedelaloi.fr/articles.html?loi=pjl15-nouvelles_libertes_entreprises_actifs Que dit le texte du #PJLTravail adopté hier suite au #49al3? Explorez les diffs ds LaFabrique img:http://framapic.org/BE2C7r79bn3v/ehm9UJ8tRnUX.png
!count Que dit le texte du #PJLTravail adopté hier suite au #49al3? Explorez les diffs ds LaFabrique http://www.lafabriquedelaloi.fr/articles.html?loi=pjl15-nouvelles_libertes_entreprises_actifs img:http://framapic.org/BE2C7r79bn3v/ehm9UJ8tRnUX.png

• Urls with "..." in the middle

!count https://github.com/regardscitoyens/Collaborateurs-Senat/compare/c1e32f9823...29addbd56f

Add users' links to digest

Include links posted by users in the digest.

Handle new TLS

diff --git a/gazouilleur/bot.py b/gazouilleur/bot.py
index 89561d7..b23e06c 100644
--- a/gazouilleur/bot.py
+++ b/gazouilleur/bot.py
@@ -1592,17 +1592,12 @@ class IRCBotFactory(protocol.ReconnectingClientFactory):
     if not hasattr(config, "SOLITARY") or str(config.SOLITARY).lower() == "false":
         channels.append("#gazouilleur")
 
-
-class ClientTLSContext(ssl.ClientContextFactory):
-    isClient = 1
-    def getContext(self):
-        return SSL.Context(SSL.TLSv1_METHOD)
-
+ssl_options = ssl.optionsForClientTLS(hostname=config.HOST)
 
 # Run as 'python gazouilleur/bot.py' ...
 if __name__ == '__main__':
     if is_ssl(config):
-        reactor.connectSSL(config.HOST, config.PORT, IRCBotFactory(), ssl.ClientContextFactory())
+        reactor.connectSSL(config.HOST, config.PORT, IRCBotFactory(), ssl_options)
     else:
         reactor.connectTCP(config.HOST, config.PORT, IRCBotFactory())
     log.startLogging(sys.stdout)
@@ -1614,7 +1609,7 @@ elif __name__ == '__builtin__':
     filelog.timeFormat = "%Y-%m-%d %H:%M:%S"
     application.setComponent(log.ILogObserver, filelog.emit)
     if is_ssl(config):
-        ircService = internet.SSLClient(config.HOST, config.PORT, IRCBotFactory(), ClientTLSContext())
+        ircService = internet.SSLClient(config.HOST, config.PORT, IRCBotFactory(), ssl_options)
     else:
         ircService = internet.TCPClient(config.HOST, config.PORT, IRCBotFactory())
     ircService.setServiceParent(application)

from lunar

[WebMonitor] Handle diffs on PDF source

Unable to start without wand installed

Even when URL_MANET is not set, gazouilleur will currently fail to start unless wand is installed. Looking at the code, it seems there's no reason it should be a hard dependency in this case. Otherwise requierements.txt probably need to be updated.

docker build fails

fyi, the docker file tries to install the mongodb package, which was removed from alpine due to copyright issues.

Stream isn't working

2016-05-19 16:32:01 [stream/#mgmtconfig] Start search streaming for: « mgmtconfig »
2016-05-19 16:32:01 [stream/#mgmtconfig] ERROR Stream crashed with <class 'twitter.api.TwitterHTTPError'>: Twitter sent status 401 for URL: 1.1/statuses/filter.json using parameters: (filter_level=none&....
        details: <html>\n<head>\n<meta http-equiv="Content-Type" content="text/html; charset=utf-8"/>\n<title>Error 401 Unauthorized</title>
        </head>
        <body>
        <h2>HTTP ERROR: 401</h2>
        <p>Problem accessing '/1.1/statuses/filter.json'. Reason:
        <pre>    Unauthorized</pre>
        </body>
        </html>

Any idea why?

Remove code that handles deprecated legacy DM tweets

cf https://twittercommunity.com/t/retiring-legacy-dm-commands-on-the-standard-tweet-api/86653