matrix-org/matrix-appservice-irc

Node.js IRC bridge for Matrix

Matrix IRC Bridge

This is an IRC bridge for Matrix. If you're upgrading from an old release, be sure to read the CHANGELOG as there may be breaking changes between releases.

This bridge will pass all IRC messages through to Matrix, and all Matrix messages through to IRC. It is highly configurable and is currently used on the matrix.org homeserver to bridge a number of popular IRC networks.

We maintain a list of bridged IRC networks here.

What does it do?

On startup, the bridge will join Matrix clients to the IRC channels specified in the configuration file. It will then listen for incoming IRC messages and forward them through to Matrix rooms Each real Matrix user is represented by an IRC client, and each real IRC client is represented by a Matrix user. Full two-way communication in channels and PMs are supported, along with a huge array of customisation options.

Usage

To learn how to use the bridge, see our usage guide.

Setting up your own bridge

You will need a Matrix homeserver to run this bridge. Any homeserver that supports the AS API should work.

See the getting started docs for instructions on how to set up the bridge.

Configuration

See the sample config file for an explanation of the configuration options available.

Documentation

Documentation can be found on GitHub Pages.

You can build the documentaion yourself by:

# Ensure that Rust is installed on your system.
# cargo install mdbook
mdbook build
sensible-browser book/index.html

Contributing

Please see the CONTRIBUTING file for information on contributing.

matrix-appservice-irc's Issues

Check connection notices don't lie

The irc bridge lies to you when it connects you and the target name is taken. It connected me as M-turt2live1 (cause M-turt2live was taken), but said it connected me as M-turt2live

By @ turt2live:matrix.org

Auto-pastebin large messages to avoid spamming

https://matrix.org/jira/browse/BOTS-64

Duplicate messages matrix-side (due to distributed IRCds)

Created by @ matthew:matrix.org.

e.g. roblabla on https://vector.im/beta/#/room/!cURbafjkfsMDVwdRDQ:matrix.org/$146192559593207XoTMY:matrix.org and half-shot a few days earlier

Provide some way of explicitly disconnecting from an IRC network

https://matrix.org/jira/browse/BOTS-94

bridged IRC users should quit the target network if they are not on any channels or offer some other way to quit the network. satdav claims he's still on moznet despite having left all the moznet channels

---

We cannot just quit the network if channels == 0 because otherwise you become impossible to PM. We should have a !quit or equivalent though.

NAMEs bucket doesn't seem to be popped

Commit: https://github.com/matrix-org/matrix-appservice-irc/tree/d1db5cc23e798bbbea97f5611c22425737543c35

Toml showed me some logs which indicated that the NAMEs bucket wasn't being processed (just pushed, and only 1 popped even though the request succeeded). It's possible that some refactoring has broken this so the bucket isn't purged, need to look into it more.

Manifests itself as initial IRC->M syncing not working even though it is enabled in the config (since it relies on that bucket).

Handle presence sensibly

https://matrix.org/jira/browse/BOTS-98
irc bridge AS appears as offline in presence, which is offputting and confusing

No idea what "sensibly" is though, or what it should really be representing.

Report prometheus stats for irc joins/parts

so we can more easily see when we're being DoSed by join-floods

Origin: https://matrix.org/jira/browse/BOTS-180

The bot may violate IRC networks' terms of use

When trying to bridge between ircs://irc.indymedia.org/#ecobytes and #ecobytes:matrix.allmende.io, it has turned out the appservice is being banned quickly when forking into the virtual Matrix users.

At least the docs should carry a fair warning above the installation instructions, and provide instructions whom to speak with from popular IRC networks.

For Indymedia, it's #ircd. If this is a naming pattern that applies to all PoPs, the better.

Support !whois

https://matrix.org/jira/browse/BOTS-182

This is specifically so users can check to see if the IRC user they want to speak to is online, given we can't do member list syncing I->M

Related: #49

`ECONNRESET` on `/initialSync` isn't gracefully handled

https://matrix.org/jira/browse/BOTS-147

This may now be fixed, need to investigate.

If we're not syncing matrix->irc users, then irc->matrix bridge traffic is lost until someone speaks on the matrix side of the bridge

https://matrix.org/jira/browse/BOTS-129

---

I think this is fixed now, need to check.

Handle external entities

If you have Github set up to splat commits into an IRC channel, it's done by using bot users which never join the channel specified, but emit notices instead. The IRC AS treats this as a real user, provisions an account and sends a message from that bot user, but never parts it again, leading to lots and lots of GitHub123456789 users on the Matrix side.
We need to think of how we can gracefully do this, possibly by sending the messages from the AS itself? @appservice-irc:domain.com

https://matrix.org/jira/browse/BOTS-107

Support +k channels

Any plans of supporting these? I really need it. :-/

Are channel redirects handled?

eg #foo -> ##foo

https://matrix.org/jira/browse/BOTS-66

Scrollback from remote newly-joined IRC bridge rooms produces blank events

https://matrix.org/jira/browse/BOTS-148

Blank as in "redacted" events:

{
  "content": {},
  "event_id": "$1451174561228692jZHfy:matrix.org",
  "origin_server_ts": 1451174561380,
  "room_id": "!SSbUDRwkvhDxtoRDXg:matrix.org",
  "type": "m.room.message",
  "user_id": "@freenode_taaem:matrix.org"
}

No idea which part of IRC bridge, Synapse, Federation is causing this.

IRC invite-only channels and bans

At the moment, when a matrix-bridged user tries to join an IRC channel through the bridge, the bridge tries to make an equivalent IRC user join the channel on the IRC network. However, if the user fails to join the channel (eg. invite-only or banned user), the user will be allowed to enter the matrix room. As a consequence, if any other IRC-bridge user is allowed onto the IRC channel, the disallowed matrix user will be able to read all the messages sent to the IRC channel ; which is unexpected.

Example 1:

• Matrix users M-A and M-B join IRC channel #I_#c
• M-B is kickbanned from #I_#c via IRC
• M-B can still read all the messages in #I_#c until it is kickbanned using matrix

Example 2:

• Matrix user M-A joins invite-only channel #I_#c
• Matrix user M-B, despite never having been invited into #I_#c, can just join the corresponding matrix room and listen to all the conversations, without any IRC user ever being aware he is there, given the equivalent IRC user cannot join the IRC channel.

I hope to be clear enough.

Thank you for all the work already done!

IRC bot shouldn't invite guest users to an Admin room as they cannot join

Created by @ matthew:matrix.org.

IRC->Matrix PMs are ignored unless the IRC nick being PMed has precisely the right case

https://matrix.org/jira/browse/BOTS-152

IRC servers on IPv6 addresses need to be entered directly

https://matrix.org/jira/browse/BOTS-136

If you type a domain, it'll do the AAAA-lookup but then use v4, obviously. If it find a valid v6, it should really be using it. This is on Node v0.10.x

Re-invite matrix users who leave their PM room

If you have a PM room with an IRC user and then the Matrix user leaves the room, any subsequent messages the IRC user sends will be black holed effectively. We should invite the user who left back to the room (or create a new room and clobber, which may be easier to avoid constantly checking room state)

Support masquerading real matrix users

The ability to link a registered IRC nick to a specific Matrix account
https://matrix.org/jira/browse/BOTS-4

Tests for bans

Really should have them.

https://matrix.org/jira/browse/BOTS-85

Inviting an IRC AS virtual user into an existing Matrix room should work.

Matthew: I've seen people do this several times and be disappointed that it doesn't spontaneously create a private IRC channel which mirrors the Matrix room in question

https://matrix.org/jira/browse/BOTS-125

Can tightloop on massive netsplits

https://matrix.org/jira/browse/BOTS-160
https://matrix.org/jira/browse/BOTS-155

This may be resolved in the big refactor on develop?

Bridge chan ops in a sane way

Ideally chan ops on the IRC side should have power in Matrix rooms too.

https://matrix.org/jira/browse/BOTS-57

Track IRC users by hostmask, not by nick

IRC -> matrix user mapping should use the IRC user's ident (usually something like nick!user@host, which is also used for bans etc) instead of the current nick to identify IRC users. This is needed for #71 among other things.

Settings are mixed up between IRC servers

Settings are mixed up between IRC servers.
For example:

ircService:
  servers:
    irc.server1.net:
      port: 9999
    irc.server2.net:
      port: 6667

matrix-appservice-irc will sometimes try to connect to irc.server1.net on port 6667, or irc.server2.net on port 9999. It seems to be order dependent; I can get it to work by reordering the servers.

IRC inverted colour scheme doesn't map to Matrix

https://matrix.org/jira/browse/BOTS-157

Notably background colours.

Check state_key of m.room.topic

https://matrix.org/jira/browse/BOTS-173

IRC AS honours errant m.room.topic events sent as non-state events, when surely it should only honour them if state events.

Erroneous nick changes on freenode

I've been trying to change my nick using the appservice (@appservice-irc:matrix.org) on the freenode server.

It doesn't change my nick, but informs me of some other nick change after a long time (3-9 minutes). I making a wild guess that the nick change request is getting timed out and NickServ is just sending the last nick change as response.

I've noticed this only on freenode. I tried on MozNet (@mozilla-irc:matrix.org) and it worked with no issues.

PS : I'm attaching a screenshot for reference.

Support sending arbitrary IRC commands through

https://matrix.org/jira/browse/BOTS-27

The use case here is to send explicit NICK and PASS irc commands.

We need to think about how this will affect the bridge semantics (doing something via raw IRC commands vs via the bridge).

Track nick changes on IRC

IRC users changing nicks should not spawn new matrix users, rather update their display names.
#72 is needed for this.

Need a way to tell Matrix users when an IRC user they are PMing has disconnected

https://matrix.org/jira/browse/BOTS-109

bridge relayed irc->matrix pm as an invite but the actual msg only appeared after a HS restart

https://matrix.org/jira/browse/BOTS-171

DB performance when using `[email protected]` is not acceptable

This issue is currently blocking a release from develop to master, and is nominally affecting the Freenode deployment on Matrix.org (the other IRC networks like Moznet are running develop happily as their workloads are less intense).

---

The issue

When you try to run develop on the Freenode deployment, things start well but quickly begin to slow down to a crawl before almost stopping. Logs begin to slow down then stop. CPU usage on the box indicates it is spinning at 100% CPU usage.

Investigation

After CPU profiling with node-inspector, it became clear this was a problem with NeDB. It was starving the event loop from executing tasks because it kept running this function: https://github.com/louischatriot/nedb/blob/master/lib/datastore.js#L309

More investigation indicated that the cause of this was that the IRC bridge was triggering tens of thousands of find queries, which NeDB was attempting to service. Adding in trace logging to the DataStore calls indicates that this is a problem with getIrcChannelsForRoomId which would be called with around 1000 room IDs on startup. This triggers 2 find queries (one to fetch the mapping, one to fetch the mapped room data).

Proposed solutions

• Batch find statements using $in - The bridge knows the full set of rooms it needs to query, but then queries them one at a time.
• Change the database format to store 1 row per complete mapping, ditching the idea of room entries. This would halve the number of queries as a single find would produce all the data required, rather than triggering another find to retrieve the rest of the room data.

Cannot --generate-config because EADDRINUSE on ident service

ircas tries to bind() the TCP port for the ident service before it generates the registration file. This means it's impossible to regenerate the registration file after editing config, if the service is already running.

#freenode_#archlinux:matrix.org doesn't seem to be bridged correctly

Created by @ matthew:matrix.org.

Ident with SSL doesn't work

https://matrix.org/jira/browse/BOTS-51

Reconnects forever because we don't get a callback in time.

Support 1:1 to group chat upgrades

https://matrix.org/jira/browse/BOTS-22

Generated regexes in appservice config accept any HS

The namespaces generated by the generate command are too liberal, in that they will also forward messages from remote rooms that have their own aliases.

For example, my AS is configured with a #freenode_ alias. I would expect it to only bridge channels that have a LOCAL alias e.g. #freenode_#matrix:cmc.im. However, the generated config :

namespaces:
  users:
    - exclusive: true
      regex: '@freenode_.*:.*'
  aliases:
    - exclusive: true
      regex: '#freenode_.*:.*'

means that my HS is sending events from #freenode_#matrixhq:matrix.org to my AS.

Generated config produces incomplete Required string fields

After generation of the appservice configuration YAML, which is to be loaded by the homeserver, the latter complains:

matrix_1 | 2016-06-07 13:39:55,498 - synapse.storage.appservice - 238 - ERROR - - Failed to load appservice from '/data/appservice-registration-irc.yaml'
matrix_1 | 2016-06-07 13:39:55,498 - synapse.storage.appservice - 239 - ERROR - - "Required string field: 'id' (/data/appservice-registration-irc.yaml)"
matrix_1 | Traceback (most recent call last):
matrix_1 |   File "/usr/local/lib/python2.7/dist-packages/synapse/storage/appservice.py", line 215, in load_appservices
matrix_1 |     hostname, yaml.load(f), config_file
matrix_1 |   File "/usr/local/lib/python2.7/dist-packages/synapse/storage/appservice.py", line 155, in _load_appservice
matrix_1 |     field, config_filename,
matrix_1 | KeyError: "Required string field: 'id' (/data/appservice-registration-irc.yaml)"
matrix_1 | Traceback (most recent call last):
matrix_1 |   File "/usr/lib/python2.7/runpy.py", line 162, in _run_module_as_main
matrix_1 |     "__main__", fname, loader, pkg_name)
matrix_1 |   File "/usr/lib/python2.7/runpy.py", line 72, in _run_code
matrix_1 |     exec code in run_globals
matrix_1 |   File "/usr/local/lib/python2.7/dist-packages/synapse/app/homeserver.py", line 750, in <module>
matrix_1 |     main()
matrix_1 |   File "/usr/local/lib/python2.7/dist-packages/synapse/app/homeserver.py", line 745, in main
matrix_1 |     hs = setup(sys.argv[1:])
matrix_1 |   File "/usr/local/lib/python2.7/dist-packages/synapse/app/homeserver.py", line 420, in setup
matrix_1 |     hs.setup()
matrix_1 |   File "/usr/local/lib/python2.7/dist-packages/synapse/server.py", line 120, in setup
matrix_1 |     self.datastore = DataStore(self.get_db_conn(), self)
matrix_1 |   File "/usr/local/lib/python2.7/dist-packages/synapse/storage/__init__.py", line 178, in __init__
matrix_1 |     super(DataStore, self).__init__(hs)
matrix_1 |   File "/usr/local/lib/python2.7/dist-packages/synapse/storage/registration.py", line 29, in __init__
matrix_1 |     super(RegistrationStore, self).__init__(hs)
matrix_1 |   File "/usr/local/lib/python2.7/dist-packages/synapse/storage/appservice.py", line 39, in __init__
matrix_1 |     hs.config.app_service_config_files
matrix_1 |   File "/usr/local/lib/python2.7/dist-packages/synapse/storage/appservice.py", line 215, in load_appservices
matrix_1 |     hostname, yaml.load(f), config_file
matrix_1 |   File "/usr/local/lib/python2.7/dist-packages/synapse/storage/appservice.py", line 155, in _load_appservice
matrix_1 |     field, config_filename,
matrix_1 | KeyError: "Required string field: 'id' (/data/appservice-registration-irc.yaml)"

and indeed the configured configuration

url: 'http://matrix.allmende.io:9999'
as_token: 
hs_token: 
sender_localpart: appservice_irc
namespaces:
  users:
    - exclusive: true
      regex: '@indymedia_.*:.*'
  aliases:
    - exclusive: true
      regex: '#indymedia_.*:.*'
  rooms:
    - exclusive: false
      regex: '#ecobytes:matrix.allmende.io'

does not bring an id.

What have I done wrongly after filling out the config.yaml respective to my case?

Does not support IRC server password

I run a personal IRC server with a server password. Currently the 'appservicebot' is able to connect, but virtual users cannot as there's nowhere to specify a password for them.

Support hot redeploys

https://matrix.org/jira/browse/BOTS-83

200 OKs transactions with an invalid HS token

https://matrix.org/jira/browse/BOTS-167

May be more of a matrix-appservice-node thing though

Bad nick change confirmation messages

Because:

• We don't handle all nick errors which the ircd can return and then cancel the NICK listener.
• We don't time out the NICK listener.

!nick bug: Handle Bahamut's 435 error code

There's been reports of !nick commands just black holing and not changing nicks. After digging into this, it turns out that Bahamut will send a non-standard 435 response of the form:

<current_nick> <new_nick> <#channel> :Cannot change nickname while banned on channel

The bridge should feed this back to the client and cancel the NICK listener. This may require the node-irc library at https://github.com/matrix-org/node-irc to be updated to recognise this IRC code.

Propagate matrix display name changes to IRC (and vice versa)

https://matrix.org/jira/browse/BOTS-52

Matrix to irc propagation is hard because the two protocols scope names differently. Matrix allows per room display names but irc is per connection (global). We could manage this by just applying the most recent name change, but we need to accept mismatches are unavoidable.
Irc to matrix propagation is possible, though fiddly to implement (since the nick change necessitates a new user id, invites to all of the rooms the user was previously in, etc).

After thinking about this more, I wonder whether it would be more accurate to map Matrix user IDs to IRC user@host instead - in which case nick changes are just display name changes. Other points still hold though.

Doesn't work with guest access

2016-01-17 03:17:10 ERROR:client-connection [email protected]: {"command":"ERROR","rawCommand":"ERROR","commandType":"normal","args":["Closing Link: ns501493.ip-142-4-215.net (Invalid username [-qkkcjkomk])"]}

Bridge WebRTC call events

E.g. by turning it into a URL which users can click to join the call. Interesting question is how we can do this without preferring one client over another in the bridge code.

https://matrix.org/jira/browse/BOTS-68

Ability to set your matrix displayname from IRC by msging MatrixBridge

https://matrix.org/jira/browse/BOTS-3