turt2live/matrix-appservice-webhooks

Slack-compatible webhooks for matrix

Planned obsolescence

Check out matrix-hookshot as a replacement for this archaic bridge :)

matrix-appservice-webhooks

Slack-compatible webhooks for Matrix. Talk about it on Matrix: #webhooks:t2bot.io

Usage

Invite the webhook bridge to your room (@_webhook:t2bot.io) and send the message !webhook. The bridge bot will then send you a link to send messages to in a private message. You must be able to configure the room in order to set up webhooks.

JSON Body (for posting messages)

{
  "text": "Hello world!",
  "format": "plain",
  "displayName": "My Cool Webhook",
  "avatarUrl": "http://i.imgur.com/IDOBtEJ.png"
}

Format can be plain or html. Emoji will be converted automatically(:heart: becomes ❤); set the emoji property to false to disable this conversion. To send a notice or emote, add "msgtype" : "notice" or "msgtype" : "emote" in your request.

Installation

Before you begin: A matrix homeserver and Node 9 or higher are required.

1. Clone this repository and install the dependencies

   git clone http://github.com/turt2live/matrix-appservice-webhooks
   cd matrix-appservice-webhooks
   npm install
   

2. Copy config/sample.yaml to config/config.yaml and fill in the appropriate fields

3. Generate the registration file

   node index.js -r -u "http://localhost:9000" -c config/config.yaml
   

   Note: The default URL to run the appservice is http://localhost:9000. If you have other appservices, or other requirements, pick an appropriate hostname and port.

4. Copy/symlink the registration file to your Synapse directory

   cd ~/.synapse
   ln -s ../matrix-appservice-webhooks/appservice-registration-webhooks.yaml appservice-registration-webhooks.yaml
   

5. Add the registration file to your homeserver.yaml

   ...
   app_service_config_files: ["appservice-registration-webhooks.yaml"]
   ...
   

6. Restart Synapse (synctl restart, for example)

Running

Using the port specified during the install (9000 by default), use node index.js -p 9000 -c config/config.yaml -f appservice-registration-webhooks.yaml from the repository directory.

The bridge should start working shortly afterwards.

Docker

A Docker image of the bridge is available to host the bridge yourself. The image can be built yourself with docker build -t matrix-appservice-webhooks . or you can use the image on docker.io:

docker run -p 9000:9000 -v /path/to/webhooks/dir:/data turt2live/matrix-appservice-webhooks

The /path/to/webhooks/dir should have an appservice-registration-webhooks.yaml file, config.yaml, and database.json. Additional bridge-related data will be stored here.

Example appservice-registration-webhooks.yaml

id: webhooks
hs_token: A_RANDOM_ALPHANUMERIC_STRING  # CHANGE THIS
as_token: ANOTHER_RANDOM_ALPHANUMERIC_STRING  # CHANGE THIS
namespaces:
  users:
    - exclusive: true
      regex: '@_webhook.*'
  aliases: []
  rooms: []
url: 'http://localhost:9002'  # you may need to change this (this should point at the bridge)
sender_localpart: _webhook
rate_limited: false
protocols: null

matrix-appservice-webhooks's Issues

No "appservice-registration-webhooks.yaml" in the docker image

The docker image will not start, mentioning the missing "appservice-registration-webhooks.yaml" file. There is no sample file in the github repository which I can modify, hence I'm stuck with that missing file.

Can this be uploaded in the repo, please?

Dockerfile

Im trying to dockerize this project but I'm a bit at loss at how to deal with registration file generation. Right now my best bet is to do it in the image and have the user mount a dir to access it. Feels wrong but generating it outside of the container means depending on node which kinda defeats the purpose..

You got any ideas?

Slack commands

• <!channel> becomes @channel
• <!channel|World> becomes @channel with display text World

Other commands:

• !group - alias for !channel
• !here
• !everyone

Deprecate port 4501

We should just use the existing appservice port.

It is actually authed, despite bugs.

PM notification should only appear if the command was issued in that room

The bridge bot joins rooms when people say !webhook in a room with a different bridge bot

1. Join a room with a webhook bridge already there (not yours)
2. Send the message !webhook
3. Your bridge will try to join the room

Only changing emoji in webhook name leads to display name changed events

Should somehow treat this as a different hook, possibly by using the raw name and not the converted name

Different Slack formats

I've tried out the webhooks with a couple of services I have and it seems that they use a bunch of different formats. Since you've been looking into Slack maybe you can explain?

Sonarr for example works OOTB.

Grafana sends something like this:

{
  "attachments": [
    {
      "color": "#D63232",
      "fallback": "[Alerting] Test notification",
      "fields": [
        {
          "short": true,
          "title": "High value",
          "value": 100
        },
        {
          "short": true,
          "title": "Higher Value",
          "value": 200
        },
        {
          "short": false,
          "title": "Error message",
          "value": "This is only a test"
        }
      ],
      "footer": "Grafana v4.4.1",
      "footer_icon": "https://grafana.com/assets/img/fav32.png",
      "image_url": "http://grafana.org/assets/img/blog/mixed_styles.png",
      "text": " Someone is testing the alert notification within grafana.",
      "title": "[Alerting] Test notification",
      "title_link": "https://example.com/",
      "ts": 1505222577
    }
  ],
  "parse": "full",
  "web_request": {
    "attachments": [
    ],
    "parse": "full"
  }
}

sentry.io gives me this:

{
  "payload": "{\"username\":\"Sentry\",\"icon_url\":\"http://myovchev.github.io/sentry-slack/images/logo32.png\",\"attachments\":[{\"color\":\"#f43f20\",\"fields\":[{\"short\":false,\"value\":\"raven.scripts.runner in main\",\"title\":\"Culprit\"},{\"short\":true,\"value\":\"My project\",\"title\":\"Project\"}],\"fallback\":\"[My project] This is an example Python exception\",\"title_link\":\"https://sentry.io/company/project/issues/123456789/?referrer=slack\",\"title\":\"This is an example Python exception\"}]}"
}

Re-enable profile settings in config

Display name and join can race

Previously there was an assumption that the profile could be set async to the join so long as both completed before the message was sent. Changes in how the synapse workers actually work changes this assumption to no longer be accurate - we need to block on profile changes before sending a join.

Issue with notification from Gitlab

Hi,

I was able to install the appservice for my HS successfully. But I have some problems with notification from GitLab (Slack notifications).
I receive the notification but there are no usable links inside but in the source of the message in Riot side I see some weird stuff...

I wonder if you could help me.

Here the string I captured with tcpdump:

payload=%7B%22username%22%3A%22Gitlab%22%2C%22attachments%22%3A%5B%5D%2C%22fallback%22%3Anull%2C%22text%22%3A%22Jakab+Gipsz+%28jgipsz%29+closed+%5Cu003chttps%3A%2F%2Fexample.com%2Fadmin%2Fgitlab%2FProject1%2Fansible%2Fmerge_requests%2F179%7C%21179+*Test+webhook*%5Cu003e+in+%5Cu003chttps%3A%2F%2Fexample.com%2Fadmin%2Fgitlab%2FProject1%2Fansible%7CProject1%2Fansible%5Cu003e%3A+*Test+webhook*%22%7D

After urldecode:

payload={ "username":"Gitlab", "attachments":[], "fallback":null, "text":"Jakab Gipsz (jgipsz) closed \u003chttps://example.com/admin/gitlab/Project1/ansible/merge_requests/179|!179 *Test webhook*\u003e in \u003chttps://example.com/admin/gitlab/Project1/ansible|Project1/ansible\u003e: *Test webhook*"}

And the msg source from Riot:

{ "origin_server_ts": 1527690275010, "sender": "@_webhook__MzjJH38dhjksWeeScb_example_com_Gitlab:example.com", "event_id": "$1525678268297yHRBu:example.com", "unsigned": { "age": 216 }, "content": { "body": "", "msgtype": "m.text", "formatted_body": "Jakab Gipsz (jgipsz) closed <https: example.com=\"\" admin=\"\" gitlab=\"\" project1=\"\" ansible=\"\" merge_requests=\"\" 179|!179=\"\" *test=\"\" webhook*=\"\"> in <https: example.com=\"\" admin=\"\" gitlab=\"\" project1=\"\" ansible|project1=\"\" ansible=\"\">: *Test webhook*</https:></https:>", "format": "org.matrix.custom.html" }, "type": "m.room.message", "room_id": "!MzjJH38dhjksWeeScb:example.com" }

And all I can see in Riot client:

Jakab Gipsz (jgipsz) closed in: *Test webhook*

Without any useful links... :-o

Broken HTML generated by from_slack_attachments.js (missing closing quote on attribute)

E.g. combinedHtml += "<blockquote data-mx-border-color='" + color + "'>";

Should be combinedHtml += "<blockquote data-mx-border-color=\"" + color + "\">";

This causes postprocess/upload_images.js to write content to matrix.event.formatted_body with nothing contained in any of the HTML tags, resulting in blank messages.

Replacing all ' with \" in from_slack_attachments.js fixes.

Markdown processing

This is what Slack wants

{
    "text": "*bold* `code` _italic_ ~strike~",
    "username": "markdownbot",
    "mrkdwn": true
}

This will use the mrkdwn property if it exists, but otherwise assume format as the default, ie:

{
    "text": "*bold* `code` _italic_ ~strike~",
    "username": "markdownbot",
    "format": "markdown"
}

icon_emoji fallback for avatar

If icon_emoji exists, use it if avatarUrl is missing.

Can be a URL or :shark: (for example)

database.json should magically work in Docker. (was: Webhook IDs are lost when service is restarted)

Webhook IDs are disappearing every time I restart the service.

Log displays the warning Using unknown data type INTEGER during database migration.

Loading config file /data/config.yaml
info [index] Preparing database...,
info [WebhookStore] Running migrations,
[INFO] Processed migration 20170708201820-create-bot-account-data-table,
[INFO] Processed migration 20170708235052-create-webhooks-table,
[WARN] Using unknown data type INTEGER,
[INFO] Processed migration 20170709024537-create-account-data-table,
[INFO] Processed migration 20170709024544-drop-bot-account-data,
[INFO] Done,

room-store.db and user-store.db files are created but empty (0kb).

20170709024537-create-account-data-table.js on line 19 I changed:

type: 'integer'

to

type: 'int'

(see https://github.com/db-migrate/shared/blob/8710b084b2b437a0bac3448eccd9d24ab08f08e1/data_type.js)

This removed the integer warning. *.db files are still empty and webhooks are lost after restart.

Slack-style user linking

<@travis:t2l.io|TravisR>

CircleCI slack webhooks doesn't works

Tried to use with CircleCI, getting Error: Hook failed. on CircleCI side.

Logs says Invalid message: missing text or attachments.

By using tcpdump I see the json as: {"attachments":[{"fallback":"Hello from CircleCI","text":"Hello from CircleCI","color":"#AAAAAA"}],"text":"","channel":""}

Shouldn't that already be covered by #17 or #32 ?

Running latest turt2live/matrix-appservice-webhooks:latest docker image.

Optional parameter for m.notice

It would be nice to be able to send notices as well

Process slack links

Blocked by #13

Only do transforms on slack endpoint.

• <https://alert-system.com/alerts/1234|Click here> becomes <a href='https://alert-system.com/alerts/1234'>Click here</a>
• <@user:matrix.org:Bob> becomes <a href='https://matrix.to/#/user/@user:matrix.org'>Bob</a>
• <#room:matrix.org|Room> becomes <a href="https://matrix.to/#/#room:matrix.org">Room</a>

Webhook naming

Handle room upgrades

When writing "!webhook" into a room which already has a webhook, an additional URL is created

I have a room where I got a webhook by posting "!webhook" after inviting the bridge. Works like a charm.

When a room admin later also writes "!webhook", he'll see a message:

I've sent you a private message with your hook information

In the private message there is a new URL anytime "!webhook" is written. The initial URL is still working (which is correct), but I'm wondering why a new URL is created every time. If a room has a webhook, the bridge should send the already existing webhook all the time instead of creating a new URL.

BTW: I did not test if the bridge answers the "!webhook" message to non-moderators or admins of the room. Normal members should not get any links where they can send messages via webhook IMHO.

Lint the project

Same webhook name in two different rooms

Using the same webhook display name, using two different rooms (webhook links) causes very strange results in the second room.

Fallback on non-Slack messages

Actually it would be nice to have a fallback for non-Slack messages too! Because sometimes stripping tags generates a weird message :)

Recalculate admin rooms when the partner leaves

Present possibility of outbound webhooks

Add a direction to everything (or similar)

Bridge changed it's own avatar without changing the config

At 10:24 (+1) the bridge changed it's own avatar for no particular reason it seems. It crashed a few minutes earlier for seemingly unrelated reasons.

Rate limit incoming webhook

Because spam

Support for authed webhooks

For services where you want to put the webhook URL in, but can't reasonably treat the whole URL as a secret

Application service has not registered this user

When following your instructions from README.MD and starting the server with node index.js -p 9000 -c config/config.yaml I get the following error output:

Nov-15-2017 11:57:05.964 +01:00 verbose [matrix-appservice-bridge] [-] GET http://localhost:8008/_matrix/client/r0/joined_rooms (AS) Body: 
Nov-15-2017 11:57:05.971 +01:00 info [WebService] API now listening on 0.0.0.0:4501
Nov-15-2017 11:57:05.995 +01:00 verbose [WebhookStore [SQL]] Executing (default): SELECT `id`, `objectId`, `key`, `value` FROM `account_data` AS `account_data` WHERE `account_data`.`objectId` = 'bridge';
Nov-15-2017 11:57:06.007 +01:00 verbose [matrix-appservice-bridge] [-] GET http://localhost:8008/_matrix/client/r0/joined_rooms (AS) HTTP 200 {"joined_rooms":[]}
Nov-15-2017 11:57:06.013 +01:00 verbose [utils] Downloading image from http://i.imgur.com/IDOBtEJ.png
Nov-15-2017 11:57:06.016 +01:00 verbose [matrix-appservice-bridge] [-] POST http://localhost:8008/_matrix/client/r0/register (AS) Body: {"auth":{},"username":"_webhook"}
Nov-15-2017 11:57:06.033 +01:00 error [matrix-appservice-bridge] [-] POST http://localhost:8008/_matrix/client/r0/register (AS) HTTP 400 Error: {"errcode":"M_USER_IN_USE","error":"User ID already taken."}
Nov-15-2017 11:57:06.036 +01:00 verbose [matrix-appservice-bridge] [-] GET http://localhost:8008/_matrix/client/r0/profile/%40_webhook%3Alocalhost/displayname (@_webhook:localhost) Body: 
Nov-15-2017 11:57:06.070 +01:00 verbose [matrix-appservice-bridge] [-] POST http://localhost:8008/_matrix/media/v1/upload (@_webhook:localhost) Body: {"type":"Buffer","data":[137,80,78,71,13,10,26,10,0,0,0,13,73,72,68,82,0,0,3,132
Nov-15-2017 11:57:06.104 +01:00 error [matrix-appservice-bridge] [-] POST http://localhost:8008/_matrix/media/v1/upload (@_webhook:localhost) HTTP 403 Error: "{\"errcode\":\"M_FORBIDDEN\",\"error\":\"Application service has not registered this user\"}"

Slack dates

https://api.slack.com/docs/message-formatting#formatting_dates

Support "slack blocks"

whatever that happens to be: https://api.slack.com/messaging/attachments-to-blocks

Emoji support

Convert things like :smile: to 😄 using Riot/Slack semantics.

Date format support

Slack does client-side date transformation. No matrix client supports similar behaviour, so the initial version of this will be to just dump a date in the message.

https://api.slack.com/docs/message-formatting#formatting_dates

Update hookUrlBase port in sample config.

https://github.com/turt2live/matrix-appservice-webhooks/blob/master/config/sample.yaml#L27 still mentions :4501 - should be :9000 now.

Help text says use displayName but the bridge wants username

Configurable localpart

Would it be possible to use an existing user (i.e. a "bot" user) as the bridge?
My use case is that I have a user (say @Superbot:example.com) with several backends connected, like go-neb and hubby, and I'd like to have webhooks as well without adding another user to the room.

HTML response on malformatted JSON

Just sent something stupid by mistake

{
  "username": "whatever",
}

And it returns 400 Bad request but with html. Maybe catch and return JSON instead?

Slack-style room linking

<#matrix:matrix.org|Matrix Chat>

Attachments

ref: https://api.slack.com/docs/message-attachments

The fallback should be treated as plain text (no formatting) and is used when the client (matrix) doesn't support the message.

Color / pretext

color: good, warning, danger, or any hex code like #439FE0

Slack renders this like:

we'll render it like this:

Attachments are otherwise rendered with a grey line. It's unfortunate that matrix doesn't support the line wrapping (pre-text) thing, but this will work for now.

This would be pretext<br>
<font color='#d9534f'>▌</font> danger is #d9534f<br>
<font color='#f0ad4e'>▌</font> warning is #f0ad4e<br>
<font color='#5cb85c'>▌</font> good is #5cb85c<br>
<font color='#439fe0'>▌</font> custom is #439fe0

Pretext is pretext

Use #f7f7f7 for default.

Authors

author_name, author_link, author_icon

<img src='mxc://t2bot.io/hcSELkhLCNMRxLLTXKffPPSn' width='16' height='16'> <a href='https://matrix.to/#/user/@voyager:t2bot.io'>Matrix Traveler</a>

Although the image may need cropping ahead of time.

Titles

title, title_link

Basically an h3 as an anchor

Fields

fields is an array of {title, value, short}. short is optional.

Images

image_url

thumb_url

Footers

footer and footer_icon (text & image). Treat much like the authors

ts (timestamp) - seconds since epoch

Provisioning API: Join room on first hook, leave on last deleted

Cannot get more hook IDs after the first one is issued

I think? Tried to add a new webhook in a room, but it kept doing nothing

add chinese support for localpart

Bot generate part of ulocalpart via display name..
But it will lead to some problem if display name container some non-English character some thing like

• github 通知 => github___
• github 大小 => github___
  Those different display name webhook will share same localpart because of failed generated of specified character
  there is a library can use to solve this problem like this one
  https://www.npmjs.com/package/limax

Project refresh

• Use matrix-js-snippets
• Convert to typescript
• Unit tests for format parsing
• Integration tests for appservice testing
• Use the appservice support in matrix-bot-sdk

Bridge does not work in rooms with encryption

The bridge works nicely in rooms without encryption. But when I enable encryption in a room, I cannot start the webhooks bridge.

Provisioning API

Bridge stays stuck at M_USER_IN_USE

The _webhook user was already created when I started the bridge so it just got stuck at the M_USER_IN_USE error. I tried deactivating the _webhook account using the Admin API but I still get the same error:

Loading config file /app/config/config.yaml
May-19-2018 07:09:59.743 +00:00 info [index] Preparing database...
May-19-2018 07:09:59.744 +00:00 info [WebhookStore] Running migrations
[INFO] No migrations to run
[INFO] Done
Sat, 19 May 2018 07:09:59 GMT sequelize deprecated String based operators are now deprecated. Please use Symbol based operators for better security, read more at http://docs.sequelizejs.com/manual/tutorial/querying.html#operators at ../node_modules/sequelize/lib/sequelize.js:242:13
May-19-2018 07:09:59.876 +00:00 info [index] Preparing bridge...
May-19-2018 07:09:59.876 +00:00 info [WebhookBridge] Constructing bridge
May-19-2018 07:09:59.877 +00:00 info [WebhookBridge] Starting bridge
May-19-2018 07:10:00.208 +00:00 info [WebhookBridge] Updating appearance of bridge bot
May-19-2018 07:10:00.226 +00:00 warn [index] No provisioning API token is set - the provisioning API will not work for this bridge
May-19-2018 07:10:00.228 +00:00 info [WebService] API now listening on 0.0.0.0:4501
May-19-2018 07:10:00.336 +00:00 error [matrix-appservice-bridge] [-] POST https://nerdsin.space/_matrix/client/r0/register (AS) HTTP 400 Error: {"errcode":"M_USER_IN_USE","error":"User ID already taken."}

Slack endpoint for url encoded form

Something like /api/v1/matrix/slack/hook/:hookId to allow for payload={"text":"This is a line of text.\nAnd this is another one."}

Add ability to label hooks

It would be great (if it's possible) to be able to optionally label a webhook, at least when using the provisioning API, so that existing hooks can be easily managed/deleted when there are more than one for a given room.

E.g.:

PUT /api/v1/provision/{roomId}/hook?userId={userId}&label=buildsystem
PUT /api/v1/provision/{roomId}/hook?userId={userId}&label=systemhealth

GET /api/v1/provision/{roomId}/hooks?userId={userId}
=>

{
  "success": true,
  "results": [
    {
      "id": "some_long_string",
      "url": "https://webhook.t2bot.io/api/v1/matrix/hook/some_long_string",
      "userId": "@someuser:someserver.org",
      "roomId": "!cURbafjkfsMDVwdRDQ:matrix.org",
      "type": "incoming",
      "label": "buildsystem"
    },
    {
      "id": "another_long_string",
      "url": "https://webhook.t2bot.io/api/v1/matrix/hook/another_long_string",
      "userId": "@someuser:someserver.org",
      "roomId": "!cURbafjkfsMDVwdRDQ:matrix.org",
      "type": "incoming",
      "label": "systemhealth"
    }
  ]
}

With the above it would then be possible to manage/delete webhooks created for a given purpose without needing to record the "id" field elsewhere at creation time.