muety/telepush

🤖 A simple bot to translate JSON HTTP requests into Telegram push messages

telepush's Issues

Feature Request: Support arm64 architecture

Hi, I built telepush on arm64 (aarch64) locally and everything seems to work fine. The build doesnt take long, but of course would be nice to just pull a published image from the project. For arm64, I did not need to make any changes to build it locally, so hopefully should not be hard. I can help testing if needed.

PS: support for e.g. arm32 (armhf) would be also nice. Both of these would open the project for the fairly numerous raspberryPi's user-base

Add request option to disable parsing text as Markdown

When a message contains reserved Markdown characters without actually being Markdown (e.g. a single _, a single [, etc.). Telegram will respond with an error.

'Bad Request: can't parse entities: Can't find end of the entity starting at byte offset 20'

There should be a request option so that suers can explicitly disable Markdown.

Sender blacklist

Introduce a way to blacklist certain user IDs for both incoming updates and outgoing messages. Update messages from them as well as outgoing messages to them will simply be discarded.

Idea: Webmentions Inlet

Middleman bot should be able to act as a receiver for Webmentions. That is, it should be able to process an x-www-form-urlencdoded POST request like the following, send out a notification and return 202.

Request Example:

POST /api/inlets/webmention HTTP/1.1
Host: middleman.yourserver.tld
Content-Type: application/x-www-form-urlencoded

source=https://blog.bob.com/some-post.html&
target=https://alice.com/blog/original_post.html

The only problem here is that the Webmention request performed by its sender will not contain any information about the Telegram recipient, i.e. there is no way to transmit a recipient_token and therefore no way to route the notification to a registered user.

Proposal: Introduce a way to configure the bot using commands, including the ability to set inlet-specific parameters. E.g. a new /config command might be introduced that requires two arguments, corresponding to key and value. These key-value pairs are persisted at the server and used for any kind of configuration – in this case for routing Webmention notifications. Specifically, a Middleman user might "subscribe" herself to incoming Webmentions for a certain domain: /config inlet.webmentions.subscription "alice.com". This will tell the bot to notify Alice about a new Webmention containing any URL under "alice.com" as a target.

One problem with the above proposal is that it doesn't feature any type of authorization. If you are Alice, hosting a middleman instance with multiple users, any of those users could practically subscribe to your Webmentions. I'd consider that acceptable for now, but if anyone comes up with a better approach, please let me know!

Program does not exit when bind fails

Refrain from sending messages to chats that have blocked the bot

Also, don't accept requests from them anymore.

Resolve messages asynchronously

Move from DockerHub to ghcr

https://github.com/muety/wakapi/blob/master/.github/workflows/docker.yml

Invalid character response using requests in Python

I'm really excited to message myself via MiddleManBot to alert me when trailhead spots open for the John Muir Trail.

I'm calling the public bot in python as follows:

url = "https://middleman.ferdinand-muetsch.de/api/messages"
payload = {
    'recipient_token': my_token,
    'text': my_text,
    'origin': my_app,
}
import requests
response = requests.post(url, data=payload)

I'm getting a response with status code 400, whose text is:

invalid character 'r' looking for beginning of value

Any idea what the issue could be?

Attach file to messages

How hard could be to implement the possibility to attach a file to the message sent to the bot?

Pass on Telegram API errors to requester by defaut

Client calling Telepush should be made aware of bad requests.

Support multiple tokens per user

A user should be able to register more than one recipient token at a time. There shall be a command to revoke active tokens.

Use specific, platform-independent data types

Hi, I build my own docker image for Raspberry Pi (armhf) using your repo. I needed to make one small change in Dockerfile (add one line - RUN apk add git - so should not change anything). The build was fine, but after starting the container and starting a chat with my bot, I get the following error (of course, ID changed):

2022/04/14 12:19:19 error getting updates: json: cannot unmarshal number 1234567890 into Go struct field TelegramUser.result.message.from.id of type int

Any idea what might be wrong? Could it be connected to a different arch?

Unit tests

failed to read store from /srv/data/store.gob

Hi!
I'm getting this error when run telepush with docker:

docker run --rm \
        -p 8080:8080 \
        -v /opt/telepush:/srv \
        --name telepush \
        telepush \
        -mode webhook \
        -token my-token \
        -metrics \
        -dataDir /srv/data \
        -urlSecret secret \
        -whitelist /srv/whitelist.txt

any advice?

Idea: GitHub Inlet

Allow disable link previews when sending message

Is it possible to disable link previews when sending a message with this bot? If not, can that functionality be added?

It looks like it's possible through the Telegram Bot API, but I can't get it to work with the webhook2telegram bot. Specifically, the sendMessage method takes a disable_web_page_preview but when I pass that parameter into my POST request for this bot, it has no effect.

Example

I'm using the Python requests library to send a message that contains a link.

import requests

payload = {
    "recipient_token": "my-token-value",
    "text": "here's [a link](https://github.com/muety/webhook2telegram)",
    "origin": "MyBot",
    "disable_web_page_preview": True,  # this has no effect
}

url = "https://apps.muetsch.io/webhook2telegram/api/messages"
mmb_response = requests.post(url, json=payload)  # this sends the message

Here's what the message looks like when I receive it. I don't want the link preview included in the message.

Provide official Docker image

Auto-reload inlet configs upon file change

Be alerted through inotify about changes inside inlets.d.

Grafana alerts inlet

Truncate messages to fit maximum length limitation

Discussed in #65. Roughly copy alertmanager's implementation. Also, add a config option so that a server admin can choose whether to reject too long messages (as it's currently the case) or truncate them.

Switch to config-based inlet adapter format

I want ease the process of adding new inlets without having to modify the actual code and recompile the app. Instead, come up with a JSON- / YAML- / INI-based config format for inlets, that can be switched out at runtime.

High-entropy tokens.

I tried to start TelepushBot and it issued me a token that consist of just six hex characters.
Does it mean that one can guess the token or just annoy random people by sending random messages though TelepushBot?
Should TelepushBot issue longer, more random tokens?

Idea: HackerNews Inlet

Idea: Web Reporting API Inlet

https://developers.google.com/web/updates/2018/09/reportingapi

Example

1. Make some web server return Report-To header

Report-To { "group": "reporting-1", "max_age": 10886400, "endpoints": [{ "url": "https://example.org/webhook2telegram/api/inlets/reporting?token=some-recipient-token" }] }

2. Make webhook2telegram accept JSON like this

[{
  "type": "csp",
  "age": 10,
  "url": "https://example.com/vulnerable-page/",
  "user_agent": "Mozilla/5.0 (X11; Linux x86_64; rv:60.0) Gecko/20100101 Firefox/60.0",
  "body": {
    "blocked": "https://evil.com/evil.js",
    "directive": "script-src",
    "policy": "script-src 'self'; object-src 'none'",
    "status": 200,
    "referrer": "https://evil.com/"
  }
}]

Bot is leaking ...

After few days of idling the bot is responding with '500 Internal Server Error', and it looks like it's leaking file descriptors:

ls /proc/25802/fd |wc -l
1024

Idea: whitelist mode

Also known as "allowlist" mode.

Currently the blacklist (also known as "blocklist") allows any user to register a key and use this service as long as they are not blocked. I'm interested in an "allowlist" mode which only allows a group of authorized users to use this service.

Rate limiting for token updates

Make `origin` parameter optional

... and, if not set, just assume a default value of Webhook2Telegram instead.

Satisfy golint

https://goreportcard.com/report/github.com/muety/telegram-middleman-bot

Store not thread safe

StorePut put is executed from messageHandler, we might have a situation with a race condition.

Allow to pass payload as query parameters of GET request

Is it possible to achieve this form:
https://domain.com/api/messages/?data={"recipient_token":"f9e1-bb79-4db2-8fff-cfbb8","type":"TEXT","origin":"somedevice"}
Because not all software supports all values, such as origin.

Request rate limit by IP

Support multiple recipients per requests

Allow to pass comma-separated list of recipients as path parameters.

Telegram API size limitation 4096?

Hey there! We try to use telepush for alertmanager notifications from our clusters. But we encounter situation, when not all notifications are delivered. After some investigations:

ts=XXXXXXXXXXXXX caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=21 err="telepush/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 400: https://telepush.dev/api/inlets/alertmanager/XXXXXX: message too long (max is 4096 characters)"

Some software deployed with Helm have multiply labels, and it`s easy-to-go to step over this limit. How this issue can be sold? Maybe force Telepush to short notifications size for Telegram eats it?

Make `go install`able

Embed all static asset, including version.txt

Support multi-part file uploads

Having to base64-encode files is not ideal.

Idea: Kubernetes example

Hey! May be useful for someone... I can share ready k8s configs to deploy.

You need to change just 2 things:

• pvc.yaml: spec.storageClassName
• kubernetes.yaml: Secret.data.token (in base64)
  ...else
• kubernetes.yaml: Deployment.spec,args (add/change as you wish)

I can create a Helm-chart for Telepush if it is needed. But everything is working well with this configs so far so good.

kubernetes.yaml

apiVersion: v1
kind: Secret
metadata:
  name: telepush-secret
  namespace: default
type: Opaque
data:
  token: XXXX
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: telepush
  namespace: default
  labels:
    app: telepush
spec:
  replicas: 1
  selector:
    matchLabels:
      app: telepush
  template:
    metadata:
      labels:
        app: telepush
    spec:
      containers:
      - name: telepush
        image: ghcr.io/muety/telepush
        args:
        - -mode
        - poll
        - -truncateMsgs
        - "true"
        volumeMounts:
        - name: telepush-data
          mountPath: /srv/data
        ports:
        - containerPort: 8080
        env:
        - name: APP_TOKEN
          valueFrom:
            secretKeyRef:
              name: telepush-secret
              key: token
      volumes:
      - name: telepush-data
        persistentVolumeClaim:
          claimName: telepush-pvc
---
apiVersion: v1
kind: Service
metadata:
  name: telepush
  namespace: default
spec:
  selector:
    app: telepush
  ports:
    - protocol: TCP
      port: 80
      targetPort: 8080
  type: ClusterIP

pvc.yaml

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: telepush-pvc
  namespace: default
spec:
  accessModes:
    - ReadWriteOnce
  storageClassName: microk8s-hostpath
  resources:
    requests:
      storage: 100Mi

telepush.zip

Add URL secret to Telegram updates route

See https://github.com/muety/anchr/blob/458b07be021594ad51faf80c044a90e2a2eaac96/app/controllers/telegram.js#L268

Idea: Stripe webhook inlet

Listen for certain Stripe events, e.g. customer.subscription.created.

Provide GitHub releases including pre-compiled binaries

Whitelist not working with group chat_id

Hi,

Whitelist doesn't work with group chat id (with negative sign), something like -123456789
It seems to work well with users chat_id that doesn't have the minus sign.

I haven't tested with Blacklist.

I'm using self-hosted bot with docker image version 3.3.0

I have tested a whitelist file with:
user chat id + group chat id (only user chat is permitted)
group chat id (all chats are permitted)

❗️ Webhook2Telegram becomes Telepush ❗️

Please refer to the release notes of version 3.

Feature: return 404 on /api/inlets/default and /api/messages in current version

(update:

Everything works as expected,
version 3 ( named telepush) removed the /api/messages endpoint,
but rejects it with 405 == "method not allowed", which is technically correct ( more or less ).
Unfortunately this isn't interpreted as failure by curl --fail

scripts used were changed to try v3 (/api/messages/YOUR-Tic-Tac-Token) first ,
but some clients ( not physically reachable ) might still try to use the old method
which does not work unless a frontend proxy is used like this ( nginx example)

    location ~ ^/api/messages$ {
         return 404;
         access_log /dev/null;
    }

so the request is:
Please return 404 on /api/messages and /api/inlets/default ( exact path),
which would be technically correct as well ( endpoint does not exist ) and make "curl --fail" work

Pull Request here: #54

Thank You

---

OLD TEXT:

(Original Text 2)

* the router seems to fail on 
* * `/api/messages/YOUR-Tic-Tac-Token`
* * `/api/inlets/default/YOUR-Tic-Tac-Token`


( solution/problem: 
in the below "original ticket" text , the endpoint `/api/messages` was blocked by nginx
to speed up the detection of v2/v3 telepush,

however , the "current" docker image does not seem to accept the urls including tokens..

so if you need to send something , the "recipient_token" has to be in json ,
then both "non-tokenized" endpoints `/api/inlets/default` and `api/messages` will work


)


(Original Text 1)
## Situation:

* had the thingy running in longpoll mode , logs filled up with "destination unreachable" and other warnings
* in a specific chat "sending links" was forbidden for a certain time
* the script used to send the messages is https://gitlab.com/the-foundation/pam-notify-telegram/
* thingy did not send to specific chat anymore , but to all the others
* thingy did still respond to "/start" command
* started thingy in webhook mode 
* manually set webhook ( will document in another ticket)
* created a new token ( for the not-working chat )
* new token works
* as many clients ( some physically hard to reach ) use the "damaged" token , some approaches were tried to make the old location work again: 

* * 301/302 redirect  /api/messages/old-token -> /api/messages/new-token : client follows to new location , sends exact same byte size , message is not sent
* * nginx rewrite with extra location /api/messages/old-token /api/messages/new-token ( verified with extra host/port and rewrite_log)


## Your Question was ?
is there any chance to 

* implement a "token re-write" easily 
* debug the internal storage 
* "reset" any error marker for that chat ( if there is smth like that )
*  delete the old token and change a new one
* make it accept 301 requests 
* or "anything that works" ?

Prometheus exporter

Asynchronously process incoming updates

Similar to what #17 did for messages sent via respective inlets.

Issue: webhook not used

• current docker image ( ghcr -> muety -> telepush)

• tried to set secretUrl for setWebhook, thingy did still want me to send /api/updates .. inacceptable (!)

• from the go code it should be
  /api/updates_YOURcustomTOKEN
  ( the default is /api/updates , just if any one googles this )

• since the thingy did not show the message from source code ( currently LINE 138 , Updates from Telegram are accepted under '/api%s'. Set the webhook accordingly (see https://core.telegram.org/bots/api#setwebhook)\n", botConfig.GetUpdatesPath( )
  nginx was (ab)used and curl was used to set the webhook

---

relevant parts (nginx):

location /api/ {

    location = /api/updates {
        deny all; return 403;
    }
    location ~ ^/api/messages$ {
         return 404;
         access_log /dev/null;
    }

      location = /api/updates_customstring {
      rewrite  ^/api/updates_customstring$ /api/updates break;
      client_max_body_size 4M;

      proxy_http_version 1.1;
      proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
      proxy_cache_bypass $http_upgrade;
      proxy_set_header Upgrade $http_upgrade;
      proxy_set_header Connection $http_connection;
      proxy_pass http://upstream_signals ;
      proxy_set_header Host $host;
      proxy_redirect     off;

      }
}

relevant parts(curl)

# clear webhook
curl -F "url=" https://api.telegram.org/bot<YOURTOKEN>/setWebhook
curl -F "url=https://<YOURDOMAIN.EXAMPLE>/api/updates_customstring" https://api.telegram.org/bot<YOURTOKEN>/setWebhook

---

Your Question was ?

• why is the thingy not using the setUrl parameter ( used in docker compose: APP_URL_SECRET: customtokenthingy)
• are PR's welcome to document at least the /api/updates endpoint ?

regards

Refactor store implementation

https://github.com/etcd-io/bbolt

Feature Request: Docker compose example

Hello, first of all thanks for this excellent tool.

I see in the README that there's the Docker run command to execute it in a container. However I was wondering if you could test and provide also a Docker Compose example.

I'm getting an issue where the endpoint to test complains that 'passed token does not relate to a valid user', and I don't know if it might be because I made the wrong conversion to Docker Compose and provided wrongly the access token. The bot works on other applications so it's not a matter of the bot

My docker compose looks currently as follows

services:
  telepush:
    image: ghcr.io/muety/telepush
    container_name: telepush
    command: '-mode webhook -token ${TELEGRAM_BOT_TOKEN}'
    restart: unless-stopped
    environment:
      - TELEGRAM_BOT_TOKEN=${TELEGRAM_BOT_TOKEN}
    networks:
      proxy:
    volumes:
      - /home/my/telepush/directory:/srv/data

Worth mentioning also, I obtained the recipient token you mention by sending a /start message to the bot then copying the Chat ID. Is that correct?

Thanks in advance