necoro/feed2imap-go

A software to convert rss feeds into mails.

feed2imap-go

A software to convert rss feeds into mails. feed2imap-go acts as an RSS/Atom feed aggregator. After downloading feeds (over HTTP or HTTPS), it uploads them to a specified folder of an IMAP mail server. The user can then access the feeds using their preferred client (Mutt, Evolution, Mozilla Thunderbird, webmail,...).

It is a rewrite in Go of the wonderful, but unfortunately now unmaintained, feed2imap. It also includes the features that up to now only lived on my own branch.

It aims to be compatible in functionality and configuration, and should mostly work as a drop-in replacement (but see Changes).

An example configuration can be found here with additional information in the wiki.

See the Installation section on how to install feed2imap-go. (Spoiler: It's easy ;)).

Features

• Support for most feed formats. See gofeed documentation for details. Feeds need not be supplied via URL but can also be yielded by an executable.
• Connection to any IMAP server, using IMAP, IMAP+STARTTLS, or IMAPS.
• Detection of duplicates: Heuristics what feed items have already been uploaded.
• Update mechanism: When a feed item is updated, so is the mail.
• Detailed configuration options per feed (fetch frequency, should images be included, tune change heuristics, ...)
• Support for custom filters on feeds
• Readability support, i.e. fetching and presenting the linked content instead of the teaser/link included in the feed itself.

Changes

Additions to feed2imap

• Groups: Have the ability to group feeds that share characteristics, most often the same parent folder in the hiearchy. It also allows sharing options between feeds. Usages: Categories ("News", "Linux") and merging different feeds of the same origin.
• Heavier use of parallel processing (it's Go after all ;)). Also, it is way faster.
• Global target and each feed only specifies the folder relative to that target. (feature contained also in fork of the original)
• Fix include-images option: It now includes images as mime-parts. An additional embed-images option serves the images as inline base64-encoded data (the old default behavior of feed2imap).
• Improved image inclusion: Support any relative URLs, including //example.com/foo.png
• Use an HTML parser instead of regular expressions for modifying the HTML content.
• STARTTLS support. As it turned out only in testing, the old feed2imap never supported it...
• item-filter option that allows to specify an inline filter expression on the items of a feed.
• Readability support: Fetch and present the linked article.
• Mail templates can be customized.

Subtle differences

• Feed rendering: Unfortunately, semantics of RSS and Atom tags are very broad. As we use a different feed parser ibrary than the original, the interpretation (e.g., what tag is "the author") can differ.
• Caching: We do not implement the caching algorithm of feed2imap point by point. In general, we opted for fewer heuristics and more optimism (belief that GUID is filled correctly). If this results in a problem, file a bug and include the X-Feed2Imap-Reason header of the mail.
• Configuration: We took the liberty to restructure the configuration options. Old configs are supported, but a warning is issued when an option should now be in another place or is no longer supported (i.e., the option is without function).

Unsupported features of feed2imap

• Maildir (issue #4)
• Different IMAP servers in the same configuration file. Please use multiple config files, if this is needed (see also issue #6).

Installation

The easiest way of installation is to head over to the releases page and get the appropriate download package. Go is all about static linking, thus for all platforms the result is a single binary which can be placed whereever you need.

Please open an issue if you are missing your platform.

Use your package manager

Arch Linux

feed2imap-go is present in the AUR.

Nix

feed2imap-go is present in nixpkgs.

Install from source

Clone the repository and, optionally, switch to the tag you want:

git clone https://github.com/Necoro/feed2imap-go
git checkout v1.7.0

The official way of building feed2imap-go is using goreleaser:

goreleaser --snapshot --rm-dist

The built binary is then inside the corresponding arch folder in dist.

In case you do not want to install yet another build tool, doing

go build

should also suffice, but does not embed version information in the binary (and the result is slightly larger).

If you are only interested in getting the latest build out of the master branch, do

go install github.com/Necoro/feed2imap-go@master

Using @latest instead of @master gives you the latest stable version.

Run in docker

Most times, putting feed2imap-go somewhere and adding a cron job does everything you need. For the times when it isn't, we provide docker containers for your convenience at Github Packages and at Docker Hub.

The container is configured to expect both config file and cache under /app/data/, thus needs it mounted there. When both are stored in ~/feed, you can do:

docker run -v ~/feed:/app/data necorodm/feed2imap-go:latest

Alternatively, build the docker image yourself (requires the feed2imap-go binary at toplevel):

docker build -t feed2imap-go .

Note that the supplied binary must not be linked to glibc, i.e. has to be built with CGO_ENABLED=0. When using goreleaser, you'll find this in dist/docker_linux_amd64.

Or you can roll your own Dockerfile, supplying a glibc...

NB: feed2imap-go employs no server-mode. Thus, each run terminates directly after a couple seconds. Therefore, the docker container in itself is not that useful, and you have to have a mechanism in place to spin up the container regularly.

Support

Thanks to JetBrains for supporting this project.

feed2imap-go's Issues

Image attachments should use original file name rather than "Mail Attachment.jpeg"

The old ruby feed2imap used the original file names. It would be nice if this version did as well.

Empty root is not allowed

As reported by @X41:
2021/10/17 03:34:33 ERROR: creating folder '': Invalid mailbox name: Name is empty (0.001 + 0.000 secs).

Up to now, feed2imap-go has (implicitly) required to use INBOX or something alike as a top folder. But this was never the intention: Storing feeds directly under the root must also be possible.

(@X41: As I'm already working on it, I took the liberty of creating the issue myself :)).

Split connection string into multiple fields

Currently, in feed2imap, the connection string is given as
imap://account:pwd@host:port/Folder.
This is ugly, especially as one has to take care to encode special chars (e.g. '@' accordingly).

--> Allow an alternative structure, f.ex.

connect:
   protocol: imap # optional
   host: foo.example
   port: 993
   user: account@foo

Sometimes feed2imap-go hangs

Sometimes, feed2imap-go hangs without apparent reason. As this locks the cache file, all future runs just abort.

TIL: https://www.tigraine.at/2020/08/14/debugging-go-iowait-hang-sometimes-its-really-not-your-code -- send SIGABRT to the running process to get a traceback.

This bug is to help my memory, when this occurs again.

print-cache docs?

I’m packaging feed2imap-go for nixpkgs at NixOS/nixpkgs#222690 and I am unsure what to do with the print-cache binary. Is that something you’d expect to be included in a distro packaging?

Maybe a small README in https://github.com/Necoro/feed2imap-go/tree/master/tools/print-cache would be helpful.

Don't run if config.yml is world-readable

config.yml by design contains IMAP account credentials. Refuse to execute if it is world-readable.
In the release tarball, the example config is mode 0644.

Do not hardcode username for https://index.docker.io/v1/

The username necorodm is hardcoded in

feed2imap-go/.github/workflows/release.yml

Line 46 in b8b56d3

username: necorodm

As suggested on https://github.com/marketplace/actions/build-and-push-docker-images as well as in #15 (comment) I would propose not to hardcode the username and use a GitHub secret instead. This would make it easier for others to share their builds as well.

Maildir support

Currently, only IMAP support is planned; add Maildir support if needed / spare time at hand

Make number of parallel IMAP connections configurable

Currently, the number of connections is hardcoded to be 5 internal/imap/client.go#L16

Make this configurable, especially when concurrency is not wanted.

Item updating

panic: send on closed channel

Every two days or so I get a email from cron with

panic: send on closed channel

goroutine 213 [running]:
github.com/Necoro/feed2imap-go/internal/imap.(*Client).connect(0x400040a090, {{0x40001912b0, 0x5}, {0x40000c2738, 0x18}, {0x40001912d8, 0x8}, {0x4000191310, 0x10}, {0x5e228a, ...}, ...})
	github.com/Necoro/feed2imap-go/internal/imap/client.go:72 +0x324
github.com/Necoro/feed2imap-go/internal/imap.Connect.func2(0x400034a540?)
	github.com/Necoro/feed2imap-go/internal/imap/imap.go:46 +0x60
created by github.com/Necoro/feed2imap-go/internal/imap.Connect
	github.com/Necoro/feed2imap-go/internal/imap/imap.go:45 +0x4ac

Not sure what to do about it, though. I guess the error could be caught and reported more prettily “E.g. saying which server failed during which operation”.

An error should probably be reported so that I know if it keeps failing.

However, a spurious error like this isn’t much to worry about; maybe feed2imap-go could retry once or twice before bugging the user?

Text Part of Mails

Currently, only the html rendering is implemented. The HTML-to-Text part is missing.

Question: how do I specify that I want all items to land in the same folder (from different sources)

Maybe clarify the documentation, or is this an enhancement request... I want all items to land in the same IMAP folder - I tried to leave out target (that creates a new folder for each source) and when I leave the target blank I get:
2020/08/15 05:00:33 INFO: Checking for folder 'Jobs/' 2020/08/15 05:00:33 ERROR: Creating folder of feed TNG: creating folder 'Jobs/': Mailbox already exists (0.001 + 0.000 secs).
...and it seems like no items are created.

Replace HTML entities by their counterpart

Example (from Dilbert): Can't Use Personal Laptop - Dilbert by Scott Adams

Unify items in folder

I have separate feeds of a newspaper that quite often share articles. This leads to having the same item thrice in the same folder.
Offer a way to merge items / have them only once.

Dependabot can't resolve your Go dependency files

Dependabot can't resolve your Go dependency files.

As a result, Dependabot couldn't update your dependencies.

The error Dependabot encountered was:

go: github.com/Necoro/[email protected]: invalid version: unknown revision d9090592eb44

If you think the above is an error on Dependabot's side please don't hesitate to get in touch - we'll do whatever we can to fix it.

View the update logs.

Ordering of cached items

The old feed2imap ordered the cached items ... but is this really necessary?

StackOverflow: Atom author tag in RSS feed

Hiya - this project looks good - coming from feed2imap (the old one)!

I am getting some odd errors in this feed:
https://stackoverflow.com/jobs/feed?location=stockholm&range=50&distanceUnits=Km&allowsremote=True

The error looks like something like:
Error applying item filter: cannot fetch Name from *gofeed.Person (1:8)
| Author.Name != "Weirdo"
| .......^

panic: runtime error: index out of range [0] with length 0

Thanks for providing a maintained version of feed2imap, I am a log-time user!

Trying to migrate I run into this panic:

~ $ feed2imap-go -f ~/.feed2imaprc
2023/03/23 10:52:14  WARN: Unknown global option 'include-images'. Ignored!
panic: runtime error: index out of range [0] with length 0

goroutine 6 [running]:
github.com/Necoro/feed2imap-go/pkg/config.(*Url).RootPath(0xc000836a68?)
	github.com/Necoro/feed2imap-go/pkg/config/url.go:177 +0x79
github.com/Necoro/feed2imap-go/internal/imap.Connect({{0xc00003e480, 0x5}, {0xc00002a9a8, 0x18}, {0xc00003e4a3, 0x8}, {0xc00003e4ac, 0x10}, {0xa4d362, 0x3}, ...})
	github.com/Necoro/feed2imap-go/internal/imap/imap.go:33 +0x2dd
main.run.func1()
	github.com/Necoro/feed2imap-go/main.go:137 +0x98
created by main.run
	github.com/Necoro/feed2imap-go/main.go:135 +0x430

Maybe due to my old-style

    target: imaps://YYYYY:[email protected]/XFeeds/Meine%20Bilder

entry?

New option `auto-target`

As per #25, it would be nice to have a new option auto-target with the default yes - which then can be set to no and which will disable setting a target for each feed/group, thus sparing you all the null-targets

Checklist:

• Implement
• Update example config
• Wiki-Entry not needed after all

Allow groups to have options too

Groups should allow options too. Those options will then apply for all feeds inside the group.

build cache only option

When changing from feed2imap to ~-go, one needs to build up the cache once in the beginning. For this, an option has to be created.

How to deal with unsupported configuration options

We won't support all configuration options from the old software. But, how are we dealing with them:
a) throw a warning
b) simply ignore

Currently preferred: a), as it will also spot typos

Support missing options

The following options should be supported:

• max-failures
• include-images (per feed)
• reupload-if-updated (per feed)
• disable-ssl-verification (reinterpret as tls-no-verify, cf. #3)
• timeout
• disable (per feed)
• always-new (per feed)
• ignore-hash (per feed)

The following options should be marked as unsupported:

• dumpdir
• debug-updated (use -d instead)
• execurl
• filter

Empty target not usable

In feed2imap, target: marked a feed to go into the top-level folder. Due to the way the Go-yaml-parser works, target: cannot be distinguished from a missing target. Atm, one has to use target: "".

Goal: Make target: work again.

Deliverability Issues

Using fastmail and the example config for XKCD, there are several issues.

The first I ran into was Message contains bare newlines. The fix was something like:

re := regexp.MustCompile(`\s.*\r`)
contents := re.ReplaceAllString(msg.Contents, "\r\n")

Once that is resolved, I then receive the error Message contains invalid header. See attached file for truncated log of the run.
log.txt

Readability support

As stated in #66 (comment):

It scrapes a website for value content only and removes all the clutter. This can be used in some instances to turn a feed with preview content only into a full feed.

Instead of using the proposed https://github.com/go-shiori/go-readability, I created a trimmed down fork that avoids too much clutter in the dependencies.

tls-no-verify option for feeds

Some feeds can return invalid certificates when talked to via HTTPS (wrong servername, self signed, missing cert chain). feed2imap used to circumvent this by always setting tls no verify -- but now we want this to be done only on a per feed basis (plus the possible global option).

Until this is implemented, feeds with broken HTTPS are inaccessible

docker pull no basic auth credentials

docker pull docker.pkg.github.com/necoro/feed2imap-go/feed2imap-go:latest Error response from daemon: Get https://docker.pkg.github.com/v2/necoro/feed2imap-go/feed2imap-go/manifests/latest: no basic auth credentials

Can you make this Docker Repo public or use docker hub?

https://ropenscilabs.github.io/r-docker-tutorial/04-Dockerhub.html

Identical target per feed/group

target is a field that is defined globally, per feed, and per group.
In my world they were used to define the folder structure.

Turns out: In feed2imap, it was always possible to specify full URIs there, thus actually sending different feeds to different mailboxes.

I'm not sure if this is still needed -- normally you'd probably have multiple configs then.

As shown below, in feed2imap it was not uncommon to repeat the server target for each feed. This shall be supported. Different servers will remain unsupported.

`reupload-if-updated` needs enhancement?

The Arch RSS puts new items into the mailbox, even though they are so old they are automatically removed again.
In theory this should be handled by reupload-if-updated, but perhaps this is broken. Investigate.

Feature Request: Modify template

Is there an option to disable the grey boxed meta data?

Otherwise, great work. I was maintaining my own fork of "feeds2imap-go" for the past years but it stopped working and the IMAP functionality was very spotty. Is it related in any way? I added readability support for non-full feeds, which would be nice in your work too.