pipes-digital/pipes

Repository for Pipes

Pipes CE

The open source CE edition of pipes.digital

The CE edition

pipes.digital is a spiritual successor to Yahoo Pipes, a graphical interface to get data from the web and to manipulate it by connecting block. It's heavily focused on the concept of feeds, that data flows item by item from block to block, with RSS as the default and internally used format. The CE edition is the FOSS core of pipes.digital, minus things like a pricing page and user plans.

To get an impression of how pipes works without installing anything, try pipes.digital first.

How to install and run

Clone this repository. You need to install Ruby and the bundler gem:

gem install bundler

Then cd into the cloned repository and download the required gems with bundler:

bundle install

Some gems won't install without some additional requirements, like sqlite3.

When everything is installed you can start the server:

bundle exec puma -e development

Logging in

Pipes CE uses portier to provide an easy passwordless login. After the installation you can immediately login with every email address you control.

Development

This blog article explains the architecture and most of the software used. It is a good starting point if you want to make changes.

The code is dual licensed: AGPL, but with the exception of not having to share code run on pipes.digital, the founding project (this so far only touches stuff like the pricing page, as explained above).

Feature and Pull Requests are welcome. If in doubt open an issue before investing the work to discuss deeper changes, though those are also welcome.

You can also join the gitter channel.

pipes's Issues

Can pipes take inputs? And am I doing `extract` correctly?

While working on filtering feeds I noticed a few things:

A pipe can't take input?

I can use a pipe as an input for another pipe, but a pipe can't take input itself. I.e. I'd like to create:

• ->[filter]->[duplicate]->[extract,extract,extract]->[build]->[out] and name it "filter merges", and then create:
  • [feed1]->[filter merges]->[out],
  • [feed2]->[filter merges]->[out],
  • [feed3]->[filter merges]->[out],

instead of

• [feed1]->[filter]->[duplicate]->[extract,extract,extract]->[build]->[out],
• [feed2]->[filter]->[duplicate]->[extract,extract,extract]->[build]->[out],
• [feed3]->[filter]->[duplicate]->[extract,extract,extract]->[build]->[out].

How do I extract links?

Extracting //item/title from the filter output works fine, but extracting //item/link doesn't extract any links.

The output of the [filter] looks like:

<item>
<title>
Merge pull request #713 from REPO/BRANCHNAME
  </title>
<link>
https://github.com/ORG/REPO/commit/ASDFSADFSADF
  </link>
[...]

but the output of the [extract //item/link] block looks like:

<item>
<title>
Extracted Content
  </title>
<pubDate>
Wed, 05 Jul 2017 11:01:49 +0000
  </pubDate>
<guid>
GUID
  </guid>
<content:encoded />
<dc:date>
2017-07-05T11:01:49.488103+00:00
  </dc:date>

    
  </item>

The output of the [extract //item/title] block is fine:

<item>
<title>
Extracted Content
  </title>
<pubDate>
Wed, 05 Jul 2017 11:05:54 +0000
  </pubDate>
<guid>
GUID
  </guid>
<content:encoded>
Merge pull request #713 from REPO/BRANCHNAME
  </content:encoded>
<dc:date>
2017-07-05T11:05:54.088334+00:00
  </dc:date>

    
  </item>

Finally, I had to duplicate the [extract //item/title] block, because the [build feed] block requires content, so I had one [extract //item/title] going to title, and another going to content. That seemed a bit unwieldy. I have [extract //item/link] going to link, but the output feed has only empty <link /> elements.

Postpone updates

Some Youtube channels have started using the "Premiere" feature, which adds unpublished videos to both the built-in Youtube "feed" and to RSS feeds.

You click on them expecting to play the video, but then are shown just the option to set a Google reminder for when the video is finally published:

It'd be nice if it was possible for you to add some way to postpone feed updates to bypass this. For example, the option to only add new items to a Pipes feed after they are one-week old.

Add yourself as a member of pipes-digital

https://github.com/pipes-digital says it has no members. You should add yourself. :)

Tagging problems

When you start typing in a tag, Pipes suggests tags you've already added.

Also, I'm currently unable to remove tags from https://www.pipes.digital/pipe/YPOdgaND (I added this set of technology-related tags at once from the tag suggestions, just to test it).

Firefox 58.0.1

Background doesn't fill the whole screen

This happens to the background when the Firefox window isn't maximized:

Firefox 65.0.1
Windows 10

Combine block does not append pubDate per item

When combining two or more feeds the resulting feed doesn't show a pubDate per item.
The sort block strips the date too.
Am I missing something?

Concatenate or just link text input to download block

I've made this to monitor new subtitles:

https://www.pipes.digital/pipe/j1O4Ve9d

Film pages have clean URLs like:

https://subscene.com/subtitles/metropolis
https://subscene.com/subtitles/the-general-1926

To avoid having to create new pipes for every new film I want to monitor, using a dynamic input block sounds a good idea.

However, input blocks can't be linked to download blocks. If you believe that letting people input full custom URLs could cause problems, what about having a way to at least concatenate a text input with a download block , so that I could have a download block like this:

https://subscene.com/subtitles/ + text input (e.g. metropolis, the-general-1926 etc.)

Unknown wrong feed shown in preview

While previewing a feed block with the URL set to [redacted], I got posts from some unrelated Russian download site I never used instead. The pipe worked as expected though.

After reloading the preview a couple of times, the correct feed was displayed.

Feed viewer

Firefox has removed the built-in feed viewer, and now it's become harder to quickly verify changes to your pipes, since external services and readers may cache them. Would it be possible for you to add a formatted viewer like this to Pipes, so that we don't have to install an extra extension just for this?

"Internal Server Error" when invalid URL is entered in feed block

Create a feed block with the URL https://www.pipes.digital/, connect it to Out, and try to save and view your pipe.

A more specific error message should be displayed on the main interface instead of generating a blank screen with the vague "Internal Server Error".

Compress display of pipes on the sidebar

With the current setup, the more pipes you create, the more you'll have to scroll down to reach the save button, especially if they have long titles. You could add the existing pipes to a unified field with a dropdown menu for the user to choose from, or at least move the save button to a fixed position at the top of the window as a temporary workaround.

401 server error when attempting to view output

I get "No result" when trying to work with feeds from Youtube. Example URL: https://www.youtube.com/feeds/videos.xml?channel_id=UCoxg3Kml41wE3IPq-PC-LQw

Twitter Support

It would be great to be able to import tweets from Twitters API, either from a search, a hashtag or from a profile.

Possible blocker for this is the state of twitters api, like https://dev.twitter.com/webhooks/account-activity being in beta and replacing user streams, also the rate limiting. Possible this would need to be combined with a profile feature to support entering a user api token.

This is a bigger development block and this issue might stay open for a long time.

pipes.digital SSL error: Let's Encrypt cert expired

The client probably just needs to be restarted ..

https://www.pipes.digital

NET::ERR_CERT_DATE_INVALID
Subject: pipes.digital
Issuer: Let's Encrypt Authority X3
Expires on: 23 Dec 2017
Current date: 23 Dec 2017

Filter by link

It would be great to be able to filter also by item link (currently title/summary/content are supported).

Titles of feeds built from scratch

Pipes that don't extract existing feeds and build a new feed from scratch could use the titles we manually entered for them instead of showing "Feed created by Pipes".

Example: https://www.pipes.digital/feed/aMOQA3Oe could have "Krita" as its title.

If you use "Feed created by Pipes" for promotion, I think it'd be better to add it to the body of feed posts instead of the title to make it easier to manage multiple feeds in a reader.

Does pipes work with webhooks?

I'd like to integrate with github in the manner described here: https://flow.microsoft.com/en-us/documentation/customapi-webhooks/

Add option to change email address

The "My Pipes" could have an option to change the current email address.

Editor working?

Now, whenever I save a block, it vanishes.
I tried in Firefox and Chrome.

Download block not connecting to Duplicate block

I'm currently unable to connect a new Download block to a Duplicate block on Firefox 58.0.1. The connectors don't get highlighted, as if it was impossible to do that.

Cloning an existing pipe where they are already connected works.

String replacement block

Have a block that allows to replace strings in the input feed (or the input data after #8).

Get post content from two different HTML elements

I'm trying to recreate this feed with Pipes, but the contents for each post come from two separate divs, and apparently you can't do that with Pipes, right?

https://feed43.com/learnersdictionary.xml

https://www.pipes.digital/feed/YPOdgaND

The source URL is: http://www.learnersdictionary.com/qa/post/latest (this always redirect to their most recent article).

Each post body should be generated from .question and .answer .

I've tried extracting .question, .answer with a single block, and extracting them separately and then using a combine block, but both resulted in two separate posts.

Error 500

Hi,

My feed doesn't work (https://www.pipes.digital/feed/j1O4yeqd ) it returns an error (Internal Server Error) but I don't know why...
It looks like it is because of the filter, maybe I am using it wrong ?

Any idea ?

Thanks

"Public Pipes" show your own pipes as if they belonged to other people

I've made my pipes public. I see a "fork" (but not "edit") button below each thumbnail on https://www.pipes.digital/pipes

Have you considered having a single unified page to show both all personal and public pipes? The user's own pipes could be shown at the top and the public third-party ones below them, or in a separate tab, section or column on the same page. This would solve this problem and that dilemma over hiding or not "My Pipes" in a menu.

Media enclosures not attached to output feed

https://www.pipes.digital/feed/REOmjQqL has no media enclosures, despite being formed just by:

Feed block (URL: [redacted]) -> Filter block (String: /breakfast/i) -> Out

Both blocks still show the mp3 files in their preview.

Page numbering problem

The first page on https://www.pipes.digital/pipes is numbered 0 instead of 1.

Edit public description

It seems that currently you can only edit the public description of a feed if you make it private then share it again, right?

I've also edited the title of this pipe to "Dark music on Youtube", but it's not displayed in the feed:

https://www.pipes.digital/editor/DpOrjQOw
https://www.pipes.digital/feed/DpOrjQOw

Add more connectors to the "Combine" block on demand

When you connect 5 blocks to a Combine block, it could automatically show a 6th connector in case you want to add another source to it, and then a 7th connector, and so on, up to the number of sources you're willingly to allow people to use in a single pipe.

I had to add a second Combine step to this feed because of that: https://www.pipes.digital/pipe/DpOrjQOw

Technical Documentation or paper/journal/scientific publication

Hi. I'm doing some preliminary research on visual programming and domain-specific languages. I aim to build a new DSL to connect some components as blocks. I found your amazing work and I would like to know if there's any scientific publication and/or some technical documentation I could read and make a reference in my research.
Best,
Carolina

Live block output preview

Some pipes are hard to build if you do not see exactly what a block outputs. Every block should have a way to show its output, either in the editor or in the sidebar.

Error from Zapier

I'm feeding a pipes feed into Zapier, and today for the first time I got a warning:

RSS: ExternalHTTPSConnectionPool(host='www.pipes.digital', port=443): Read timed out. (read timeout=39)

Is that a sign that Pipes is getting traction? 😁

Allow setting arbitrary children of <channel> element

I'm using Pipes to combine the free and premium feeds of a podcast into a single feed. It works great, except that it doesn't use the channel/image element from either feed, leaving me with no cover art in my podcast player.

It'd be nice if I could just manually add that element in there to get the metadata for the entire feed just the way I like it.

"No Result" from "View Output"

When I try to "View Output", I only get "No Result". Here is my feed:
https://www.pipes.digital/editor/ZnqxJ8N8

The feed does give a valid output, so I know it is working.
https://www.pipes.digital/feed/ZnqxJ8N8

The same is true when I try to create a new feed. I tried in both Firefox and Chrome, and got the same result.

Pipes as blocks in other pipes

To be able to create more complex pipes, it should be possible to use a pipe as a block in another pipe.

Support json

The extract block should support jsonpath, http://goessner.net/articles/JsonPath/, or a similar alternative like http://jmespath.org/.

This would also help #8.

Undo option

Sometimes I disconnect or delete a block accidently, and need to discard the other changes I've made to get it back.

Would it be possible to add an "undo" option?

Authorship lost in manipulated pipes

Compare these:

https://www.pipes.digital/pipe/l9vz5191 - no manipulations after the Combine block

https://www.pipes.digital/pipe/MOlEE5qD - full pipe with manipulations

Public Pipes

This is a long term goal.

So far, Pipes is focussing on creating the actual pipe, the editor and their blocks. But it would also be good to mirror the community features: To have public pipes that can be described, shared, upvoted, commented and, most important, forked.

This would also help new users, by showing them how Pipes can be used.

Input blocks that translate to rest API

One important feature of pipes I actually used quite a bit were text input blocks. They could be connected to the input fields of other blocks, which allowed for a lot of programmability. And they they could be controlled when calling the pipe by settings GET parameters, which effectively made each pipe a microservice.

Pipes should support those text input blocks as well.

Work with non-RSS data

Currently, all block but the download block generate and expect RSS/XML as input. It would be helpful for some use cases if blocks could also work with less structured data. For example, each POST to a webhook could be an array item that that the other blocks can work with. Downloaded text files could be treated line by line.

For this to work, all blocks would need to accept non-RSS input, and implicit RSS conversion (like with the webhook and extract block) would need to be disabled.

Automatically parse Youtube URLs to get channel or user's feed

A useful idea from Inoreader is to convert Youtube URLs in feed blocks to their appropriate feed URLs automatically on your server side, since Youtube doesn't embed the addresses in its pages, and people would need to look it up manually every time they create a new pipe.

Examples of URLs formats that may be entered in a feed block, and their corresponding feeds:

Channel:

https://www.youtube.com/channel/EXAMPLE/videos (featured|videos|about|etc.)
Actual feed: https://www.youtube.com/feeds/videos.xml?channel_id=EXAMPLE

User page:

https://www.youtube.com/user/EXAMPLE/videos (featured|videos|about|etc.)
Actual feed: https://www.youtube.com/feeds/videos.xml?user=EXAMPLE

Playlist:

https://www.youtube.com/playlist?list=EXAMPLE / https://www.youtube.com/watch?v=FOO&list=EXAMPLE
Actual feed: https://www.youtube.com/feeds/videos.xml?playlist_id=EXAMPLE

Extracting plain dates from downloaded source

https://www.pipes.digital/editor/aMOQA3Oe

This page has dates in the American format (9/21/2017, 9/16/2017 etc.), but the "Build Feed" block can't currently parse them. Have you considered adding this feature? Users would have to be able to tell if strings are in the international or American format, though.

Errors with CraigsList RSS feeds

Really excited if Pipes is back!

I used Y's Pipes to look through craigslist, but pipes.digital is having a bit of trouble with their RSS feeds:
I see "No Result" at Combine block's output (even though the Feed block output has content), and "Internal Server Error" for the pipe output (https://www.pipes.digital/feed/aNg2Aa9J)

I think I also saw a message somewhere as I was experimenting with the blocks, letting me know that it couldn't parse the source.

Is there anything you can do to help me fix it?

The certificate expired on Saturday, April 21, 2018

Error code: SEC_ERROR_EXPIRED_CERTIFICATE

Case-insensitive toggle for filters

It'd be nice to have a checkbox to make filters case-insensitive to avoid having to create complex duplicate entries to cover all possibilities.

The Youtube channel I told you about before (https://www.youtube.com/feeds/videos.xml?channel_id=UCoxg3Kml41wE3IPq-PC-LQw) is an example of a feed not following a pattern and mixing up cases in different post titles. I could shorten the filter I've created for it to only three keywords if it wasn't case-sensitive:

[T|t]railer|TRAILER|[I|i]nterview|INTERVIEW|[P|p]lay.?through|PLAY.?THROUGH

Export/backup pipes

Have you considered adding an option for accounts with regular and above plans to export/backup their pipes?

Back button not working

Open a pipe page like https://www.pipes.digital/pipe/DpOrjQOw , click on Edit then go back to the previous page. It shows the raw code of the pipe.

Firefox 62.0

Combine block failure

Anything I put into the Combine block results in "Internal Server Error" for the output feed.

New update made a filter stop working

Hey. You've recently added a new "search in fields:" option to the filter block, right? It let a lot of posts with "trailer" in the title, for example, to get through my pipe and appear on my RSS reader, even though the keyword "trailer" is listed among my blacklisted terms.

I've solved this by just opening the pipe and pressing "Save" without changing anything. After reloading the feed, those posts were filtered again. Anyway, I thought that you might like to know about this problem.

Inverse Filter

Many of the sites I read put their podcast announcements on the same feeds as their articles. I'd love to be able to build something that says "Remove any post that matches..." to remove these! Sort of a block that inverts the current "Keep any post that matches..." provided by Filter.