turbot/steampipe-plugin-rss

Use SQL to instantly query RSS channels and Atom Feeds. Open source CLI. No DB required.

RSS Plugin for Steampipe

Use SQL to query RSS channels and Atom Feeds.

• Get started →
• Documentation: Table definitions & examples
• Community: Join #steampipe on Slack →
• Get involved: Issues

Quick start

Install the plugin with Steampipe:

steampipe plugin install rss

Run a query:

select
  title,
  link,
  description
from
  rss_item
where
  feed_link = 'https://steampipe.io/blog/feed.xml';

Engines

This plugin is available for the following engines:

Engine	Description
Steampipe	The Steampipe CLI exposes APIs and services as a high-performance relational database, giving you the ability to write SQL-based queries to explore dynamic data. Mods extend Steampipe's capabilities with dashboards, reports, and controls built with simple HCL. The Steampipe CLI is a turnkey solution that includes its own Postgres database, plugin management, and mod support.
Postgres FDW	Steampipe Postgres FDWs are native Postgres Foreign Data Wrappers that translate APIs to foreign tables. Unlike Steampipe CLI, which ships with its own Postgres server instance, the Steampipe Postgres FDWs can be installed in any supported Postgres database version.
SQLite Extension	Steampipe SQLite Extensions provide SQLite virtual tables that translate your queries into API calls, transparently fetching information from your API or service as you request it.
Export	Steampipe Plugin Exporters provide a flexible mechanism for exporting information from cloud services and APIs. Each exporter is a stand-alone binary that allows you to extract data using Steampipe plugins without a database.
Turbot Pipes	Turbot Pipes is the only intelligence, automation & security platform built specifically for DevOps. Pipes provide hosted Steampipe database instances, shared dashboards, snapshots, and more.

Developing

Prerequisites:

• Steampipe
• Golang

Clone:

git clone https://github.com/turbot/steampipe-plugin-rss.git
cd steampipe-plugin-rss

Build, which automatically installs the new version to your ~/.steampipe/plugins directory:

make

Configure the plugin:

cp config/* ~/.steampipe/config
vi ~/.steampipe/config/rss.spc

Try it!

steampipe query
> .inspect rss

Further reading:

• Writing plugins
• Writing your first table

Open Source & Contributing

This repository is published under the Apache 2.0 (source code) and CC BY-NC-ND (docs) licenses. Please see our code of conduct. We look forward to collaborating with you!

Steampipe is a product produced from this open source software, exclusively by Turbot HQ, Inc. It is distributed under our commercial terms. Others are allowed to make their own distribution of the software, but cannot use any of the Turbot trademarks, cloud services, etc. You can learn more in our Open Source FAQ.

Get Involved

Join #steampipe on Slack →

Want to help but don't know where to start? Pick up one of the help wanted issues:

• Steampipe
• RSS Plugin

steampipe-plugin-rss's Issues

Add support for default feed links in config file.

Is your feature request related to a problem? Please describe.
The user should be able to set up the default feed links in the config file, or they should be able to pass it through the optional column.

Update steampipe-plugin-sdk to v1.3.1

cannot see items for aws and talend status of services

Describe the bug
Cannot get teh list of itens

Steampipe version (steampipe -v)
latest

Plugin version (steampipe plugin list)
latest

To reproduce

No records for

select title from rss_channel where feed_link = 'http://status.aws.amazon.com/rss/all.rss' limit 1

I get only 1 record for

select * from rss_channel where feed_link = 'https://trust.talend.com/history.atom';
+----------------------------------------+--------------------------+--------------+--------------+------------+-----------+-------------+------------+---------------->
| title                                  | link                     | author_email | author_name  | categories | copyright | description | extensions | feed_link      >
+----------------------------------------+--------------------------+--------------+--------------+------------+-----------+-------------+------------+---------------->
| Talend Cloud Status - Incident History | https://trust.talend.com |              | Talend Cloud | <null>     | <null>    | <null>      | <null>     | https://trust.t>
+----------------------------------------+--------------------------+--------------+--------------+------------+-----------+-------------+------------+---------------->

Queries with `where...in` return no results

These work individually, but not both via where ... in

> select * from rss_item where feed_link in ('https://steampipe.io/blog/feed.xml', 'http://feeds.feedburner.com/AmazonWebServicesBlog')
+-------+------+--------------+-------------+------------+---------+-------------+------------+------------+-----------+------+-------------+-----------+-----------+---------+
| title | link | author_email | author_name | categories | content | description | enclosures | extensions | feed_link | guid | image_title | image_url | published | updated |
+-------+------+--------------+-------------+------------+---------+-------------+------------+------------+-----------+------+-------------+-----------+-----------+---------+
+-------+------+--------------+-------------+------------+---------+-------------+------------+------------+-----------+------+-------------+-----------+-----------+---------+
> 
> select * from rss_item where feed_link in ('https://steampipe.io/blog/feed.xml')
+---------------------------------------------------+--------------------------------------------------------------------------------+--------------+-------------+------------+---------+------
| title                                             | link                                                                           | author_email | author_name | categories | content | descr
+---------------------------------------------------+--------------------------------------------------------------------------------+--------------+-------------+------------+---------+------
| Top 10 Checks: IAM Credential Report              | https://steampipe.io/blog/aws-iam-credential-report-top-10-checks              | <null>       | <null>      | <null>     | <null>  | Learn
| Analysis of Cloud Provider Market Share – 2021    | https://steampipe.io/blog/aws-azure-gcp-where-to-build                         | <null>       | <null>      | <null>     | <null>  | Using
| Corral your untagged cloud cattle                 | https://steampipe.io/blog/finding-untagged-resources                           | <null>       | <null>      | <null>     | <null>  | How t
| The Hitchhiker's Guide to IAM Policy Wildcards    | https://steampipe.io/blog/aws-iam-policy-wildcards-reference                   | <null>       | <null>      | <null>     | <null>  | A qui
| New: AWS Multi-Region Queries and Query Caching   | https://steampipe.io/blog/release-0-2-0                                        | <null>       | <null>      | <null>     | <null>  | Learn
| Monitor compliance of the AWS CIS 1.15 benchmark. | https://steampipe.io/blog/aws-cis-1-15-ensure-iam-users-have-group-permissions | <null>       | <null>      | <null>     | <null>  | Ensur
| Normalizing AWS IAM Policies for Automation       | https://steampipe.io/blog/normalizing-aws-iam-policies-for-automated-analysis  | <null>       | <null>      | <null>     | <null>  | Uncov
| Steampipe ❤ Parliament                            | https://steampipe.io/blog/steampipe-builds-on-parliament-aws-iam               | <null>       | <null>      | <null>     | <null>  | How S
| Steampipe: select * from cloud;                   | https://steampipe.io/blog/introducing-steampipe                                | <null>       | <null>      | <null>     | <null>  | The e
+---------------------------------------------------+--------------------------------------------------------------------------------+--------------+-------------+------------+---------+------
> select * from rss_item where feed_link in ('http://feeds.feedburner.com/AmazonWebServicesBlog' )
+-------------------------------------------------------------------------------------------------------------+----------------------------------------------------------------------+----------
| title                                                                                                       | link                                                                 | author_em
+-------------------------------------------------------------------------------------------------------------+----------------------------------------------------------------------+----------
| AWS Fault Injection Simulator – Use Controlled Experiments to Boost Resilience                              | http://feedproxy.google.com/~r/AmazonWebServicesBlog/~3/5dAf2jYNI4o/ |          
|                                                                                                             |                                                                      |          
...

Add example of querying multiple feeds from a CSV file

Here is a full example of how to query multiple feeds in a CSV file. We should add a simpler version of this to the docs:

/tmp/demo $ cat feeds.csv 
uri
https://steampipe.io/blog/feed.xml
http://scripting.com/rss.xml
https://rss.art19.com/apology-line
http://rss.art19.com/the-daily
https://feeds.fireside.fm/bibleinayear/rss
https://feeds.megaphone.fm/ADL9840290619
http://feeds.wnyc.org/experiment_podcast
/tmp/demo $ steampipe query
Welcome to Steampipe v0.14.4-rc.0
For more information, type .help
> select * from feeds
+--------------------------------------------+---------------------------+
| uri                                        | _ctx                      |
+--------------------------------------------+---------------------------+
| https://feeds.fireside.fm/bibleinayear/rss | {"connection_name":"csv"} |
| https://rss.art19.com/apology-line         | {"connection_name":"csv"} |
| https://steampipe.io/blog/feed.xml         | {"connection_name":"csv"} |
| http://feeds.wnyc.org/experiment_podcast   | {"connection_name":"csv"} |
| http://scripting.com/rss.xml               | {"connection_name":"csv"} |
| https://feeds.megaphone.fm/ADL9840290619   | {"connection_name":"csv"} |
| http://rss.art19.com/the-daily             | {"connection_name":"csv"} |
+--------------------------------------------+---------------------------+

Time: 80.980125ms
> select published, title, author_name, feed_link
  from feeds join rss_item on feed_link = uri
  limit 5;
+---------------------------+------------------------------------------------+------------------------------------+-----
| published                 | title                                          | author_name                        | feed
+---------------------------+------------------------------------------------+------------------------------------+-----
| 2022-04-21T09:30:00+05:30 | Day 111: Saul Tries to Kill David (2022)       | Ascension Catholic Faith Formation | http
| 2022-04-29T09:30:00+05:30 | Day 119: David's Wisdom (2022)                 | Ascension Catholic Faith Formation | http
| 2022-05-07T09:30:00+05:30 | Day 127: Kindness for the House of Saul (2022) | Ascension Catholic Faith Formation | http
| 2022-04-18T09:30:00+05:30 | Day 108: Saul's Vanity (2022)                  | Ascension Catholic Faith Formation | http
| 2022-04-27T09:30:00+05:30 | Day 117: Reverence and Faithfulness (2022)     | Ascension Catholic Faith Formation | http
+---------------------------+------------------------------------------------+------------------------------------+-----

Time: 12.359159375s
> select date_trunc('year', published) as pub_date, count(*)
  from feeds join rss_item on feed_link = uri
  group by pub_date
  order by pub_date desc;
+---------------------------+-------+
| pub_date                  | count |
+---------------------------+-------+
| 2022-01-01T05:30:00+05:30 | 351   |
| 2021-01-01T05:30:00+05:30 | 819   |
| 2020-01-01T05:30:00+05:30 | 382   |
| 2019-01-01T05:30:00+05:30 | 330   |
| 2018-01-01T05:30:00+05:30 | 323   |
| 2017-01-01T05:30:00+05:30 | 240   |
+---------------------------+-------+

Time: 11.790391792s
> 

v0.0.2