Planet Venus is an awesome ‘river of news’ feed reader. It downloads news feeds published by web sites and aggregates their content together into a single combined feed, latest news first.
Home Page: http://intertwingly.net/code/venus/docs/index.html
License: Other
Legacy Planet software encoded the feed URLs.
On Venus, this syntax is broken:
[http://hyperboree-apollon.blogspot.com/feeds/posts/default/-/wikipédia]
It would be nice if spider.py could read the string as UTF-8 and encode it properly, to be compatible with Planet .ini files.
dbhash module is deprecated:
https://docs.python.org/2.7/library/dbhash.html
This functionality still exists in bsddb3 (outside of standard library).
bsddb3 also supports Python 3.
--- planet/idindex.py
+++ planet/idindex.py
@@ -13,8 +13,8 @@
cache = config.cache_directory()
index=os.path.join(cache,'index')
if not os.path.exists(index): return None
- import dbhash
- return dbhash.open(filename(index, 'id'),'w')
+ import bsddb3
+ return bsddb3.hashopen(filename(index, 'id'),'w')
except Exception, e:
if e.__class__.__name__ == 'DBError': e = e.args[-1]
from planet import logger as log
@@ -35,8 +35,8 @@
cache = config.cache_directory()
index=os.path.join(cache,'index')
if not os.path.exists(index): os.makedirs(index)
- import dbhash
- index = dbhash.open(filename(index, 'id'),'c')
+ import bsddb3
+ index = bsddb3.hashopen(filename(index, 'id'),'c')
try:
import libxml2
--- tests/test_idindex.py
+++ tests/test_idindex.py
@@ -67,8 +67,8 @@
self.assertEqual(12,len(doc.getElementsByTagName('planet:name')))
try:
- module = 'dbhash'
+ import bsddb3
except ImportError:
- planet.logger.warn("dbhash is not available => can't test id index")
+ planet.logger.warn("bsddb3 is not available => can't test id index")
for method in dir(idIndexTest):
if method.startswith('test_'): delattr(idIndexTest,method)
1st of all, thanks for the great tool!
As reported in ros-infrastructure/planet.ros.org#15, I have just updated my blog that was inactive for a year and a half. And the latest posts are not pulled (to Venus-powered http://planet.ros.org/, the website the ticket above is for).
Is there any chance that a website that is once labeled as inactive won't become active even though there are new entries, unless specifically operated so on the Venus-host side?
feedburner changed what they are inserting into people's feeds. Need to update it.
Probably <div class="feedflare">.*</div> would work...
(note: this is a somewhat long winded stream of consciousness rant, because i'm not really sure what the best solution is ... i'm not sure i even understand all the problems)
"ignore_in_feed: updated" seems really handy, but it's an incredibly painful double edged sword.
For my purposes, what i really (truly) want is for entries to never be reordered in my planet, even if they are updated in the source feed.
If you use "ignore_in_feed: updated" you get something close to this behavior, except that:
...that last point sounds like a positive, except that anytime you add a new feed w/o "published" dates, every item in that feed suddenly floods the "top" of the planet. Likewise if you need to purge your cache for some reason (ie: if you've added a new filter and want all the existing items to be updated) suddenly all sorts of old items appear at the top of the planet.
What i'd like is to have a "sort_date" which is defined along the lines of:
if entry has never been seen before and is not in the cache:
set entry.sort_date = min(now_if_null(entry.published_date), now_if_null(entry.updated_date))
else:
set entry.sort_date = cache.get(entry.id).sort_date
...and then have venus use that sort_date for the sorted list (i'm happy to still display the published & updated dates in the templates, although it would be nice to have the sort_date in the template as well)
Having already realized that venus sets the "mtime" of the cache files based on the "updated" date of each entry, I set out to write a plugin filter that would implement that logic, but using the "updated" date of each entry as the sort so i wouldn't have to modify any of the main venus code for dealing with the cache file mtimes.
To start with, i just made my filter replace the "updated" date with the "published" date if it existed and was earlier -- but then i realized that even though my filter was setting the "updated" just fine, it wasn't being reflected in the "mtime" of the cache file. This was perplexing, but I figured it wasn't a big deal: i was going to have my filter read from the cache anyway, i could also make it touch the cache file with the appropriate mtime value -- but when i looked closer at the venus code i realized that when spider.py's writeCache method is looping over the feed entries, it seems to go out of it's way to save out the "updated" date of the entry before running any filters -- and then it uses that saved "updated" value to set the mtime on the final cache file after the filters run -- so even if my plugin did reach into the cache to touch the file, spider.py would undo that.
hence this writeup.
It seems like the flow of control in spider.py is kind of backwards, it seems like filters should be able to modify just about anything in the entry, including the "updated" date, and spider.py should respect that final computed value and use it as the mtime of the file -- but the flow of that method almost seems intentional, like there is a deliberate reason why that's not allowed, and if there is i'm definitely curious as to why that is.
Alternately: if anyone has any suggestions on how to achieve the goal stated above (to have the date used for sorting entries come from the feed the first time the entry is encountered, but never change once it's in the cache) I'd love to hear them as well. so far the best i've come up with is to use "ignore_in_feed: updated" when i let venus run as a cron, but comment it out any time I add a new feed or after flushing the cache. (but that's error prone, and a pain in the ass)
See e.g. http://planet.webcompat.com/ (which uses Venus), Dec 13th. The title should include a literal <dialog> but it's removed.
We have the following setup:
http://planet.symlink.ch/ is subscribed to http://noone.org/blog/index.rss
http://planet-commandline.org/ is subscribed to http://noone.org/blog/English/Computer/Debian/CoolTools/index.rss and http://noone.org/blog/English/Computer/Shell/index.rss which both contain subsets of the postings in http://noone.org/blog/index.rss
Both planets run from the same PlanetVenus installation (git master HEAD as of now) including the same cache, just with different configuration files.
Now the posting http://noone.org/blog/Deutsch/Computer/Internet/Mutationen%2520auf%2520Planet%2520Symlink.html shows up on http://planet-commandline.org/ as soon as it has been updated once after http://planet.symlink.ch/ has cached it, despite it's only in http://noone.org/blog/index.rss but neither in http://noone.org/blog/English/Computer/Debian/CoolTools/index.rss nor in http://noone.org/blog/English/Computer/Shell/index.rss
Looks to me as if the cache evaluation is broken with regards to checking if a cached posting belongs to a planet or not.
planet.gnome.org currently autoplays the video in this post: http://zaheer.merali.org/articles/2010/05/24/more-exciting-live-streaming-from-flumotion-with-vp8-and-webm/
Should this be fixed in planet venus, or in the feedparser?
Hi
I am currently seeing that the articles that have embed gist are not being display. Just wondering is this feature available?
Greetings!
In XMPP, using PubSub(1) (XEP-0060), it is possible to store posts as Atom Syndication(2) entries into XMPP(3).
There are several XMPP extensions that apply it, namely XEP-0227 (4) and XEP-0472 (5) which make node xmpp:microblog:0 as a common choice to manage news for client accounts (e.g. [email protected]), yet a component account (e.g. pubsub.horizontal.land) may use all of its nodes for this purpose.
pubsub.horizontal.land/
|
|-- updates/
|-- Sombre
`-- Varied
|-- releases/
`-- events/
pubsub service/
|
|-- node/
|-- item
`-- item
|-- node/
`-- node/
I have made a script that realizes viewing PubSub node items as Atom XML feeds over HTTP(6).
Hey @rubys
I've got a couple of webhooks in this repo that I'd like to add. I'd like to setup Codeship (CI) and ReadTheDocs (Sphinx documentation). Do you have accounts on either? I am unable to get these setup without admin rights to this repo because both need to add webhooks to the repo..
I am NOT asking for admin rights just for you to hit the go button on the various bits to get things running. :)
ReadTheDocs is currently setup here: https://readthedocs.org/projects/planet-venus/
I just need a username from you to add you as a maintainer and then the webhooks will get set. This will allow an automated documentation host.
Codeship is a little more complicated. We could also use TravisCI, Semaphore, etc. You'd need to create a project there and add me as a collaborator. Then we get the testing running and the badge and the rest.
Hi! The website referenced in the code and the README is not reachable. Is there an alternative? I'd be interested in the mailinglist as well.
$ ping planetplanet.org
PING planetplanet.org (70.85.31.216) 56(84) bytes of data.
[no further output]
Hi could the dev branch be made the default please? Only because master has not been updated since 2011.
I cannot find a python package called RDF that is used in the foaf2config functions. I believe it has been replaced with rdflib
Some feeds, especially feeds generated by Drupal, seem to break the layout of the planet website. See for example blog entries from http://spatialguru.com/blog/feed/, http://www.blogger.com/feeds/5176900881057973693/posts/default and http://www.geops.de/blog/feed/Technology on http://planet.osgeo.org/. I guess this is an issue at feed parsing level?
I cloned the git repo as of "9de21094a8cf565bdfcf75688e121a5ad1f5397b" and tried to follow the general instructions but encountered an error with "python runtests.py" Full output and the details i could think of that might be relevant (i'm not really a python guy) are below...
hossman@bester:~/venus [master] $ python runtests.py
Genshi is not available => can't test genshi filters
Redland RDF is not available => can't test FOAF reading lists
................................................................................................................E............................................................................................................
======================================================================
ERROR: test_content_tag_soup (tests.test_reconstitute.ReconstituteTest)
----------------------------------------------------------------------
Traceback (most recent call last):
File "/home/hossman/venus/tests/test_reconstitute.py", line 43, in <lambda>
func = lambda self, name=root: self.eval(name)
File "/home/hossman/venus/tests/test_reconstitute.py", line 26, in eval
results = feedparser.parse(data)
File "/home/hossman/venus/planet/vendor/feedparser.py", line 3602, in parse
feedparser.feed(data)
File "/home/hossman/venus/planet/vendor/feedparser.py", line 1723, in feed
sgmllib.SGMLParser.feed(self, data)
File "/usr/lib/python2.6/sgmllib.py", line 104, in feed
self.goahead(0)
File "/usr/lib/python2.6/sgmllib.py", line 143, in goahead
k = self.parse_endtag(i)
File "/usr/lib/python2.6/sgmllib.py", line 320, in parse_endtag
self.finish_endtag(tag)
File "/usr/lib/python2.6/sgmllib.py", line 360, in finish_endtag
self.unknown_endtag(tag)
File "/home/hossman/venus/planet/vendor/feedparser.py", line 580, in unknown_endtag
method()
File "/home/hossman/venus/planet/vendor/feedparser.py", line 1540, in _end_content
value = self.popContent('content')
File "/home/hossman/venus/planet/vendor/feedparser.py", line 867, in popContent
value = self.pop(tag)
File "/home/hossman/venus/planet/vendor/feedparser.py", line 782, in pop
mfresults = _parseMicroformats(output, self.baseuri, self.encoding)
File "/home/hossman/venus/planet/vendor/feedparser.py", line 2279, in _parseMicroformats
p = _MicroformatsParser(htmlSource, baseURI, encoding)
File "/home/hossman/venus/planet/vendor/feedparser.py", line 1884, in __init__
self.document = BeautifulSoup.BeautifulSoup(data)
File "/usr/lib/pymodules/python2.6/BeautifulSoup.py", line 1499, in __init__
BeautifulStoneSoup.__init__(self, *args, **kwargs)
File "/usr/lib/pymodules/python2.6/BeautifulSoup.py", line 1230, in __init__
self._feed(isHTML=isHTML)
File "/usr/lib/pymodules/python2.6/BeautifulSoup.py", line 1263, in _feed
self.builder.feed(markup)
File "/usr/lib/python2.6/HTMLParser.py", line 108, in feed
self.goahead(0)
File "/usr/lib/python2.6/HTMLParser.py", line 148, in goahead
k = self.parse_starttag(i)
File "/usr/lib/python2.6/HTMLParser.py", line 226, in parse_starttag
endpos = self.check_for_whole_start_tag(i)
File "/usr/lib/python2.6/HTMLParser.py", line 301, in check_for_whole_start_tag
self.error("malformed start tag")
File "/usr/lib/python2.6/HTMLParser.py", line 115, in error
raise HTMLParseError(message, self.getpos())
HTMLParseError: malformed start tag, at line 1, column 14
----------------------------------------------------------------------
Ran 221 tests in 19.080s
FAILED (errors=1)
hossman@bester:~/venus [master] $ uname -a
Linux bester 2.6.31-23-generic #75-Ubuntu SMP Fri Mar 18 18:16:06 UTC 2011 x86_64 GNU/Linux
hossman@bester:~/venus [master] $ python --version
Python 2.6.4
hossman@bester:~/venus [master] $ dpkg --list | grep python
ii libpython2.6 2.6.4-0ubuntu3 Shared Python runtime library (version 2.6)
ii python 2.6.4-0ubuntu1 An interactive high-level object-oriented la
ii python-apport 1.9.3-0ubuntu4.2 apport crash report handling library
ii python-apt 0.7.13.2ubuntu4 Python interface to libapt-pkg
ii python-aptdaemon 0.10+bzr264-0ubuntu1 Python module for the server and client of a
ii python-aptdaemon-gtk 0.10+bzr264-0ubuntu1 Python GTK+ widgets to run an aptdaemon clie
ii python-avahi 0.6.25-1ubuntu5.3 Python utility package for Avahi
ii python-beautifulsoup 3.1.0.1-2 error-tolerant HTML parser for Python
ii python-brlapi 4.0-7ubuntu2 Python bindings for BrlAPI
ii python-cairo 1.8.6-1ubuntu1 Python bindings for the Cairo vector graphic
ii python-central 0.6.11ubuntu9 register and build utility for Python packag
ii python-cherrypy3 3.1.2-1 Python web development framework - version 3
ii python-chm 0.8.4-1 Python binding for CHMLIB
ii python-clientform 0.2.10-2 module for handling HTML forms on the client
ii python-configglue 0.2dev-0ubuntu2 Glues together optparse.OptionParser and Con
ii python-couchdb 0.6-1 library for working with Apache CouchDB
ii python-crypto 2.0.1+dfsg1-4ubuntu1 cryptographic algorithms and protocols for P
ii python-cssutils 0.9.5.1-1 CSS Cascading Style Sheets parser and builde
ii python-cups 1.9.46-0ubuntu2 Python bindings for CUPS
ii python-cupshelpers 1.1.12+git20090826-0ubuntu8 Python modules for printer configuration wit
ii python-dateutil 1.4.1-3 powerful extensions to the standard datetime
ii python-dbus 0.83.0-1ubuntu2 simple interprocess messaging system (Python
ii python-debian 0.1.14ubuntu1 Python modules to work with Debian-related d
ii python-desktopcouch 0.5-0ubuntu1 Python Desktop CouchDB
ii python-desktopcouch-records 0.5-0ubuntu1 Desktop CouchDB Records API
ii python-dev 2.6.4-0ubuntu1 Header files and a static library for Python
ii python-django 1.1.1-1ubuntu1.2 High-level Python web development framework
ii python-django-tagging 0.2.1+svn154-0ubuntu2 A generic tagging application for Django pro
ii python-encutils 0.9.5.1-1 Encoding detection collection for Python
ii python-fstab 1.4-0ubuntu1 read, manipulate, and write /etc/fstab files
ii python-gconf 2.28.0-0ubuntu1 Python bindings for the GConf configuration
ii python-gdbm 2.6.3-0ubuntu1 GNU dbm database support for Python
ii python-glade2 2.16.0-0ubuntu1 GTK+ bindings: Glade support
ii python-gmenu 2.28.0.1-0ubuntu1 an implementation of the freedesktop menu sp
ii python-gnome2 2.28.0-0ubuntu1 Python bindings for the GNOME desktop enviro
ii python-gnomeapplet 2.28.0-0ubuntu1 Python bindings for the GNOME panel applet l
ii python-gnomecanvas 2.28.0-0ubuntu1 Python bindings for gnomecanvas (debug exten
ii python-gnomekeyring 2.28.0-0ubuntu1 Python bindings for the GNOME keyring librar
ii python-gnupginterface 0.3.2-9ubuntu2 Python interface to GnuPG (GPG)
ii python-gobject 2.18.0-0ubuntu2 Python bindings for the GObject library
ii python-gst0.10 0.10.17-1 generic media-playing framework (Python bind
ii python-gtk2 2.16.0-0ubuntu1 Python bindings for the GTK+ widget set
ii python-gtkhtml2 2.25.3-3ubuntu1.9.10.1 Python bindings for the GtkHTML 2 library
ii python-gtksourceview2 2.8.0-1 Python bindings for the GtkSourceView widget
ii python-httplib2 0.4.0-0ubuntu2 A comprehensive HTTP client library written
ii python-ibus 1.2.0.20090927-2ubuntu2 New input method framework using dbus
ii python-imaging 1.1.6-3ubuntu1 Python Imaging Library
ii python-launchpad-integration 0.1.26 library for launchpad integration
ii python-launchpadlib 1.5.1-0ubuntu1 Launchpad web services client library
ii python-lazr-restfulclient 0.9.3-0ubuntu3 client for lazr.restful-based web services
ii python-lazr-uri 1.0-0ubuntu1 library for parsing, manipulating, and gener
ii python-libxml2 2.7.5.dfsg-1ubuntu1.2 Python bindings for the GNOME XML library
ii python-louis 1.7.0-1ubuntu1 Python bindings for liblouis
ii python-lxml 2.1.5-1ubuntu2 pythonic binding for the libxml2 and libxslt
ii python-mechanize 0.1.11-0ubuntu1 stateful programmatic web browsing
ii python-minimal 2.6.4-0ubuntu1 A minimal subset of the Python language (def
ii python-newt 0.52.10-4ubuntu1 A NEWT module for Python
ii python-notify 0.1.1-2build2 Python bindings for libnotify
ii python-numpy 1:1.3.0-3 Numerical Python adds a fast array facility
ii python-oauth 1.0a~svn1124-0ubuntu2 implementation of the OAuth protocol
ii python-openoffice 0.1+r34-1 Python libraries for interacting with OpenOf
ii python-openssl 0.9-1 Python wrapper around the OpenSSL library
ii python-pam 0.4.2-12ubuntu3 A Python interface to the PAM library
ii python-papyon 0.4.3-1ubuntu1 MSN client library written in Python
ii python-pkg-resources 0.6c9-0ubuntu5 Package Discovery and Resource Access using
ii python-problem-report 1.9.3-0ubuntu4.2 Python library to handle problem reports
ii python-protobuf 2.0.3-2.2ubuntu2 Python bindings for protocol buffers
ii python-pyatspi 1.28.1-0ubuntu1 Assistive Technology Service Provider Interf
ii python-pyinotify 0.8.6-2ubuntu2 simple Linux inotify Python bindings
ii python-pyorbit 2.24.0-0ubuntu3 A Python language binding for the ORBit2 COR
ii python-pypdf 1.12-2 PDF toolkit implemented solely in Python
ii python-qt4 4.6-1 Python bindings for Qt4
ii python-rdflib 2.4.0-5ubuntu1 RDF library containing an RDF triple store a
ii python-renderpm 2.3-0ubuntu1 python low level render interface
ii python-reportlab 2.3-0ubuntu1 ReportLab library to create PDF documents us
ii python-reportlab-accel 2.3-0ubuntu1 C coded extension accelerator for the Report
ii python-rsvg 2.28.0-0ubuntu1 Python bindings for the RSVG library
ii python-serial 2.3-1 pyserial - module encapsulating access for t
ii python-sexy 0.1.9-1ubuntu2 python language bindings for libsexy
ii python-simplejson 2.0.9-1 Simple, fast, extensible JSON encoder/decode
ii python-sip4 4.9.1-snapshot-20091015-0ubuntu1 Python/C++ bindings generator runtime librar
ii python-smbc 1.0.6-0ubuntu2 Python bindings for Samba clients (libsmbcli
ii python-software-properties 0.75.4 manage the repositories that you install sof
ii python-speechd 0.6.7+git20090914~unofficial-0ubuntu4 Python interface to Speech Dispatcher
ii python-support 1.0.3ubuntu1 automated rebuilding support for Python modu
ii python-telepathy 0.15.11-1 Python language bindings for telepathy
ii python-tk 2.6.3-0ubuntu1 Tkinter - Writing Tk applications with Pytho
ii python-twisted-bin 8.2.0-3 Event-based framework for internet applicati
ii python-twisted-core 8.2.0-3 Event-based framework for internet applicati
ii python-twisted-names 8.2.0-1ubuntu2 A DNS protocol implementation with client an
ii python-twisted-web 8.2.0-2ubuntu1 An HTTP protocol implementation together wit
ii python-ubuntuone-client 1.0.3-0ubuntu1 Ubuntu One client Python libraries
ii python-ubuntuone-storageprotocol 1.0.1-0ubuntu1 Python library for Ubuntu One file storage a
ii python-uniconvertor 1.1.4-1 Universal vector graphics translator
ii python-uno 1:3.2.0-7ubuntu1~karmic1 Python-UNO bridge
ii python-virtkey 0.50ubuntu2 Library to emulate keyboard keypresses.
ii python-vte 1:0.22.2-0ubuntu2.1 Python bindings for the VTE widget set
ii python-wadllib 1.1.2-0ubuntu1 Python library for navigating WADL files
ii python-webkit 1.1.5-1 WebKit/Gtk Python bindings
ii python-xapian 1.0.14-1build1 Xapian search engine interface for Python
ii python-xdg 0.15-1.1ubuntu5 A python library to access freedesktop.org s
ii python-xkit 0.4.2 library for the manipulation of the xorg.con
ii python-zope.interface 3.5.2-1 Zope 3 Interface Infrastructure
ii python2.4 2.4.6-1ubuntu3.2.9.10.1 An interactive high-level object-oriented la
ii python2.4-minimal 2.4.6-1ubuntu3.2.9.10.1 A minimal subset of the Python language (ver
ii python2.5 2.5.4-1ubuntu6.1 An interactive high-level object-oriented la
ii python2.5-minimal 2.5.4-1ubuntu6.1 A minimal subset of the Python language (ver
ii python2.6 2.6.4-0ubuntu3 An interactive high-level object-oriented la
ii python2.6-dev 2.6.4-0ubuntu3 Header files and a static library for Python
ii python2.6-minimal 2.6.4-0ubuntu3 A minimal subset of the Python language (ver
It seems that there is a bug in the current version of Venus where images that are very large span beyond the page margin. Ideally images would be automatically resized within the current template. Is there a fix for this planned?
Thanks!
There doesn't seem to be any activity in Github for a long time, is the project being maintained elsewhere?
The web site points to Github:
http://www.intertwingly.net/code/venus/planet/
If you are not maintaining the project, do you want somebody else to do so or is there a fork or official successor to the project?
WP Engine (and perhaps other blog hosts) reject HTTP requests that don't have a user-agent field in the header. Planet Venus issues such requests for feeds when spider_threads > 0.
Case in point, mathjax from blogger feeds.
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
Django
The Web framework for perfectionists with deadlines.
Laravel
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
Microsoft
Open source projects and samples from Microsoft.
Google
Google ❤️ Open Source for everyone.
Alibaba
Alibaba Open Source for everyone
D3
Data-Driven Documents codes.
Tencent
China tencent open source team.