othernet-project/librarian

Content indexer for Outernet receivers

librarian

Librarian is a web server that allows users to view and download files stored on Outernet receivers. In essence it is similar to a typical static web server's directory index with bells and whistles.

Generating documentation

To generate the documentation, you need to have Python and Sphinx installed. Once you have the prerequisites run make:

$ make docs

The generated documentation will be found in docs/build/html.

Running the test suite

To run the tests first prepare the development environment according to the documentation:

$ make prepare

You will need to have Tox installed. Then run the tests by simply executing the tox command:

$ tox

Contributing interface translations

You can either translate using the .po files in the project directory, or use the hosted version on POEditor (recommended). Note that Librarian is always under active development, so strings will change from time to time, and sometimes the contributed translations may be completely removed.

If you are contributing, or considering contributing translations, please swing by our forums and say hello.

List of translation contributors

Special thanks to all those who have contributed their valuable time and knowledge:

• Kurei-Z
• Haveson Florvil Alphanet
• Susruthan Seran
• Ignas Sklerius
• Dani Álvarez
• Mogens From
• necklinux
• Stefan Ionita
• Lars Lillo Ulvestad
• Max Boone
• Eric Alexander
• Rafael Leda
• Daniel Antal
• Emanuelis Adomaitis
• Filippo
• zaid
• Manish Rastogi
• cavit
• Surit
• Chengtao Hua
• Josep Manel
• Saman rajaei
• patrik
• Aadarsh
• Kajcha
• André Daniel
• Andrey
• Ismail Al Ahmad
• Morten
• Morteza
• Max Sundström
• Viktoria
• David Marques
• Warodom Phaenthong
• Baris Sekerciler
• Tolga Evcimen
• Will Martin
• Manoel Junior
• Krabbs
• Vibhor Dubey
• Nelson Macchi
• 13. Ammar
• Gustaf
• Hammad Tariq
• Romaric FAVE
• Peer Oliver Schmidt
• javad katooli
• dan
• Gerry Galactic
• Varshan BP
• Frédéric Uhrweiller
• Pengan Zhou
• Arthur
• Navina
• Nicky Kouffeld, Dutchwebs
• Sakara
• Bruno Nogueira
• Mohammad Reza Golaghaee
• Luis Fuentes
• Marco Rubio
• Rodrigo
• Kasparas
• Frederik
• zanghel
• Benoît Casanova
• Gonzalo
• Anirban Chatterjee
• Christian Novrup
• Janberk Genç
• Mario Lopez
• Dana Tierney
• Terrence
• Ahmed
• daming_99
• Steffie
• Francesca
• HM
• Julien
• Sebastian Borg
• Ronald Philipsen
• Rebeca Virgo
• Ciprian
• Gabriel
• Salutlolo
• Souhaïl BOUGRINE
• Baris Kilic
• Tori Arbaugh
• Hamza Siddiqui
• Thibaut
• Alireza Keshavarz
• Jannis A. K.
• Miguel Maldonado
• Daem0n
• tommaso
• Moe Ihab
• Massimiliano CARNEMOLLA
• Behzad
• Mehmet Mallı
• Buddha Burman
• Zipper
• Sai Chakradhar Araveti
• Klara Milena Hirscher
• Andrew
• soukayna
• Zakaria Bendali
• Siddharth Nair
• Bruno
• Francis
• Roman
• ix
• Christoph Nebendahl
• Алексей
• behzad
• Chase Burgess
• Slandgkearth
• DURAIRAJAA N

Reporting bugs and feature requests

Bugs and feature requests can be posted either in our forums or in the GitHub issue tracker.

License

Librarian and supporting code are released under GPLv3 or later. Please see COPYING file in the source tree.

librarian's Issues

Add extensive logging support

We also need a place on the dashboard where logs would be displayed.

Dashboard

Show following info:

• Free drive space in spool directory
• Number of content bundles in archive
• Number of downloaded .sig and .zip files in the spool directory

Broken datestamp

The timestamp for files imported a day ago (Aug 14) read '12-31', whereas timestamps of files imported today (Aug 15) say '01-01'.

Dashboard shows date of first updated file instead of last

Content archive cleanup

The automated cleanup is done by removing the oldest items with least views first. The UI should contain a single button to perform this action.

There should be a configurable minimum free space that needs to be freed using this feature, and the system should calculate how much space can be freed before proceeding. This implies there should be no action if there is already sufficient free space.

Add time zone support/configuration

Allow user to set time zone in which all times will be shown. Make that a configuration option.

Depends on #5

Prepare the development environment

Reorganize data files

Move all data files under librarian package to facilitate smooth installation.

Increase librarian log maxbytes to 10M

Unpack zipballs to a separate directory

Do not leave zip files in spool directory. Instead, use a separate directory where zip files should be temporarily stored.

Content stylesheet

Show number of updates on dashboard

Upload unsigned zipballs

Importing content with same ID creates a new entry in archive

The expected behavior is for the existing entry to be updated.

User configuration

• Allow the app to write user configuration as an .ini file.
• Provide both high-level and low-level interfaces for editing configuration.
• Guard against invalid configuration
• Prepare a help file with instructions on how to configure manually and troubleshoot errors

Move ArtExIn's article extraction functions to Librarian

Because of legal restrictions it is not possible to modify HTML of the pages before broadcast. Therefore, the ArtExIn's functionality needs to be moved to Librarian. We still need to do artex on the server so we can preview, but the actual zip file should contain unmodified HTML and Librarian should do the article extraction.

The extraction function should run on content when it is being imported into archive.

We also need to keep an eye on CPU usage to make sure we're not killing the device.

Wrap dvb tools to provide basic tuning capability

Ability to unpack complete directory paths when serving from zipballs

For instance, if the directory contains a file images/foo.jpg the content path should accept the full path and serve it.

Dashboard sidebar for messages

The dashboard needs to have a small area which displays messages. Messages should be downloaded in a separate file messages.json, which should be prepared by ArtExIn. Each message should have the following keys:

• author: Name of the message author
• lang: Language code (two-letter code)
• created: Date and time of creation in UTC
• text: The body of the message

The message file must be encoded UTF8.

Dashboard should (re)load the JSON file on every request, and it should display only messages for the currently active locale, sorted reverse-chronologically.

Include styles for <b> and <i> tags

Code editor

Editing librarian's code from within the web UI

Check disk space before copying to content directory

System update

Linux packages

To simplify installation and updating, create Linux packages for supported platforms. Currently these platforms are:

• Raspbian (deb)
• Arch Linux ARM (pkg.tar.xz)

The packages are platform independent, though, so support for Debian proper, and Arch Linux proper are implied.

In translations, values should always go together with labels

So instead of saying {{ x }} {{ _('updates') }} it should say {{ str(_('%s updates')) % x }}.

Delete content

Next to each item in content library, there should be a button to remove it from archive permanently.

Logging

Add support for user-viewable logs

Overwrite existing file

  File "/vagrant/librarian/downloads.py", line 298, in add_to_archive
    shutil.move(path, target_dir)
  File "/usr/lib/python3.4/shutil.py", line 520, in move
    raise Error("Destination path '%s' already exists" % real_dst)
shutil.Error: Destination path '/srv/outernet/ccf9a2fb7e9d7e32ce5b6a03f186594d.zip' already exists

Fix date labels and show download date

Figure out a way to get the date when file was downloaded and show it next to title on download page.

Also change labels on dates so that it's clear what they mean. Should be 'published' on updates page, and 'added' or something along those lines on archive page.

"total free space" instead of "total space" on Dashboard

Identifying sponsored content

Users should very easily know which content is a part of our basic library (like Wikipedia), which content has been selected by the community, and which content is sponsored.

Sponsored content should be highlighted in feeds, and should also have it's own section.

Find online translation service, and upload translation files

Use TVHeadend API and provide status report

Show device used, whether it has acquired a lock, and signal strength. If possible, also allow user to pick a transponder and tune in.

API is undocumented, so it will need to be reverse-engineered. Also, the TVHeadend version will need to be fixed so API changes don't affect Librarian.

Updates page stops working after crash

Not sure what happend, but looks like some kind of filesystem corruption or the other.

Logs say:

[2014-08-18 02:51:45 (-0600)] ERROR    </var/spool/downloads/content/5c5b3d61e512ec82f6883f79901a494b.zip> invalid zip file
[2014-08-18 02:51:45 (-0600)] DEBUG    </var/spool/downloads/content/5c5b3d61e512ec82f6883f79901a494b.sig> verifying

For some reason the invalid zip file cannot be unlinked.

Traceback (most recent call last):
  File "/usr/lib/python3.4/site-packages/bottle.py", line 862, in _handle
    return route.call(**args)
  File "/usr/lib/python3.4/site-packages/bottle.py", line 1729, in wrapper
    rv = callback(*a, **ka)
  File "/usr/lib/python3.4/site-packages/librarian/lib/squery.py", line 107, in plugin
    body = callback(*args, **kwargs)
  File "/usr/lib/python3.4/site-packages/librarian/lib/i18n.py", line 201, in wrapper
    return callback(*args, **kwargs)
  File "/usr/lib/python3.4/site-packages/bottle.py", line 3612, in wrapper
    result = func(*args, **kwargs)
  File "/usr/lib/python3.4/site-packages/librarian/routes/downloads.py", line 39, in list_downloads
    extracted, errors = downloads.decrypt_all(decryptables)
  File "/usr/lib/python3.4/site-packages/librarian/lib/downloads.py", line 145, in decrypt_all
    os.unlink(zipball)
FileNotFoundError: [Errno 2] No such file or directory: '/var/spool/downloads/content/5c5b3d61e512ec82f6883f79901a494b.zip'

Do not fail if content of the signed file is not a valid zip file

  File "/vagrant/librarian/downloads.py", line 206, in get_file
    raise ContentError("'%s' is not a valid zipball" % path, path)
librarian.downloads.ContentError: '/var/spool/downloads/1c2a9d2077b3717bfff0b4fad2d0961f.zip' is not a valid zipball

Rename "Archive" to "Library"

Implement i18n

Librarian user manual

On-the-fly patching of HTML content

UI for browsing archive contents

Add support for right-to-left layout (Arabic and similar languages)

The page layouts are currently only for left-to-right languages. We need support for right-to-left, based on selected language.

The backend would need to set this information in the base template and provide selectors for this purpose.

Do not run as root

Application API

The application API needs to provide the following:

• read-only access to content database
• read-write access to a database/table/whatever belonging to the application
• ability to identify individual users (sessions, authentication, etc)
• place to store user files
• ability to include application icons in the zip file or info.json in a way that Librarian can recognize

We also need:

• tools for publishing and verifying apps
• SDK

Any other feature requests should be made as comments to this ticket.

Tags

Each info.json metadata file in content bundles may include a tags key, which should be an array of strings, each representing a tag. To support tags, we need either a fixed list of possible tags, or provide tag translations in the form of gettext .mo files in the system updates regularly as new tags appear. Tags are optional and metadata files do not need to contain the tags key, and, if present, it may be empty (null, empty string, etc). Tags are informational labels that are shown next to the content in content listings so that user can determine the origin (e.g., 'sponsored').

Paging on archive and updates pages

HTTP 500 when refreshing uploads page while uploads are still being processed

When user requests the uploads page, then goes to another page, and comes back to the uploads page, the following exception is raised:

Traceback (most recent call last):
  File "/usr/lib/python3.4/site-packages/bottle.py", line 862, in _handle
    return route.call(**args)
  File "/usr/lib/python3.4/site-packages/bottle.py", line 1729, in wrapper
    rv = callback(*a, **ka)
  File "/usr/lib/python3.4/site-packages/librarian/lib/squery.py", line 107, in plugin
    body = callback(*args, **kwargs)
  File "/usr/lib/python3.4/site-packages/librarian/lib/i18n.py", line 201, in wrapper
    return callback(*args, **kwargs)
  File "/usr/lib/python3.4/site-packages/bottle.py", line 3612, in wrapper
    result = func(*args, **kwargs)
  File "/usr/lib/python3.4/site-packages/librarian/routes/downloads.py", line 39, in list_downloads
    extracted, errors = downloads.decrypt_all(decryptables)
  File "/usr/lib/python3.4/site-packages/librarian/lib/downloads.py", line 137, in decrypt_all
    os.unlink(signedf)
FileNotFoundError: [Errno 2] No such file or directory: 'PATH'

This issue should be fixed once Outernet switches to unencrypted broadcasts, but we'll keep this ticket here until it does so others know about this issue.

Add paging to content lists

Channels

There should be four channels:

• updates: System updates
• content: Content (we already have this)
• nonfree: Non-free Content (any content that is not published under a permissive or open-documentation license)
• files: Big files (files that are not packaged as content bundles)

System updates channel contains files necessary to perform system updates. For now, we will perform updates manually. Each update must be accompanied by either a system message (preferable) or a piece of content that contains the description.

The content channel requires no explanation.

The non-free channel is necessary to transmit content that cannot be processed and cleaned up before broadcast (due to copyright issues). Content from this channel may be lightly touched up by Librarian itself (e.g., strip external and internal links except links to fragment identifiers), stripping of stylesheets, etc.

The big files channel is used for distributing large files which cannot be held in memory and must be streamed straight from disk (e.g., DVD iso, larger software packages, etc). Files in this channel can be directly accessed by URL, but should, generally be accompanied by matching content in the content channel, which contains an HTML file with the link to locally stored big file, copyright and license information, and so on.

UI for importing downloaded files

The files should be downloaded to /var/spool/outernet/. For now, we will simply simulate the 'download' by manually fetching zip files on the artexin zip file page, and placing them into the folder. We assume that the actual code that fetches the files from broadcasts will do the same.

The metadata files (info.json) should be extracted from the zipballs, and added to the database of available files. UI should provide a way to review all 'available' items and either add them to the local archive, or discard. Either way, the entry is removed from 'available' table.

In case user chose to add the item to archive, the zipball should be extracted into /srv/outernet/. In case user chose to discard the item, it should be moved to /tmp/deleted/. For now, we won't bother with undelete feature.

Better icons

The current icons are truly ugly. We could use better icons.