shemanaev/dotopds

Lightweight .NET OPDS server

DotOPDS: the lightweight OPDS server

DotOPDS is an OPDS server designed for large libraries like full archive of lib.rus.ec or Flibusta.

Features

• Full-text search through Lucene.net
• Plugins support
• Support for external converters
• OPDS catalog localization
• Cover and annotation extraction (experimental)
• Web interface

Getting started

Download latest release and extract somewhere. Copy appsettings.json to appsettings.Production.json and customize it or use any other method to configure ASP.NET Core Application

Settings description:

{
  "Presentation": {
    "DefaultLanguage": "en", // language used to index texts
    "PageSize": 10, // how many items per page to show
    "Title": "DotOPDS Library", // OPDS title
    "LazyInfoExtract": false, // try to extract cover and annotation from book before displaying
    "Converters": [
      {
        "From": "fb2", // file extension
        "To": "epub", // file extension
        "Command": "fb2epub.exe", // path to converter
        "Arguments": "{0} {1}" // {0} - from, {1} - to
      }
    ],
  },
  "IndexStorage": {
    "Path": "./database" // path to database storage
  }
}

Import library index from .inpx file:

DotOPDS import inpx C:\library C:\lib.inpx

To see available import plugins type:

DotOPDS import help
DotOPDS import help inpx # plugin help

Now just start server:

DotOPDS.Server

and OPDS will be available at http://localhost:5000/opds

You always can use help command to get more info.

Docker

Build the image

docker build . -t dotopds

Run the image

docker run -it -p 5000:80 -v databasePath:/app/database dotopds

Manage data in docker

Find container id with docker ps and get into it

docker exec -it CONTAINER_ID /bin/bash

Included plugins

• BookProvider.Inpx - import .inpx files
• FileFormat.Fb2 - extract annotation and cover from fb2 books

Preparing release

• Make tag with version number
• Wait for ci build completed and edit draft description
• Publish

dotopds's Issues

Store library paths mapping near index

It's stored in settings, that's just wrong.

Add support for plugins (import and parsers especially)

It's hard to add custom formats support right now. Consider adding plugin system for importers and parsers.

Paging doesn't work in search results

1. Connect with a eBook reader to the DotOPDS server (tested on Moon Reader Pro on Android).
2. Search for a term (e.g. an author) to get more that one page of results.
3. Try to go to next page by tapping ">" or by tapping the pages number and specifying a page number.
4. Result list refreshes, but contains the same list.

This behavior doesn't depend on a pagination specified in the config file (but of course you'll less likely get more that one page with larger pagination setting). Another interesting thing: when you try to get to specific page, it shows a number equal to actual page number - 1 (e.g. when you are on the first page it shows "0"), and if you enter e.g. "3" it tries to go to 2-nd page (but displays the same list as on 1-st page).

Search results aren't sorted

Use a reader (tested on Moon Reader Pro).

1. Search for a term, or browse via Genres.
2. Search results aren't sorted, it's a bit difficult to find a book you need (e.g. you've searched for an author and then tried to find specific book by that author).

Error during import

Bad input file with latest lib.rus.ec dump. MyHomeLib with the same file works fine.

cmd: DotOPDS.exe import D:\library librusec_local_fb2.inpx

Any ideas?

DotOPDS v1.0.8-7b980a8-master, win 8.1 x64.

How to add an Importer for pdf files?

I would like to add to your implementation an Importer for pdf files. It would get meta data from the PDF file itself. I envisage using multiple source directories (I don't want to move the files from where they are located.) and recurse into them as deep as necessary.

Do I need to only implement something like LuceneImporter or do I need to change code elsewhere to allow choosing an Importer? E.g. in ImportTask.cs the importer is hardcoded to Lucene.

Fix the hardcoded `RussianAnalyzer`

Use some language detection library (ntextcat) or directly specify language code in import command.

PDF Importer updated

https://github.com/gerritv/DotOPDS/releases/tag/V0.1.0.1

This provides a general purpose document library tool for pdf files. Each file has a Cover (the file thumbnail) included in the index.

Cleanup `Book` model

There are a lot of unnescessary fields that really doesn't matter for almost anyone.
Maybe there should be somthing like 'meta' fields. Some indexed key-value fields under meta. namespace, so it'll be searchable with query like meta.language:ru.
It can be placed under custom namespace tag in OPDS model (like series field in FeedEntry).
Fields like Size, LibId, Del, Language, Keywords can be placed there.

Fresh release

Any chance to make a fresh GitHub release as well as publish it on Docker Hub?

Enhance WebController to support a web site

I need/want to add some web pages to the server. This will lead to using that web pages to show statistics of my libraries and to also Update/Remove etc. books and documents so I don't have to use Command window. To that end I added an empty index.html to the project. And then enhanced the webcontroller to support a general web site. If you issue http://:8080 it will get index.html (as default page)

public class WebController : ApiController

{
    /// <summary>
    /// Handles web site if present, otherwise redirects to /opds
    /// </summary>
    /// <param name="filename"></param>
    /// <returns>Http Response message</returns>
    //[Route("{*filename}")]
    [HttpGet]
    public HttpResponseMessage ServeIndex(string filename)
    {
        var file = GetFile("index.html");
        if (filename == null)
        {
            if (file == null)
            {
                file = Request.CreateResponse(HttpStatusCode.Moved);
                file.Headers.Location = new Uri("/opds", UriKind.Relative);
            }
            else
            {
                return file;
            }
        }
        else
        { 
            if ((file = GetFile(filename)) == null)
            {
                file = Request.CreateResponse(HttpStatusCode.NotFound);
            }
        }
        return file;
    }

Question about Category and Genre

I am trying to fit my needs in with the existing structures to avoid making too many code changes.
My document trees are of the form e.g.:

\CircularStore\eBooks\Cooking\Chinese\Dim Sum\ or
\CircularStore\eBooks\Machining\Tools\Tool to Make\

I am thinking that if I add a Root parameter to my pdf importer I could end up with a Genre list using \CircularStore\eBooks as Root. The Genre list could then be parsed from the remaining directory structure. The top level would have category_ prepended

category_Cooking
Chinese
Dim Sum

Keep in mind that I will be extracting (already have this working) the list of Genre actually used from the Index. I have the top level working, I use IndexReader to get all the Genre and pull out the category_ ones for the top level. I have no reasonable other way to infer Genre/Category from the Pdf files.

Is that along the lines of the two or more level Genre you have implemented?

Pdf tree import plugin added to https://github.com/GerritV/DotOPDS

FYI, I updated my repository with the working pdf tree importer. This uses iTextSharp to retrieve Info from pdf file.
It expects a Genre on command line for the top level. Each directory in the path to pdf file is added to the Genre list. Keywords, if any in the pdf, are indexed but not stored.

Delete + Add document on import

Right now you forced to delete imported library to avoid duplicates on re-import. That's definetly not the way to go.
Records should be updated on unique index (LibraryId + File + Ext + Archive) instead.

After reaching last page in a search result, paging no longer available

Use a reader (tested on Moon Reader Pro on Android).

1. Search for some term (or alternatively browse via genres).
2. Browse to the last page by entering its number (or alternatively go the last page but one, i.e. 108 of 109, and then tap ">").
3. At this point, paging is no longer available, page number is displayed as 1, despite it seems you are actually on the last page instead.

Please see attached screenshots for details (1 - the last page but one, 2 - the "last" page).

Search term is incorrect?

Using OPDSBrowser on Android, I am entering a Search for 'Harold', the author of one of my pdf files. This finds no result. The reason is that the search term 'harold' doesn't get passed. In fact the Request looks badly formatted. (I would try with Aldiko but there the Search button doesn't appear :-( ) I am guessing that {searchTerm} should be part of the initial page sent to the client?

The command screen for DotOPDS shows:

6362165145178012834 - Response: GET http://192.168.0.117:8080/opds
Content-Type: application/atom+xml; charset=utf-8
<feed xmlns:os="http://a9.com/-/spec/opensearch/1.1/" xmlns:opds="http://opds-spec.org/2010/catalog" xmlns:dc="http://purl.org/dc/terms/" xmlns:dotopds="urn:dotopds:v1.0" xmlns="http://www.w3.org/2005/Atom">
  <updated>2017-02-02T21:57:31</updated>
  <id>tag:root:root</id>
  <title>DotOPDS Library</title>
  <link rel="search" type="application/atom+xml" href="/opds/search?q={searchTerms}" />
  <link rel="start" type="application/atom+xml;profile=opds-catalog;kind=navigation" href="/opds" />
  <link rel="self" type="application/atom+xml;profile=opds-catalog;kind=navigation" href="/opds" />
  <entry>
    <updated>2017-02-02T21:57:31</updated>
    <id>tag:root:genres</id>
    <title>Books by genres</title>
    <dc:issued>0</dc:issued>
    <content type="text">Browse books by genres</content>
    <link type="application/atom+xml;profile=opds-catalog;kind=acquisition" href="/opds/genres" />
  </entry>
</feed>
6362165150697394674 - Request: GET http://192.168.0.117:8080/opds/search?q={searchTerms}&q=harold
User-Agent: Dalvik/1.6.0 (Linux; U; Android 4.4.2; SM-T310 Build/KOT49H)
Host: 192.168.0.117:8080
Connection: Keep-Alive
Accept-Encoding: gzip

Breakpointing in SearchController.cs in Feed Search I see q is {searchTerms}, not harold. The query of course finds nothing.