h3rald/litestore

A lightweight, self-contained, RESTful, searchable, multi-format NoSQL document store.

LiteStore is a lightweight, self-contained, RESTful, multi-format NoSQL document store server written in Nim and powered by a SQLite backend for storage. It aims to be a very simple and lightweight backend ideal for prototyping and testing REST APIs and single-page applications.

For more information, see:

• Getting Started
• Usage

Or read the full LiteStore User Guide.

litestore's Issues

Pack/unpack directories

Implement methods to:

• recursively "pack" the contents of a directory into a data store
• "unpack" a previously-packed directory (removing also the corresponding contents from the datastore)

Serve directory contents

By default, when running LiteStore with -d option specified, the directory contents should be served by LiteStore web server.

Implement common logger

Implement a simple (and thread-safe) logger.

App: No feedback is given to user in case of request errors

Make sure that all requests provide a feedback to the user in case of errors (e.g. via a flash message).

Regressions: sorting and retrieving single documents

As of the latest minor version:

• it is no longer possible to sort by id, created, modified.
• an error is returned when retrieving a single element, in some cases

Minor mistakes in dir documentation

There are some mistakes/typos in titles when copying and pasting from info page.

--address, --port, and --store don't work

Some command line options have no effect.

App: Display image documents

If a document is an image, display it instead of showing its base64-encoded contents.

App: Upload files

Allow file uploading by browsing local filesystem and/or drag & drop.

Cannot view/edit .json file containing dots in filename

Document tagging

Implement tagging/untagging operatins

Automatically remove special character sequences from full text index

**, --, __, etc.

Datastore management

Implement methods to manage datastores.

CORS doesn't work for API v4

• Access-Control-Allow-Origin is not set correctly.
• Access-Control-Allow-Headers does not allow authorization.

App: Search

Implement support for searching text files within the app.

Document search

Implement fulltext search query methods for documents

App: Markdown enhancements

Override some of marked rendering methods to support Discount class blocks and pseudo-protocols.

Linux x64 build

Error when searching within folder

Search does not work if a folder is specified (ambiguous column: id).

Great Job

Man, what a gem of a project.

Thank you for open sourcing this.

😻

Upgrade to SQLite v3.22.0

"Index out of bounds" error returned when removing all user tags (2 or more)

Steps to reproduce the bug:

1. Add two tags to a document
2. Remove both tags

A 400 - Bad Request "Index out of bounds" error is returned.

mirror operation

Enable LiteStore to "mount" a local directory and keep it mirrored both on the database and the filesystem.

Open default document in browser

• Add default_page column to info table
• Increment db Schema version
• Add --default:<doc_id> option to set/unset default page
• Add logic to open default document in browser at startup if default is set

Document management

Implement document management (simple key/value store and simple queries)

Refactor Mithril components

Unable to update content type and system tags when updating document

Unable to modify or cancel file upload

At present, it is not possible to edit the document ID in the upload dialog after it is set for the first time, and it is not possible to "cancel" an upload (clearing all fields of the form).

Support multiple order clauses

?sort=-modified~created

Search enhancements

• Sort by relevance
• Retrieve highligting

App: Detect editor language by document ID when creating document

When creating a new document, auto-set the language of the editor based on the document ID.

Use with Nim web-framework?

Is it possible to use Litestore as a prototype-DB for a Nim web-framework?

I am currently tinkering with Jester and I would like to add/edit/delete JSON (via the Jester-backend).

So instead of having Litestore be a client-facing datastore (as the documentation shows), I would instead like to use litestore in this manner:

Client <-> Jester-REST-API <-> litestore-DB

Is this possible to achieve?

If yes, are you able to share a small example in the readme file? It may help a lot of people that want to prototype small webapps with Jester+Litestore

Thanks!

Add info table

• Keep track of datastore version
• Cache document count

-s:<db> does not work on windows

The executable defaults to data.db no matter what the -s or --store value is
Is the same for both the x86 and x64 builds.
Version 1.0.1

Test failing

/Users/h3rald/Development/litestore/test/http_api.nim(194, 37): Check failed: json["results"][3]["data"] == testdata
   json["results"][3]["data"] was {"age":null,"email":null}
   testdata was {"age":36,"email":"[email protected]"}
   /Users/h3rald/Development/litestore/test/http_api.nim(200, 23): Check failed: json["data"] == testdata
   json["data"] was {"age":null}
   testdata was {"age":31}

Admin App is not fully responsive

• Tag lists do not wrap.
• Insufficient margin for content.

Whitespace Cleanup

Was wondering if you'd be ok with a PR that cleaned up whitespace in *.nim?

Thanks!

Command-line interface

Implement a CLI for LiteStore

App: Improve datastore information layout

Display data in the Info page in a more user-friendly way.

Document queries with time ranges on created and modified

?modified-before=<timestamp>
?modified-after=<timestamp>
?created-before=<timestamp>
?created-after=<timestamp>

Performance Improvements

Apply the following changes to improve performance for large (> 1GB) SQLite database.

Modify compilation flags

   {.passC: "-DSQLITE_ENABLE_FTS4=1 -DSQLITE_DEFAULT_LOCKING_MODE=1".}

Configure PRAGMAs when opening the database

   discard result.db.tryExec("PRAGMA locking_mode = exclusive;".sql)
   discard result.db.tryExec("PRAGMA journal_mode = WAL;".sql)
   discard result.db.tryExec("PRAGMA page_size = 4096;".sql)
   discard result.db.tryExec("PRAGMA cache_size = 10000;".sql)
   discard result.db.tryExec("PRAGMA synchronous = NORMAL;".sql)

Execute commands on data store via command line

• vacuum
• optimize/rebuild

Import optimizations

1. DROP INDEX for docid
2. Import data
3. CREATE INDEX for docid
4. OPTIMIZE
5. VACUUM

Additional indexes

     CREATE INDEX tags_document_id ON tags(document_id);
     CREATE INDEX tags_tag_id ON tags(tag_id);
     CREATE INDEX documents_id ON documents(id);

Implement basic CORS support

Access-Control-Allow-Origin: *

Built-in web server and REST API

Implement a built-in web server for LiteStore and expose all the most common operations via a REST API.

Improve tag pages

• Add paginator
• Use panels instead of tables

Documentation

Provide complete documentation, both as standalone document and integrated in the app.

App: Add/remove tags from documents

Provide the possibility to add/remove non-system tags from a document.

Total documents not updated and searchdata not deleted by delete operation

Running litestore -d:dir delete deletes documents and tags but not searchdata contents. Additionally, the total documents count is not updated.

Document new tags resource and update admin app menu with new docs

/tags/?like=$format*

Invalid nimble file

Move all custom fields out of the nimble file.

Error: unhandled exception: Invalid field: appame [ENimble]

Machine Tags

LiteStore should support Flickr-style machine tags.

Examples:

  http://localhost:9500/docs?sys.type=text;*.subtype=*;
  http://localhost:9500/docs?sys.format=binary;myapp.something=%22something%20else%22