lodestonehq/lodestone

Personal Document Archiving (DMS, EDMS for Personal/Home Office use)

lodestone's Issues

Add a status page

Status page should include

• the status of the Elastic Search Watch folder collector.
• status of thumbnail generator
• status of email collector
• status of elasticsearch api.

Publisher FSNotify events missing on Mac via Docker

fsnotify/fsnotify#306

Support for XML-files

I have som XML-files that i want to organize in a DMS. Not parsing the XML-data, but just a place to save xml and index the name for fast searching of file name.

Thanks! Good Luck!

-Viktor

Add ability to rename document title

Add logo

• https://thenounproject.com/term/rock/481051
• https://thenounproject.com/term/rock/1092727
• https://thenounproject.com/term/crystal/570596
• https://thenounproject.com/term/crystal/570601
• https://thenounproject.com/term/quartz-crystal/570585

Rock
Mineral
Geode
Crystal
Mine
Gem

Find similar documents to the Details page.

Zoomable images

https://www.npmjs.com/package/ngx-image-zoom

highlighted search results should have an expand/collapse button to hide more than 2 entries.

Generate Thumbnails for all files

Add popover with help text for query bar.

update screenshot in README.md

UI is much more polished

• thumbnails
• slider units
• structured tags
• bookmarking

update folder structure to break up each component into its own directory

Root directory should be simple, and only contain docker-compose basicaly.

embed the processor version in the elasticsearch document metadata

we might want to be able to re-process all documents that were created with an old version.

Add watchdog that verifies all files on the filesystem have associated entries in the DB.

Should run every x hour(s)

Export/Backup system or FUSE filesystem for accessing files in a tagged folder structure.

Multiple types of backup/export

• FUSE filesystem
• Cloud storage export
  • https://github.com/lytics/cloudstorage
  • https://github.com/google/go-cloud
  • https://github.com/graymeta/stow
  • https://github.com/ncw/rclone
  • https://gocloud.dev/howto/blob/
  • https://github.com/Xuanwo/storage
  • https://github.com/spf13/afero

Add a logs page for messages from collectors

Publisher should have a crawler function to ensure that all documents in FS have associated DB entry.

show the correct number of available documents in filetype aggregation.

Storage publisher disconnects from Rabbitmq after computer put to sleep

storage_1        | 2020/01/11 15:53:32 error: Exception (504) Reason: "channel/connection is not open"
storage_1        | 2020/01/11 15:53:32 event: "/data/documents/filetypes/sample.txt": CHMOD
storage_1        | 2020/01/11 15:53:32 Ignoring event:  "/data/documents/filetypes/sample.txt": CHMOD
storage_1        | 2020/01/11 15:53:32 event: "/data/documents/filetypes/sample.rtf": CREATE
storage_1        | 2020/01/11 15:53:32 error: Exception (504) Reason: "channel/connection is not open"
storage_1        | 2020/01/11 15:53:32 event: "/data/documents/filetypes/sample.rtf": CHMOD
storage_1        | Publishing event..```

Add Viewer for non PDF documents

• Spreadsheets: https://sheetjs.com/demos
• Docx: https://github.com/mwilliamson/mammoth.js?files=1
• Images: inside an Iframe.

paid

• https://www.kukudocs.com/jsViewer
• https://www.pdftron.com/office-sdk/office-document-viewer/

Generate API with GraphQL or API.

Add support for auto-tagging using AI and customizable keywords

https://www.reddit.com/r/selfhosted/comments/cvlbng/lodestone_a_personal_digital_file_cabinetedms/ey50psz/

processor/publisher should ignore files that start with `.` or have 0 bytes

this is so we dont get weird issues when deleting files on network shared .smbdeleteAAA2a6f4.4\

• ignore all files with size 0
• ignore all files that end with '/'
• ignore all files that start with .

Basic installation how to

Hi Jason,
I like your project ans i try to install on my macbook pro.
Ok i have understand to use docker in folder project and run : docker-compose up
It create the container and run ok. But i haven't any file. I download your sample folder AnalogJ/lodestone-test-docs and put all in folder ./docs but i have no file when i run localhost:4000
And another thing, where i try to make a search, i write my search word but i can't click enter.
But when I click on enter nothing happens.
Have you more information for all that, because i like to try completely the project.
Have a nice day.
Best Regards
Robert

Footer links and content should be fixed.

Fix File Size Label - should be humanized text.

https://github.com/angular-slider/ng5-slider/blob/master/STYLING.md

Limit fields that are returned by Search

in search page we dont care about most data returned by ElasticSearch. Makes results more complicated and heavyweight.

Lowercase (to make consistent) all file type/extensions in processor before storing.

Add bookmark field to the documents.

Update the Details page to fix header

Extract title information from content embedded in images

• https://grobid.readthedocs.io/en/latest/Introduction/
• https://cwiki.apache.org/confluence/display/tika/GrobidJournalParser
• http://apache-tika-users.1629097.n2.nabble.com/Analysing-a-document-sections-with-Apache-Tika-td7574322.html
• https://github.com/miso-belica/sumy
• https://github.com/jdkato/prose
• https://medium.com/@vasista/extract-title-from-the-image-documents-in-python-application-of-rlsa-58f91237901f

reverse proxy fscrawler missing port

Hi,

Thanks for sharing this project. I have been looking for something like this.

When I spin up on Ubuntu 18.04 I see constant

reverse-proxy_1  | time="2019-11-02T02:33:48Z" level=error msg="port is missing" providerName=docker container=fscrawler-lodestone-d8525a5f5cdc3439c626c7757935b2f7b1aadff8c2a05cd5f04a07353031f627

And can't search for anything in the /web UI. The status dashboard suggests it has seen the documents I added to the watchfolder.

I tried removing --api.insecure=true for Traefik in the docker-compose.yml but it didn't solve anything.

Please let me know if I can assist with further logs.

Thanks.

publisher/processor should handle renaming files.

Add view for empty document list.

Add a Sort By Direction dropdown

Import metadata from MayanEDMS/Paperless?

Needs a full API endpoint.

https://www.reddit.com/r/selfhosted/comments/cvlbng/lodestone_a_personal_digital_file_cabinetedms/ey5j8g5/

fix the minio event notifications (for web UI uploads)

minio no longer supports notifications for NAS Gateway mode.

Pick a reverse proxy to act as the frontdoor to the application components

Choices?

• Nginx
• Traefik

Should be simple to maintain, and lightweight as possible.

Account management system.

Currently anyone with the url can login to Lodestone.

add support for runtime configuration options via JSON

Default API url needs to be customizable.
We should also be able to preload/configure the built in tags.
https://juristr.com/blog/2018/01/ng-app-runtime-config/

Add support for ebook file formats

https://www.reddit.com/r/selfhosted/comments/cvlbng/lodestone_a_personal_digital_file_cabinetedms/ey5ixzu/

Add support for highlighted query snippets.

Update the lode document type to include tags.

use the https://fscrawler.readthedocs.io/en/latest/admin/fs/elasticsearch.html#creating-your-own-mapping-analyzers

Processor should correctly delete entries from DB.

Additional repositories

lodestone-sync

• ability to sync files to dropbox/gdrive/fuse filesystem.

lodestone-watchdog

• run a watchdog command on a cron, verifying that all files in the FS are present in ES.

Add File size slider

Create a document list of various documents to be used by crawler.

Support Gif and other image types.

Add tabular view option to dashboard search results

On the main dashboard, it would be nice to have the option to view the search results in a tabular format instead of the default card view. This will help more easily deal with large numbers of results as our document libraries go.

We can add a control to the result header to switch between tabular and card view.

Settings persistenance in Lodestone

https://github.com/etcd-io/bbolt