bevuta/pepa

A document management system

Pepa DMS

What is Pepa?

Pepa is a document management system primarily designed for archiving and retrieving your digital documents. It's implemented in Clojure and ClojureScript. Pepa can be used from within your browser and - this goes without saying - you can view all your documents there.

How does Pepa work?

Any document management system (any archive whatsoever really) would be pretty useless if it would only store documents. To be more than just a black hole for documents, a DMS should aid you in organizing the documents. You should be able to not only store your documents, but to search for them and - even more importantly - to find them.

Pepa uses a tagging system inspired by Notmuch to achieve this. A tag is a text label that the user adds to a document. This could be anything from invoice to university or work or doctor or contracts or papers or ... Of course you can add more than one tag to a document. A document tagged with invoice can additionally be tagged with unpaid or paid for example.

Those tags can be used to find a document. You want to find every document that's an unpaid invoice? Easy, let Pepa search for documents which are tagged with invoice and unpaid - once you have paid those invoice, you can remove the unpaid tag and add the paid tag instead.

Of course, Pepa allows not only searching by tag. Every document in Pepa is available for full-text search. You can combine this search with the tag search to increase your chances of finding just the right document.

Current features

• Supported file formats
  • PDF
• Import channels
  • SMTP / email (attachments are extracted)
  • HTTP (API and upload via web interface)
  • Virtual Network Printer (LPD)
• Manual tagging
• Full-text search
• OCR (via Tesseract and/or CuneiForm)
• Interactive document dissection

Planned features

• User management and groups/organisations
• Rule-based (automatic) tagging
• Self-learning tagging
• Additional document types
  • JPG, PNG, TIFF
  • Plain text
  • HTML
  • Email

Features not planned

Since we want to keep our focus, the following features are not planned to become a part of Pepa:

• Support for proprietary and/or very complex file formats. notably Microsoft Word, Excel, and Powerpoint. Neither are we planning to implement support for documents from LibreOffice or OpenOffice. Those formats are not well suited for archiving and good support would be too much effort.

Since Pepa is open source, you are of course free to implement those features yourself.

Project status

Pepa is currently in an early development stage which means that it's not ready for production use, yet, as there will be frequent breaking changes still. You are very welcome to give it a try, though, and we are happy to receive feedback either via GitHub issues or by email via [email protected]. Note that since there is no official release, yet, you will have to build Pepa from sources to run it (see build instructions below).

Screenshots

Installation

See installation.md in the doc/ directory.

Icons

Icons found under the resources/public/img/material/ folder are based on the "Material Design Icons" by Google and are released under an Attribution 4.0 International license.

License

Copyright © 2015 bevuta IT GmbH

This program is free software: you can redistribute it and/or modify it under the terms of the GNU Affero General Public License as published by the Free Software Foundation, either version 3 of the License, or (at your option) any later version.

This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU Affero General Public License for more details.

You should have received a copy of the GNU Affero General Public License along with this program, see LICENSE. If not, see https://gnu.org/licenses/agpl.html.

In addition, as a special exception, the copyright holders of the software give you permission to link its code with the following libraries and distribute linked combinations including them:

• ch.qos.logback/logback-classic
• ch.qos.logback/logback-core
• cider/cider-nrepl
• cljs-tooling
• clout
• com.cemerick/piggieback
• com.facebook/react
• com.stuartsierra/dependency
• compliment
• compojure
• crypto-equality
• crypto-random
• garden
• instaparse
• io.clojure/liberator-transit
• liberator
• medley
• om
• org.clojure/clojure
• org.clojure/clojurescript
• org.clojure/core.async
• org.clojure/core.cache
• org.clojure/core.match
• org.clojure/core.memoize
• org.clojure/data.csv
• org.clojure/data.json
• org.clojure/data.priority-map
• org.clojure/java.classpath
• org.clojure/java.jdbc
• org.clojure/test.check
• org.clojure/tools.analyzer
• org.clojure/tools.analyzer.jvm
• org.clojure/tools.macro
• org.clojure/tools.namespace
• org.clojure/tools.nrepl
• org.clojure/tools.reader
• org.clojure/tools.trace
• org.tcrawley/dynapath
• prismatic/plumbing
• prismatic/schema
• ring-transit
• sablono
• secretary

If you modify this file, you may extend this exception to your version of the file, but you are not obligated to do so. If you do not wish to do so, delete this exception statement from your version.

pepa's Issues

Dockerized version

Hey guys,

Would you consider adding a docker image version of Pepa (could be in a separate repository if you don't believe it's within scope). Alternatively, I can pull on this if you are open to the idea, but don't have the time to setup.

Feature Request

bibtex support

Hi,

Despite all the work that people have done on this problem, I'm still not satisfied with document management for people who also write papers with bibliographies. Is bibtex support in your plans? Does it fit with the goals of pepa?

Thanks,
Gideon.

[Bug] Download of PDF Files concats ".pdf" in any case

".pdf" is appended to the filename of a download (see: https://github.com/bevuta/pepa/blob/master/src/pepa/web/handlers.clj#L124)

a.pdf -> a.pdf.pdf

Not working with Java 7

The installation guide lists Java 7 as the minimum requirement, but on a Debian 8 (jessie) system, lein uberjar fails with

Exception in thread "main" java.lang.ExceptionInInitializerError
        at clojure.main.<clinit>(main.java:20)
Caused by: java.lang.UnsupportedClassVersionError: org/postgresql/util/PGobject : Unsupported major.minor version 52.0, compiling:(pepa/db.clj:1:1)
        at clojure.lang.Compiler.load(Compiler.java:7441)

So it seems the PostgreSQL driver needs Java 8, which require Backports on a stable Debian system...

Stacktrace

Hey guys,

wanted to try out pepa.

I believe I installed all the required prerequisites (basically by copy-pasting your apt-get install line from the Readme), still I'm getting this stacktrace when I upload a document. The document is not visible in the UI after this.

Any ideas? Must there something (Poppler?) be configured or anything?

I'm using Google Chrome on a Linux Mint system, Leiningen 2.5.1 on Java 1.8.0_66 Java HotSpot(TM) 64-Bit Server VM.

22:57:56.215 INFO  [FilePageExtractor] Start processing file 3 (10498 bytes, origin: web)
22:57:56.289 INFO  [FilePageExtractor] Processing page 0
22:57:56.353 ERROR [FilePageExtractor] Exception in `process-item'.
java.io.IOException: Stream closed
    at java.io.BufferedInputStream.getBufIfOpen(BufferedInputStream.java:170) ~[na:1.8.0_66]
    at java.io.BufferedInputStream.read(BufferedInputStream.java:336) ~[na:1.8.0_66]
    at sun.nio.cs.StreamDecoder.readBytes(StreamDecoder.java:284) ~[na:1.8.0_66]
    at sun.nio.cs.StreamDecoder.implRead(StreamDecoder.java:326) ~[na:1.8.0_66]
    at sun.nio.cs.StreamDecoder.read(StreamDecoder.java:178) ~[na:1.8.0_66]
    at java.io.InputStreamReader.read(InputStreamReader.java:184) ~[na:1.8.0_66]
    at java.io.BufferedReader.fill(BufferedReader.java:161) ~[na:1.8.0_66]
    at java.io.BufferedReader.read(BufferedReader.java:182) ~[na:1.8.0_66]
    at clojure.core$slurp.doInvoke(core.clj:6650) ~[pepa-0.1.0-SNAPSHOT-standalone.jar:na]
    at clojure.lang.RestFn.invoke(RestFn.java:410) [pepa-0.1.0-SNAPSHOT-standalone.jar:na]
    at pepa.pdf.PopplerReader.extract_page_text(pdf.clj:75) ~[na:na]
    at pepa.processor.file_page_extractor$extract_pages.invoke(file_page_extractor.clj:15) ~[na:na]
    at pepa.processor.file_page_extractor.FilePageExtractor$fn__18280$fn__18281.invoke(file_page_extractor.clj:34) ~[na:na]
    at pepa.processor.file_page_extractor.FilePageExtractor$fn__18280.invoke(file_page_extractor.clj:33) [na:na]
    at clojure.java.jdbc$db_transaction_STAR_.doInvoke(jdbc.clj:595) [na:na]
    at clojure.lang.RestFn.invoke(RestFn.java:521) [pepa-0.1.0-SNAPSHOT-standalone.jar:na]
    at clojure.java.jdbc$db_transaction_STAR_.doInvoke(jdbc.clj:611) [na:na]
    at clojure.lang.RestFn.invoke(RestFn.java:425) [pepa-0.1.0-SNAPSHOT-standalone.jar:na]
    at pepa.processor.file_page_extractor.FilePageExtractor.process_item(file_page_extractor.clj:32) [na:na]
    at pepa.processor$process_next$fn__18242.invoke(processor.clj:22) [na:na]
    at pepa.processor$process_next.invoke(processor.clj:21) [na:na]
    at pepa.processor$start$fn__18246.invoke(processor.clj:34) [na:na]
    at clojure.core.async$thread_call$fn__5893.invoke(async.clj:405) [na:na]
    at clojure.lang.AFn.run(AFn.java:22) [pepa-0.1.0-SNAPSHOT-standalone.jar:na]
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [na:1.8.0_66]
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [na:1.8.0_66]
    at java.lang.Thread.run(Thread.java:745) [na:1.8.0_66]
java.io.IOException: Stream closed
    at java.io.BufferedInputStream.getBufIfOpen(BufferedInputStream.java:170)
    at java.io.BufferedInputStream.read(BufferedInputStream.java:336)
    at sun.nio.cs.StreamDecoder.readBytes(StreamDecoder.java:284)
    at sun.nio.cs.StreamDecoder.implRead(StreamDecoder.java:326)
    at sun.nio.cs.StreamDecoder.read(StreamDecoder.java:178)
    at java.io.InputStreamReader.read(InputStreamReader.java:184)
    at java.io.BufferedReader.fill(BufferedReader.java:161)
    at java.io.BufferedReader.read(BufferedReader.java:182)
    at clojure.core$slurp.doInvoke(core.clj:6650)
    at clojure.lang.RestFn.invoke(RestFn.java:410)
    at pepa.pdf.PopplerReader.extract_page_text(pdf.clj:75)
    at pepa.processor.file_page_extractor$extract_pages.invoke(file_page_extractor.clj:15)
    at pepa.processor.file_page_extractor.FilePageExtractor$fn__18280$fn__18281.invoke(file_page_extractor.clj:34)
    at pepa.processor.file_page_extractor.FilePageExtractor$fn__18280.invoke(file_page_extractor.clj:33)
    at clojure.java.jdbc$db_transaction_STAR_.doInvoke(jdbc.clj:595)
    at clojure.lang.RestFn.invoke(RestFn.java:521)
    at clojure.java.jdbc$db_transaction_STAR_.doInvoke(jdbc.clj:611)
    at clojure.lang.RestFn.invoke(RestFn.java:425)
    at pepa.processor.file_page_extractor.FilePageExtractor.process_item(file_page_extractor.clj:32)
    at pepa.processor$process_next$fn__18242.invoke(processor.clj:22)
    at pepa.processor$process_next.invoke(processor.clj:21)
    at pepa.processor$start$fn__18246.invoke(processor.clj:34)
    at clojure.core.async$thread_call$fn__5893.invoke(async.clj:405)
    at clojure.lang.AFn.run(AFn.java:22)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    at java.lang.Thread.run(Thread.java:745)
22:57:56.368 WARN  [FilePageExtractor] Failed to render file 3
22:57:56.394 INFO  [FilePageExtractor] Finished processing of file 3

Can arbitrary network locations be mapped for import?

I'm interested in using pepa, but this feature would be necessary. Specifically, I'm looking for a tagging/search interface that will watch folders, leave the files where they are, and allow tagging and searching. Am I right to be looking at pepa, or should I look elsewhere. Thanks.

Problem with pdfunite

Hey guys,

I can now upload and view documents, but when I try to download them, I'm getting this stacktrace:

clojure.lang.ExceptionInfo: pdfunit didn't terminate correctly {:exit-code 99, :args ("/tmp/pdfseparate-2586469553236351662.pdf" "/tmp/pdfunite-123464094108447981.pdf"), :pages (#object[java.io.File 0x61dc0674 "/tmp/pdfseparate-2586469553236351662.pdf"])}
    at clojure.core$ex_info.invoke(core.clj:4591)
    at pepa.pdf$merge_pages.invoke(pdf.clj:177)
    at pepa.model$document_pdf.invoke(model.clj:480)
    at pepa.web.handlers$document_download$fn__15620$fn__15623.invoke(handlers.clj:219)
    at liberator.core$decide.invoke(core.clj:80)
[...]

This is what I see in the repl:

21:13:37.990 INFO  [Database] Generating PDF for 5
pdfunite version 0.24.5
Copyright 2005-2013 The Poppler Developers - http://poppler.freedesktop.org
Copyright 1996-2011 Glyph & Cog, LLC
Usage: pdfunite [options] <PDF-sourcefile-1>..<PDF-sourcefile-n> <PDF-destfile>
  -v             : print copyright and version info
  -h             : print usage information
  -help          : print usage information
  --help         : print usage information
  -?             : print usage information

Feels like the process is not correctly invoked. I hope the pdfunite version I have is recent enough?

And why is it actually merging pdfs? I thought the file I uploaded would be stored as it is, and then downloaded as it is?

java.lang.Error: java.sql.SQLException: An attempt by a client to checkout a Connection has timed out.

[root@izwz90ajic0vbw7o4pj6xyz pepa2]# java -jar pepa-0.1.0-SNAPSHOT-standalone.jar

15:46:06.747 INFO [pepa.bus] Starting Bus
15:46:06.810 INFO [pepa.db] Starting database
15:46:07.128 INFO [pepa.web] Starting HTTP Server
15:46:07.128 INFO [pepa.web] Started web server on http://localhost:8070/
15:46:07.204 INFO [org.xnio] XNIO version 3.3.6.Final
15:46:07.227 INFO [org.xnio.nio] XNIO NIO Implementation Version 3.3.6.Final
15:46:07.420 INFO [org.projectodd.wunderboss.web.Web] Registered web context /
15:46:07.423 INFO [pepa.processor.page-ocr] Starting page OCR processor
15:46:07.457 INFO [pepa.processor.file-page-extractor] Starting file page extractor
15:46:07.484 INFO [pepa.processor.page-renderer] Starting page renderer
Exception in thread "async-thread-macro-1" java.lang.Error: java.sql.SQLException: An attempt by a client to checkout a Connection has timed out.
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1155)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.sql.SQLException: An attempt by a client to checkout a Connection has timed out.
at com.mchange.v2.sql.SqlUtils.toSQLException(SqlUtils.java:118)
at com.mchange.v2.sql.SqlUtils.toSQLException(SqlUtils.java:77)
at com.mchange.v2.c3p0.impl.C3P0PooledConnectionPool.checkoutPooledConnection(C3P0PooledConnectionPool.java:690)
at com.mchange.v2.c3p0.impl.AbstractPoolBackedDataSource.getConnection(AbstractPoolBackedDataSource.java:140)
at clojure.java.jdbc$get_connection.invokeStatic(jdbc.clj:298)
at clojure.java.jdbc$get_connection.invoke(jdbc.clj:193)
at clojure.java.jdbc$db_transaction_STAR_.invokeStatic(jdbc.clj:628)
at clojure.java.jdbc$db_transaction_STAR_.invoke(jdbc.clj:585)
at clojure.java.jdbc$db_transaction_STAR_.invokeStatic(jdbc.clj:598)
at clojure.java.jdbc$db_transaction_STAR_.invoke(jdbc.clj:585)
at pepa.processor$next_item_STAR_.invokeStatic(processor.clj:14)
at pepa.processor$next_item_STAR_.invoke(processor.clj:13)
at pepa.processor$process_next.invokeStatic(processor.clj:21)
at pepa.processor$process_next.invoke(processor.clj:19)
at pepa.processor$start$fn__20979.invoke(processor.clj:35)
at clojure.core.async$thread_call$fn__7338.invoke(async.clj:439)
at clojure.lang.AFn.run(AFn.java:22)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
... 2 more
Caused by: com.mchange.v2.resourcepool.TimeoutException: A client timed out while waiting to acquire a resource from com.mchange.v2.resourcepool.BasicResourcePool@703c90bc -- timeout at awaitAvailable()
at com.mchange.v2.resourcepool.BasicResourcePool.awaitAvailable(BasicResourcePool.java:1467)
at com.mchange.v2.resourcepool.BasicResourcePool.prelimCheckoutResource(BasicResourcePool.java:644)
at com.mchange.v2.resourcepool.BasicResourcePool.checkoutResource(BasicResourcePool.java:554)
at com.mchange.v2.c3p0.impl.C3P0PooledConnectionPool.checkoutAndMarkConnectionInUse(C3P0PooledConnectionPool.java:758)
at com.mchange.v2.c3p0.impl.C3P0PooledConnectionPool.checkoutPooledConnection(C3P0PooledConnectionPool.java:685)
... 17 more
Exception in thread "async-thread-macro-3" java.lang.Error: java.sql.SQLException: An attempt by a client to checkout a Connection has timed out.
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1155)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.sql.SQLException: An attempt by a client to checkout a Connection has timed out.
at com.mchange.v2.sql.SqlUtils.toSQLException(SqlUtils.java:118)
at com.mchange.v2.sql.SqlUtils.toSQLException(SqlUtils.java:77)
at com.mchange.v2.c3p0.impl.C3P0PooledConnectionPool.checkoutPooledConnection(C3P0PooledConnectionPool.java:690)
at com.mchange.v2.c3p0.impl.AbstractPoolBackedDataSource.getConnection(AbstractPoolBackedDataSource.java:140)
at clojure.java.jdbc$get_connection.invokeStatic(jdbc.clj:298)
at clojure.java.jdbc$get_connection.invoke(jdbc.clj:193)
at clojure.java.jdbc$db_transaction_STAR_.invokeStatic(jdbc.clj:628)
at clojure.java.jdbc$db_transaction_STAR_.invoke(jdbc.clj:585)
at clojure.java.jdbc$db_transaction_STAR_.invokeStatic(jdbc.clj:598)
at clojure.java.jdbc$db_transaction_STAR_.invoke(jdbc.clj:585)
at pepa.processor$next_item_STAR_.invokeStatic(processor.clj:14)
at pepa.processor$next_item_STAR_.invoke(processor.clj:13)
at pepa.processor$process_next.invokeStatic(processor.clj:21)
at pepa.processor$process_next.invoke(processor.clj:19)
at pepa.processor$start$fn__20979.invoke(processor.clj:35)
at clojure.core.async$thread_call$fn__7338.invoke(async.clj:439)
at clojure.lang.AFn.run(AFn.java:22)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
... 2 more
Caused by: com.mchange.v2.resourcepool.TimeoutException: A client timed out while waiting to acquire a resource from com.mchange.v2.resourcepool.BasicResourcePool@703c90bc -- timeout at awaitAvailable()
at com.mchange.v2.resourcepool.BasicResourcePool.awaitAvailable(BasicResourcePool.java:1467)
at com.mchange.v2.resourcepool.BasicResourcePool.prelimCheckoutResource(BasicResourcePool.java:644)
at com.mchange.v2.resourcepool.BasicResourcePool.checkoutResource(BasicResourcePool.java:554)
at com.mchange.v2.c3p0.impl.C3P0PooledConnectionPool.checkoutAndMarkConnectionInUse(C3P0PooledConnectionPool.java:758)
at com.mchange.v2.c3p0.impl.C3P0PooledConnectionPool.checkoutPooledConnection(C3P0PooledConnectionPool.java:685)
... 17 more
Exception in thread "async-thread-macro-2" java.lang.Error: java.sql.SQLException: An attempt by a client to checkout a Connection has timed out.
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1155)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.sql.SQLException: An attempt by a client to checkout a Connection has timed out.
at com.mchange.v2.sql.SqlUtils.toSQLException(SqlUtils.java:118)
at com.mchange.v2.sql.SqlUtils.toSQLException(SqlUtils.java:77)
at com.mchange.v2.c3p0.impl.C3P0PooledConnectionPool.checkoutPooledConnection(C3P0PooledConnectionPool.java:690)
at com.mchange.v2.c3p0.impl.AbstractPoolBackedDataSource.getConnection(AbstractPoolBackedDataSource.java:140)
at clojure.java.jdbc$get_connection.invokeStatic(jdbc.clj:298)
at clojure.java.jdbc$get_connection.invoke(jdbc.clj:193)
at clojure.java.jdbc$db_transaction_STAR_.invokeStatic(jdbc.clj:628)
at clojure.java.jdbc$db_transaction_STAR_.invoke(jdbc.clj:585)
at clojure.java.jdbc$db_transaction_STAR_.invokeStatic(jdbc.clj:598)
at clojure.java.jdbc$db_transaction_STAR_.invoke(jdbc.clj:585)
at pepa.processor$next_item_STAR_.invokeStatic(processor.clj:14)
at pepa.processor$next_item_STAR_.invoke(processor.clj:13)
at pepa.processor$process_next.invokeStatic(processor.clj:21)
at pepa.processor$process_next.invoke(processor.clj:19)
at pepa.processor$start$fn__20979.invoke(processor.clj:35)
at clojure.core.async$thread_call$fn__7338.invoke(async.clj:439)
at clojure.lang.AFn.run(AFn.java:22)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
... 2 more
Caused by: com.mchange.v2.resourcepool.TimeoutException: A client timed out while waiting to acquire a resource from com.mchange.v2.resourcepool.BasicResourcePool@703c90bc -- timeout at awaitAvailable()
at com.mchange.v2.resourcepool.BasicResourcePool.awaitAvailable(BasicResourcePool.java:1467)
at com.mchange.v2.resourcepool.BasicResourcePool.prelimCheckoutResource(BasicResourcePool.java:644)
at com.mchange.v2.resourcepool.BasicResourcePool.checkoutResource(BasicResourcePool.java:554)
at com.mchange.v2.c3p0.impl.C3P0PooledConnectionPool.checkoutAndMarkConnectionInUse(C3P0PooledConnectionPool.java:758)
at com.mchange.v2.c3p0.impl.C3P0PooledConnectionPool.checkoutPooledConnection(C3P0PooledConnectionPool.java:685)
... 17 more