papis/papis

Powerful and highly extensible command-line based document and bibliography manager.

Papis

Papis is a powerful and highly extensible CLI document and bibliography manager.

With Papis, you can search your library for books and papers, add documents and notes, import and export to and from other formats, and much much more. Papis uses a human-readable and easily hackable .yaml file to store each entry's bibliographical data. It strives to be easy to use while providing a wide range of features. And for those who still want more, Papis makes it easy to write scripts that extend its features even further.

Features

• Add documents and automatically fetch their metadata.
• Search by author, title, tags, and so on.
• Synchronize your library with whatever software you're already using.
• Share your documents with colleagues without having to force some proprietary service onto them.
• Import your data from other bibliography managers.
• Export to BibTeX and other formats.
• Integrate with your editor with plugins for (Neo)vim and Emacs.
• TUIs make it easy to get a quick overview of your library.
• Use the web app when the CLI doesn't quite cut it (for example on your tablet).
• Hacking Papis is easy! Use the API to easily create your own custom python scripts.

Quick tour

Install Papis with pip (or one of the alternatives):

pip install papis

Let's download a couple of documents:

wget http://www.gnu.org/s/libc/manual/pdf/libc.pdf
wget http://www.ams.org/notices/201304/rnoti-p434.pdf

We can now add these to the (default) library. This will automatically query for the metadata associated with the doi.

papis add --from doi 10.1090/noti963 rnoti-p434.pdf

You can also use --set to add information:

papis add libc.pdf --set author "Sandra Loosemore" \
                   --set title "GNU C reference manual" \
                   --set year 2018 \
                   --set tags programming \
                   --confirm

Now open an attached file or edit an entry:

papis open
papis edit

Or export them to bibtex:

papis export --all --format bibtex > mylib.bib

Papis also includes a web app that you can start with:

papis serve

You can then open the indicated address (http://localhost:8888) in your browser.

All papis commands come with help messages:

papis -h      # General help
papis add -h  # Help with a specific command

Installation & setup

Information about installation and setup can be found in the docs, for example in the these sections:

• Installation
• Configuration
• Import
• Editor integration

Questions?

The docs cover Papis' features and discuss possible work flows. If you still have questions, head to our GitHub discussions — we're more than happy to help. If you've found a bug, please open an issue and help make Papis even better!

Reviews and blog posts

• Blog post about getting a paper's references with papis explore.
• Blog post about using Papis with Zotero and Syncthing.
• GNU/Linux Switzerland wrote about Papis (in German).
• The folks at OSTechNix wrote a review of Papis.
• A review of Papis by Ubunlog (in Spanish).

Contributing

Contributions are very welcome! Take a look at CONTRIBUTING.md for general rules and HACKING.md for additional code-related information. We encourage you to also check out, contribute to, or even help maintain the other projects in the Papis ecosystem mentioned below 😉.

The Papis ecosystem

Papis has grown over the years and there are now a number of projects that extend Papis' features or integrate it with other software.

Project	Maintained by
papis (core)	Alejandro Gallo, Julian Hauser, Alex Fikl
papis-rofi	Etn40ff
papis-dmenu	you?
papis-vim	you?
papis.nvim	Julian Hauser
papis-emacs	Alejandro Gallo
papis-zotero	Alex Fikl
papis-libgen	you?
papis-firefox	wavefrontshaping

Related software

Papis isn't the only fish in the pond. You might also be interested in:

• bibman (opensource)
• cobib (opensource)
• jabref (opensource)
• Mendeley (proprietary)
• pubs (opensource)
• Xapers (opensource)
• Zotero (opensource)

papis's Issues

Webapplication for papis

What about trying to get paperhero work with papis?

https://github.com/alejandrogallo/paperhero

gui --rofi crashing on delete

As per the title, when deleting something from within the rofi GUI papis crashes. The selected item is deleted, in any case.

Traceback:

Traceback (most recent call last):
  File "/usr/bin/papis", line 11, in <module>
    load_entry_point('papis==0.1.3', 'console_scripts', 'papis')()
  File "/usr/lib/python3.4/site-packages/papis-0.1.3-py3.4.egg/papis/main.py", line 24, in main
    papis.commands.main()
  File "/usr/lib/python3.4/site-packages/papis-0.1.3-py3.4.egg/papis/commands/__init__.py", line 210, in main
    commands["default"].main()
  File "/usr/lib/python3.4/site-packages/papis-0.1.3-py3.4.egg/papis/commands/default.py", line 137, in main
    commands[self.args.command].main()
  File "/usr/lib/python3.4/site-packages/papis-0.1.3-py3.4.egg/papis/commands/gui.py", line 53, in main
    return self.rofi_main()
  File "/usr/lib/python3.4/site-packages/papis-0.1.3-py3.4.egg/papis/commands/gui.py", line 36, in rofi_main
    return papis.gui.rofi.Gui().main(self.documents)
  File "/usr/lib/python3.4/site-packages/papis-0.1.3-py3.4.egg/papis/gui/rofi.py", line 151, in main
    self.delete(self.documents[i])
  File "/usr/lib/python3.4/site-packages/papis-0.1.3-py3.4.egg/papis/gui/rofi.py", line 167, in delete
    self.documents = self.fetch_documents()
AttributeError: 'Gui' object has no attribute 'fetch_documents'

file-name option and adding files with incomplete metadata

If I set

file-name = {doc[author]}{doc[year]}

in my config, I can't add new documents using a command like,

papis add file.pdf --name "John Smith" --title "An Article"

presumably because papis tries to access the doc[year], but the metadata does not yet include a year. Seems like it should have a way of gracefully falling back to an incomplete file-name in situations like this.

Traceback (most recent call last):
  File "/usr/local/bin/papis", line 11, in <module>
    sys.exit(main())
  File "/usr/local/lib/python3.6/site-packages/papis/main.py", line 24, in main
    papis.commands.main()
  File "/usr/local/lib/python3.6/site-packages/papis/commands/__init__.py", line 210, in main
    commands["default"].main()
  File "/usr/local/lib/python3.6/site-packages/papis/commands/default.py", line 167, in main
    commands[self.args.command].main()
  File "/usr/local/lib/python3.6/site-packages/papis/commands/add.py", line 509, in main
    suffix=string_append
  File "/usr/local/lib/python3.6/site-packages/papis/commands/add.py", line 228, in get_file_name
    "." + papis.utils.guess_file_extension(original_filepath)
  File "/usr/local/lib/python3.6/site-packages/papis/utils.py", line 68, in format_doc
    return python_format.format(**{doc: document})
KeyError: 'year'

Typo in help message

https://github.com/alejandrogallo/papis/blob/4b9e0c549fa4d205708e740968ab1c1cef98c70d/doc/build/man/papis.1#L1754

the selected

backspace not detected

in papis open, backspace does nothing while C-H and C-backspace are working.

I think it depends on the terminal emulator (there's plenty of discussiong like
https://stackoverflow.com/questions/44943249/detecting-key-backspace-in-ncurses)

In my case urxvt the code is 127.

With terminology it is working

Project specific libraries

It wasn't entirely clear from the documentation how to add files to a specific library. Is this possible? I would like to have all the related to a project contained within the project (i.e have the library inside a repository for a paper I'm writing, for example). To do this it would be convenient to have a local configuration file so that papis commands ran inside a project only affect the project specific library.

Possibly pulling BibTeX information from PubMed via HubMed

Even if there is already an API available for PubMed via the Entrez retrieval system its output is in RIS format, as I already wrote briefly about on Reddit. But luckily it looks like there's a 3rd party interface available, HubMed (Wikipedia page, help page), which allows for search and download of bibliographic data in BibTeX format - it looks like it's meant to be used interactively, and I couldn't find much in the way of description/help/ToS, but the data can be accessed directly.

As papis already uses a BibTeX library downloading and adding this information for any PMID should be about as easy as for a DOI.

Example:

curl "http://pubmed.macropus.org/articles/?format=text%2Fbibtex&id=16006400"

Resulting in

@Article{Lester2004,
author="Lester, David",
title="The suicide rate in the concentration camps was extraordinarily high: a comment on Bronisch and Lester.",
journal="Archives of suicide research : official journal of the International Academy for Suicide Research",
year="2004",
volume="8",
number="2",
pages="199--201",
abstract="Previous reports, based on limited reports from inmates who survived the Nazi concentration camps, have claimed that suicide was rare in the concentration camps. Using slightly more detailed, but nonetheless still limited, data from survivors of the camps, it is estimated that the suicide rates in the camp were most likely 25,000 per 100,000 per year or higher and, therefore, enormous!",
issn="1381-1118",
doi="10.1080/13811110490271425",
url="http://www.ncbi.nlm.nih.gov/pubmed/16006400",
language="eng"
}

pdf import: rename file in a meaningful way

When sharing all the papers, with zotero I did "find zotero_dir -iname "*.pdf" -exec cp {} output_dir ;".
Using ZotFile, the files where named after the paper year - author - title.

Now with papers downloaded from scihub, for example, I have a lot of "output.pdf". This is ugly also if using the "history" function of pdf viewers and so on, since it's difficult to remind which paper was it (on emacs interleave, notes are named like the source pdf, so having almost-unique name is a must).

I'd suggest to name them with doc[ref] (especially if it will be possible to customize the ref format) upon import and then don't touch them anymore

Multiple authors in bibtex

Currently, papis exports multiple authors as a list separated with commas, e.g.,

{ Barack Obama, Donald Trump }

But BibTeX doesn't understand this: it interprets commas as separating last name from first, and expects multiple authors to be separated by the word 'and'. So papis should export multiple authors either as,

{ Barack Obama and Donald Trump }

or

{ Obama, Barack and Trump, Donald }

`papis zotero` fails if bibtex entry contains %

Possibly related to #4. See below example. Notice that the {} in Li2014 author key work fine, however abstract {} do not. After further testing it is in fact the %.

fail.txt
success.txt

    $ papis zotero fail.txt

    INFO: Processing  |   ref: Li2014
    Traceback (most recent call last):
    File "/home/mike/.config/papis/scripts/papis-zotero", line 51, in <module>
    print('                  |Author: %s' % entry['author'])
    KeyError: 'author'

    $ papis zotero success.txt
    
    INFO: Processing  |   ref: Ren2015
                  |Author: Ren, Shuhua and Chen, Ruiyong and Maawad, Emad and Dolotko, Oleksandr and Guda, Alexander a. and Shapovalov, Viktor and Wang, Di and Hahn, Horst and Fichtner, Maximilian
                  | Title: Improved Voltage and Cycling for Li + Intercalation in High-Capacity Disordered Oxyfluoride Cathodes
                  |
INFO: Processing  |   ref: Li2014
                  |Author: Li, Yiyang and {El Gabaly}, Farid and Ferguson, Todd R and Smith, Raymond B and Bartelt, Norman C and Sugar, Joshua D and Fenton, Kyle R and Cogswell, Daniel A and Kilcoyne, a L David and Tyliszczak, Tolek and Bazant, Martin Z and Chueh, William C
                  | Title: Current-induced transition from particle-by-particle to concurrent intercalation in phase-separating battery electrodes.
                  |
INFO: Processing  |   ref: Harry2014
                  |Author: Harry, Katherine J and Hallinan, Daniel T and Parkinson, Dilworth Y and MacDowell, Alastair a and Balsara, Nitash P
                  | Title: Detection of subsurface structures underneath dendrites formed on cycled lithium metal electrodes.
                  |

    $ diff fail.txt success.txt

    18c18
< abstract = {... to 100 {\%}(concurrent intercalation)...},
---
> abstract = {... to 100 (concurrent intercalation)...},

papis zotero does not honour --pick-lib or -l my_lib

Problem

papis --pick-lib zotero my_bib.bib or papis -l my_lib zotero my_bib.bib currently does not honour the library options and will add the contents of the bib file to the default library defined in the config file.

Solution

Add -l papis.api.get_lib() option to examples/scripts/papis-zotero

Allow `ref` preferences to be set in config

Some of the larger bibliography programs allow the style of the ref flag to be set in Preferences. The ref flag is used in referencing entries from bibtex files, and would be useful to remember off hand. Personally, I use the style AuthorYear e.g. Plews2019. It may be useful to allow users to add a preferred format which is used if the information is available when using the --from-doi flag (or similar).

An Example:

[settings]
ref-format = {doc[author]}{doc[year]}

Right now it defaults to the doi, which is good fallback behaviour.

I have a branch I was going to PR where I added a --ref flag (310e096), but this might be worth adding. I'm happy to work on this, I just can't figure out where/how you pull information from the config file, and how I would translate it into papis add.

Why does papis browse not default to using the doi?

Is there a reason that papis browse tries to use different methods (pubmed, airvx, etc.) to open the paper in a browser rather than using the digital object identifier (DOI)? DOIs are designed to last for a very long time, if not essentially forever. It also, in some cases, does not redirect to the original article location.

Example:

$ papis browse Cha2008
# opens https://www.ncbi.nlm.nih.gov/pubmed/18062687

This is only the abstract, not the full article.

$ papis export Cha2008 --yaml | grep 'doi:' | awk '{print "https://doi.org/"$2}'

https://doi.org/10.1021/ic701570z

This redirects to the original article

EDIT: At the time of writing, pubs.acs.org is down.

'ref' tag with spaces causes errors on LaTeX compiling

Bibtex needs refs that do not contain whitespace. Refs should be collapsed to remove whitespace during papis add

I can take care of this.

use-cache option broken

setting use-cache = False (either in settings or in library)

❯papis check

Traceback (most recent call last):
  File "/nix/store/mfgz8nf4a1i35had90zxii4cx8dpy9v2-papis-0.5.2/bin/.papis-wrapped", line 12, in <module>
    sys.exit(main())
  File "/nix/store/mfgz8nf4a1i35had90zxii4cx8dpy9v2-papis-0.5.2/lib/python3.6/site-packages/papis/main.py", line 24, in main
    papis.commands.main()
  File "/nix/store/mfgz8nf4a1i35had90zxii4cx8dpy9v2-papis-0.5.2/lib/python3.6/site-packages/papis/commands/__init__.py", line 236, in main
    commands["default"].main()
  File "/nix/store/mfgz8nf4a1i35had90zxii4cx8dpy9v2-papis-0.5.2/lib/python3.6/site-packages/papis/commands/default.py", line 167, in main
    commands[self.args.command].main()
  File "/nix/store/mfgz8nf4a1i35had90zxii4cx8dpy9v2-papis-0.5.2/lib/python3.6/site-packages/papis/commands/check.py", line 46, in main
    self.get_args().search
  File "/nix/store/mfgz8nf4a1i35had90zxii4cx8dpy9v2-papis-0.5.2/lib/python3.6/site-packages/papis/api.py", line 216, in get_documents_in_lib
    return papis.api.get_documents_in_dir(directory, search)
  File "/nix/store/mfgz8nf4a1i35had90zxii4cx8dpy9v2-papis-0.5.2/lib/python3.6/site-packages/papis/api.py", line 197, in get_documents_in_dir
    return papis.utils.get_documents(directory, search)
  File "/nix/store/mfgz8nf4a1i35had90zxii4cx8dpy9v2-papis-0.5.2/lib/python3.6/site-packages/papis/utils.py", line 298, in get_documents
    folders = get_folders()
TypeError: get_folders() missing 1 required positional argument: 'folder'

add --from-isbn

It would be nice to be able to add books given their ISBN, like it's possible to use PMID for papers. Zotero does this with they "magic add" tool

papis add fails with FileNotFoundError

papis add --from-doi 10.1021/nn4005022 /tmp/paper.pdf 

Results in:

Traceback (most recent call last):
  File "/usr/local/bin/papis", line 9, in <module>
    load_entry_point('papis==0.5.0', 'console_scripts', 'papis')()
  File "/home/mike/Software/papis/papis/main.py", line 24, in main
    papis.commands.main()
  File "/home/mike/Software/papis/papis/commands/__init__.py", line 210, in main
    commands["default"].main()
  File "/home/mike/Software/papis/papis/commands/default.py", line 167, in main
    commands[self.args.command].main()
  File "/home/mike/Software/papis/papis/commands/add.py", line 509, in main
    suffix=string_append
  File "/home/mike/Software/papis/papis/commands/add.py", line 228, in get_file_name
    "." + papis.utils.guess_file_extension(original_filename)
  File "/home/mike/Software/papis/papis/utils.py", line 601, in guess_file_extension
    if eval("is_%s" % ext)(file_description):
  File "/home/mike/Software/papis/papis/utils.py", line 585, in is_pdf
    return file_is(file_description, 'pdf')
  File "/home/mike/Software/papis/papis/utils.py", line 565, in file_is
    r".*%s.*" % fmt, magic.from_file(file_description, mime=True),
  File "/usr/local/lib/python3.5/dist-packages/magic.py", line 131, in from_file
    return m.from_file(filename)
  File "/usr/local/lib/python3.5/dist-packages/magic.py", line 81, in from_file
    with open(filename):
FileNotFoundError: [Errno 2] No such file or directory: 'paper.pdf'

Is the full filepath not being passed to the magic function?

Scihub downloader failing

https://github.com/alejandrogallo/papis/blob/1f88cdc3bbb6cfdae7de72d3c8646fbb9ddacea3/examples/scripts/papis-scihub#L29

Sci-hub.io seems down, while sci-hub.cc is ok. Maybe we can have a list of valid url, check for the first being alive, and use it.

❯curl sci-hub.io
curl: (6) Could not resolve host: sci-hub.io                                                                                                                                                      
❯curl sci-hub.cc
<index.html here>

Rofi GUI doesn't honour 'editor' option in .config/papis/config

Trying to edit the info.yaml from rofi defaults to xdg-open even if the editor option is set in .config/papis/config.

Example:

$ papis gui --rofi
# rofi opens
# Alt+e opens info.yaml in xdg-open

$ papis config editor 
nano

Rofi GUI open wrong file, sometimes

Hi, I don't know how to debug this right now. However, opening a file with the gui --rofi not always open the correct paper (both for edit/show file) while the vim GUI for the same entries open the correct file.

I'll look if I can reproduce this with a smaller library

Force https as standard

It would be nice (and an easy fix) to change every URL to https to encourage security.

get_clean_doi regex tweak?

Here is a url that contains a doi:

http://onlinelibrary.wiley.com/doi/10.1111/nous.12244/abstract

Papis "cleans" this to:

10.1111/nous.12244/abstract

But that isn't recognized by Crossref. It should clean it to:

10.1111/nous.12244

which Crossref recognizes.

Update database model to allow for whoosh backend

Using Whoosh would enable a more powerful query language and probably make the queries more performant. How is the querying done currently and what would need to be done to add whoosh?

gracefully treat conflicting export config

I tried exporting the biblography without reading too much the help info

papis export --json --folder /tmp/newexp -a

Traceback (most recent call last):                                                                                                                                                               
  File "/home/IITaudio/tmp/bin/papis", line 11, in <module>
    load_entry_point('papis==0.4.6', 'console_scripts', 'papis')()
  File "/home/IITaudio/tmp/lib/python3.4/site-packages/papis/main.py", line 24, in main
    papis.commands.main()
  File "/home/IITaudio/tmp/lib/python3.4/site-packages/papis/commands/__init__.py", line 210, in main
    commands["default"].main()
  File "/home/IITaudio/tmp/lib/python3.4/site-packages/papis/commands/default.py", line 153, in main
    commands[self.args.command].main()
  File "/home/IITaudio/tmp/lib/python3.4/site-packages/papis/commands/export.py", line 135, in main
    return self.args.out.write(                                                                                                                                                                  
AttributeError: 'str' object has no attribute 'write'                                                                                                                                            
(tmp)                         

Crash when giving an empty string to papis rename

Papis crashes when simply pressing enter at the papis rename prompt as it tries to perform a mv folder/ folder/, likely because of default=document.get_main_folder_name(). It should check for equality (or empty/invalid string) and quietly exit instead.

Enter new folder name:
> (paper_name):
[...]
Message: 'Path /home/mftrhu/Documents/papers/paper_name already exists!'
Arguments: ()
mv: cannot move '/home/mftrhu/Documents/papers/paper_name' to a subdirectory of itself, '/home/mftrhu/Documents/papers/paper_name/paper_name'

Rofi GUI cannot open files if there's a space anywhere in the path name

If there's a space either in the file name or in any of the parent directories, the rofi GUI will fail to open the PDF with the following error when run through the terminal:

xdg-open: unexpected argument [part after space]
Try 'xdg-open --help' for more information.

doi resolution is buggy

Hi, papis is failing add the paper from this doi:
10.1145/3139513.3139518

doing it manually, both crossref and doi.org are working.
I almost implemented papis add --automatic [ISBN/DOI/PMID] but I'm having problems with both isbnplus and doi resolvers working just for some papers/books

papis-python-rofi wrong name

Hi, I could not open the issue on your python-rofi page because it's a fork.

the setup file has the line

name='python-rofi',

while in pip it is called papis-python-rofi.
Could you update it in the repo (and tag the release?), else installing from github does not satifsy the dependency

Thanks

(info: I'm adding papis to nixpkgs (nix package manager))

Import bib (zotero) fails if field is missing

https://github.com/alejandrogallo/papis/blob/1f88cdc3bbb6cfdae7de72d3c8646fbb9ddacea3/examples/scripts/papis-zotero#L50

I'd do something like:

    print('INFO: Processing  |   ref: %s' % entry.get('ref',''))
    print('                  |Author: %s' % entry.get('author','Unknown'))
    print('                  | Title: %s' % entry.get('title','Untitled'))

Logo design

Hi Alejandro,

I like the idea in ROADMAP.md of a logo, so I went ahead and designed one. Perhaps this thread would be a good one to post logo ideas until you decide on one you like.

Thanks!

or

Search citations

I see that some papers I add --from-doi have a list of citations in the info.yaml. It would be awesome if, from gui or command line, could be able to see references with the picker. Hit enter, and, if you already have them in your library open the file, else open a web browser of https://dx.doi.org/[DOI].

Might take a bit of work for the extra functionality but I wanted to add it while I remembered.

Full-text search implementation

I'm looking at the roadmap.
The first point ("PDF or general document type content indexing") is what I'm referring here to "full-text search".
One way would be to just extract text and use something like xapian (used by notmuch for mail indexing) that is a wonderful library. However, the already existing recoll search tool already uses it. It is capable of converting a variety of file formats including pdf files (by using poppler pdftotext).

IMHO taking advantage of their python API could do everything we need.

I wanted to hear here opinions
Nicolò

tk dependency not declared

https://github.com/alejandrogallo/papis/blob/3293d18de1a751bb1033e3899b923012adf56cce/papis/gui/tk/__init__.py#L4

Tk is neither optional nor required in setup.py
(it is required to have tests passing)

Duplication warnings on different documents

For some reason I'm getting duplication warnings when I try to add two different documents.

(papis) ➜  papis mkdir lib
(papis) ➜  papis papis -l lib add --from-url https://arxiv.org/abs/1712.02621v1
(papis) ➜  papis papis -l lib add --from-url https://arxiv.org/abs/1711.09534v1

WARNING:Command:                    
abstract:   Generating novel, yet realistic, images of persons is a challenging task due
to the complex interplay between the different image factors, such as the
foreground, background and pose information. In this work, we aim at generating
such images based on a novel, two-stage reconstruction pipeline that learns a
disentangled representation of the aforementioned image factors and generates
novel person images at the same time. First, a multi-branched reconstruction
network is proposed to disentangle and encode the three factors into embedding
features, which are then combined to re-compose the input image itself. Second, three corresponding mapping functions are learned in an adversarial manner in
order to map Gaussian noise to the learned embedding feature space, for each
factor respectively. Using the proposed framework, we can manipulate the
foreground, background and pose of the input image, and also sample new
embedding features to generate such targeted manipulations, that provide more
control over the generation process. Experiments on Market-1501 and Deepfashion
datasets show that our model does not only generate realistic person images
with new foregrounds, backgrounds and poses, but also manipulates the generated
factors and interpolates the in-between states. Another set of experiments on
Market-1501 shows that our model can also be beneficial for the person
re-identification task.
archiveprefix:   arXiv
author:   Liqian Ma and Qianru Sun and Stamatios Georgoulis and Luc Van Gool and Bernt Schiele and Mario Fritz
eprint:   1712.02621v1
file:   1712.02621v1.pdf
files:   ['tmpbm24n56y']
month:   Dec
primaryclass:   cs.CV
ref:   1712.02621v1
title:   Disentangled Person Image Generation
type:   article
url:   http://arxiv.org/abs/1712.02621v1
year:   2017

WARNING:Command:DUPLICATION WARNING
WARNING:Command:The document above seems to be already in your libray: 

WARNING:Command:(Hint) Use the update command if you just want to update the info.
Really add? (Y/n): y

Making the metadata more structured

While having no schema for the metadata is very convenient, I think it might be good to add some structure to the data. I think this would greatly improve interoperability. It also make it easier to query the data. Perhaps it is possible to add YAML-LD contexts to the metadata files? Datalad uses json-ld for metadata and whoosh for searching it, maybe we could do something similar.

Improve bash completion and add zsh completion

I have added a script to create a bash completion script
in

scripts/shell_completion

just hit make and source the script at build/bash-completion.sh.
Maybe someone uses zsh and wants to update it to create a build/zsh-completion.zsh script?

Use XDG base directory specification for paths

In order to cleanup dotfile clutter, making backup easier and giving a standard, XDG Base directory standard was made.
The standard uses a common prefix for user dirs having either by default a well known path or either an env var chosen by the user, allowing flexibility if necessary ( env var can be overridden on a per-program basis if one needs so ).
Implementation of the standard should be rather easy, if necessary there's a sample library from freedesktop itself.
For a third party project that tracks adoption, see the Arch Wiki.

Papis tests don't work with bibtexparser 1.0.1

When I run the test suite with bibtexparser 1.0.1, I get the following error:

_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

self = <bibtexparser.bibdatabase.BibDatabase object at 0x7ffff184aeb8>
name = 'jul'

    def expand_string(self, name):
        try:
            return BibDataStringExpression.expand_if_expression(
                self.strings[name])
        except KeyError:
>           raise(UndefinedString(name))
E           bibtexparser.bibdatabase.UndefinedString: 'jul'

/nix/store/s3141s829l75k7m7vafrh4hijiyd3dc0-python3.6-bibtexparser-1.0.1/lib/python3.6/site-packages/bibtexparser/bibdatabase.py:104: UndefinedString
----------------------------- Captured stderr call -----------------------------
DEBUG:bibtex:Reading in file /build/tmphy9b3_1h
DEBUG:bibtex:Removing comments...
DEBUG:bibtex:Removing empty lines...
------------------------------ Captured log call -------------------------------
bibtex.py                  145 DEBUG    Reading in file /build/tmphy9b3_1h
bibtex.py                  149 DEBUG    Removing comments...
bibtex.py                  151 DEBUG    Removing empty lines...
===================== 1 failed, 10 passed in 3.16 seconds ======================

@nico202 This breaks the papis build on Nixpkgs' staging branch.

parse stdin

It would be useful to have something like

papis list | head -1 | papis export --bibtex

working

Improve urwid tui

I have pushed today a new branch called urwid.
Some time ago I saw the very nice project called xapers,
and I saw their very nice TUI based on urwid.
I have modified it so that we can use it with papis, I think it has a nice potential of becoming
a very capable tui similar to the tui used in tig.
It is in a very rough state right now but in order to use it I have created also a database class
for papis, that might be used to handle different search-engine backends related to the
issues #31 and so on.

The tui is relatively usable but it is i nthe very early stages where everyone can hack in
and discuss about the architecture thereof. So feel free to pull the urwid branch and hack in.
I will create also an acknowledgement document where we acknowledge the great work done in.

https://asciinema.org/a/xfu7qMYaS22N2Tx83D3uqPmbp

Choose option screen broken by left/right arrow

The screen to choose papers or files entered via papis open, papis edit, or by choosing open in the rofi GUI when more than one file is present, allows for both the right and left arrows to be pressed - netting me ąĄ and erasing the content of the screen minus the Pick: prompt - which then results in a IndexError if this "choice" is confirmed.

Full traceback:

Traceback (most recent call last):
  File "/usr/bin/papis", line 11, in <module>
    load_entry_point('papis==0.1.3', 'console_scripts', 'papis')()
  File "/usr/lib/python3.4/site-packages/papis-0.1.3-py3.4.egg/papis/main.py", line 24, in main
    papis.commands.main()
  File "/usr/lib/python3.4/site-packages/papis-0.1.3-py3.4.egg/papis/commands/__init__.py", line 211, in main
    commands["default"].main()
  File "/usr/lib/python3.4/site-packages/papis-0.1.3-py3.4.egg/papis/commands/default.py", line 137, in main
    commands[self.args.command].main()
  File "/usr/lib/python3.4/site-packages/papis-0.1.3-py3.4.egg/papis/commands/open.py", line 44, in main
    document = self.pick(documents)
  File "/usr/lib/python3.4/site-packages/papis-0.1.3-py3.4.egg/papis/commands/__init__.py", line 307, in pick
    pick_config
  File "/usr/lib/python3.4/site-packages/papis-0.1.3-py3.4.egg/papis/api.py", line 116, in pick
    return papis.pick.pick(options, **pick_config)
  File "/usr/lib/python3.4/site-packages/papis-0.1.3-py3.4.egg/papis/pick.py", line 246, in pick
    return picker.start()
  File "/usr/lib/python3.4/site-packages/papis-0.1.3-py3.4.egg/papis/pick.py", line 223, in start
    return curses.wrapper(self._start)
  File "/usr/lib64/python3.4/curses/__init__.py", line 94, in wrapper
    return func(stdscr, *args, **kwds)
  File "/usr/lib/python3.4/site-packages/papis-0.1.3-py3.4.egg/papis/pick.py", line 216, in _start
    return self.run_loop()
  File "/usr/lib/python3.4/site-packages/papis-0.1.3-py3.4.egg/papis/pick.py", line 188, in run_loop
    return self.get_selected()
  File "/usr/lib/python3.4/site-packages/papis-0.1.3-py3.4.egg/papis/pick.py", line 78, in get_selected
    return self.get_filtered_options()[self.index]
IndexError: list index out of range

Import bib (zotero) fails finding file

https://github.com/alejandrogallo/papis/blob/1f88cdc3bbb6cfdae7de72d3c8646fbb9ddacea3/examples/scripts/papis-zotero#L56

why the ":" split? for entries exported by zotero 5 at least, the file list separator is ";".

Also, you are taking the second ([1]) element, while on my export, pdf is always the first, followed (eventally) by an html. Maybe we should split on the separator (";" for me) and then iterate finding pdfs in file extension/mime?

Add zotero conversion

Does someone has a bibtex dump that I could use to develop it?

Add unicode to latex support for bibtex export

maybe recycle the code from here

https://github.com/pkgw/bibtools/blob/master/bibtools/unicode_to_latex.py

unable to open notes.tex

running papis edit, or pressing the e key in the gui results in the info.yaml file being opened, instead of the notes.tex file, which is the behavior described in the documentation.

Export multiple elements

right now it is possible to either export all elements (-a) or a single one (query)

Discussion platform (irc ? gitter ?, both?)

I feel that it's time to define a place where we can discuss some
future directions and also talk to users, and maybe not always the issues
are the place to do it, what do you guys use/propose, we could just use a freenode
channel called #papis.

Parsing of BibTeX breaks with nested curly brackets

Apparently BibTeX allows for nested curly brackets, seemingly to hint at names that shouldn't be decomposed or titles made into lowercase (see here), and Zotero will happily export BibTeX files containing them.

A minimal example file showcasing the problem, test.bib:

@article{haddaway_1993,
    title = {What is love? {Baby} don't hurt me},
    author = {Haddaway},
    month = may,
    year = {1993},
}

Running papis add --from-bibtex test.bib --no-document will thus result in a broken title. Further, it looks like the parsing somehow stops when encountering month, as if the positions of year and month are swapped year will appear in the output file.

The resulting info.yaml file:

author: Haddaway
files:
- info.yaml
ref: haddaway_1993
title: What is love? {Baby
type: article

Additional bibtex keys

When I export my library to bibtex some entries I specified in the info.yaml don't show up. I figured out that only bibtex keys listed in bibtex.py are exported.

bibtex_keys = [
  "address",
  "annote",
  "author",
#...
]

Is it possible to customize this list in a permanent way?

Add bibtex-journal-key option

Problem

Currently, papis export --all --bibtex shows the full journal title in the journal key. When writing articles, some publishers prefer journal abbreviations in the bibliography.

Solution

There should be a config option bibtex-journal-key, and, possibly, a command line option, to export bibtex with the journal key defined by the user, allowing bibtex files to be made with the journal as either full_journal_title or abbrev_journal_title. Default should just be journal seeing as it will need to be backwards compatible with entries not containing the new fields.

I can take care of this