casouri/xeft

Fast, interactive Emacs note searching

NOTE The primary repository is now at SourceHut.

Usage

To use Xeft the note searching interface, install it and type M-x xeft RET to bring up the panel. If the dynamic module doesn’t already exists, you are prompted to download or compile it automatically. If you choose to download the module, no more action is required. If you want to compile the module locally, refer to the next section for prerequisites for compiling the module.

Once the Xeft buffer is up, type the search phrase in the first line. Press C-n and C-p to go through each file. You can preview a file in another window by pressing SPC on a file, or click the file with the mouse. Press RET to open the file in the current window.

Directory xeft-directory stores note files, directory xeft-database stores the database. Xeft uses xeft-default-extension to create new files, and it ignores files with xeft-ignore-extension.

By default, Xeft only searches for first level files in xeft-directory, to make it search recursively, set xeft-recursive to t. Also beware that Xeft ignores search phrases shorter than three characters, unless they are CJK characters.

See the “xeft” customize group for more custom options and faces.

Queries

On search queries:

Since Xeft uses Xapian, it supports the query syntax Xapian supports:

AND, NOT, OR, XOR and parenthesizes
+word1 -word2      which matches documents that contains word1 but not
                   word2.
word1 NEAR word2   which matches documents in where word1 is near word2.
word1 ADJ word2    which matches documents in where word1 is near word2
                   and word1 comes before word2
"word1 word2"      which matches exactly “word1 word2”

Xeft deviates from Xapian in one aspect: consecutive phrases have implied AND between them. So word1 word2 word3 is actually seen as word1 AND word2 AND word3.

See https://xapian.org/docs/queryparser.html for Xapian’s official documentation on query syntax.

Further customization

You can customize the following faces

• xeft-selection
• xeft-inline-highlight
• xeft-preview-highlight
• xeft-excerpt-title
• xeft-excerpt-body

Functions you can customize to alter Xeft’s behavior:

• xeft-filename-fn: How does Xeft create new files from search phrases.
• xeft-file-filter: Which files does Xeft include/exclude from indexing.
• xeft-directory-filter: When xeft-recursive is t, which sub-directories does Xeft include/exclude from indexing.
• xeft-title-function: How does Xeft find the title of a file.
• xeft-file-list-function: If xeft-file-filter and xeft-directory-filter are not flexible enough, this function gives you ultimate control over which files to index.

Building the dynamic module

If the dynamic module is not installed yet, Xeft will automatically prompt you to either download a prebuilt module or build the module locally. Prebuilt modules don’t require xapian, but to build the module locally and run it, you need to install xapian.

You can also build by command line:

make PREFIX=/opt/local

Here /opt/local is the default prefix of macports, which is what I used to install Xapian. Homebrew and Linux users probably can leave it empty.

I can’t test it but on windows you can get msys2 and mingw-w64-x86_64-xapian-core and make should just work. Thanks to pRot0ta1p for reporting this.

notdeft

I owe many thanks to the author of notdeft. I don’t really know C++ or Xapian, without reading his code I wouldn’t be able to write Xeft.

Also, if you want a more powerful searching experience, you will be happier using notdeft instead.

Xapian dynamic module

I wrote a xapian dynamic module that you can use too. Check it out at https://git.sr.ht/~casouri/xapian-lite

Q & A

Why do I still see files that should’ve been ignored after changing xeft-ignore-extension?

You need to delete the database and let xeft rebuild it, because files already included into the database are not automatically deleted.

How to exclude a directory?

You can customize xeft-directory-filter or xeft-file-list-function to do it.

How to use the file name as excerpt title instead?

(setq xeft-title-function #'file-name-nondirectory)

How to include files in the subdirectories?

;; Don't follow symlinks.
(setq xeft-recursive t)
;; Follow symlinks.
(setq xeft-recursive 'follow-symlinks)

How to make the preview pane to show up automatically?

(defvar-local xeft--displayed-by-xeft-p nil)

(defun xeft--eager-preview()
  (when-let* ((button (button-at (point)))
              (path (button-get button 'path)))
    ;; Kill previously displayed buffer.
    (when (window-live-p xeft--preview-window)
      (with-selected-window xeft--preview-window
        (when xeft--displayed-by-xeft-p
          (kill-buffer))))
    ;; Show preview of current selection.
    (xeft--preview-file path)))

(add-hook 'xeft-find-file-hook
          (lambda () (setq xeft--displayed-by-xeft-p t)))

(advice-add 'xeft-next :after #'xeft--eager-preview)
(advice-add 'xeft-previous :after #'xeft--eager-preview)

xeft's Issues

Is there an intent to make it reads a directory recursively?

Thanks for the efforts here. Can we have an option like this in deft
(setq deft-recursive t)

some extensions are not ignored, and some non-ignored extensions are ignored

Some entries en xeft-ignore-extension are ignored (and other extensions not in that variable are ignored).

Minimal reproducible example

(package-initialize)
(add-to-list 'load-path "~/Sources/xeft")
(setq xeft-directory "~/tmp2")
(setq xeft-database "~/.deft/db")
(setq xeft-ignore-extension '("txt~" "tat" "org~"))
(setq xeft-default-extension "txt")

(require 'xeft)

Now, create the xeft-directory and add only these files as follows:

echo aaaaa1 > f1.org~
echo bbbbb2 > f2.txt~
echo ccccc3 > f3.tat
echo ddddd4 > f4.tat~
echo eeeee5 > f5.uuu
echo fffff6 > f6.org
echo ggggg7 > f7.txt

xeft shows:

aaaaa1  # should not be here, as it is coming from a org~ file
bbbbb2  # should not be here, as it is coming from a txt~ file
eeeee5 # OK
fffff6 # OK
ggggg7 # OK

But ddddd4 (which is in f4.tat~, a non-excluded extension) is not shown (ccccc3 is not shown, as it should).

The problem arises in line 709 to 710 of xeft.el:

(not (member (file-name-extension file)
                    xeft-ignore-extension))))

because it uses file-name-extension which, as per documentation, "Return FILENAME’s final "extension" sans any backup version strings." We can check

(file-name-extension "cucu.org~")

returns org , not org~.

An error while trying to build the module

xeft-module.cc:332:33: error: invalid conversion from ‘emacs_subr’ {aka ‘emacs_value_tag* (*)(emacs_env_28*, long int, emacs_value_tag**, void*)’} to ‘emacs_value_tag* (*)(emacs_env*, ptrdiff_t, emacs_value_tag**, void*) noexcept’ {aka ‘emacs_value_tag* (*)(emacs_env_28*, long int, emacs_value_tag**, void*) noexcept’} [-fpermissive]
  332 |     (env, min_arity, max_arity, function, documentation, NULL);
      |                                 ^~~~~~~~
      |                                 |
      |                                 emacs_subr {aka emacs_value_tag* (*)(emacs_env_28*, long int, emacs_value_tag**, void*)}
make: *** [Makefile:9: xeft-module.so] Error 1

Numbers/digits in query are ignored

-<token> does not work

• m1 mbp
• compiling xapian against /opt/homebrew
• claims compiled successfully
• search doesn't work

Description typo

Should Xapian

Publish to MELPA

Hi casouri, I'd like to know if it were possible to publish the package to MELPA? I've never done so but I'd like to help build the recipe if possible. What do you think?

xeft-full-reindex: Xapian library error: "DatabaseCorruptError: Expected block 76 to be level 1, not 0"

How to reproduce:

Here is the config:

;;; -*- lexical-binding: t -*-

(setq straight-check-for-modifications '(check-on-save find-when-checking))
(setq straight-vc-git-default-clone-depth 1)

(defvar bootstrap-version)
(let ((bootstrap-file
       (expand-file-name "straight/repos/straight.el/bootstrap.el" user-emacs-directory))
      (bootstrap-version 5))
  (unless (file-exists-p bootstrap-file)
    (with-current-buffer
        (url-retrieve-synchronously
         "https://raw.githubusercontent.com/raxod502/straight.el/develop/install.el"
         'silent 'inhibit-cookies)
      (goto-char (point-max))
      (eval-print-last-sexp)))
  (load bootstrap-file nil 'nomessage))

(straight-use-package '(xeft
                        :type git :host github :repo "casouri/xeft"
                        :pre-build ("make")
                        :files (:defaults "Makefile" "*.h" "*.cc" "*.so")))

(setq xeft-directory "~/Dropbox/org/roam"
      xeft-database "~/Dropbox/org/roam/db")

And M-x xeft, then get this error message in minibuffer.

Emacs version: GNU Emacs 29.0.50 (build 1, x86_64-apple-darwin19.6.0, NS appkit-1894.60 Version 10.15.7 (Build 19H1715)) of 2022-03-03 (Install from brew install emacs-app-good)
OS: macOS 12.3

Do I need to check anything else like xapian?

Somes files are not ignored

I tried creating new files either with find-file or with xeft and it seems that some files are not ignored, even though they are in the xeft-ignore-extension list.

Steps to reproduce :

Create a new org file in the repository, save it and then modify it. Emacs will create a backup org~ file. When searching for files, the backup file will show up even though it's in the ignored list. Removing the database and launching xeft again did not fix the issue.

face highlighting on random entry when opening left

starting xeft sometimes highlights an entry which is not selected. i'm not sure why this is happening

Module could not be opened

Emacs on latest Master branch.
I can load the package fine. I can build the module fine. No errors.
When I run M-x xeft, I have the following error

if: Module could not be opened: "/home/xxx/.emacs.d/opt/xeft/xeft-module.so", "/home/xxx/.emacs.d/opt/xeft/xeft-module.so\
: undefined symbol: _ZN6Xapian16WritableDatabaseD1Ev"Unable to load color "light golenrod"

How to use xeft on macOS? Thank you very much.

I installed xeft using git submodule

(add-to-list 'load-path "~/.config/emacs/extensions/xeft")
(require 'xeft)
(setq xeft-directory "~/Documents/GitHub/Chaos")
(setq xeft-database "~/Documents/GitHub/Chaos/.xeft-db")

And installed xapian using Homebrew

brew install xapian 

13:14:15 in emacs/extensions/xeft
➜ make
g++ module/xapian-lite.cc -o xapian-lite.dylib -shared -fPIC -I/usr/local/include -L/usr/local/lib -lxapian
module/xapian-lite.cc:19:10: fatal error: 'xapian.h' file not found
#include <xapian.h>
         ^~~~~~~~~~
1 error generated.
make: *** [xapian-lite.dylib] Error 1

Creating files with a different extension than ".txt" does not prompt for the correct extension

Hi, I'm willing to correct this issue in a pull request, in the mean time, I think it'd be better to have a a customizable file extension to create new files.

(defcustom xeft-default-file-extension ".txt"
  "The default file extension for new file creation."
  :type 'string)

Also, this allows this to modify the following functions :

(defcustom xeft-filename-fn
  (lambda (search-phrase)
    (concat search-phrase xeft-default-file-extension))
  "A function that takes the search phrase and returns a filename."
  :type 'function)

And avoid the bug in xeft-create-note which was defaulting to asking for a text extension even though it was not the case. I also removed inserting the file name and create an empty buffer instead.

(defun xeft-create-note ()
  "Create a new note with the current search phrase as the title."
  (interactive)
  (let* ((search-phrase (xeft--get-search-phrase))
         (file-path (expand-file-name
                     (funcall xeft-filename-fn search-phrase)
                     xeft-directory))
         (exists-p (file-exists-p file-path)))
    ;; If there is no match, create the file without confirmation,
    ;; otherwise prompt for confirmation. NOTE: this is not DRY, but
    ;; should be ok.
    (when (or (search-forward "Press RET to create a new note" nil t)
    ;; changed here for the prompt
              (y-or-n-p (format "Create file %s%s? " search-phrase xeft-default-file-extension)))
      (find-file file-path)
    ;; changed here for new file creation
      (unless exists-p
	(write-region "" nil file-path))
      (run-hooks 'xeft-find-file-hook))))

Cheers, thanks for the great package!

How to use xeft-file-list-function correctly?

Hello! Thanks for you contribution.

I have xeft-directory like "~/Dropbox/org", and there are some subdirectories under "org", like "book" "paper"...

How to use xeft-file-list-function for hide the contents of a folder "book"?

Prefer allowed extensions over ignored extensions

Hi casouri,

I'd like to modify the behavior of the ignored extensions and rather prefer allowed extensions. I think it'll be easier to use because it helps filter out the noise rather than picking out all the ignored extensions. For example, I convert files through LaTeX which generates a lot of intermediate files which I don't need. This is the behavior I'd implement instead :

(defcustom xeft-allowed-extension '("txt")
  "Files with extensions in this list are allowed.
All other files are ignored."
  :type '(list string))

It requires only a small change in the function xef--file-list, like so :

(defun xeft--file-list ()
  "Return a list of all files in ‘xeft-directory’."
  (cl-remove-if-not
   (lambda (file)
     (and (file-regular-p file)
          (not (string-prefix-p
                "." (file-name-base file)))
          (member (file-name-extension file)
                  xeft-allowed-extension)))
   (directory-files xeft-directory t nil t)))

Finally, every extension is is specified without a dot beforehand, like "txt" except for the xeft-default-extension which defaults to ".txt", I think it would be more consistent to use this instead :

(defcustom xeft-default-extension "txt"
  "The default extension for new files created by xeft."
  :type 'string)

(defcustom xeft-filename-fn
  (lambda (search-phrase)
    (concat search-phrase "." xeft-default-extension))
  "A function that takes the search phrase and returns a filename."
  :type 'function)

Cheers =)

xapian didn't support long file path

 (xapian-lite-lib-error "InvalidArgumentError: Term too long (> 245): Q/Users/suliveevil/Documents/GitHub/Chaos/000图书-中图法/A马克思主义、列宁主义、*****、***理论/MaoZeDongAnthology/articles/**发言人关于命令国民党反动政府重新逮捕前日本侵华军总司令冈村宁次和逮捕国民党内战罪犯的谈话.md")

xeft-internal.h: No such file or directory

Hey, pretty cool demo. I am a notdeft user so would love to try this.

Currently I can't compile though (on linux):

$ make PREFIX=/usr
g++ -c -fPIC -I/usr/include -L/usr/lib -lxapian xeft-internal.cc -o xeft-internal.o
gcc -shared -fPIC -I/usr/include -L/usr/lib -lxapian xeft-module.c -o xeft-module.so
xeft-module.c:2:10: fatal error: xeft-internal.h: No such file or directory
    2 | #include "xeft-internal.h"
      |          ^~~~~~~~~~~~~~~~~
compilation terminated.
make: *** [Makefile:11: xeft-module.so] Error 1

But that header file isn't part of the repo, so I am confused as to how to proceed.

xeft isn't finding the dynamic's module path

there could be a variable to set the path to the .so file, would prevent this

how to use it with org-roam

when a prompt is not matched in the notes xeft allow to make a new note. I want to use it but it should create new note just like org-roam created new note with the default template , i read the org-roam-node-find function and xeft-filename-fn but i am not expert enough in elisp make it work, would you please help me out on this ?

Could xeft support arm64?

When I install Xeft, it needs the dynamic module to work. However it gives error like below?

xapian-lite.so' (mach-o file, but is an incompatible architecture (have 'x86_64', need 'arm64'))"

:PROPERTIES: or --- Extracted as Title on all files listed rather than #+TITLE or yaml frontmatter title:

(apologies, src.ht does not have an issue tracking functionality... :-/ Unless I missed it when I signed up.).

Loving xeft and how fast it is. However, as I use org-roam, I have a :PROPERTIES: block at the top of every org-mode file. There is also a #+TITLE in each file (as well as most of my md files have yaml frontmatter with title: in them.

I'm guessing xeft takes the first line of the file as title so I'm seeing a lot of :PROPERTIES: which makes it impossible to distinguish the files well even after filtering (unless for some reason it does look for titles, but PROPERTIES somehow throws it off.

So two possible things which would be great:

1. Use filename as title with "shortpath" for directories ie (~/Documents/org/areas/blog/ becomes ~D/o/a/b in display)
2. Extract title blocks from the documents when they're indexed for major modes like org, md, adoc

Thanks for your hard work on this. It's really great. Enjoying it and the way it works with org-roam... =]

(ps> also, if you could add the cutomization options possible in the README that would be great. I tried changing the title font for example by hooking it into my nano theme, but failed ).

Example top of template file in case thta helps if it is a bug

:PROPERTIES:
:ID:       6B58FF53-C8A1-47F3-B374-CD67FB95C327
:END:
#+TITLE: The Rust Programming Language
#+CREATED: [2023-01-22 Sun]
#+MODIFIED: [2023-01-30 Mon 16:26]

* The Rust Programming Language
:PROPERTIES:
:Type: book
:Start: <2023-01-22 Sun>
:Fin: 
:Killed: 
:Rating: 
:Digested: 
:Creator: Steve Klabnik
:URL:
:END:

Stuffy stuff notes I took on the book... 

Show preview for all files as one goes through each one

Thanks a lot for the package. Fast and intuitive to use. :-)

Would it be possible to enable a setting such that:

1. The file is always previewed
2. Ideally, as one moves to the next file that buffer is removed (so as not to fill the buffer list with files).

Ignored files are not kept after searches

Hi again,

I think the ignored files are only ignored when xeft is first called but not afterwards.

For example, create a directory with one accepted file test.txt and another ignored file like test.log. When you first run M-x xeft then only test.txt will show up but if you type something in the search bar which matches both files then test.log will also show up!

how to exclude a given directory

Thanks for the package!

I just tried it with my notes directory which is under git version control, it turned out xeft generated quite big index (around 400Mb) even though my notes takes about 22Mb in space total.

I think it indexes the .git directory as well, is there anyway to exclude certain directory when xeft-recursive is set to t?

How to exclude indexing folders? Thank you very much.

xeft database is so big 😂

Having trouble previewing files with xeft?

When previewing a file, is it possible to open the file at the location of the searched keyword?
Currently using c+n will open the file at the beginning of the file.

backup usage

it's sort of tedious that xeft does not, either by default or by configuration, ignore emacs backup files. namely, the specification of xeft-ignore-extension to quoted ~ does not ignore the "extension" as the implementation uses file-name-extension, which ignores trailing ~ by default. it would be nice if you had some interface for ignoring these. i leave the design of this and its configurability to you -- if you think these semantics would be useful

Show file name in xeft buffer

Thanks for the package!

Is it possible to show the file names in the xeft searching buffer? Some of my org-mode files have other information in the first line rather than the title, which makes it unclear where the search results come from.

Thanks.

Add MS-Windows support ?

For ms-windows I tried using msys2 to compile the module , and it turns out not hard to do.

All I need todo is first install package mingw-w64-x86_64-xapian-core and change the xapian-lite.so to xapian-lite.dll in Makefile.

Can you change the Makefile so it produce a .dll file in ms-windows instead of a .so file? This and the msys2 dependency is all it need to make xeft work on windows.