Dynamic candidate set filtering

[oantolin]

Dynamic candidate set filtering

[oantolin]

Oh, it furthermore only seems to work if basic comes first, and then you only get basic matching after "a " or "b ". I'm starting to think the example can be rewritten to work for all completion styles.

[clemera]

The reason is that other places use minibuffer-contents and minibuffer-prompt-end to check for the input string and rely on that to be always the case.

[oantolin]

So I think we can close this now, right? Let me know if I'm wrong.

[oantolin]

That's an interesting example! Thanks to @clemera for coming up with it and to you for bringing it to my attention.

I agree that orderless doesn't handle this example, but neither does substring or partial-completion.

I tested with completion-styles set to various subsets of '(basic substring partial-completion orderless) and the example only seems to work if basic is included among the completion-styles.

I'd have to think a bit more to decide whether this is a bug in all of substring, partial-completion and orderless, or whether it is a bug in the example.

For now, a quick workaround seems to be to include basic in your completion-styles. I normally have that set to '(orderless) but see no harm in setting it to '(basic orderless).

[oantolin]

It bothered me I didn't test systematically. But here's quick proof that among all subsets of '(basic substring partial-completion orderless), the example only works on those subsets that include basic:

(defun subsets (set)
  (if (null set)
      (list nil)
    (cl-loop with x = (car set)
             for sub in (subsets (cdr set))
             collect (cons x sub) collect sub)))

(let ((colla (all-completions "a " #'test)))
  (setcdr (last colla) 0)
  (cl-loop for styles in (subsets
                          '(basic substring partial-completion orderless))
           when (equal colla
                       (let* ((completion-styles styles))
                         (completion-all-completions "a " #'test nil 2)))
           collect styles))

If you evaluate that you should get exactly the 8 subsets that include basic.

[minad]

Oh, it furthermore only seems to work if basic comes first, and then you only get basic matching after "a " or "b ". I'm starting to think the example can be rewritten to work for all completion styles.

That would be ideal. Since up to now we have not been able to come up with something else.

[oantolin]

I can't seem to get this to work with substring. I'll keep trying, but in the meantime have you considered just prefixing the candidates with a special string? Like prefix all buffers with b:, all files with f:, etc. That should make it easy to narrow to the ones you want. You can use a plain list instead of a fancy completion table, and you just need to strip the prefix off the result before using it. It's a low-tech solution.

[minad]

I'll keep trying, but in the meantime have you considered just prefixing the candidates with a special string?

Yes, I have. I could also combine that with a 'display property if it is preferred to hide the prefix (as I do for consult-line for example). But then it is not functionally equivalent since the prefix will match on other stuff too. So I like the "high-tech" solution more. It is also some kind of proof of concept/prototype before trying out other more complicated stuff with dynamic candidate sets. Note that Selectrum for example does not yet support dynamic tables at all as far as I understood. So it would nice to have something like this as an example for dynamic sets and the continue from there. And since Icomplete is the reference implementation it should first work there. Unfortunately it is not that simple it seems. I think in general the idea of restricting Consult/Marginalia/Embark and friends to the standard completing read API is a really good idea since it ensures that we can easily replace components and it ensures that we restrict ourselves to the boundaries of the API. But as of now I have to use selectrum-read for consult-buffer (with a fallback to completing-read).

[clemera]

If there isn't an approach without hacks the following works by tricking minibuffer-contents and minibuffer-prompt-end into believing the prefix belongs to the prompt:

(let ((colla '("this" "is" "the" "colla"))
      (collb '("now" "showing" "collb")))
  (defun test (str pred action)
    (if (eq action 'metadata)
        `(metadata
          (cycle-sort-function . identity)
          (display-sort-function . identity))
      (let* ((ap (save-excursion
		   (goto-char (minibuffer-prompt-end))
		   (get-text-property (1- (point)) 'ap)))
	     (bp (save-excursion
		   (goto-char (minibuffer-prompt-end))
		   (get-text-property (1- (point)) 'bp)))
	     (coll (cond (ap
			  colla)
			 (bp
			  collb)
			 (t
			  (append colla collb)))))
	(complete-with-action action coll str pred)))))

(defun self-insert-command+ (n)
  (interactive "p")
  (setq this-command 'self-insert-command)
  (call-interactively 'self-insert-command)
  (save-excursion
    (goto-char (minibuffer-prompt-end))
    (let ((ap (looking-at "a \\'"))
	  (bp (looking-at "b \\'")))
      (when (or ap bp)
	(add-text-properties
	 (point) (+ (point) 2)
	 `(field
	   t
	   ,(if ap 'ap 'bp)
	   t
	   rear-nonsticky t))))))


(minibuffer-with-setup-hook
    (lambda ()
      (let ((map (make-sparse-keymap)))
	(define-key map " " 'self-insert-command+)
	(use-local-map (make-composed-keymap map (current-local-map)))))
  (completing-read "Check: " 'test))

[minad]

Yes, but what is the deeper reason that the thing given example above cannot work? I am very wary of building anything on top of dynamic completion tables in consult if it is that complicated and hacky. In particular I am not a fan of an effectful access touching something external, while I see the candidate set function as pure.

[minad]

So they are making exactly the mistake I am pointing out, breaking things for everyone else. But my point is superficial and abstract, probably there are deeper reasons. In marginalia we also have to look at the minibuffer contents at some point in the annotator function in order to get the full candidate. Breaks purity too minad/marginalia@4ba9804

[clemera]

There would have to be a way to let the dynamic candidate function tell what the current input is, this is what we do in selectrum.

[clemera]

Maybe this could be fixed upstream by proposing a change of minibuffer-contents and minibuffer-prompt-end to check this info from metadata of the completion table? But I don't now what other consequences this could have would probably break a bunch of things.

[minad]

Well from my pov (and I am not deep in the details as you) and with regards to what I would like to achieve: I would like to start using dynamic candidate sets in consult, if there is a way. These are the roadblocks:

• Right now I have consult-buffer which falls back to selectrum-read. Maybe this is such an edge case that it simply cannot be supported as of now without hack/changing upstream/changing orderless and what else. Sounds rather like we want something which just does not work. Given the discussion in this very issue...
• Selectrum has no support for dynamic candidates yet
• I have to write a few minimal prototypes which work well with dynamic candidates in incomplete and selectrum. I thought the consult buffer could be that kind of prototype

Given the status quo I consider just giving up on the current consult buffer implementation and reconsider doing the low tech solution mentioned by @oantolin above. I wonder if the low tech solution could be enhanced by modifying the completion style locally in order for the "a SPC" prefix to only match at the beginning. This would somehow circumnavigate the issue here. Combined with the display property the prefix could even be hidden.

[clemera]

This sums it up well, good to have a clearer picture on this now, thank you for pushing on this!

Selectrum has no support for dynamic candidates yet

To provide some more detail on this: Selectrum does support it via selectrum-read but for completing-read it currently assumes the completion table returns all the candidates at once when called with (all-completions "" coll) and the table doesn't get queried further afterwards.

[minad]

Yes, that's right. The goal is to work within the confines of the completing-read API. I would prefer if I can from now on extend consult only with generic functionality. And since you are interested in improving the api support in selectrum (e.g., fixing the annotation issues recently), I am hopeful that we can make this work!