kungsgeten/hypothesis

Import data from hypothes.is into Emacs

hypothesis's Issues

Use for archiving?

Thank you for that. It's very helpful. Could we consider a system that allows you to periodically "download" annotations and comments to an "org" file... to build up a kind of archive ?
By the way, why is it limited to 50 ?

Row order inside Site

Hi @Kungsgeten,

thanks for this nice little package! I just discovered it and hypothes.is in general a couple of days ago. What I feel to be a little inconvenient is the fact, that the highlights are sorted from youngest to oldest according to 'update date'. This leads to the highlights being in the opposite way, I would expect them to be. I realize that I do not know what your use case is and therefore your need may be different...
Looking at the API, I realized it is possible to sort them according to location in the actual site as well. I added this functionality to my clone and was wondering if you'd like to support it? If so I'd be happy to provide this.

Cheers
Peter

Some suggestions of enhancement

Hello and thanks for your work. I've been using your package for a few days to generate an "org" file containing my hypothes.is annotations. Like bepolymathe (see this issue) I put the file generated by hypothesis.el in my org-roam directory which allows me to link my annotations to my notes and vice versa.

After a couple of days of use I have some suggestions for enhancement (sorry for my poor English). I would have made a pull request, but I don't master elisp (I'm still going to put the code of my hacks).

Allow configuration of the file's header

To be able to use an org file with org-roam (or deft which I also use) it is better to have some extra information in the file header, like #+TITLE: which allows deft and a org-roam to name the file by its title (If this information does not exist deft replaces it with the file name, for org-roam I don't know) or #ROAM_TAGS: which allows org-roam to filter the files by tags.

To allow the insertion of a personalised header I have created a new variable, hypothesis-file-header, whose content is inserted at the creation of the hypothesis-archive file.

(defvar hypothesis-file-header "#+TITLE: Hypothesis\n#+ROAM_TAGS: Quotes Hypothesis\n" "File header")

For this I created a new function hypothesis-update, inspired by the hypothesis-to-archive function:

;;;###autoload
(defun hypothesis-update ()
  (interactive)
  (let ((last-update (hypothesis-last-archive-update)))
    (hypothesis-request
     (lambda (sites)
       (find-file hypothesis-archive)
       (goto-char (point-max))
       ;; Write or update header
       (if (eq (point-max) 17)
           (save-excursion
             (goto-char (point-min))
             (kill-whole-line)
             (insert (format "%s" hypothesis-file-header))
             (insert "#+LAST_UPDATE:" hypothesis--last-update "\n")
             (insert "\n")
             )
         (save-excursion
           (re-search-backward "^#\\+LAST_UPDATE:\\(.*\\)")
           (kill-whole-line)
           (insert "#+LAST_UPDATE:" hypothesis--last-update "\n")
           )
         )
       ;; Write content
       (goto-char (point-max))
       (if (seq-empty-p sites)
           (message "Nothing new since last import.")
         (save-excursion
           (org-insert-time-stamp (current-time) nil t "* Imported on " "\n\n")
           (let ((hypothesis--site-level 2))
             (mapc #'my/hypothesis-insert-site-data sites)
             )
           )
         )
       (save-buffer)
       )
     (append (when last-update `(("search_after" . ,last-update))) `(("limit" . 200) ("order" . "asc"))))))

As I don't know anything about elisp it's a bit confusing, sorry. Basically the function adds the header if the file is empty and updates #+LAST_UPDATE: if the file already exists, then the content is added.

I had to do a bit of an ugly hack to make it work. Indeed, to check that the file is newly created I use (eq (point-max) 17) and not (eq (point-max) 1) which would be more logical. The problem is that the function hypothesis-last-archive-update inserts the string #+LAST_UPDATE: at the top of the file as soon as it is created and I haven't managed to change this behaviour, so if the file contains 17 characters (i.e. #+LAST_UPDATE:) I empty it and insert the header, which is not satisfying, but it works for my personal use.

Add a hierarchical level in the file

Currently all the annotations of a single site (if imported at the same time) are grouped together in a level 2 title. I think it's a pity that each of them doesn't have its own title. So I created a function my/hypothesis-insert-site-data (Adapted from the function hypothesis-insert-site-data) which allows to obtain an archive file like this :

* Import date
** Website annoted
*** Annotation 1
*** Annotation 2

Add some informations in the file

There is some information transmitted by the hypothesis api that I would like to have in the archive file. So I built the my/hypothesis-insert-site-data function to add this information and I modified the hypothesis-data function to retrieve more information.

Here is my version of hypothesis-data :

(defun hypothesis-data (row)
  "Parse data from ROW into an alist."
  (let (
        (id (alist-get 'id row))
        (text (alist-get 'text row))
        (highlight (hypothesis--selector-key "TextQuoteSelector" 'exact row))
        (location-start (hypothesis--selector-key "TextPositionSelector" 'start row))
        (location-end (hypothesis--selector-key "TextPositionSelector" 'end row))
        )
    `((id . ,id)
      (uri . ,(alist-get 'uri row))
      (title . ,(elt (alist-get 'title (alist-get 'document row)) 0))
      (text . ,text)
      (highlight . ,highlight)
      (creation-time . ,(hypothesis-parse-iso-date (alist-get 'created row)))
      (update-time . ,(hypothesis-parse-iso-date (alist-get 'updated row)))
      (user . ,(alist-get 'user row))
      (tags . ,(alist-get 'tags row))
      (group . ,(alist-get 'group row))
      (location-start . ,location-start)
      (location-end . ,location-end)
      (incontext-link . ,(alist-get 'incontext (alist-get 'links row)))
      (type . ,(cond
                ((string-empty-p text) 'highlight)
                (highlight 'annotation)
                (t 'page-note))))))

I don't use all the information that the hypothesis-data function returns, but I think it would be interesting to exploit it.

Here is my my/hypothesis-insert-site-data function:

(defun my/hypothesis-insert-site-data (site)
  "Insert the data from SITE as `org-mode' text."
  (insert (format "%s [[%s][%s]]\n\n"
                  (make-string hypothesis--site-level ?*)
                  (car site)
                  (alist-get 'title (cadr site))))
  (dolist (x (sort (cdr site)
                   (lambda (row1 row2)
                     (< (or (alist-get 'location-start row1) 0)
                        (or (alist-get 'location-start row2) 0)))))
    (insert (format "%s [[%s][%s]]\n"
                    (make-string (+ hypothesis--site-level 1) ?*)
                    (alist-get 'incontext-link x)
                    (alist-get 'title (cadr site))
                    )
            )
    (org-set-property "Hypothesis_ID" (alist-get 'id x "error"))
    (org-set-property "Hypothesis_user" (alist-get 'user x "error"))
    (org-set-property "Hypothesis_group" (alist-get 'group x "error"))
    
    (org-insert-time-stamp (alist-get 'update-time x) t t nil "\n")
    (when-let ((highlight (alist-get 'highlight x)))
      (insert (format "%s\n%s\n%s"
                      hypothesis-quote-prefix
		      highlight
                      hypothesis-quote-sufix)))
    (when (eq 'annotation (alist-get 'type x))
      (insert "\n\n- "))
    (insert (concat (alist-get 'text x) "\n\n\n"))))

There are some modifications to the initial hypothesis-insert-site-data function.

Firstly, I am adding a new hierarchical level, as I said earlier. I said to myself that it was not necessary to allow the parameterisation of this level. So the number of "*" is the value of the parameter hypothesis--site-level plus 1. The title of this new level is a hyperlink whose value is the variable incontext-link. Clicking on the annotation can therefore be found in the context of the page (if the user is logged). On the other hand I did not find a relevant title, I thought of the id of the annotation on hypothesis, but it is not very readable, so I left the title of the page, which makes double with the higher hierarchical level. Maybe a solution would be to compute a local id of the form PageTitle_AnnotationNumber.

Secondly I have added some information in org-mode properties, such as id, group and user. For the moment I don't use them, but I think it's interesting to keep them for more advanced uses (sorting, selection, etc.).

To give an example of the modifications, here is what my archive file looks like :

#+TITLE: my_title
#+ROAM_TAGS: tag1 tag2 tag3
#+LAST_UPDATE:2021-01-06T17:48:00.572906+00:00

* Imported on [timestamp]
** [[WEBPAGE_URL][WEBPAGE_TITLE]]

*** [[ANNOTATION_INCONTEXT_URL][WEBPAGE_TITLE]]
    :PROPERTIES:
    :Hypothesis_ID: ANNOTATION_ID
    :Hypothesis_user: acct:[email protected]
    :Hypothesis_group: GROUPID
    :END:
[TIMESTAMP]
#+BEGIN_QUOTE
MY QUOTE
#+END_QUOTE

- MY COMMENT


*** [[ANNOTATION_INCONTEXT_URL][WEBPAGE_TITLE]]
    :PROPERTIES:
    :Hypothesis_ID: ANNOTATION_ID
    :Hypothesis_user: acct:[email protected]
    :Hypothesis_group: GROUPID
    :END:
[TIMESTAMP]
#+BEGIN_QUOTE
MY QUOTE
#+END_QUOTE

- MY COMMENT

Other proposals, not implemented

I also have some other proposals but they are much more complex and I am not able to propose an implementation.

Add tags

The Hypothesis api allows you to export the tags, but I haven't managed to use them, I think it would be interesting to exploit this information, especially since org-mode offers tag management.

Changing the file organisation

Currently the annotations are grouped by date of import. I would love to have them grouped by site, but that would require rewriting the file each time you import.

Allow the parameterization of the writing order

To extend the previous proposal it would be really interesting to propose to the user to choose how the first hierarchical level is constructed, by date of import, by site, by user, by group, etc.

Allow sorting and filtering

It would be really cool to have functions to select only some annotations, for example only a group or the annotations having a particular tag. I don't know if it's possible with org-mode (it seems to me that it is but I don't know) or if it's necessary to build a particular interface as proposed by the pocket-reader.el package.

Fragmenting files

It might be interesting for users with a lot of annotations to be able to create multiples archive files, for example one per group or per site.

Read user and password from authinfo.gpg

Hi @Kungsgeten,

is it possible to read the credentials from the .authinfo.gpg.
I'm an Elisp noob, therefore I can't tell from the code...

Thanks for creating this! Looking forward to integrating it into my Building A Second Brain workflow 😄

Usage Problem about the 200 limit

I usually this package downloading periodically all the notes and saving them to a file. It worked well until I reached the hard limit of 200. Now only the 200 latest annotations are synced. How could I import all the annotations?

noob setup problem

I tried to modify the .el file like this:
(defvar hypothesis-username "example")
(defvar hypothesis-token "66354345")
(defvar hypothesis-archive (expand-file-name "hypothesis.org" org-directory)

Where of course in place of "example" i put my username... Then I M-x and load the file. Then if I try to launch hypothesis-to-org i get the following error:

user-error: ‘hypothesis-token’ and ‘hypothesis-username’ needs to be set

Am I missing something?

Best!