dotslash/recent2

Swiss army knife to query bash history

• Knows a little bit of everything
• enjoys golang, python
  • IMO scala is gross
• you should use sqlite more often

Use recent2 and thank me later. Its a lot better than ctrl-r for searching bash history. I have easy access to 2+ yrs of bash history thanks to recent2

Misc recent things

• https://github.com/dotslash/cloudsync: 2 way file sync between Google Cloud Storage and local machine
• wordle solver. Also hosted as a google cloud function. It works except the date is behind by a few days because NYT started curating wordle words and removed a few words which NYT thinks are in appropriate
• https://storage.googleapis.com/yesteapea/gcp_publicload.py : Uploads individual file from your local machine to google cloud storage and makes them publicly available. https://storage.googleapis.com/yesteapea/gcp_publicload.py was uploaded to gcp by running gcp_publicload.py gcp_publicload.py -kn

recent2's Issues

Do performance testing

I dont know yet, about how many commands can sqlite take before making the tool "visibly slow".

Add some perf tests which record latencies for various db sizes.

Add logs to track operation latencies

1. time to log a history entry
2. extra latency due to recent2 incurred when i open a new shell

Also dont keep the logs for too long.

Check the latest version of recent once in while and suggest users to upgrade

Handle unicode chars properly

(bash) >15:01:56 $ å
bash: å: command not found
Traceback (most recent call last):
  File "/usr/local/bin/log-recent", line 10, in <module>
    sys.exit(log())
  File "/usr/local/lib/python2.7/site-packages/recent2.py", line 173, in log
    return_value, pwd, session.id])
sqlite3.ProgrammingError: You must not use 8-bit bytestrings unless you use a text_factory that can interpret 8-bit bytestrings (like text_factory = str). It is highly recommended that you instead just switch your application to Unicode strings.

Setup CI/CD

Allow PROMPT_COMMAND to invoke a function

I'd like to do the following in .bashrc:

log_on_success() {
	if [ $? -eq 0 ]
	then
        log-recent -r $? -c "$(HISTTIMEFORMAT= history 1)" -p $$
    fi
}

export PROMPT_COMMAND='log_on_success'

When I invoke the recent command, I get this error:

$ recent -h
PROMPT_COMMAND env variable is not set. Add the following line to .bashrc or .bash_profile
export PROMPT_COMMAND='log-recent -r $? -c "$(HISTTIMEFORMAT= history 1)" -p $$' 

Remove all uses of os.path in favor of pathlib.Path

pathlib is modern and much more easy to work with

Case sensitive search

sqlite like clause is case insensitive i.e a like "wow" is same as a like "WoW". Use PRAGMA case_sensitive_like to support this.

Incorrect number of bindings supplied

recent2/recent2.py

Line 284 in 3f14633

session.id, ""])

Traceback (most recent call last):
  File "/usr/local/bin/recent-import-bash-history", line 8, in <module>
    sys.exit(import_bash_history_entry_point())
  File "/usr/local/lib/python3.8/site-packages/recent2.py", line 228, in import_bash_history_entry_point
    import_bash_history()
  File "/usr/local/lib/python3.8/site-packages/recent2.py", line 280, in import_bash_history
    c.execute(SQL.INSERT_ROW_CUSTOM_TS, [
sqlite3.ProgrammingError: Incorrect number of bindings supplied. The current statement uses 6, and there are 7 supplied.

Python 3.8.2 (default, Feb 28 2020, 00:00:00)
[GCC 10.0.1 20200216 (Red Hat 10.0.1-0.8)] on linux

Distributor ID: Fedora
Description:    Fedora release 32 (Thirty Two)
Release:        32
Codename:       ThirtyTwo

workaround

            # exit status=-1, working directory=/unknown
            -1, "/unknown",
            session.id,])
    conn.commit()
    conn.close()

Dont use python f-strings

f-strings are introduced in python 3.6. So someone installing recent2 with an older python (lets say 3.4) will see errors.

Introduced in 03c6b7d

recent2 returns wrong number of results when --return_self is not set

The way Im filtering recent commands from output results in the util printing wrong number of results

recent -n 10 # returns 10 results
recent -n 10 # returns 9 results
recent -n 8 # returns 8 results

Capture whitelised env variables

1. Capture env vars whitelisted in a config file (~/.recentcfg)
2. Capture env vars that start with REV_ (recent env var)

Add an option to filter out duplicates

My database contains many log entries for commands that I use frequently. However, I'm only interested is the most recent use of any given command. I'd like to request that an option be added to the recent command to list only the most recent use of each command in the result set.

Exclude long commands by default

Due to copy paste errors our bash history might be corrupt. Commands that have more than 300(400?? 500??) characters most likely fall into this bucket.

Provide an option to set the max length of the commands to return. By default this will be something like 300.

Make default output copy friendly

Very often I run recent and then recent -ht to copy-paste the output. Make the default output copy-friendly and also contain the timestamp

Print each record like this

command1 # time1
longer_command1 # time2
cmd3 # time3

instead of

time1 command1
time2 longer_command1
time3 cmd3

Provide a way to return an equivalent sqlite command

Idea is that this sqlite command will serve as a starting point for further slice/dice/customizations

Link to discussions in the help message

bash: /data/users/miniconda3/envs/her2/bin/log-recent: No such file or directory

I uninstall the package and then i got the message every time I type anything. Do you know how to solve this?

Add support for zsh

Describe the bug
Logging does not occur correctly when using zsh instead of bash.

To Reproduce
Steps to reproduce the behavior:

1. pip install recent2
2. Add export PROMPT_COMMAND='log-recent -r $? -c "$(HISTTIMEFORMAT= history 1)" -p $$' to .zshrc
3. Start new shell

$ recent
# Output:

PROMPT_COMMAND env variable is not set. Add the following line to .bashrc or .bash_profile
export PROMPT_COMMAND='log-recent -r $? -c "$(HISTTIMEFORMAT= history 1)" -p $$'

Expected behavior

$ recent
# Output:
recent: building schema

Desktop
Darwin MacBook-Pro.lan 20.4.0 Darwin Kernel Version 20.4.0: Fri Mar 5 01:14:14 PST 2021; root:xnu-7195.101.1~3/RELEASE_X86_64 x86_64

Provide option to delete entries from history

Search for history around a command

I want to have an equivalent of grep's -C.
But it is hard to support "-C" for multiple results. But it might suffice to say "give me 10 commands in the session around the timestamp x"

Ability to turn off and turn on history logging

Indicate whether the command succeeded or failed

Options

• failed commands would be highlighted in red
• for failed commands the timestamp is red
• provide a generic way to return more command columns
  (e.g return status, pwd etc)

Modify log-recent to support direct calls

I wrote a bash script that can execute commands using the eval built-in. I'd like to log these commands by calling log-recent directly but it looks like this capability isn't supported.

Provide a way to return commands from the current session only

Add unit tests

Deprecation warnings during dev install

Not sure what these warnings mean:

recent2 (master) $ pip3 install -e .
DEPRECATION: Configuring installation scheme with distutils config files is deprecated and will no longer work in the near future. If you are using a Homebrew or Linuxbrew Python, please see discussion at https://github.com/Homebrew/homebrew-core/issues/76621
Obtaining file:///Users/Core/Develop/Tools/recent2
Requirement already satisfied: tabulate in /Users/Core/homebrew/lib/python3.9/site-packages (from recent2==1.0.2) (0.8.9)
Installing collected packages: recent2
  Attempting uninstall: recent2
    Found existing installation: recent2 1.0.2
    Uninstalling recent2-1.0.2:
      Successfully uninstalled recent2-1.0.2
  DEPRECATION: Configuring installation scheme with distutils config files is deprecated and will no longer work in the near future. If you are using a Homebrew or Linuxbrew Python, please see discussion at https://github.com/Homebrew/homebrew-core/issues/76621
  Running setup.py develop for recent2
Successfully installed recent2-1.0.2

Support searching via ctrl-r backed by recent

Feature improvements and enhancements for recent2.

On Ubuntu 20.04.2 LTS, I use the recent2 to log and query my bash history, which uses a sqlite3 database to store the history information. The datetime of running a command is inserted in the sqlite3 database with a unixepoch format, and it will be converted into localtime accordingly when retrieved and displayed later.

But I found that it did not perform the correct conversion according to the time zone setting on the machine, as shown below:

werner@X10DAi-00:~$ rm 222222222222222222
rm: cannot remove '222222222222222222': No such file or directory
werner@X10DAi-00:~$ recent -fo -w . 2222222222222
rm 222222222222222222                              # rtime@ 2021-08-29 10:57:13
werner@X10DAi-00:~$ date 
Sun 29 Aug 2021 06:57:22 PM CST

I also filed an issue here. Any hints for this problem?

Regards,
HY