jinyyu/tlsh_gist

tlsh_gist

A PostgreSQL plugin for tlsh hash, whitch is a fuzzy matching program and library, Similar files will have similar hash values which allows for the detection of similar objects by comparing their hash values TLSH

see: https://tlsh.org/

doc: https://zhuanlan.zhihu.com/p/497732848

build

    git clone https://github.com/trendmicro/tlsh.git
    cd tlsh
    mkdir build
    cd build
    cmake .. -D TLSH_SHARED_LIBRARY=1
    make && make install

    # make sure that pg_config is in PATH
    git clone https://github.com/jinyyu/tlsh_gist.git
    cd tlsh_gist
    USE_PGXS=1 make
    USE_PGXS=1 make install

usage

    create extension tlsh_gist;
    create table test_tlsh(v tlsh);
    CREATE INDEX tlsh_idx ON test_tlsh USING GIST (v gist_tlsh_ops);

    -- index scan
    SELECT v, '30e5be217761c8b1c27213314d18b62a58bdbdf02b35d2ababe9352c5e780c1b677795' <-> v AS dist FROM test_tlsh ORDER BY dist LIMIT 1;

    --seq scan
    select v, tlsh_dist(v, '30e5be217761c8b1c27213314d18b62a58bdbdf02b35d2ababe9352c5e780c1b677795') as dist FROM test_tlsh ORDER BY dist LIMIT 1;

    -- % operator
    select v from test_tlsh where v % '30e5be217761c8b1c27213314d18b62a58bdbdf02b35d2ababe9352c5e780c1b677795';

tlsh_gist's Issues

build failed on pg14

USE_PGXS=1 make

g++ -Wall -Wpointer-arith -Wendif-labels -Wmissing-format-attribute -Wimplicit-fallthrough=3 -Wcast-function-type -Wformat-security -fn
o-strict-aliasing -fwrapv -g -g -O2 -flto=auto -ffat-lto-objects -flto=auto -ffat-lto-objects -fstack-protector-strong -Wformat -Werror
=format-security -fPIC -I/usr/local/include -DBUCKETS_128 -I. -I./ -I/usr/include/postgresql/14/server -I/usr/include/postgresql/intern
al  -Wdate-time -D_FORTIFY_SOURCE=2 -D_GNU_SOURCE -I/usr/include/libxml2   -c -o tlsh_gist.o tlsh_gist.cpp                             
In file included from /usr/include/postgresql/14/server/port/atomics.h:70,                                                             
                 from /usr/include/postgresql/14/server/utils/dsa.h:17,                                                                
                 from /usr/include/postgresql/14/server/nodes/tidbitmap.h:26,                                                          
                 from /usr/include/postgresql/14/server/access/genam.h:19,                                                             
                 from /usr/include/postgresql/14/server/access/amapi.h:15,                                                             
                 from /usr/include/postgresql/14/server/access/reloptions.h:22,                                                        
                 from tlsh_gist.cpp:4:                                                                                                 
/usr/include/postgresql/14/server/port/atomics/arch-x86.h: In function ‘bool pg_atomic_test_set_flag_impl(volatile pg_atomic_flag*)’:  
/usr/include/postgresql/14/server/port/atomics/arch-x86.h:143:23: warning: ISO C++17 does not allow ‘register’ storage class specifier 
[-Wregister]                                                                                                                           
  143 |         register char _res = 1;                                                                                                
      |                       ^~~~                                                                                                     
In file included from tlsh_gist.h:7,                                                                                                   
                 from tlsh_gist.cpp:1:                                                                                                 
tlsh_gist.cpp: At global scope:                                                                                                        
/usr/include/postgresql/14/server/fmgr.h:424:12: error: conflicting declaration of ‘int no_such_variable’ with ‘C’ linkage             
  424 | extern int no_such_variable                                                                                                    
      |            ^~~~~~~~~~~~~~~~                                                                                                    
tlsh_gist.cpp:12:5: note: in expansion of macro ‘PG_FUNCTION_INFO_V1’                                                                  
   12 |     PG_FUNCTION_INFO_V1(tlsh_mean);                                                                                            
      |     ^~~~~~~~~~~~~~~~~~~                                                                                                        
In file included from /usr/include/postgresql/14/server/catalog/pg_attribute.h:25,                                                     
                 from /usr/include/postgresql/14/server/access/tupdesc.h:18,                                                           
                 from /usr/include/postgresql/14/server/utils/relcache.h:17,                                                           
                 from /usr/include/postgresql/14/server/access/genam.h:21,                                                             
                 from /usr/include/postgresql/14/server/access/amapi.h:15,                                                             
                 from /usr/include/postgresql/14/server/access/reloptions.h:22,                                                        
                 from tlsh_gist.cpp:4:                                                                                                 
/usr/include/postgresql/14/server/catalog/genbki.h:81:61: note: previous declaration with ‘C++’ linkage                                
   81 | #define DECLARE_UNIQUE_INDEX_PKEY(name,oid,decl) extern int no_such_variable                                                   
      |                                                             ^~~~~~~~~~~~~~~~                                                   
/usr/include/postgresql/14/server/catalog/pg_attribute.h:211:1: note: in expansion of macro ‘DECLARE_UNIQUE_INDEX_PKEY’                
  211 | DECLARE_UNIQUE_INDEX_PKEY(pg_attribute_relid_attnum_index, 2659, on pg_attribute using btree(attrelid oid_ops, attnum int2_ops)
);                                                                                                                                     
      | ^~~~~~~~~~~~~~~~~~~~~~~~~                                                                                                      
make: *** [<builtin>: tlsh_gist.o] Error 1                                                                                             

License?

Could you add a license to your project? I'd like to incorporate it in MalwareDB, and use this to learn how to do something similar for other similarity hashes, if possible.

Refs: malwaredb/malwaredb-rs#165