simonhf/sharedhashfile

Share Hash Tables With Stable Key Hints Stored In Memory Mapped Files Between Arbitrary Processes

sharedhashfile's Issues

possible to provide golang binding for this?

possible to provide golang binding for this?

thx in advance!

trying to write golang binding but faced an issue

i can contribute the golang binding client but what's happening to this? i've tried generating
gcc -shared -fPIC -ftls-model=initial-exec -o libSharedHashFile.so shf.c

and it doesnt work. i've already done the libSharedHashFile.a into /usr/local/lib in ubuntu 22.04 and having same issues too

CGO_CFLAGS="-I/usr/local/include" CGO_LDFLAGS="-L/usr/local/lib -lSharedHashFile -lstdc++ -lm -lz -pthread -lrt -ldl" go build main.go
# command-line-arguments
/usr/local/go/pkg/tool/linux_amd64/link: running gcc failed: exit status 1
/usr/bin/ld: shf_val: TLS definition in /usr/local/lib/libSharedHashFile.so section .tbss mismatches non-TLS reference in /tmp/go-link-3683685780/go.o
/usr/bin/ld: /usr/local/lib/libSharedHashFile.so: error adding symbols: bad value
collect2: error: ld returned 1 exit status

shf_put_key_val

Hi,

shf_put_key_val, can we insert by "key", without knowing the uuid? And also, how to get the uuid from a Key? Please post just a few examples.

Would be really helpful! Thanks so much

Key value pairs

Is it possible to store value as a struct ? Like in glib ghashtable does?

Donation for an iterator

Hi Simon, how are you?

Are you willing to build a solid & fast Iterator if ever I'll send you a good donation? :)
My company is asking me but I'm not sure I would have the talent to do it myself.

Thanks!

python bindings progress

It seems that there was interest for this project to provide python bindings. any progress on that front? (I'd vote for using cffi to do this).

Also: Does this project support hash tables without using mmap. Pure named shared memory implementation? I don't need persistence, but would be nice to have write locks on particular key, value pairs.

SHF_MAKE_HASH

Hi,

Can you give me slighty more infos about : SHF_MAKE_HASH
I can insert a value and then getting the Uuid and retrieve it, but how do i use the SHF_MAKE_HASH?

I mean, can I retrieve DIRECTLY a value from a Key without having to compute the hash?
Thanks so much again

Also, how do I directly retrieve a value from a uuid I have?

use xxh3 instead of murmur

use xxh3 instead of murmur

possible to resolve the issue for golang?

i can sponsor a few coffees for this.

also, the server is running in "pre-fork mode", means one process per thread. so there's thread isolation "goroutine is using gnet's ants" https://github.com/panjf2000/ants or bytedance's gopool

pls do a golang binding for this and reduce the issues for golang with the use case scenario i've mentioned above.

with ref to this:
#22

thx in advance! really appreciate this. will repay ur kindness in future in other ways.

Updating values

Morning!

How do you actually update values of a specific key, after I use SHF_MAKE_HASH.
How do I replace my value "HELLO" to "GOODB".

I've noticed in your samples "shf_upd_key_val", but I don't see any parameters to it outside the SHF set.
Have a great day Simon.

what happen if row is full and all keys are in same logic table

Add 16 keys into shf. It's possible that all the keys locate in same row and with same logic table.
In this situation, split is useless. if i add another key, still locate in same row and with same logic table, it will fail.

compilation error on ubuntu 22.04

make and make clean failed with ubuntu 22.04

make
make: 0 line(s) with eol whitespace
make: 0 line(s) with tab
make: compiling: release-gcc/main.shf.log.o
make: compiling: release-gcc/murmurhash3.o
make: compiling: release-gcc/shf.o
In file included from src/shf.private.h:29,
                 from src/shf.c:46:
src/shf.c: In function ‘shf_put_key_val’:
src/shf.c:805:45: error: taking address of packed member of ‘struct SHF_WIN_MMAP’ may result in an unaligned pointer value [-Werror=address-of-packed-member]
  805 |     if (shf->is_lockable) { SHF_LOCK_WRITER(&shf->shf_mmap->wins[win].lock); }
      |                                             ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
src/shf.lock.h:299:111: note: in definition of macro ‘SHF_LOCK_WRITER’
  299 | #define SHF_LOCK_WRITER(LOCK)   SHF_DEBUG("- rw locking for writer %s()\n", __FUNCTION__); shf_rw_lock_writer(LOCK)
      |                                                                                                               ^~~~
src/shf.c:838:47: error: taking address of packed member of ‘struct SHF_WIN_MMAP’ may result in an unaligned pointer value [-Werror=address-of-packed-member]
  838 |     if (shf->is_lockable) { SHF_UNLOCK_WRITER(&shf->shf_mmap->wins[win].lock); }
      |                                               ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
src/shf.lock.h:300:113: note: in definition of macro ‘SHF_UNLOCK_WRITER’
  300 | #define SHF_UNLOCK_WRITER(LOCK)                                                            shf_rw_unlock_writer(LOCK); SHF_DEBUG("- rw unlocked for writer %s()\n", __FUNCTION__);
      |                                                                                                                 ^~~~
src/shf.c:850:47: error: taking address of packed member of ‘struct SHF_WIN_MMAP’ may result in an unaligned pointer value [-Werror=address-of-packed-member]
  850 |     if (shf->is_lockable) { SHF_UNLOCK_WRITER(&shf->shf_mmap->wins[win].lock); }
      |                                               ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
src/shf.lock.h:300:113: note: in definition of macro ‘SHF_UNLOCK_WRITER’
  300 | #define SHF_UNLOCK_WRITER(LOCK)                                                            shf_rw_unlock_writer(LOCK); SHF_DEBUG("- rw unlocked for writer %s()\n", __FUNCTION__);
      |                                                                                                                 ^~~~
src/shf.c: In function ‘shf_find_key_internal’:
src/shf.c:907:98: error: taking address of packed member of ‘struct SHF_WIN_MMAP’ may result in an unaligned pointer value [-Werror=address-of-packed-member]
  907 |     ||     (SHF_FIND_KEY_OR_UID_AND_ATOM_ADD == what)) { if (shf->is_lockable) { SHF_LOCK_READER(&shf->shf_mmap->wins[win].lock); }}
      |                                                                                                  ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
src/shf.lock.h:297:111: note: in definition of macro ‘SHF_LOCK_READER’
  297 | #define SHF_LOCK_READER(LOCK)   SHF_DEBUG("- rw locking for reader %s()\n", __FUNCTION__); shf_rw_lock_reader(LOCK)
      |                                                                                                               ^~~~
src/shf.c:908:98: error: taking address of packed member of ‘struct SHF_WIN_MMAP’ may result in an unaligned pointer value [-Werror=address-of-packed-member]
  908 |     else /* SHF_FIND_KEY_OR_UID_AND_(DELETE|UPDATE) */ { if (shf->is_lockable) { SHF_LOCK_WRITER(&shf->shf_mmap->wins[win].lock); }}
      |                                                                                                  ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
src/shf.lock.h:299:111: note: in definition of macro ‘SHF_LOCK_WRITER’
  299 | #define SHF_LOCK_WRITER(LOCK)   SHF_DEBUG("- rw locking for writer %s()\n", __FUNCTION__); shf_rw_lock_writer(LOCK)
      |                                                                                                               ^~~~
src/shf.c:1024:100: error: taking address of packed member of ‘struct SHF_WIN_MMAP’ may result in an unaligned pointer value [-Werror=address-of-packed-member]
 1024 |     ||     (SHF_FIND_KEY_OR_UID_AND_ATOM_ADD == what)) { if (shf->is_lockable) { SHF_UNLOCK_READER(&shf->shf_mmap->wins[win].lock); }}
      |                                                                                                    ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
src/shf.lock.h:298:113: note: in definition of macro ‘SHF_UNLOCK_READER’
  298 | #define SHF_UNLOCK_READER(LOCK)                                                            shf_rw_unlock_reader(LOCK); SHF_DEBUG("- rw unlocked for reader %s()\n", __FUNCTION__);
      |                                                                                                                 ^~~~
src/shf.c:1025:100: error: taking address of packed member of ‘struct SHF_WIN_MMAP’ may result in an unaligned pointer value [-Werror=address-of-packed-member]
 1025 |     else /* SHF_FIND_KEY_OR_UID_AND_(DELETE|UPDATE) */ { if (shf->is_lockable) { SHF_UNLOCK_WRITER(&shf->shf_mmap->wins[win].lock); }}
      |                                                                                                    ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
src/shf.lock.h:300:113: note: in definition of macro ‘SHF_UNLOCK_WRITER’
  300 | #define SHF_UNLOCK_WRITER(LOCK)                                                            shf_rw_unlock_writer(LOCK); SHF_DEBUG("- rw unlocked for writer %s()\n", __FUNCTION__);
      |                                                                                                                 ^~~~
src/shf.c: In function ‘shf_tab_copy_iterate’:
src/shf.c:1154:45: error: taking address of packed member of ‘struct SHF_WIN_MMAP’ may result in an unaligned pointer value [-Werror=address-of-packed-member]
 1154 |     if (shf->is_lockable) { SHF_LOCK_READER(&shf->shf_mmap->wins[win].lock); }
      |                                             ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
src/shf.lock.h:297:111: note: in definition of macro ‘SHF_LOCK_READER’
  297 | #define SHF_LOCK_READER(LOCK)   SHF_DEBUG("- rw locking for reader %s()\n", __FUNCTION__); shf_rw_lock_reader(LOCK)
      |                                                                                                               ^~~~
src/shf.c:1175:47: error: taking address of packed member of ‘struct SHF_WIN_MMAP’ may result in an unaligned pointer value [-Werror=address-of-packed-member]
 1175 |     if (shf->is_lockable) { SHF_UNLOCK_READER(&shf->shf_mmap->wins[win].lock); }
      |                                               ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
src/shf.lock.h:298:113: note: in definition of macro ‘SHF_UNLOCK_READER’
  298 | #define SHF_UNLOCK_READER(LOCK)                                                            shf_rw_unlock_reader(LOCK); SHF_DEBUG("- rw unlocked for reader %s()\n", __FUNCTION__);
      |                                                                                                                 ^~~~
src/shf.c: In function ‘shf_q_flush’:
src/shf.c:1709:49: error: taking address of packed member of ‘struct SHF_Q_LOCK_MMAP’ may result in an unaligned pointer value [-Werror=address-of-packed-member]
 1709 |     shf_debug_verbosity_less(); SHF_LOCK_WRITER(&shf->q.q_lock->lock); shf_debug_verbosity_more();
      |                                                 ^~~~~~~~~~~~~~~~~~~~
src/shf.lock.h:299:111: note: in definition of macro ‘SHF_LOCK_WRITER’
  299 | #define SHF_LOCK_WRITER(LOCK)   SHF_DEBUG("- rw locking for writer %s()\n", __FUNCTION__); shf_rw_lock_writer(LOCK)
      |                                                                                                               ^~~~
src/shf.c:1790:51: error: taking address of packed member of ‘struct SHF_Q_LOCK_MMAP’ may result in an unaligned pointer value [-Werror=address-of-packed-member]
 1790 |     shf_debug_verbosity_less(); SHF_UNLOCK_WRITER(&shf->q.q_lock->lock); shf_debug_verbosity_more();
      |                                                   ^~~~~~~~~~~~~~~~~~~~
src/shf.lock.h:300:113: note: in definition of macro ‘SHF_UNLOCK_WRITER’
  300 | #define SHF_UNLOCK_WRITER(LOCK)                                                            shf_rw_unlock_writer(LOCK); SHF_DEBUG("- rw unlocked for writer %s()\n", __FUNCTION__);
      |                                                                                                                 ^~~~
src/shf.c: In function ‘shf_log_prefix_get’:
src/shf.c:2057:33: error: taking address of packed member of ‘struct SHF_LOG_MMAP’ may result in an unaligned pointer value [-Werror=address-of-packed-member]
 2057 |                 SHF_LOCK_WRITER(&shf->log->lock);   /* vvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvv */
      |                                 ^~~~~~~~~~~~~~~
src/shf.lock.h:299:111: note: in definition of macro ‘SHF_LOCK_WRITER’
  299 | #define SHF_LOCK_WRITER(LOCK)   SHF_DEBUG("- rw locking for writer %s()\n", __FUNCTION__); shf_rw_lock_writer(LOCK)
      |                                                                                                               ^~~~
src/shf.c:2062:35: error: taking address of packed member of ‘struct SHF_LOG_MMAP’ may result in an unaligned pointer value [-Werror=address-of-packed-member]
 2062 |                 SHF_UNLOCK_WRITER(&shf->log->lock); /* ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ */
      |                                   ^~~~~~~~~~~~~~~
src/shf.lock.h:300:113: note: in definition of macro ‘SHF_UNLOCK_WRITER’
  300 | #define SHF_UNLOCK_WRITER(LOCK)                                                            shf_rw_unlock_writer(LOCK); SHF_DEBUG("- rw unlocked for writer %s()\n", __FUNCTION__);
      |                                                                                                                 ^~~~
src/shf.c: In function ‘shf_log_thread’:
src/shf.c:2178:25: error: taking address of packed member of ‘struct SHF_LOG_MMAP’ may result in an unaligned pointer value [-Werror=address-of-packed-member]
 2178 |         SHF_LOCK_WRITER(&shf->log->lock);   /* vvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvv */
      |                         ^~~~~~~~~~~~~~~
src/shf.lock.h:299:111: note: in definition of macro ‘SHF_LOCK_WRITER’
  299 | #define SHF_LOCK_WRITER(LOCK)   SHF_DEBUG("- rw locking for writer %s()\n", __FUNCTION__); shf_rw_lock_writer(LOCK)
      |                                                                                                               ^~~~
src/shf.c:2181:27: error: taking address of packed member of ‘struct SHF_LOG_MMAP’ may result in an unaligned pointer value [-Werror=address-of-packed-member]
 2181 |         SHF_UNLOCK_WRITER(&shf->log->lock); /* ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ */
      |                           ^~~~~~~~~~~~~~~
src/shf.lock.h:300:113: note: in definition of macro ‘SHF_UNLOCK_WRITER’
  300 | #define SHF_UNLOCK_WRITER(LOCK)                                                            shf_rw_unlock_writer(LOCK); SHF_DEBUG("- rw unlocked for writer %s()\n", __FUNCTION__);
      |                                                                                                                 ^~~~
src/shf.c:2198:33: error: taking address of packed member of ‘struct SHF_LOG_MMAP’ may result in an unaligned pointer value [-Werror=address-of-packed-member]
 2198 |                 SHF_LOCK_WRITER(&shf->log->lock);   /* vvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvv */
      |                                 ^~~~~~~~~~~~~~~
src/shf.lock.h:299:111: note: in definition of macro ‘SHF_LOCK_WRITER’
  299 | #define SHF_LOCK_WRITER(LOCK)   SHF_DEBUG("- rw locking for writer %s()\n", __FUNCTION__); shf_rw_lock_writer(LOCK)
      |                                                                                                               ^~~~
src/shf.c:2202:35: error: taking address of packed member of ‘struct SHF_LOG_MMAP’ may result in an unaligned pointer value [-Werror=address-of-packed-member]
 2202 |                 SHF_UNLOCK_WRITER(&shf->log->lock); /* ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ */
      |                                   ^~~~~~~~~~~~~~~
src/shf.lock.h:300:113: note: in definition of macro ‘SHF_UNLOCK_WRITER’
  300 | #define SHF_UNLOCK_WRITER(LOCK)                                                            shf_rw_unlock_writer(LOCK); SHF_DEBUG("- rw unlocked for writer %s()\n", __FUNCTION__);
      |                                                                                                                 ^~~~
src/shf.c: In function ‘shf_log_append’:
src/shf.c:2359:25: error: taking address of packed member of ‘struct SHF_LOG_MMAP’ may result in an unaligned pointer value [-Werror=address-of-packed-member]
 2359 |         SHF_LOCK_WRITER(&shf->log->lock);   /* vvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvv */
      |                         ^~~~~~~~~~~~~~~
src/shf.lock.h:299:111: note: in definition of macro ‘SHF_LOCK_WRITER’
  299 | #define SHF_LOCK_WRITER(LOCK)   SHF_DEBUG("- rw locking for writer %s()\n", __FUNCTION__); shf_rw_lock_writer(LOCK)
      |                                                                                                               ^~~~
src/shf.c:2372:27: error: taking address of packed member of ‘struct SHF_LOG_MMAP’ may result in an unaligned pointer value [-Werror=address-of-packed-member]
 2372 |         SHF_UNLOCK_WRITER(&shf->log->lock); /* ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ */
      |                           ^~~~~~~~~~~~~~~
src/shf.lock.h:300:113: note: in definition of macro ‘SHF_UNLOCK_WRITER’
  300 | #define SHF_UNLOCK_WRITER(LOCK)                                                            shf_rw_unlock_writer(LOCK); SHF_DEBUG("- rw unlocked for writer %s()\n", __FUNCTION__);
      |                                                                                                                 ^~~~
cc1: all warnings being treated as errors
make: *** [GNUmakefile:107: release-gcc/shf.o] Error 1

how to change the value

I should use "SHF_MAKE_HASH() "and "shf_put_key_val()" when need to add a key-value pair. but how to change a value ? If I use "shf_get_key_val_addr()" then change the value, it seems unsafe :(

Can't find documentation

Hi!

Thanks for this awesome lib.
I can't find the documentation for your lib? GET/PUT...?

Thanks Simon

Persistent storage ?

Hi,
Is there any options for persistent storage ?

Regards

possible to provide nim binding for this?

possible to provide nim binding for this?

Please give examples

Can you give some useful examples？I didn’t know how to use this awesome library?

API?

Is it a pure HashTable?
If yes why there is no api such:

void insert(Key key, Value value);

and

Value find(Key key);

And also where is the api doc?

Update values (append)

Hi,

Me again :)
I have my key:value, i'd like to update the value, add a string at the end.

Example, key : HELLO
Value: TEST-

I like to update my key HELLO with +VALUE TEST2, HELLO VALUE become TEST-TEST2
How can I do it without having to retrieve entirely the string? I'd like to concat at the end. I don't mind reallocating always.

PS: My values can be quite long, 100KB +

Would be very useful for me.

Thanks! :)

Support for LRU with expire

Thanks for the high performance sharedhashfile. Any plans to support LRU with expire similar to Nginx shared cache ?

Iterator

Hello Simon!

Do you happen to have an example of an Iterator? Reading the whole DB in an lexico order?
I'm considering switching from Lmdb but I can't without this feature.

Thank you

shf_put_key_val (bytes)

Hi,

i'm inserting a string of 155 char, but sometimes it could go up to 200, so i'm putting ", 200)" in the bytes.
But when I update with a value of 160 char for example and put 200 again, it gives me some weird character "�uIv�U" when i retrieve it again.

is there anyway to pre-allocate a larger amount of char?

Thanks :)

How to query entries in this database from php?

As the title says, how to query information in this database from php?

How do we do:

SELECT ID FROM table where col2=asdf;

or

SELECT * FROM table 2 left join table1 1 where col2=asdf;

Multithread

Hi,

When you lock shf_set_is_lockable, how do you unlock afterward? I don't find the function in test.9.shf.c.

I have a hard time doing some Get into 2 different threads. it always says:
mremap(): 14: Bad address

If i'm retrieving from a single thread, I have no issues.

Thanks again :)