shivin9/fpstream

Mining Frequent Data patterns for Anytime streams

fpstream's Issues

Empty buffers one level down before pruning

Implement Coalescing buffers

No need to push all the buffers down every time as it slows things down... collect a lot of itemsets and then push them down in a batch.

no pruning for FP-Trees

fp trees are not getting pruned based on the epsilon rule.

Buffer operations are slow

Better to use some form of hash table.

Notes

• Max_buffer_size depends on tree and the level within it are getting filled up much
• Intermittent pruning is a must to decrease insertion time (200s vs 136s for 1MD100 eps = 0.005)
• Breaking up the atomic operation leads to performance gain
• BufferSize doesn't seem to impact the recall, rather it increases insertion time
• Removing continuous subtree pruning does wonders to time and memory

prune_helper function is very slow

It would be good if we can use the header table for pruning the tree. Problem is of invalid pointers though.

Fix memory leaks

Lots of memory is being leaked

Changes in hash_buffer

Remove DLL, remove oldest buffer item
Cleaning time of buffer table

• if size of buffer > 90% max_size --> prune
• if delta(time) > eps*N

Delayed insertion criteria (insufficient count) (theta on support count)

• if less, wont insert the node in the queue
• deferring with time (due to less time)

Later

• Make transactions as dynamic arrays