long claim filter makes the process extremly slow

[maxlath]

long claim filter makes the process extremly slow

[hsaif]

Agree, a Hash-like structure should a quick and sufficient solution!

[maxlath]

so I wrote the hash implementation in ff1da9f, published in v2.3.1, and added documentation to parallelize the operations

[hsaif]

Thanks Max,

I ran few tests with the hashed claims. The code run very fast in the beginning, but it gets significantly slower after processing the first 1M entities

I tried to run the process with parallelism as you described, but it didn't run for some reason. I'll try to debug and gets back to you

[maxlath]

I think the parallelization is just a boost to get things done faster but if it doesn't work with a single process, that's there is something wrong already, putting several CPU at it won't help

[hsaif]

Looks like I wasn't clear enough in my previous comment. Apologies for that.

Basically, I first ran the process without parallelisation, and that's where the performance slowed down after processing the first ~1M entities

In another test, I ran process with parallelism, but it didn't work.

Hope it's clear now :)

[maxlath]

could you share the code you're using so that I could try to reproduce the issue?

[hsaif]

cat latest-all.json.gz | gzip -d | wdfilter --claim ./claims_interest.txt > entity_filtered.json

With parallelism:

nice -n+19 pigz -d < latest-all.json.gz | nice -n+19 load-balance-lines wdfilter --claim ./claims_interest.txt > entity_pra_filtered.json

Attached the claims_interest.txt

claims_interest.txt