kemonomachi / experhash Goto Github PK
View Code? Open in Web Editor NEWPerceptual hashing of images in Elixir
License: Do What The F*ck You Want To Public License
Perceptual hashing of images in Elixir
License: Do What The F*ck You Want To Public License
When using this package as a dependency in another project, the priv
dir is not created.
This causes the symlink into the build directory to not be created by mix
.
The solution is:
.keep
file in the priv
dir.priv
from .gitignore
and instead explicitly ignore the generated bin file priv/experhash
.Hey I'm not really sure where to ask this, or if this is the appropriate place but figured you may have the most experience to address this or at least point me in the right direction.
I installed Imagemagick(7) through brew on MacOS Sierra (10.12) with seemingly no issue. I'm able to run magick
in the command line etc. However when I try to run Experhash I receive the following error:
dyld: Library not loaded: /usr/local/lib/libMagick++-6.Q16.5.dylib
Referenced from: /Users/plato/Desktop/test_app/api/_build/test/lib/experhash/priv/experhash_port
Reason: image not found
When I go and check the /usr/local/lib/
directory I see the following files exist:
libMagick++-7.Q16HDRI.4.dylib
libMagick++-7.Q16HDRI.dylib
but not libMagick++-6.Q16.5.dylib
which I assume makes sense since I installed 7. Now I guess the confusion I have is that it seems the Experhash results are different than running Imagemagick directly from the command line which leads me to believe it may have to do with the makefile (https://github.com/kemonomachi/experhash/blob/master/Makefile#L10).
If it helps at all when I run,
Magick++-config --cppflags --cxxflags --ldflags --libs
in the command line I get the following back:
-DMAGICKCORE_HDRI_ENABLE=1 -DMAGICKCORE_QUANTUM_DEPTH=16 -DMAGICKCORE_HDRI_ENABLE=1 -DMAGICKCORE_QUANTUM_DEPTH=16 -DMAGICKCORE_HDRI_ENABLE=1 -DMAGICKCORE_QUANTUM_DEPTH=16 -I/usr/local/Cellar/imagemagick/7.0.7-30/include/ImageMagick-7
-DMAGICKCORE_HDRI_ENABLE=1 -DMAGICKCORE_QUANTUM_DEPTH=16 -DMAGICKCORE_HDRI_ENABLE=1 -DMAGICKCORE_QUANTUM_DEPTH=16 -DMAGICKCORE_HDRI_ENABLE=1 -DMAGICKCORE_QUANTUM_DEPTH=16 -I/usr/local/Cellar/imagemagick/7.0.7-30/include/ImageMagick-7
-L/usr/local/Cellar/imagemagick/7.0.7-30/lib -lMagick++-7.Q16HDRI -lMagickWand-7.Q16HDRI -lMagickCore-7.Q16HDRI
-L/usr/local/Cellar/imagemagick/7.0.7-30/lib -lMagick++-7.Q16HDRI -lMagickWand-7.Q16HDRI -lMagickCore-7.Q16HDRI
And if it helps here are my environment variables within ~/.zshrc.
export PATH="/usr/local/bin:$PATH"
export MAGICK_HOME=“/usr/local/Cellar/imagemagick/7.0.7-30”
export PATH="$MAGICK_HOME/bin:$PATH"
export DYLD_LIBRARY_PATH="$MAGICK_HOME/lib/"
export PKG_CONFIG_PATH="$MAGICK_HOME/lib/pkgconfig"
Tl;DR
It seems like the Makefile is referencing the wrong dylib and I'm not sure how to make it point to the "right" Imagemagick libraries which work when called straight from the command line.
I wrote a Hamming distance function that doesn't need to go all the way to GenServer, C++ and back. Maybe you want to use it? It allows to run the image comparison in parallel and is much faster.
def hamming_distance(hash1, hash2) do
{:ok, hamming_value_distance(:binary.bin_to_list(hash1), :binary.bin_to_list(hash2), 0)}
end
defp hamming_value_distance([], [], sum), do: sum
defp hamming_value_distance([val1 | tail1], [val2 | tail2], sum) do
value = val1 ^^^ val2
%{^value => distance} = %{0 => 0, 1 => 1, 2 => 1, 3 => 2, 4 => 1, 5 => 2, 6 => 2, 7 => 3, 8 => 1, 9 => 2, 10 => 2, 11 => 3, 12 => 2, 13 => 3, 14 => 3, 15 => 4, 16 => 1, 17 => 2, 18 => 2, 19 => 3, 20 => 2, 21 => 3, 22 => 3, 23 => 4, 24 => 2, 25 => 3, 26 => 3, 27 => 4, 28 => 3, 29 => 4, 30 => 4, 31 => 5, 32 => 1, 33 => 2, 34 => 2, 35 => 3, 36 => 2, 37 => 3, 38 => 3, 39 => 4, 40 => 2, 41 => 3, 42 => 3, 43 => 4, 44 => 3, 45 => 4, 46 => 4, 47 => 5, 48 => 2, 49 => 3, 50 => 3, 51 => 4, 52 => 3, 53 => 4, 54 => 4, 55 => 5, 56 => 3, 57 => 4, 58 => 4, 59 => 5, 60 => 4, 61 => 5, 62 => 5, 63 => 6, 64 => 1, 65 => 2, 66 => 2, 67 => 3, 68 => 2, 69 => 3, 70 => 3, 71 => 4, 72 => 2, 73 => 3, 74 => 3, 75 => 4, 76 => 3, 77 => 4, 78 => 4, 79 => 5, 80 => 2, 81 => 3, 82 => 3, 83 => 4, 84 => 3, 85 => 4, 86 => 4, 87 => 5, 88 => 3, 89 => 4, 90 => 4, 91 => 5, 92 => 4, 93 => 5, 94 => 5, 95 => 6, 96 => 2, 97 => 3, 98 => 3, 99 => 4, 100 => 3, 101 => 4, 102 => 4, 103 => 5, 104 => 3, 105 => 4, 106 => 4, 107 => 5, 108 => 4, 109 => 5, 110 => 5, 111 => 6, 112 => 3, 113 => 4, 114 => 4, 115 => 5, 116 => 4, 117 => 5, 118 => 5, 119 => 6, 120 => 4, 121 => 5, 122 => 5, 123 => 6, 124 => 5, 125 => 6, 126 => 6, 127 => 7, 128 => 1, 129 => 2, 130 => 2, 131 => 3, 132 => 2, 133 => 3, 134 => 3, 135 => 4, 136 => 2, 137 => 3, 138 => 3, 139 => 4, 140 => 3, 141 => 4, 142 => 4, 143 => 5, 144 => 2, 145 => 3, 146 => 3, 147 => 4, 148 => 3, 149 => 4, 150 => 4, 151 => 5, 152 => 3, 153 => 4, 154 => 4, 155 => 5, 156 => 4, 157 => 5, 158 => 5, 159 => 6, 160 => 2, 161 => 3, 162 => 3, 163 => 4, 164 => 3, 165 => 4, 166 => 4, 167 => 5, 168 => 3, 169 => 4, 170 => 4, 171 => 5, 172 => 4, 173 => 5, 174 => 5, 175 => 6, 176 => 3, 177 => 4, 178 => 4, 179 => 5, 180 => 4, 181 => 5, 182 => 5, 183 => 6, 184 => 4, 185 => 5, 186 => 5, 187 => 6, 188 => 5, 189 => 6, 190 => 6, 191 => 7, 192 => 2, 193 => 3, 194 => 3, 195 => 4, 196 => 3, 197 => 4, 198 => 4, 199 => 5, 200 => 3, 201 => 4, 202 => 4, 203 => 5, 204 => 4, 205 => 5, 206 => 5, 207 => 6, 208 => 3, 209 => 4, 210 => 4, 211 => 5, 212 => 4, 213 => 5, 214 => 5, 215 => 6, 216 => 4, 217 => 5, 218 => 5, 219 => 6, 220 => 5, 221 => 6, 222 => 6, 223 => 7, 224 => 3, 225 => 4, 226 => 4, 227 => 5, 228 => 4, 229 => 5, 230 => 5, 231 => 6, 232 => 4, 233 => 5, 234 => 5, 235 => 6, 236 => 5, 237 => 6, 238 => 6, 239 => 7, 240 => 4, 241 => 5, 242 => 5, 243 => 6, 244 => 5, 245 => 6, 246 => 6, 247 => 7, 248 => 5, 249 => 6, 250 => 6, 251 => 7, 252 => 6, 253 => 7, 254 => 7, 255 => 8}
distance + hamming_value_distance(tail1, tail2, sum)
end
Hamming distance tests could use some love. At least make sure the tests hit all edge cases. Perhaps do exhaustive testing on ultra-short hashes (8 bits? 16 bits?)
A bit neater than having to use the Github version.
Make sure documentation can be generated and looks good.
Remove this manual step.
Don't commit any binary files (like #2 suggested)
When make
fails with a compile error or similar, mix test
just keeps on trucking, using the old executable. This annoys me.
The non-wrapping version seems to be slightly more accurate. From Neal Krawetz's blog:
"I have tried a few variations of the dHash algorithm. For example, David's initial proposal used an 8x8 image and wrapped the last comparison (computing the pixel difference between P[0] and P[7] for the 8th comparison). This actually performs a little worse than my 9x8 variation (more false-positives), but only by a little."
Email Convo about the issue
For DD-Hash.
for(int i = 0; i < SIZE; ++i) {
for(int j = 0; j < SIZE; ++j) {
row_hash[i] |= (pixels[i*SIZE + j] < pixels[i*SIZE + j+1]) << (7-j);
col_hash[i] |= (pixels[i + j*SIZE] < pixels[(i + (j+1)*SIZE)%PIXEL_COUNT + j/8]) << (7-j);
}
}
here you are shifting by (7-j)
If we change the size from 8 to 32 or 64
, how will that effect on the algorithm.
Hi Zubair,
I'm using vectors of 8-bit integers to assemble the hash, which means
that the shifting distance (7-j) should be independent of the size.
Currently it is not, and it is not possible to change the size because
of that. This needs fixing.
I'll need to look it over before I can give you a definite answer. It's
been a while since I wrote this. If possible, could you open an issue
about this on Github? [1] Feel free to include this mail.
I'm going out of town for the Easter holiday soon, so I won't have time
until next week. I'll get back to you next weekend. If you have any
additional questions, please feel free to ask.
- Need to work on that
Experhash is working fine locally, but in production I am getting an error. I'm using Distillery 2 and Docker to make releases on Ubuntu 16.04. Imagemagick is installed on both the docker image and on the production server, and mix compile
runs during the build.
Genserver times out, but I believe that is just a symptom.
experhash-0.1.1/priv/experhash_port: Syntax error: word unexpected (expecting ")")
Phoenix 1.4.3
Elixir 1.8.1
Distillery 2.0 using the new Elixir provider
11:07:51.097 request_id=FZXvw-ylKvX9vlEAAADx [debug] Processing with DistanceWeb.DocumentController.new/2
Parameters: %{}
Pipelines: [:browser]
11:07:51.098 request_id=FZXvw-ylKvX9vlEAAADx [info] Sent 200 in 3ms
11:08:02.951 request_id=FZXvxq9fFd2k4d0AAAEB [info] POST /documents
11:08:02.955 request_id=FZXvxq9fFd2k4d0AAAEB [debug] Processing with DistanceWeb.DocumentController.create/2
Parameters: %{"_csrf_token" => "RysAfnR6Yk4BenoDIB0KAgxlDDU8JgAA3n//DH7xH77LcEr0EP8EVQ==", "_utf8" => "✓", "document" => %{"file" => %Plug.Upload{content_type: "image/png", filename: "a028.png", path: "/tmp/plug-1555/multipart-1555412882-15036320304116-1"}}}
Pipelines: [:browser]
/home/deploy/lib/experhash-0.1.1/priv/experhash_port: 1: /home/deploy/lib/experhash-0.1.1/priv/experhash_port: Syntax error: word unexpected (expecting ")")
11:08:12.959 request_id=FZXvxq9fFd2k4d0AAAEB [info] Converted exit {:timeout, {GenServer, :call, [#PID<0.1680.0>, {:command, {:hash, :dd, "media/1555412882-15036320304116-1-a028-orig.png", 8}}, 10000]}} to 500 response
11:08:12.961 request_id=FZXvxq9fFd2k4d0AAAEB [error] Process #PID<0.1678.0> terminating
** (exit) an exception was raised:
** (ErlangError) Erlang error: {{:timeout, {GenServer, :call, [#PID<0.1680.0>, {:command, {:hash, :dd, "media/1555412882-15036320304116-1-a028-orig.png", 8}}, 10000]}}, {DistanceWeb.Endpoint, :call, [%Plug.Conn{adapter: {Plug.Cowboy.Conn, :...}, assigns: %{}, before_send: [], body_params: %Plug.Conn.Unfetched{aspect: :body_params}, cookies: %Plug.Conn.Unfetched{aspect: :cookies}, halted: false, host: "redacted", method: "POST", owner: #PID<0.1678.0>, params: %Plug.Conn.Unfetched{aspect: :params}, path_info: ["documents"], path_params: %{}, port: 80, private: %{}, query_params: %Plug.Conn.Unfetched{aspect: :query_params}, query_string: "", remote_ip: {127, 0, 0, 1}, req_cookies: %Plug.Conn.Unfetched{aspect: :cookies}, req_headers: [{"accept", "text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8"}, {"accept-encoding", "gzip, deflate, br"}, {"accept-language", "en-US,en;q=0.5"}, {"connection", "upgrade"}, {"content-length", "30026"}, {"content-type", "multipart/form-data; boundary=---------------------------7719490031198181894888934539"}, {"cookie", "_distance_key=SFMyNTY.g3QAAAABbQAAAAtfY3NyZl90b2tlbm0AAAAYdEUvUTAyVTZJTU1PQ1h4Mkk1NHBqdz09.Ojpdcrhb9AN8xoU4Sjxl5rR-uCsb6wFd-FJlt6vuz4Y"}, {"host", "redacted"}, {"referer", "https://redacted/documents/new"}, {"upgrade-insecure-requests", "1"}, {"user-agent", "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.14; rv:68.0) Gecko/20100101 Firefox/68.0"}, {"x-cluster-client-ip", "97.89.62.78"}, {"x-forwarded-for", "97.89.62.78"}], request_path: "/documents", resp_body: nil, resp_cookies: %{}, resp_headers: [{"cache-control", "max-age=0, private, must-revalidate"}], scheme: :http, script_name: [], secret_key_base: nil, state: :unset, status: nil}, []]}}
(phoenix) lib/phoenix/endpoint/cowboy2_handler.ex:49: Phoenix.Endpoint.Cowboy2Handler.init/2
(cowboy) /redacted/distance/deps/cowboy/src/cowboy_handler.erl:41: :cowboy_handler.execute/2
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.