Comments (6)
Yes, I think that's a sensible change. I wonder if Context should have the same treatment
from cryptonite.
I'm a bit puzzled actually. Just was looking at the code, and trying couple of things to try to reproduce but just couldn't; is the problem not in the part that transform a Digest x -> String
and lack of evaluation of String
?
from cryptonite.
Vincent Hanquez wrote:
I'm a bit puzzled actually. Just was looking at the code, and trying couple of
things to try to reproduce but just couldn't; is the problem not in the code
that transform a Digest x -> String ?
I had some difficulty reproducing it too, only saw the problem on some
systems and not others, and only in some use patterns (comparison with
known checksums).
I don't see how Digest x -> String could need 3 terabytes of memory, so
I assume the memory use is in a partially evaluated Digest keeping
references to parts of the lazy ByteString, so that the whole input data
remains in memory.
see shy jo
from cryptonite.
sorry for being unclear, but when I say the problem is in the Digest x -> String
part, I meant it's actually not forcing any evaluation of the Digest anyway unless you're actually looking at least at the first char which ! is not going to do.
i.e. the problem is:
!hashAsString <- md5hasher <$> L.readline foo
return hashAsString
Whereas if you write:
!hash <- md5 <$> L.readFile foo
return $ show hash
I think the !
is doing the right thing in the second case, and only the digest -> String
would be done lazily. Whereas in the first case, the ! on a String would do nothing.
from cryptonite.
although, it doesn't quite explain needing so much memory, maybe some context are not folded properly to the next one..
from cryptonite.
Yes, part of the problem was indeed that bang doesn't fully force String.
It forced it enough to fix my FD-remaining-open problem, but not enough
to avoid buffering the file in ram. While that was my mistake to make,
it wouldn't have mattered that I made that mistake if Digest was itself
strict.
Also, it's not really clear from the public API whether bang fully forces
a Digest. newtype Digest a = Digest Bytes, and the implementation of
Bytes is not exposed so it's difficult to reason about how to force it,
other than using the handy NFData instance of course.
see shy jo
from cryptonite.
Related Issues (20)
- aarch64 - How to remove/override dependency on Scrypt - ubuntu aarch64 on pi4 provides its own include file and library and executable HOT 2
- On AVX2 systems testsuite fails on unaligned stores in cbits/decaf/include/constant_time.h:159 HOT 4
- Unable to build on FreeBSD aarch64: cbits/decaf/p448/f_generic.c:56:8: error: loop not unrolled HOT 2
- Fails to build with GHC 9.2.1 HOT 5
- Fails to compile on CentOS with stack lts-18.16 HOT 1
- test-cryptonite fails to link HOT 6
- master fails to build on macOS HOT 4
- Argon2 KATs test failure on aarch64-darwin (apple silicon) HOT 1
- 4-8x sha256 performance deficit compared to sha256sum/openssl (SHA instructions) HOT 2
- Feature: add support for NIST PQC finalists HOT 1
- Test suite segfaults on armv7l-linux
- Is this package the right location for the xxhash non-cryyptographic hash function HOT 2
- Sign function taking way too much time HOT 2
- error: instruction requires: sha3 HOT 4
- Return V Value From ECDSA Signing
- Verify argon2
- Write permission HOT 4
- Expose the bytes underlying an AES128/AES192/AES256 HOT 1
- Unable to build on MacOS 13.4.1 (aarch64) HOT 1
- constant time comparison for `Digest`? HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from cryptonite.