wilyarti/hashtree

S3 compatible data deduplication and client side encryption program

hashtree's Issues

Excessive memory consumption while doing push

The memory climbs up to 1.9GiB when using the push function.

Using pprof the cause seams to be the minio package:

(pprof) top
Showing nodes accounting for 1.12GB, 99.59% of 1.13GB total
Dropped 56 nodes (cum <= 0.01GB)
      flat  flat%   sum%        cum   cum%
    1.12GB 99.59% 99.59%     1.13GB 99.68%  github.com/minio/minio-go.Client.putObjectMultipartStreamNoLength
         0     0% 99.59%     1.13GB 99.68%  github.com/minio/minio-go.Client.PutObject
         0     0% 99.59%     1.13GB 99.68%  github.com/minio/minio-go.Client.PutObjectWithContext
         0     0% 99.59%     1.13GB 99.68%  github.com/minio/minio-go.Client.putObjectCommon
         0     0% 99.59%     1.13GB 99.73%  hashtree/uploadFiles.UploadFile

The cause is this line (195):
size, err := s3Client.PutObject(bucket, hash, encrypted, -1, minio.PutObjectOptions{})

The -1 flag cause minio to treat the file as a stream with an unknown length.