Comments (22)
Thank you. I let the issue open. I will spend some hours on reviewing the code around "limit reached" to find the problem. The file creating the error would be however extremely helpful.
from xz.
Can you confirm that this error happens during compression? Your message doesn't tell me that. The error shouldn't occur and is very likely a bug. It hasn't been reported so far and did never occur in the tests I'm running before a release. Can you provide me with a single sample file of the data that triggers the error? That would allow me to reproduce the problem and analyze it.
from xz.
We have our own issue tracking this problem with more details. See kubevirt/containerized-data-importer#335, especially the comments near the end...
from xz.
We are wondering if we should increase DictCap
or Size
fields?
from xz.
Hi @ulikunitz
Our sample data is generated on the fly for tests from a single know source (tinyCore.iso: https://github.com/kubevirt/containerized-data-importer/tree/master/tests/images).
In these instances where we see the limit reached
error, our generator package tars and xz compresses tinyCore.iso for consumption in the test suite. The operations are performed using archive/tar
and this project for xz.
The code that results in this error is a little long to paste in its entirety here so I'll snippet just the relative function (the package is at https://github.com/kubevirt/containerized-data-importer/blob/master/tests/utils/fileConversion.go). Hopefully it's just a matter of mis-configuring the writer on our part.
func toXz(src, tgtDir string) (string, error) {
// createTargetFile opens a new file in the target directory, using the basename
// of src and appending ".xz" e.g. /path/to/tgtDir/srcBaseName.xz
// it returns the *File and path
tgtFile, tgtPath, err := createTargetFile(src, tgtDir, image.ExtXz)
defer tgtFile.Close()
w, err := xz.NewWriter(tgtFile)
if err != nil {
return "", errors.Wrapf(err, "Error getting xz writer for file %s", tgtPath)
}
defer w.Close()
srcFile, err := os.Open(src)
if err != nil {
return "", errors.Wrapf(err, "Error opening file %s", src)
}
defer srcFile.Close()
_, err = io.Copy(w, srcFile)
if err != nil {
return "", errors.Wrapf(err, "Error writing to file %s", tgtPath)
}
return tgtPath, nil
}
from xz.
Thanks. What file creates the "limit reached" error? I assume I can find it in tinyCore.iso.
It is not an configuration error. The XZ format has blocks that have a maximum size. The compressor has to stop before this maximum size is reached. ErrLimit ("limit reached") would indicate that this maximum size is reached without the encoder stopping before. Currently I don't know, how this can happen and for that reason I ask for the file that creates the error to reproduce the issue.
from xz.
The file is tinyCore.iso.tar.xz
that we generate on the fly from tinyCore.iso
. Is that what you're referring to? If so, I'll create it with our generator package, upload it to a personal git repo and post the link.
from xz.
@ulikunitz Thanks for your help and interest in this! At a high level, we take a base file, tinyCore.iso and convert it on-the-fly to various tar'd and compressed formats, and use these converted files to test that we can reverse the process with our CDI importer code. The limit error occurs intermittently (but is not rare) only when we are using xz.Writer
as the Writer in the io.Copy
call where the src file is the just-converted tinyCore.iso.tar. So tinyCore.xz works and tinyCore.tar works, but tinyCore.tar.xz occasionally fails. This occurs in our Travis CI environment, although Jon has seen the error in his local dev environment (but this is much rarer).
from xz.
Please provide me the link to the tar file. I will look at it at the weekend.
from xz.
@ulikunitz Here is a .tar that we create with archive/tar
as well as the .xz
that we generate from that
https://s3.amazonaws.com/cdi-xz-debug/tinyCore.iso.tar
https://s3.amazonaws.com/cdi-xz-debug/tinyCore.iso.tar.xz
Thanks for your interest in helping us debug this!
from xz.
Hi, I tested against the master and dev branch and I don't get any error for the tar file. Are you sure that this file generates the issue? Apparently you were able to create an xz file out of it.
Can you send me the output of go version? I want to check whether this is specific for a go version.
from xz.
Okay, thanks for giving it a shot. We see the error maybe 1/20 builds, so it's not a constant event. It's difficult to provide the specific .tar
that results in the limit reached
error since it's created dynamically by our CI and destroyed almost immediately when the test ends. We don't have access to the CI environment to reach in an grab it either.
[root@b260faf6ac3f /]# go version
go version go1.10 linux/amd64
from xz.
@ulikunitz I can provide the docker container and/or Dockerfile in which this error occurs. It would probably be helpful since it provides the identical environment. Running the container in a for loops until it errors would be one way to catch it.
from xz.
I would run it, if the files causing the error would be written into a specific directory.
from xz.
I've made a change to the dev branch that should minimize the risk of an ErrList. Can you please check whether it prevents the error from reoccuring?
from xz.
@ulikunitz We'll pull in the branch to our test env and see if we can reproduce the error. Thank you!
from xz.
I have not received any feedback for 17 days now, There has never been any file produced that is actually triggering the reported behavior. I have never observed this behavior myself, neither has anybody else reported it. Since there is nothing I can do right now, I'm closing the issue.
from xz.
I am seeing this issue today 100% of the time with the following file
https://drive.google.com/file/d/19eMvQaalynVlGj4PtuGKXhRg4WTkcilH/view?usp=sharing
the following is the command i use
gxz -k -c -9 rootfs-sumo-fslc-mx6.tar > rootfs-sumo-fslc-mx6.tar.xz
strangely other files don't have the problem
from xz.
Many thanks for the file. I could reproduce the issue for the master branch. Please test the dev branch (commit d1e248f). You may use it as a workaround.
from xz.
Cool thanks, I'll give it a test on Monday. Many thanks :)
from xz.
I increased the margin constant in master. I can compress the sample file now without an issue and released the fix as v0.5.5. Please report to me, whether it works for you.
from xz.
Have re-ran the test this morning and all is working now, many thanks :)
from xz.
Related Issues (20)
- Unzipping is too slow HOT 12
- use of internal package github.com/ulikunitz/xz/internal/xlog not allowed HOT 2
- Not clear how to trade most resources for most compression HOT 3
- Checksum None is valid HOT 4
- expose blockreader HOT 4
- Current maturity of project (and other semantics) HOT 3
- Achieving maximum xz compression HOT 2
- memory leak HOT 2
- [SECURITY] Implementation of readUvarint vulnerable to CVE-2020-16845 HOT 17
- How to use multi CPU work for compression? HOT 3
- Panic with invalid input HOT 2
- missing match limit, was "lzip" HOT 11
- How to write compress & decompress data in file? HOT 2
- Don't worry about this lssue HOT 1
- Missing common APIs like Reader:Close() Writer:Flush() HOT 3
- Out of Memory bug when using a large reader HOT 3
- high allocation ratio HOT 3
- Expose `processFile` function HOT 2
- Plan for rewrite branch HOT 4
- How i can compressed folder? HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from xz.