Comments (17)
I just got the same error while running ubuntu 16.04 inside docker on macOS.
from bootsnap.
xattr appear to be working OK here:
require 'ffi-xattr'
`touch a`
xattr = Xattr.new("a")
str = 'x'*16000 + 'y'*16000
xattr['user.foo'] = str
xattr = Xattr.new("a")
puts xattr['user.foo'].length
puts xattr['user.foo'] == str
32000
true
on ext4 you are limited on size but stuff seems unlimited on xfs
from bootsnap.
I added some puts statements
The file it is failing on is:
/home/sam/.rbenv/versions/2.4.1/lib/ruby/2.4.0/psych/visitors/to_ruby.rb
from bootsnap.
this can be cause #define XATTR_SIZE_MAX 65536
on linux... will check
... confirmed :( ...
I guess the simplest workaround would be to store multiple values for cases where size exceeds size max.
So for a 100K file
user.autocc.value == 64k of the data
user.autocc.value1 == rest of data
Then when you get a value you check if it is == to max size and if it is you try for value1 and so on.
from bootsnap.
Interesting. The maximum is rather a lot larger on darwin. Probably we should just transition away from xattr somehow.
from bootsnap.
I was thinking, one way of saving a bit of storage is to run the iseq stream via snappy
from bootsnap.
Yeah, that's definitely true. It would help. The major issue though is that the limit is 64MB on darwin and 64kB on linux. Inflating that limit to an effective ~90kB is certainly going to help in some cases, but not enough.
The other complicating factor is that to decompress a string would require a roundtrip through ruby or to link libsnappy into the bootsnap bundle somehow.
from bootsnap.
yeah a round trip to ruby would be out of the question here, but linking libsnappy should be straight forward I think, it compiles quite fast.
there is a trick though that can make this quite workable
If you read user.autocc.value
and get XATTR_SIZE_MAX back you can proceed to read user.autocc.value1
and so on. 99% of the time it would fit into one xattr (with libsnappy) but the 1% case would be easily catered for by splitting.
You only write XATTR_SIZE_MAX in cases where you have to split across multiple attributes.
from bootsnap.
Yeah, actually, that seems like a very reasonable way to solve this.
from bootsnap.
The major issue though is that the limit is 64MB on darwin and 64kB on linux.
actually the xattr limit is 64kB on linux only for JFS
, XFS
, or Reiserfs
filesystems. The limit is 16kB for Btrfs
, and for ext2
/3
/4
the limit on all of a file's xattrs is only 4kB, so a user.autocc.value1
approach won't work.
See man xattr(7)
for details:
The VFS imposes limitations that an attribute names is limited to 255
bytes and an attribute value is limited to 64 kB. [...]
In the current ext2, ext3, and ext4 filesystem implementations, the
total bytes used by the names and values of all of a file's extended
attributes must fit in a single filesystem block (1024, 2048 or 4096
bytes, depending on the block size specified when the filesystem was
created).
In the Btrfs, XFS, and Reiserfs filesystem implementations, there is
no practical limit on the number of extended attributes associated
with a file, and the algorithms used to store extended attribute
information on disk are scalable.
In the Btrfs filesystem implementation, the total bytes used for the
name, value, and implementation overhead bytes is limited to the
filesystem nodesize value (16 kB by default).
from bootsnap.
from bootsnap.
Yeah, the thing here is that I went down a bit of a rabbit hole with the xattr thing and it worked well enough in practice on our macOS dev machines that I haven't walked back out of it yet. It's probably not a great long-term feature simply because it doesn't behave nicely on linux, though the attribute it has of tying cache lifecycle to the source file is quite nice.
from bootsnap.
Personally, I'd love to see some brainstorming on a way to transition from xattr
to something that is supported by at least mac OS and Linux. I also know the Rails team has put effort into making everything work on Windows, so if it works for that too, that'd be grand.
Anyone have any ideas?
from bootsnap.
likely the right solution is the boring one: cache things at tmp/cache/bootsnap-compile-cache/A/B
where B is a hash over the file path and A is the first 2 chars of B.
from bootsnap.
Should be fixed by #43
from bootsnap.
curious to know if 0.3.0.pre works for you.
from bootsnap.
0.3.0 released
from bootsnap.
Related Issues (20)
- /usr/share/ruby/time.rb is not `.stable?` - path_test.rb failing HOT 9
- YAML cache "bypasses" permitted_classes in some cases HOT 4
- ibf_dump_object_unsupported […] T_NONE (NotImplementedError) HOT 26
- bootsnap compile-cache-yaml folder contents file permissions issue HOT 4
- windows rails demo project error HOT 2
- sharing remote caching bootsnap cache dir across machines HOT 2
- Facing `compile_file': "\\xE2" on US-ASCII (Encoding::InvalidByteSequenceError) error when trying to run any rails command HOT 14
- Bootsnap causes server to hang in Ruby 3.2.1 HOT 6
- undefined method `driver_path=' for Selenium::WebDriver::Chrome:Module HOT 1
- Testsuite fails with ruby3.3.0dev HOT 6
- warning: method redefined; discarding old require HOT 2
- Bootsnap LoadError for Prism HOT 8
- `bin/bootsnap precompile` not compiling everything HOT 7
- 1.18.x extension build failed HOT 11
- 1.18.x: `compile_cache/iseq.rb:64:in 'fetch': Invalid argument - (Errno::EINVAL)` HOT 1
- EOFError: end of buffer reached HOT 9
- Shopify IP allowlist restricting access to non-sensitive resource? HOT 3
- I feel like everybody is doing `gem "bootsnap", require: false` so why not make bootsnap.rb do nothing? HOT 2
- `bootsnap -v` or `bootsnap --version` could return the version of bootsnap? HOT 1
- Exception raised when upgraded to ruby 3.3.1 and deployed to Heroku HOT 6
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from bootsnap.