Fails to load iseq cache in Linux

[shopify]

Fails to load iseq cache in Linux

[jdrago999]

I just got the same error while running ubuntu 16.04 inside docker on macOS.

[SamSaffron]

xattr appear to be working OK here:

require 'ffi-xattr'
`touch a`
xattr = Xattr.new("a")
str = 'x'*16000 + 'y'*16000
xattr['user.foo'] = str
xattr = Xattr.new("a")
puts xattr['user.foo'].length
puts xattr['user.foo'] == str

32000
true

on ext4 you are limited on size but stuff seems unlimited on xfs

[SamSaffron]

I added some puts statements

The file it is failing on is:

/home/sam/.rbenv/versions/2.4.1/lib/ruby/2.4.0/psych/visitors/to_ruby.rb

[SamSaffron]

this can be cause #define XATTR_SIZE_MAX 65536 on linux... will check

... confirmed :( ...

I guess the simplest workaround would be to store multiple values for cases where size exceeds size max.

So for a 100K file

user.autocc.value == 64k of the data
user.autocc.value1 == rest of data

Then when you get a value you check if it is == to max size and if it is you try for value1 and so on.

[burke]

Interesting. The maximum is rather a lot larger on darwin. Probably we should just transition away from xattr somehow.

[SamSaffron]

I was thinking, one way of saving a bit of storage is to run the iseq stream via snappy

[burke]

Yeah, that's definitely true. It would help. The major issue though is that the limit is 64MB on darwin and 64kB on linux. Inflating that limit to an effective ~90kB is certainly going to help in some cases, but not enough.

The other complicating factor is that to decompress a string would require a roundtrip through ruby or to link libsnappy into the bootsnap bundle somehow.

[SamSaffron]

yeah a round trip to ruby would be out of the question here, but linking libsnappy should be straight forward I think, it compiles quite fast.

there is a trick though that can make this quite workable

If you read user.autocc.value and get XATTR_SIZE_MAX back you can proceed to read user.autocc.value1 and so on. 99% of the time it would fit into one xattr (with libsnappy) but the 1% case would be easily catered for by splitting.

You only write XATTR_SIZE_MAX in cases where you have to split across multiple attributes.

[burke]

Yeah, actually, that seems like a very reasonable way to solve this.

[wjordan]

The major issue though is that the limit is 64MB on darwin and 64kB on linux.

actually the xattr limit is 64kB on linux only for JFS, XFS, or Reiserfs filesystems. The limit is 16kB for Btrfs, and for ext2/3/4 the limit on all of a file's xattrs is only 4kB, so a user.autocc.value1 approach won't work.

See man xattr(7) for details:

   The VFS imposes limitations that an attribute names is limited to 255
   bytes and an attribute value is limited to 64 kB. [...]

   In the current ext2, ext3, and ext4 filesystem implementations, the
   total bytes used by the names and values of all of a file's extended
   attributes must fit in a single filesystem block (1024, 2048 or 4096
   bytes, depending on the block size specified when the filesystem was
   created).

   In the Btrfs, XFS, and Reiserfs filesystem implementations, there is
   no practical limit on the number of extended attributes associated
   with a file, and the algorithms used to store extended attribute
   information on disk are scalable.

   In the Btrfs filesystem implementation, the total bytes used for the
   name, value, and implementation overhead bytes is limited to the
   filesystem nodesize value (16 kB by default).

[SamSaffron]

Yeah, it would suck on ext3/4 and not be a usable workaround. on XFS though it is workable, but yeah getting people to mount home as XFS just cause they want bootsnap is a tough call. Sadly without xattr it would mean an additional seek per file and all the nightmare that comes with managing a cache file per rb file.

…

On Wed, May 3, 2017 at 6:02 PM, Will Jordan ***@***.***> wrote: The major issue though is that the limit is 64MB on darwin and 64kB on linux. actually the xattr limit is 64kB on linux only for JFS, XFS, or Reiserfs filesystems. The limit is 16kB for Btrfs, and for ext2/3/4 the limit on *all* of a file's xattrs is only 4kB, so a user.autocc.value1 approach won't work. See man xattr(7) <http://man7.org/linux/man-pages/man7/xattr.7.html> for details: The VFS imposes limitations that an attribute names is limited to 255 bytes and an attribute value is limited to 64 kB. [...] In the current ext2, ext3, and ext4 filesystem implementations, the total bytes used by the names and values of all of a file's extended attributes must fit in a single filesystem block (1024, 2048 or 4096 bytes, depending on the block size specified when the filesystem was created). In the Btrfs, XFS, and Reiserfs filesystem implementations, there is no practical limit on the number of extended attributes associated with a file, and the algorithms used to store extended attribute information on disk are scalable. In the Btrfs filesystem implementation, the total bytes used for the name, value, and implementation overhead bytes is limited to the filesystem nodesize value (16 kB by default). — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#20 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAAUXVrtnB9g1vU08t0hrP0iLhqgvFd6ks5r2PmSgaJpZM4NMZEz> .

[burke]

Yeah, the thing here is that I went down a bit of a rabbit hole with the xattr thing and it worked well enough in practice on our macOS dev machines that I haven't walked back out of it yet. It's probably not a great long-term feature simply because it doesn't behave nicely on linux, though the attribute it has of tying cache lifecycle to the source file is quite nice.

[jules2689]

Personally, I'd love to see some brainstorming on a way to transition from xattr to something that is supported by at least mac OS and Linux. I also know the Rails team has put effort into making everything work on Windows, so if it works for that too, that'd be grand.

Anyone have any ideas?

[burke]

likely the right solution is the boring one: cache things at tmp/cache/bootsnap-compile-cache/A/B where B is a hash over the file path and A is the first 2 chars of B.

[burke]

Should be fixed by #43

[burke]

curious to know if 0.3.0.pre works for you.

[burke]

0.3.0 released