Update makefile to use / fall back on built-in "ar"

[forthrin]

Update makefile to use / fall back on built-in "ar"

[kddnewton]

@ParadoxV5 do you think you could take a look at this?

[eregon]

I used GitHub Actions (with mxschmitt/action-tmate@v3) to check and it's a clear issue in Homebrew ruby bottles since 3.2:

$ for v in 3.0 3.1 3.2 3.3; do echo $v; /opt/homebrew/opt/ruby@$v/bin/ruby -e 'pp RbConfig::CONFIG["CC"]'; done       
3.0
"clang -fdeclspec"
3.1
"clang"
3.2
"clang"
3.3
"clang"

$ for v in 3.0 3.1 3.2 3.3; do echo $v; /opt/homebrew/opt/ruby@$v/bin/ruby -e 'pp RbConfig::CONFIG["AR"]'; done       
3.0
"ar"
3.1
"ar"
3.2
"/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/ar"
3.3
"/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/ar"

[kddnewton]

To be clear, did this work with 0.25.0?

[forthrin]

Running gem update daily for quite a while and first time this happened, so assume yes.

[kddnewton]

Also @forthrin that value is coming from RbConfig::CONFIG. Is it possible that you compiled Ruby with an ar and then somehow removed it?

[ParadoxV5]

Also @forthrin that value is coming from RbConfig::CONFIG. Is it possible that you compiled Ruby with an ar and then somehow removed it?

⬆. The new release switched from make-default AR & etc. to those specified by RbConfig::CONFIG, so this sort of problem arise if the current setup differs that built Ruby with.

This might be the same issue as the one failing CI in #2711. [EDIT: It’s different.]
I’ll try some research on the Mac building environment. [Update: Homebrew/brew#17114 is reporting an incompatibility from Ubuntu.]
In the mean time, p RUBY_DESCRIPTION, RbConfig::CONFIG.slice('AR', 'ARFLAGS', 'CC') data could help us with confirmations.

[forthrin]

I'm on macOS 14. gem says it's a "default gem". I have never installed it manually or used it directly. So everything that has happened with it has done so by default behind the scenes without my intervention or knowledge.

["ruby 3.3.0 (2023-12-25 revision 5124f9ac75) [arm64-darwin23]",
 {"AR"=>
   "/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/ar",
  "ARFLAGS"=>"rcu ",
  "CC"=>"clang"}]

[ParadoxV5]

gem says it's a "default gem"

Prism 0.19 is a default gem for Ruby 3.3, though gem lets us update default gems to never versions (e.g., Prism 0.26) if we wish.

[ParadoxV5]

 {"AR"=>
   "/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/ar",
  "ARFLAGS"=>"rcu ",
  "CC"=>"clang"}]

This CC would search for “built-in clang” on the PATH, but AR is the full path to specifically the ar from Xcode.

---

Help us out @forthrin (and thank you for your patience):

• How did you install Ruby 3.3?

  • Is it possible that you compiled Ruby with an ar and then somehow removed it?

    • Did you switch toolsets after installing Ruby?

    • $ mkdir -p /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin

      Had to create a directory that’s supposed to be managed by the Xcode app? 🤔

• Are other C extension gems affected (without this workaround in-place)?

@kddnewton any other thoughts?

---

[arm64-darwin23]

So we support ARM? 👌

[nickmccurdy]

I'm still getting this error after trying the workaround.

[ParadoxV5]

I'm still getting this error after trying the workaround.

@nickmccurdy
Your setup is probably different then.
Neither OP nor the lead dev have replied, so while I (we?) ponder about how things work, can you also get us your p … data and question responses?

---

This is an issue indeed (a recurring one, sounds like), but I don’t want to just push some workaround (like what the title says). I want to cure it at its root.
Just that I’m not yet certain if it’s my fault or rather similar to Homebrew/homebrew-portable-ruby#187.

[forthrin]

@ParadoxV5: Happy to help

1. How did you install Ruby 3.3?

$ brew install ruby # before 3.3
$ brew upgrade # runs daily

2. Is it possible that you compiled Ruby with an ar and then somehow removed it?

Never compiled Ruby, at least not manually or intentionally. It's a cask.

3. Did you switch toolsets after installing Ruby? Had to create a directory supposed to be managed by Xcode?

Don't have Xcode, just CLT. Might have experimentally run xcode-select as suggested workarounds for various issues.

4. Are other C extension gems affected (without this workaround in-place)?

Like which? (Generally haven't had problems with Ruby or gems)

[ParadoxV5]

$ brew install ruby # before 3.3

[…]
Never compiled Ruby, at least not manually or intentionally. It's a cask.

Google says this’ll brew a formula, not a cask?
I don’t have experience in Mac development nor Homebrew, so this debugging is quite a challenge.

Might have experimentally run xcode-select as suggested workarounds for various issues.

That could be something. https://www.unix.com/man-page/OSX/1/xcode-select (from 2013?) implies that this Xcode util manages building tools including but not limited to /usr/bin/ar and /usr/bin/clang. But 🤔

• Your ar exists in /usr/bin/ but not in Xcode where /usr/bin/* refers to.
• Your CC searches the PATH while your AR is pinned to Xcode.

Like which?

Anything that says “Building native extensions. This could take a while” when gem installed (or gem updated).

Does brew upgrade ruby include gem update?

[Edit]
Actually, it look like a malformed AR entry won’t affect most C extension gems because they are dynamically linked (i.e., built as .so) which does not need ar.
Otherwise, mkmf doesn’t seem to have any mechanism to fall back to PATH ar if the rbconfig AR is broken (like what this issue requested (originally)).

[ParadoxV5]

Actually, it look like a malformed AR entry won’t affect most C extension gems because they are dynamically linked (i.e., built as .so) which does not need ar.
Otherwise, mkmf doesn’t seem to have any mechanism to fall back to PATH ar if the rbconfig AR is broken (like what this issue requested (originally)).

@kddnewton
What was the motivation of building Prism core as a static library (which the C extension part of Prism includes in the dynamic library)?
Would it be reasonable to build Prism core as a dynamic lib (and thus avoids using AR)?
[P.S.] Or maybe Ruby mkmf should accept user preferences rather than what’s used by the possibly-prebuilt Ruby.

[forthrin]

@ParadoxV5:

Yes, formula. Ruby is not compiled on the local machine.

All brews on the local machine are freshened daily by brew update && brew upgrade.

"where /usr/bin/* refers to"

$ ls -l /usr/bin/ar
-rwxr-xr-x  76 root  wheel  119008 Mar 21 07:13 /usr/bin/ar
$ file /usr/bin/ar
/usr/bin/ar: Mach-O universal binary with 2 architectures: [x86_64:Mach-O 64-bit executable x86_64] [arm64e:Mach-O 64-bit executable arm64e]
/usr/bin/ar (for architecture x86_64):	Mach-O 64-bit executable x86_64
/usr/bin/ar (for architecture arm64e):	Mach-O 64-bit executable arm64e
$ man ar
<snip>
STANDARDS
     The ar utility is expected to offer a superset of the IEEE Std 1003.2
     (“POSIX.2”) functionality.
Darwin                           July 27, 2005                          Darwin

[kddnewton]

So if I understand correctly, the issue is that the Ruby is compiled on a different machine and contains config settings for the machine that compiled it, which may or may not function correctly on the actual machine installing prism.

In this case, I think it makes sense to check if the AR exists, and if it doesn't then fall back to ar.

[kddnewton]

@forthrin does #2725 fix your system?

[forthrin]

Step by step terminal commands for testing this? Normally just running gem update (or would gem install prism).

[kddnewton]

Ensure current Ruby is >= 3.0.0 with ruby -v, then:

git clone https://github.com/ruby/prism.git
cd prism
ruby templates/template.rb
gem build
gem install prism-0.26.0.gem

[forthrin]

Sure the above commands include the PR? Lamentably:

/bin/sh: /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/ar: No such file or directory

[kddnewton]

Oh sorry, I should have added, please run git checkout fallback after the cd prism step

[forthrin]

Ditto. git branch is fallbackand git diff main fallback does show the PR! Keep hitting me. How about:

cmd = ([ENV['AR']] + `which -a ar`.lines.map(&:chomp)).compact.find { |e| File.exist?(e) }

[kddnewton]

Ahh maybe the issue is not in extconf.rb but in Rakefile, let me update it there

[kddnewton]

Alright, can you pull the latest version of that branch and try again with gem build && gem install --verbose prism-0.26.0.gem?

[forthrin]

Houston, we have lift-off!

[kddnewton]

Fixed on main

[forthrin]

What date will this be available in the central Gem repository? (It's breaking gem update atm.)

[kddnewton]

I'll make a new release today, I'll update the ticket when I do.

[kddnewton]

Released with v0.27.0

[forthrin]

$ gem install prism
Successfully installed prism-0.27.0

Great detective work, you guys.

[ParadoxV5]

you guys.

No thank you, @kddnewton did all the thing.

Though I’m still puzzled on how your rbconfig ended up like that.

Also, the PR logs told that you needed the AR ?= ar line, but it was supposed to not matter for GNU make.
What’s your make --version? Could that be a possiblility?
(The README.md specifies GNU make.)

[eregon]

@forthrin I have been trying to understand this issue in more details (here).
Could you share the output of ruby -ve 'pp RbConfig::CONFIG'?
And also the output of these 5 commands:

xcode-select --print-path
ls $(xcode-select --print-path)
ls $(xcode-select --print-path)/bin
ls $(ruby -e 'puts RbConfig::CONFIG["CC"]')
ls $(ruby -e 'puts RbConfig::CONFIG["AR"]')

?

I wonder if maybe an incorrect xcode-select or an XCode update removing an older toolchain could cause this.
The part about CC existing and AR not is really puzzling.

[forthrin]

$ xcode-select --print-path
/Library/Developer/CommandLineTools

$ ls $(xcode-select --print-path)
Library SDKs    usr

$ ls $(xcode-select --print-path)/bin
ls: /Library/Developer/CommandLineTools/bin: No such file or directory

$ ls $(ruby -e 'puts RbConfig::CONFIG["CC"]')
ls: clang: No such file or directory

$ ls $(ruby -e 'puts RbConfig::CONFIG["AR"]')
ls: /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/ar: No such file or directory

FYI:

$ ls -l /Library/Developer/CommandLineTools/usr/bin
/Library/Developer/CommandLineTools/usr/bin/ar
/Library/Developer/CommandLineTools/usr/bin/clang
# snip

[eregon]

Thank you, that confirms that RbConfig::CONFIG["CC"] is clang and so that just works.
And RbConfig::CONFIG["AR"] is /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/ar, which is a broken value as (from what I understood) you never had a file at that path on that machine.

This seems very much a case of Homebrew/homebrew-portable-ruby#187 or very similar.
IOW it's a bug of that precompiled ruby you got that RbConfig::CONFIG["AR"] is an invalid path.
Although ruby in your PATH is most likely brew install ruby from what you said earlier, not that homebrew-portable-ruby.
So then it'd be a bug of https://github.com/Homebrew/homebrew-core/blob/master/Formula/r/ruby.rb, specifically that the binary (bottle) contains such invalid paths.

[eregon]

I filed Homebrew/homebrew-core#170978

[forthrin]

@eregon: Kudos for digging deeper and locating the core issue!