Crashes on Apple M1

[siddhsql]

Crashes on Apple M1

[siddhsql]

okay I will try and update the thread with my findings but it might be a while.

[siddhsql]

---------------  S U M M A R Y ------------

Command Line:

Host: "Mac14,3" arm64, 8 cores, 8G, Darwin 22.6.0, macOS 13.5 (22G74)
Time: Tue Aug 29 15:19:52 2023 PDT elapsed time: 0.027915 seconds (0d 0h 0m 0s)

---------------  T H R E A D  ---------------

Current thread (0x000000010f1ec200):  JavaThread "Unknown thread" [_thread_in_vm, id=259, stack(0x000000016e05c000,0x000000016f05c000)]

Stack: [0x000000016e05c000,0x000000016f05c000],  sp=0x000000016f057ca0,  free space=16367k
Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code)
V  [libjvm.dylib+0x4492c0]  CodeHeap::allocate(unsigned long)+0x1c8
V  [libjvm.dylib+0x27e6cc]  CodeCache::allocate(int, CodeBlobType, bool, CodeBlobType)+0xa4
V  [libjvm.dylib+0x2763cc]  BufferBlob::create(char const*, int)+0xf0
V  [libjvm.dylib+0x8e7a90]  StubRoutines::initialize1()+0x94
V  [libjvm.dylib+0x4796f0]  init_globals()+0x28
V  [libjvm.dylib+0x942668]  Threads::create_vm(JavaVMInitArgs*, bool*)+0x340
V  [libjvm.dylib+0x51f094]  JNI_CreateJavaVM+0x6c
C  [jnius.cpython-39-darwin.so+0x1d29c]  __pyx_f_5jnius_get_jnienv+0x1468

I see same exact crash using jpype so is this a bug in JNI then?

[cmacdonald]

Does it only occur on Java 20? Can you try older Java?

[siddhsql]

i was also thinking the same. i am not able to try older java but there is very similar issue filed for jpype by someone else that uses JDK 17: jpype-project/jpype#1143

…

On Wed, Aug 30, 2023 at 2:48 AM Craig Macdonald ***@***.***> wrote: Does it only occur on Java 20? Can you try older Java? — Reply to this email directly, view it on GitHub <#667 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/A6NWEK3EZTAEQQNQYLVIAD3XX4D77ANCNFSM6AAAAAA4DTWRFI> . You are receiving this because you authored the thread.Message ID: ***@***.***>

[cmacdonald]

We've not been able to reproduce this on new apple silicon tests under #679. Some google suggests that the error goes away if macOS is upgraded (Homebrew/homebrew-core#124306)?

[misl6]

As @cmacdonald said, we're not able to reproduce the issue on latest pyjnius release.
Please upgrade to 1.6.0 and open a new issue if the issue persists on your side.

[siddhsql]

for me its a repro:

(myenv) ➜  ~/code/venv  pip install pyjnius
Collecting pyjnius
  Downloading pyjnius-1.6.1-cp39-cp39-macosx_10_9_universal2.whl (513 kB)
     |████████████████████████████████| 513 kB 5.3 MB/s
Installing collected packages: pyjnius
Successfully installed pyjnius-1.6.1

(myenv) ➜  ~/code/venv  python3
Python 3.9.6 (default, May  7 2023, 23:32:44)
[Clang 14.0.3 (clang-1403.0.22.14.1)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> from jnius import autoclass
#
# A fatal error has been detected by the Java Runtime Environment:
#
#  SIGBUS (0xa) at pc=0x0000000106f13d70, pid=10269, tid=259
#
# JRE version:  (20.0.2+9) (build )
# Java VM: Java HotSpot(TM) 64-Bit Server VM (20.0.2+9-78, mixed mode, sharing, tiered, compressed oops, compressed class ptrs, g1 gc, bsd-aarch64)
# Problematic frame:
# V  [libjvm.dylib+0x447d70]  CodeHeap::allocate(unsigned long)+0x1c8
#
# No core dump will be written. Core dumps have been disabled. To enable core dumping, try "ulimit -c unlimited" before starting Java again
#
# An error report file with more information is saved as:
# /Users/xxx/code/venv/hs_err_pid10269.log
#
#
[1]    10269 abort      python3

I am on Mac OS 13.5.2 (22G91) (Ventura)

[cmacdonald]

anything useful in the hs_err_pid10269.log file?

[cmacdonald]

https://stackoverflow.com/questions/75496223/macos-13-3-beta-killed-java talks about brew JDKs vs Oracle.

Silly question - does Java work normally on this machine (i.e. can you try some Java programs, jshell etc), so that we can establish if the problem is Java or Jnius?

[siddhsql]

Yes Java works normally on the machine.

[siddhsql]

log has been pasted earlier in the thread

[cmacdonald]

ok, I have a theory based on random searches that this somehow relates to MAP_JIT security that is enabled on Apple Silicon.

Assuming you have clang installed, could you try to compile and execute the attached C++ code:
testJ.cpp.txt
(you'll have to remove the .txt extension).

This compilation should work

clang++ -I `/usr/libexec/java_home`/include -I `/usr/libexec/java_home`/include/*/ -L `/usr/libexec/java_home`/lib/server -ljvm testJ.cpp

On my Intel mac, executing the ./a.out binary, it gives the following output:

% ./a.out
pthread_jit_write_protect_supported_np 0
jvm started
java version obtained from JVM: 18

If yours gives the SIGBUS error, could you uncomment line 22 and try again, i.e. enabling pthread_jit_write_protect_np(1);

[cmacdonald]

(Not sure the above is going to work)

Could you paste the XML output of codesign --display --verbose=4 --xml --entitlements - /path/to/bin/java and the same for your Python executable?

[cmacdonald]

On the Github Action runner for apple-silicon, Python is signed and has some security features disabled (see below)

<?xml version="1.0" encoding="UTF-8"?><!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "https://www.apple.com/DTDs/PropertyList-1.0.dtd"><plist version="1.0"><dict><key>com.apple.security.automation.apple-events</key><true/><key>com.apple.security.cs.allow-dyld-environment-variables</key><true/><key>com.apple.security.cs.disable-executable-page-protection</key><true/><key>com.apple.security.cs.disable-library-validation</key><true/></dict></plist> 

My bet is your Python does not have this.

[siddhsql]

codesign --display --verbose=4 --xml --entitlements - $JAVA_HOME/bin/java
Executable=/Library/Java/jdk-20.0.2.jdk/Contents/Home/bin/java
Identifier=com.oracle.java.java
Format=Mach-O thin (arm64)
CodeDirectory v=20500 size=768 flags=0x10000(runtime) hashes=13+7 location=embedded
VersionPlatform=1
VersionMin=720896
VersionSDK=721152
Hash type=sha256 size=32
CandidateCDHash sha256=b59d03b829920870dc190869a7f80229bd9f03a1
CandidateCDHashFull sha256=b59d03b829920870dc190869a7f80229bd9f03a137eee6d68d40231e19532318
Hash choices=sha256
CMSDigest=b59d03b829920870dc190869a7f80229bd9f03a137eee6d68d40231e19532318
CMSDigestType=2
Executable Segment base=0
Executable Segment limit=16384
Executable Segment flags=0x1
Page size=4096
Launch Constraints:
	None
CDHash=b59d03b829920870dc190869a7f80229bd9f03a1
Signature size=9011
Authority=Developer ID Application: Oracle America, Inc. (VB5E2TV963)
Authority=Developer ID Certification Authority
Authority=Apple Root CA
Timestamp=Jun 14, 2023 at 3:19:58 AM
Info.plist entries=5
TeamIdentifier=VB5E2TV963
Runtime Version=11.1.0
Sealed Resources=none
Internal requirements count=1 size=180
<?xml version="1.0" encoding="UTF-8"?><!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "https://www.apple.com/DTDs/PropertyList-1.0.dtd"><plist version="1.0"><dict><key>com.apple.security.cs.debugger</key><true/><key>com.apple.security.cs.allow-jit</key><true/><key>com.apple.security.device.audio-input</key><true/><key>com.apple.security.cs.disable-library-validation</key><true/><key>com.apple.security.cs.allow-dyld-environment-variables</key><true/><key>com.apple.security.cs.allow-unsigned-executable-memory</key><true/></dict></plist>

codesign --display --verbose=4 --xml --entitlements - /Users/xxx/code/venv/myenv/bin/python3
Executable=/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.9/bin/python3.9
Identifier=com.apple.python3
Format=Mach-O universal (x86_64 arm64)
CodeDirectory v=20400 size=629 flags=0x0(none) hashes=9+7 location=embedded
VersionPlatform=1
VersionMin=786432
VersionSDK=852736
Hash type=sha256 size=32
CandidateCDHash sha256=377708588a6e2a74b5f2a861e627f4c98d380fe0
CandidateCDHashFull sha256=377708588a6e2a74b5f2a861e627f4c98d380fe09199fc25b219ab0e080e2c52
Hash choices=sha256
CMSDigest=377708588a6e2a74b5f2a861e627f4c98d380fe09199fc25b219ab0e080e2c52
CMSDigestType=2
Executable Segment base=0
Executable Segment limit=16384
Executable Segment flags=0x1
Page size=4096
Launch Constraints:
	None
CDHash=377708588a6e2a74b5f2a861e627f4c98d380fe0
Signature size=4412
Authority=Software Signing
Authority=Apple Code Signing Certification Authority
Authority=Apple Root CA
Info.plist=not bound
TeamIdentifier=59GAB85EFG
Sealed Resources=none
Internal requirements count=1 size=68
<?xml version="1.0" encoding="UTF-8"?><!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "https://www.apple.com/DTDs/PropertyList-1.0.dtd"><plist version="1.0"><dict><key>com.apple.application-identifier</key><string>com.apple.python3</string></dict></plist>

[cmacdonald]

I think the problem here is that your Python does not include com.apple.security.cs.allow-jit or com.apple.security.cs.disable-executable-page-protection entitlements

python/cpython#84379 and https://bugs.python.org/issue43964 suggests that the correct entitlements have been present since Python 3.7.7 and 3.8.2.

On my (Intel) macOS 13.5.1, /usr/bin/python3 is version 3.7.6 and has no entitlements.
My Anaconda installed version of Python 3.8.18 on intel is not signed at all (perhaps as there is no need for entitlements on that platform)

Could you try installing a more recent version of Python and verify it has the relevant entitlements?

[cmacdonald]

AND, to verify that we can identify such a problem, could you try to install the Pyjnius version in #698 on the original non-working Python:

i.e.

pip install git+https://github.com/cmacdonald/pyjnius.git@issue_667

should fail to install on your /Users/xxx/code/venv/myenv/bin/python3

[siddhsql]

Hi Craig, thanks for investigating this issue but I don't think entitlements is the cause of the problem. I am not an expert but here are the reasons I think so:

• You say On my (Intel) macOS 13.5.1, /usr/bin/python3 is version 3.7.6 and has no entitlements. but i am assuming it works. So then, entitlements is not the problem.
• the error is a SIGBUS error. if it was related to security permissions, i would expect to see some other error.
• the link you provide complains about not being able to find alibrary. That is not the case here. the program is able to find libjvm but the invocation fails.

this is what my best friend had to say:

A SIGBUS error is a signal in Unix and Unix-like operating systems (including Linux and macOS) that indicates a bus error. This error occurs when the program tries to perform an unaligned memory access at the hardware level, specifically referring to accessing memory on a computer's bus.

Here are a few common causes of a SIGBUS error:

Unaligned memory access: Some architectures, especially older ones, require data to be aligned in memory. If a program attempts to access memory in a way that violates alignment requirements, it can result in a SIGBUS error.

Hardware failure: In rare cases, a SIGBUS error might be an indication of a hardware problem, such as issues with the computer's memory or other low-level components.

Operating system issues: Bugs or errors in the operating system's kernel or low-level libraries could also lead to SIGBUS errors.

When a program encounters a SIGBUS error, it typically terminates and produces a core dump, which is a file containing a snapshot of the program's memory at the time of the crash. Analyzing the core dump can help developers diagnose and fix the issue.

If you encounter a SIGBUS error, it's important to review your code for potential memory access issues, such as unaligned data access. Additionally, checking for hardware issues and ensuring that your software is compatible with the architecture and platform it's running on can be helpful in resolving SIGBUS errors.

[siddhsql]

also I am confused about how this whole thing works. does python call C++ code (using ctypes) and the C++ code then call Java (using JNI)? could you explain?

[cmacdonald]

You say On my (Intel) macOS 13.5.1, /usr/bin/python3 is version 3.7.6 and has no entitlements. but i am assuming it works. So then, entitlements is not the problem.

Entitlements are only activated for Apple Silicon. I have an intel.

the link you provide complains about not being able to find a library.

Similarly, again entitlements are only activated on Apple Silicon (where this cannot find library error was detected). That version of Python was another error where there was missing entitlements.

A SIGBUS error is a signal in Unix and Unix-like operating systems (including Linux and macOS) that indicates a bus error. This error occurs when the program tries to perform an unaligned memory access at the hardware level, specifically referring to accessing memory on a computer's bus.

SIGBUS is mentioned in unicorn-engine/unicorn#1470 about jit_write, MAP_JIT etc.

I wont comment on the genericity of ChatGPT.

I suggested a method of working around the problem - given how late I worked on it last night, I'd appreciate if you can attempt it.

[cmacdonald]

@siddhsql any news on using a more recent version of Python?

[siddhsql]

I will have to hold off on it for a bit. I think you might be correct in your diagnosis:

• the error happens with jpype also (and only on apple silicon) which suggests issue might not be with jpype or pyjnius or Java for that matter, but the environment

Should I use pyenv to install newer version of Python? which one? could you provide more clear instructions as to next steps?

[cmacdonald]

Should I use pyenv to install newer version of Python? which one? could you provide more clear instructions as to next steps?

Can you try installing 3.11.5 from:
https://www.python.org/downloads/macos/

the error happens with jpype also (and only on apple silicon) which suggests issue might not be with jpype or pyjnius or Java for that matter, but the environment

Entitlements are held by the executable binary. You already showed that the Java executable has entitlements; Python does not. Entitlements are needed by the JVM (its a JIT compiler), so when the JVM is initialised from the Python binary these entitlements are missing, therefore causing the SIGBUS.

[cmacdonald]

Any news @siddhsql?

[siddhsql]

Hi Craig, unfortunately I won't be able to get back anytime soon. I am parking this issue for later.

…

On Fri, Nov 24, 2023 at 3:29 AM Craig Macdonald ***@***.***> wrote: Any news @siddhsql <https://github.com/siddhsql>? — Reply to this email directly, view it on GitHub <#667 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/A6NWEK5T32FTJ4A2MXETNSLYGCAIJAVCNFSM6AAAAAA4DTWRFKVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTQMRVGUZTOOBVG4> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[cmacdonald]

An update from my side: I now have access to an Apple M1 mac. I can confirm that the Apple provided Python (/usr/bin/python3) does not work, but an Anaconda version of Python does work (as far back as Python 3.8). Neither appear to have entitlements!

[Julian-O]

Closing as not planned.