Build: Add Java 25 support by adding arrow-memory-unsafe to dependencies#15931
Closed
wangyum wants to merge 1 commit intoapache:mainfrom
Closed
Build: Add Java 25 support by adding arrow-memory-unsafe to dependencies#15931wangyum wants to merge 1 commit intoapache:mainfrom
wangyum wants to merge 1 commit intoapache:mainfrom
Conversation
…lity Arrow's NettyAllocationManager static initializer fails on Java 25 (JEP 471) because PooledByteBufAllocatorL calls EmptyByteBuf.memoryAddress() via sun.misc.Unsafe, which is no longer supported. Both arrow-memory-netty and arrow-memory-unsafe ship org.apache.arrow.memory.DefaultAllocationManagerFactory and when the netty variant loads first the JVM crashes at startup. Changes: - Remove arrow-memory-netty from iceberg-arrow; use arrow-memory-unsafe exclusively as the Arrow allocator - Add defensive netty excludes to arrow-memory-unsafe (consistent with arrow-vector exclusions in the same block) - Add netty-common library alias to libs.versions.toml (version.ref = netty-buffer, since Netty publishes all modules at the same version) - Use libs.netty.common catalog alias for ArrowReaderTest instead of an inline string interpolation against the wrong version key - Remove dead arrow-memory-netty alias from libs.versions.toml Closes apache#15930 Co-Authored-By: Claude (claude-sonnet-4-6) <<EMAIL_ADDRESS>>
Member
Author
|
Fixed it by adding |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Background
The Problem: Default Netty Allocator Fails on Java 25
When running Apache Iceberg / Spark with Apache Arrow on Java 25, the default memory allocator (
arrow.allocation.manager.type=Netty) will crash our jobs with anExceptionInInitializerErrorandUnsupportedOperationException.This happens because the older version of Netty bundled inside Iceberg relies heavily on deep JVM internals to access memory. Specifically, Java 25 completely removed the SecurityManager (JEP 486) and changed internal memory structures (JEP 471) that Netty was looking for. Because these internal classes and fields no longer exist, no amount of --add-opens flags will fix the crash.
The Solution: Use the Unsafe Allocator
To avoid this issue on Java 25, we must bypass Netty entirely by adding the following to our Spark configuration:
But it will throw
java.lang.ClassNotFoundException: org.apache.arrow.memory.UnsafeAllocationManager. So we need to add arrow-memory-unsafe to dependencies.Closes #15930
How to test
/Library/Java/JavaVirtualMachines/zulu-17.jdk/Contents/Home/bin/javac -cp ".:*" Reproducer.java/Library/Java/JavaVirtualMachines/zulu-25.jdk/Contents/Home/bin/java -cp ".:*" --add-opens=java.base/java.nio=ALL-UNNAMED Reproducer, it will throw exception. We should add-Darrow.allocation.manager.type=Unsafeto run it:/Library/Java/JavaVirtualMachines/zulu-25.jdk/Contents/Home/bin/java -cp ".:*" --add-opens=java.base/java.nio=ALL-UNNAMED -Darrow.allocation.manager.type=Unsafe Reproducer