The jpype from thrameos

forName()

The jpype documentation specifically gives a workaround because forName() produces the wrong result. It would be a good idea to place a customizer that implements the work around automatically so that other team members don't hit the problem.

Str/Unicode

In the devel version I make it automatically select between unicode and str for Python 2.7 where there is a difference. Python 3 is always str (which is unicode). But Python 2 expects __str__ to always produce a str regardless of whether it is really a unicode rep or not.

We need to add specific methods that force it to the correct type for toString on Python 2.x.

Segfault on double startJVM()

To reproduce call

import jpype
jpype.startJVM(jpype.getDefaultJVMPath())
try:
   jpype.startJVM(jpype.getDefaultJVMPath())
except RuntimeError:
  pass
a=jpype.JArray(jpype.JInt, 1)(10)

isinstance()

One of the most challenging pieces of starting work with jpype was the tendency to need to get into the inner jpype private classes to be able to determine if something was an java object, class or method. The base classes for these are not exposed because those classes have no public members. It would be a good idea to expose a stub version for each of the Java types without an functionality to support isinstance().

Jython interaction

The addition of the imports system seems to have created a minor issue with certain packages which have hooks to Jython. Our import system was sufficiently compatible with the Jython system that some packages are picking up Jpype and assuming it is Jython leading to somewhat odd behavior. I have changed the Exceptions issued to that the error handling for Jython in most packages is sufficiently clean to work. But not every package may be handle this well. Need to update the documentation in Jpype to make others aware of the issue.

Better error reporting on overload

When executing a complicated command while using a builder pattern such as

sg=sgb.add(bkg).includeWeightedSNR(sg1,10.0).miscalibrate(0, 0.01).create()

it is not clear which overloaded method is causing an error.

RuntimeError: No matching overloads found. at native\common\jp_method.cpp:117

At bare minimum the name of the method should appear in the error message. Although the native line number is useful for the developer it is likely meaningless for the user.

Investigate exception reported in testDoc.py

=========================================================
Exception while removing reference.

Get name of arrays

It is difficult to get the name of an array type from the Jpype object. It lacks a __name__ field. We have patched this with a customizer, but it would be much easier to just port this back to jpype.

Module reload

Currently the module can only be started once. If it is started a second time the resources from the previous run will become active but in a bad state.

To fix this each resource should have a pointer to the jvm they were created under. If that jvm is different than the live one they should through a runtime error. This would make it so that we can handle a restart. However, in addition we need to make sure that shutdown gets called on a module reload.

Automatic type conversion via python

Currently all the logic regarding deciding if a type can be converted must be hard coded into C++. This means we are very limited in our handling. Adding a type conversion table to each class that can register a piece of python code which will convert to a valid convertible type would solve that issue.

See about making List satisfy Sequence contract

Like 1107 in documentation

String integration

As noted in the project section, we need to consider the integration of java and python strings.

Elementwise access for buffers?

Line 1144 in user doc.

Exceptions in initialization

If an exception happens during initialization, the system tries to convert it to a java exception to return to python. But if the system is not fully this can lead to a segfault. Add appropriate safety check to prevent this type of error.

Stack trace integration

Fake a traceback object when emitting a python exception so that the stacktrace from C++ and Java is included in the python rather than appending it to the message string.

Class table overrun bug

If multiple threads call JClass for the same class at the same time, they can end up getting different copies of class because it is possible to interrupt during the time of class cache check and class creation.

To replicate this create a custom class. Add a class customizer with a wait call. Start multiple threads and then each request the custom class. Compare the resulting classes and they will not all be the same.

epypj master list

Getting started?

Hi,
I'm a Java guy and only a casual Python (and pyjnius) user. I saw @Thrameos's issue on the pyjnius project and am interested in comparing the two myself, but I'm having a hard time getting started. Is there a document for Python newbs that explains how to set up JPype that I'm missing?

Spyder inspect

Spyder is able to create a segmentation fault when calling inspect on a Jpype class. Once in a blue moon it actually dumps a python stack trace showing which entry point in jpype is faulty. I have added bullet proofing code for all of the major entry points when the JVM is not started, but this is clearly another failure mode. We can dodge this one as it is a corner case for now, but it may give cause to force us to use an alternative solution if our other developers feel this is not a stable solution.

Examine what is in tcalmant branch

I believe tcalmant is mostly outdated and should run out of the box with our master, but I need to verify it.

Looking over the contents here the differences from the last branch point

Okay so next mark the stuff that we have covered.

JArray lacks a convert method.

If you have a sequence and you need a java array, it has be be converted each time it is passed rather than converted and held. Consider overloading JArray(type, values=) to create the array initializing from a python list.

This should not be hard to support as there is already a convertToJava method that can give the java array instance.

Auto Proxy

Any callable should be able to go into a proxy if it requires only one callable method such as a @FunctionalInterface. This could be driven by checking for this attribute on the class then checking if the method is callable.

Make JChar derive from unicode

To make it more consistent we should find a way to make JChar derive from str.

Check if Java primitives are closed

Cygwin integration

Unfortunately, my best debugging tools for jpype are in cygwin as the Visual Studio version I am using for other projects are restricting the windows build. Unfortunately, as cygwin is not yet on the head I have to keep backporting the fixes from the cygwin port to the main. Poor marscher must be taking a look at the pull requests and then his watch and deciding maybe another time. Need to find a way to purchase marscher a beer.

No numpy checklist

Attempt to deal with two (seemingly) mutually exclusive requests.

Make JPype integrate with numpy better for better speed performance.
Make JPype not require numpy to operate.

Unicode bypass

Several sections of code are using special methods in python3 and python2. They are properly marked so that they only activate on the correct version, but this means somewhat arbitrary differences which can lead to errors. Either the methods need to work on both versions of python, or they need to be documented to be specific to a version.

Thunk code missing

Several parts of jpype depend on precompiled classes which are stored as arrays in C++. There appears to be no code that compiles these byte code nor convert it to the source code. Thus there is no way to verify what it in the code.

Numpy to array conversion

Numpy arrays of are not being properly handled and converted into java arrays when passed to a constructor, resulting in a "No matching overloads found" error. Small example code and java class is below:

import numpy as np
arr = np.arange(0, 10, dtype=np.int32)
n2a = Numpy2Array(arr)
Traceback (most recent call last):

  File "<ipython-input-11-b39ee7ec0b3b>", line 1, in <module>
    n2a = Numpy2Array(arr)

  File "/Users/monterial1/anaconda3/lib/python3.6/site-packages/JPype1-0.6.3-py3.6-macosx-10.7-x86_64.egg/jpype/_jclass.py", line 111, in _javaInit
    *args)

RuntimeError: No matching overloads found for [init in find. at native/common/jp_method.cpp:127

And the java class is:

public class Numpy2Array
{

  private int[] data = null;
  
  public Numpy2Array(int[] data)
  {
    this.data = data;
  }
  
  public void PrintArray()
  {
    for (int i = 0; i < this.data.length; i++)
    {
      System.out.println(this.data[i]);
    }
  }
}

JPype crashing with "AttributeError: _hints can't be set"

Hi,

With this new version of JPype and with the case below, I have this error

File "venvs/pyenv/lib/python3.7/site-packages/JPype1-0.7.2-py3.7-linux-x86_64.egg/jpype/_jclass.py", line 141, in _JClassPost
    res._hints = hints

Below, an example of code to reproduce the problem, the code may seem weird but originally it is in Kotlin and the enum entry holds an instance of a sealed class

TestField.java:

package com.company;

public abstract class TestField {

    private final String id;

    private TestField(String id) {
        this.id = id;
    }

    public static final class FieldImpl extends TestField {

        public static final FieldImpl INSTANCE;

        private FieldImpl(String id) {
            super(id);
        }

        static {
            INSTANCE = new FieldImpl("FIELD_1");
        }
    }
}

DFAlias.java

package com.company;

public enum DFAlias {
    TEST_FIELD_1(TestField.FieldImpl.INSTANCE);

    private final TestField field;

    private DFAlias(TestField field) {
        this.field = field;
    }

    public TestField getField() {
        return field;
    }
}

test.py

import jpype
jpype.addClassPath("/home/bjuin_exd/workspace/MyJar/out/artifacts/MyJar_jar/MyJar.jar")
jpype.startJVM()
from com.company import DFAlias
alias = DFAlias.valueOf("TEST_FIELD_1")
print(alias)
print(alias.getField())

startDefaultJVM()

Most of our usage of jpype is interactive with spyder. As used the jvm is started by startup.py. However, sometimes the script must also be used independently. Having a simple safe call to the start the jvm with default settings which cannot fail would help support both.

Closed classes

Currently calling a setattr on a java object can succeed even if there is nothing on the java side to support the call. This has been found to a be a source of bugs in our development. A static member variable may be exposed and set in the python code. Later we remove that static. As the python code silently accepts this setting, we end up not seeing the problem or worse think that the action has the desired effect. We have patched the base class with a customizer, but depending on the order that classes are constructed it is not always getting called. Either customizers need to be enhanced to work on classes later, or we need to get this change pushed to the jpype base. How much code this will break remains to be seen.

Item check list

Integration of java.util.Date and python datetime

In order to facilitate plotting of data, we need to wrap java.util.Date so that numpy can use java.util.Date directly. This will need to bit flexible as the datetime is immutable and java.util.Date is not. No sure if this is possible. Will need to hide all deprecated members and hide setTime() so that it meets the python contract need for dict. The only downside to this someone may return a Date with the intent to set it. Will likely document the issue.

Task list for JPype 0.8 Release

Get all pull requests pulled.
Finish the documentation.
Scan documentation for api issues that require action.
Scan issue list for api issues that require action.
Resolve api issues.
Remove deprecated code.
Test.
Check with stakeholders.

Looks like about two weeks of work. Apply a factor of PI and we are on target for a June release.

java.io.Closeable

Brandon would like to make sure he can use this structure to close our resources.

with MyResource() as resource:
   resource.doSomething()
#  resource.close() called here

Too many layers to the onion.

The separation by HostRef to support ruby leads to massive amounts of code replication and difficult to follow code. Some of the replicated paths have diverged sufficiently that it is unclear as to which copy is correct.

Segfault on bad JVM heap size

It appears that setting the size of the JVM incorrectly is a segfault. No trace is obtainable so this will be hard to track down. It may be the JVM itself.

from jpype import *
print('hello')
startJVM(getDefaultJVMPath(), '-Xms1024')
print('there')

Support for class operations

Jpype seems to have to levels of classes in the getClass() and javaclass. Neither appears to be a complete copy of the functions of java classes. One is the python class which is structurally different as we can't add methods to it that won't appear in the object and the other is the hooks needed for jpype. We should enhance the jpype one to give access to all of the things we use for reflection such as getting an individual method and accessing the annotations. This is likely a heavier this lift than can be supported right now.

math operations

We have integrated gov.llnl.math into jpype as a customizer which allows use of numpy like operations for java classes. This greatly increases run speed when performing prototyping and testing as we can perform math without ever moving the arrays from the java jvm. Unfortunately, it will take heavy lifting to get this modification out of the laboratory IP release system. Also the jpype does not currently include an ant build to create a org.jpype jar which would be able to hold the required enhancement hooks. The downside is the existing customizer may under certain situations create an unenhanced array class which leads to a method error. Fortunately, this is sufficiently rare we can work around easily by reordering calls.

Speed

@xhochy With regard to https://uwekorn.com/2019/11/17/fast-jdbc-access-in-python-using-pyarrow-jvm.html, have you made any attempts to measure the speed after the JPype conversion to a CPython-API?

The blog you posted was dated November 2019 and since then we have had a factor of 3 speed up in return path, a factor a 4 on the call dispatch, and another factor of 20 to 400 on array transfers depending on the type. I am sure that marshalling arguments over one at a time is still much slower than a dedicated batch transfer but I would hope that we are substantially faster than 2 hours at this point.

Docstring generation

Replace the docstring for class and methods so that it gives arguments and overloads when using

Class?
object?
object.method?
object.field?

Paths.get()

One of the common patterns we are working around is the need to call Paths.get() so that we don't end up adding load(String) overloads everywhere we want to use Path. The key issue here is the lack of varargs support in jpype. I have submitted a patch in for this. But until it makes in to a release, I am going to have to continue to wage this battle. Unfortunately, unless matlab also comes up with an easier way to access varargs java methods, this may be a losing one for now.

Segfault on call abstract method

In order to allow jpype to call the method on a base class the method lookup calls invoke with an option to not call anything deeper than requested method. This is fine unless somehow you manage to invoke an abstract method. This currently goes down in flames on the develop branch while handling the exception on the java side.

Tasks:

Test this on master
Stop the segfault.
Add to test cases.

Customizer rework

The current customizer system works by adding or removing methods and base classes prior to the creation of the class. This is allowed to run arbitrary user code.

Limiting the types of actions to a fixed set of prescribed actions would make it much safer and ensure those actions take place consistently. It would also make it more future proof as the actions would be executed by the library code rather than by conventions.

Verify the behavior of underbar with Java symbols

These should always be hidden as privates on members and fields.

Implements

Make it such that proxies look like a java declaration.

# Declare a python class to implement an interface
class MyProxy(implements("java.util.Function")):

    # Explicitly declare what method that we are overloading/
    @Override("java.lang.Object")
    def apply(self, argument):
        return 0

Implements would create a new proxy class for that interface. It would verify that all of the interfaces are satisfied by checking to see that all required methods are implemented. It would use the proxy interface to find the best overload given the arguments. The @OverRide decorator would be used to allow multiple functions. They would be renamed by decorator to non-conflicted.

Place python project in jar.

We will often need to ship a jars only to users, but some of the visualization software will be in python. Shipping a separate python package means the potential for version shear. It would be best if we can get a hook into the jpype.imports system to check to see if init.py exists in the jar and load the python module from there. There are some existing examples for loading a module from memory in stackoverflow and the python docs. Perhaps one of them would be a working base.

thrameos / jpype Goto Github PK

jpype's People

Contributors

Watchers

Forkers

jpype's Issues

Recommend Projects

Recommend Topics

Recommend Org