juliapy/pythoncall.jl

Python and Julia in harmony.

pythoncall.jl's Issues

Set Jill upstream when checking versions

Make it so tryconvert rules can assume S is a DataType

... or nested UnionAll of DataType.

Split any Unions into their parts and pass each part separately into the rule.

Any TypeOfBottom can be ignored.

Test on earlier versions of Julia and Python

How can we mix Python and Julia code without relying on finalizers?

e.g. we could have a macro which transforms ordinary Julia code to track which variables are Python objects and decrefs them as appropriate. These Python objects could just be pointers, as if we were writing CPython directly. We'd need to ensure that any Julia code mixed in could not raise uncaught exceptions.

_type_lb and _type_ub are wrong

e.g. _type_ub(Vector{Vector{T}} where {T<:Int}) == Vector{Vector{Int}}, when the intention is that it is Vector{Vector{T} where {T<:Int}}

coming from PyCall section in docs

Rename the PyCall page to include both the comparison and some migration tips.

Use timestamp of juliacallmeta.json

Get module load time down

Currently about 7 seconds, 80% compilation

Release on pypi

Pickle doesn't work on julia modules

Bit of a strange request, but there is a use case. A minimal example is:

import pickle
from juliacall import Base
pickle.dumps(Base)
# Traceback (most recent call last):
#   File "<stdin>", line 1, in <module>
# TypeError: cannot pickle 'juliacall.ModuleValue' object

This would be useful for defining functions that are passed around in Python. I initially ran into this when using ray, so a slightly more complex example that has the same behaviour:

import ray
ray.init()
from juliacall import Base
@ray.remote
def v(x):
    Base.println(x)
x = ray.get(v.remote(1))

I had a look at PyCall as well, but they've got a similar issue here JuliaPy/PyCall.jl#863, with a few comments on how to fix it.

Any ideas? I'll try to have a look into this further, but any help getting this working would be appreciated.

Using Python.jl and PyCall.jl together

If I do:

using Python
using PyCall

the latter segfaults (at least on 1.6-rc1). For the reverse order I do get an error.

I don't personally need both to work, just curious if they could (probably not). That's not really the end-goal, but some packages do have PyCall as a dependency, e.g. PyPlot.jl and Pandas.jl. Is there a good way to support such, as it seems this package has a similar API as PyCall?

Faster Py_IncRef/Py_DecRef

There is a considerable overhead from calling a C function (~15ns).

In the case of Py_IncRef/Py_DecRef we could modify the ref count directly from Julia, only calling Py_DecRef once the ref count is <2.

"writing packages" section in docs

Export pynull, pycopy and pyisnull. Rename to unsafe_pycopy. Document these in a section about writing packages.

InitError

Hi, interesting project

I'm getting this error even when using a fresh Julia installation (1.6.3). OS: Mac OS 10.13.6.

julia> using PythonCall
ERROR: InitError: MethodError: no method matching project_file_name_uuid(::Bool, ::String)
Closest candidates are:
  project_file_name_uuid(::String, ::String) at loading.jl:405
Stacktrace:
 [1] init_context()
   @ PythonCall.C ~/.julia/packages/PythonCall/YSeRU/src/cpython/context.jl:30
 [2] __init__()
   @ PythonCall.C ~/.julia/packages/PythonCall/YSeRU/src/cpython/CPython.jl:19
 [3] _include_from_serialized(path::String, depmods::Vector{Any})
   @ Base ./loading.jl:696
 [4] _require_search_from_serialized(pkg::Base.PkgId, sourcepath::String)
   @ Base ./loading.jl:782
 [5] _require(pkg::Base.PkgId)
   @ Base ./loading.jl:1020
 [6] require(uuidkey::Base.PkgId)
   @ Base ./loading.jl:936
 [7] require(into::Module, mod::Symbol)
   @ Base ./loading.jl:923
during initialization of module C

I've tried both:

pkg> add PythonCall

and

pkg> add PythonCall#master

Don't understand how to pass numpy arrays to Julia function

Let's consider the code fragment

import numpy as np

X = np.ones((10,10))
Y = np.ones((10,10))

x, y = jl.MyModule.compute_something(X, Y, -1.0, 1.0, -1.0, 1.0)

where I call a Julia function defined in the MyModule module. I am getting the error

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: Julia: MethodError: no method matching compute_something(::PythonCall.PyArray{Float64, 2, Float64, true, false}, ::PythonCall.PyArray{Float64, 2, Float64, true, false}, ::Float64, ::Float64, ::Float64, ::Float64)
Closest candidates are:
  compute_something(!Matched::Matrix{T}, !Matched::Matrix{T}, ::T, ::T, ::T, ::T) where T<:Real at /opt/julia/compute_something:362

Does this mean that I need to define a new method for compute_something() with ::PythonCall.PyArray arguments? Can I avoid doing that? From the documentation it is not clear what to do in this situation.

Document unsafe API

pynew, pydel!, pycopy!, pyisnull, unsafe_pynext

Run Aqua tests

Highlight the differences with PyCall

Hi! Thanks for this project. It would be nice if the README could highlight the difference between PythonCall and PyCall, so users can easily decide which package to use; or if both can be used together for particular tasks. Thanks!

Possible legal issues with the logo

I once wanted to use a logo in colors of python in order to highlight that my project is intended to be used with python (it is essentially python bindings to some third-party code), but feeled like they can be trademarked.

I have contacted the email and asked for clarifiaction and permission. I was told that currently they don't intend to trademark python colors and that if I start using my logo, I'd "be" a my trademark (damn this legal shit, I wish there would be no trademark laws at all!). But the colors are already a de-facto trademark.

I asked if they can either give me a digitally signed text that the use of the colors in my logo is not considered trademark violation, or change their trademark policy to clarify which uses of colors are explicitly allowed. I was answered that this is the decision which is up to the board, and the board won't do that for me unless maybe for a lot of money.

IMHO this FUD situation is a bit annoying.

https://pyright.blogspot.com/2011/02/svg-xml-and-python-logo.html
Nuitka/Nuitka-website#19
PyAV-Org/PyAV#728
derb12/mcetl#13
https://github.com/ReagentX/Logria-py/issues/147
python-trio/trio#1828
amcc1996/symbeam#7
#3
fatiando/pooch#222
aiortc/aiortc#442
MatthijsKamstra/haxepython#3

Reloading code

Hi there,

Just wondering - is there a way to reload the code from a Python script without restarting the current Julia session?

Thanks in advance! :)

More test coverage

Is rewrite complete now ?

Is rewrite complete now ?

If finished rewrite , can you release a version as soon as possible?

Support __copy__ and __deepcopy__?

juliacall.AnyValue.__copy__ and __deepcopy__ could call Julia's copy and deepcopy.

Conversely, copy(::Py) and deepcopy_internal(::Py, ::IdDict) could call Python's copy.copy and copy.deepcopy.

https://docs.python.org/3/library/copy.html

pymacro next, print, int, ...

pymacro functions, classes, decorators

Todo list

Py<Wrapper>

• PyIterable
• PyList
• PyDict
• PySet
• PyBuffer
  • General parsing of format strings.
  • Unaligned types.
  • Byte-reversed types.
• PyArray
  • Support the numpy array interface (__array__, __array_interface__).
  • Allow conversion to/from tuples containing python objects.
• PyPandasDataFrame
  • Tables interface
  • TableTraits interface
  • Customise column types and index
  • Convert pandas.core.indexes.range.RangeIndex to a StepRange{Int}?
• PyIO
  • Currently limited to byte-based IO and extremely slow. Perhaps need to manually buffer each way?
• PyObjectArray <: AbstractArray{PyObject}, represented as an array of CPyPtr, used by default as output of pyjulia(::AbstractArray).__array__().

pyjulia

• Sequence interface (tuples and vectors)
• Buffer protocol (strided arrays)
• Numpy array protocol (__array_interface__)
• Numpy array conversion (__array__) returning PyObjectArray by default unless __array_interface__ already supported.
• Generalize pytypestrformat to work on any type.
• Mapping interface (dicts, named tuples?)
• Set interface (sets)
• Number interface
• IO interface (IO=IOBase, then subclasses for raw, buffered and text IO)
• An argument parsing function or macro
• Separate pyjlraw from pyjl
• Have a BufferedIterator which iterates over 1024 (say) values at a time and saves them as a buffer of PyObject.

pyconvert

• dates, times
• ctypes
• to named tuple
• arrays (buffer protocol, array interface, and __array__)
• IO
• numbers
• numpy scalars (np.float32 etc)

Others

• README
• Documentation
• Logo
• Testing & CI
• Migrate CI away from Travis. GitHub Actions?
• Move configuration (finding libpython) all to run-time & dynamically load/cache function pointers as required.
• Multimedia display (MIMEs and corresponding _repr_MIME_ methods from ipython)
• Hook into Julia documentation system
• QT_PLUGIN_PATH hack
• GUI event loops to support interactive plotting (e.g. with matplotlib)
• @pyeval, @pyexec, py"..." for calling pyeval and pyexec with Julia value interpolation
• ?some_python_function should show the function name and signature
• Add GIL manipulation functions (pygiloff, pygilon, etc)
• When running inside an IPython kernel, change the default output display to go the kernel
• When running inside an IJulia kernel, change sys.stdout to go the the kernel
• When running inside an IPython or IJulia kernel, start up the other kind of kernel and link them together.

DSL avoid checking embedded Python objects

We embed PyInternedStrings and PyLazyObjects into the AST, but accessing them still does a NULL check and IncRef each time.

Instead, we could keep a list of all the lazy objects used, and a bool indicating whether or not they have all been checked, and check this once at the start of the DSL block. So the first time, all these objects are initialised, and thereafter we can just extract the pointer value directly.

Error when calling Julia function defined in module

I am trying to call a Julia function defined in a module (say MyModule, defined in MyModule.jl) in a Python script. I did something along the lines of

from juliacall import Main as jl

jl.seval('push!(LOAD_PATH, "<path to MyModule.jl>")')
jl.seval("using MyModule")

ret = jl.MyModule.some_function(arg1,arg2,arg3)

however I am getting an error:

juliacall.Error: ArgumentError: Package MyModule not found in current path:
- Run `import Pkg; Pkg.add("MyModule")` to install the MyModule package.

What am I doing wrong?

Precompile statements (and error)

Hi, I CAN install your package on a clean install (not otherwise, see below), but as you know startup is rather slow.

A.
I added precompile statements and got down to (still higher than what I would like to see):

$ ~/julia-1.6.0-rc2/bin/julia -O0 --compile=min --startup-file=no -q
julia> @time using PythonCall
  2.384556 seconds (2.29 M allocations: 143.823 MiB, 1.21% gc time)

Still rather bad:
$ ~/julia-1.6.0-rc2/bin/julia -O0  --startup-file=no
julia> @time using PythonCall
 17.399345 seconds (44.16 M allocations: 2.541 GiB, 6.10% gc time, 0.02% compilation time)

and even worse:
$ ~/julia-1.6.0-rc2/bin/julia --startup-file=no -q
julia> @time using PythonCall
 26.929030 seconds (44.16 M allocations: 2.541 GiB, 3.96% gc time, 0.02% compilation time)

compared to your package as is:

julia> @time using PythonCall
 31.716683 seconds (55.21 M allocations: 3.177 GiB, 5.03% gc time, 0.02% compilation time)

$ ~/julia-1.6.0-rc2/bin/julia -O0 --compile=min --startup-file=no -q
julia> @time using PythonCall
  6.888381 seconds (19.04 M allocations: 1.109 GiB, 9.42% gc time)

I used a naive way, not SnoopCompile.jl, so maybe it's better. I'm also thinking, do you need to do this much at init? Can you postpone some things, and do later lazily?

B.
The above worked with the new Julia 1.6.0-rc2, but with my original .julia folder I get (on that version and later, including 1.53):

(@v1.6) pkg> add PythonCall.jl
    Updating registry at `~/.julia/registries/General`
ERROR: refusing to add package `PythonCall [6099a3de]`: package `PythonCall = "6099a3de-0909-46bc-b1f4-468b9a2dfc0d"` already exists as a direct dependency

Do you have any idea what it means (this is before my work with precompile statements)?

At least I thought you would like to know, as your users might encounter this, and I have no idea where to begin to debug this. Except, while it doesn't seem to strictly be about your package (or is it?), only an interaction with my (broken?) setup:

(@v1.6) pkg> st
      Status `~/.julia/environments/v1.6/Project.toml`
  [7d9fca2a] Arpack v0.5.1
  [1dc0ca97] ArrayTools v0.1.0 `https://github.com/emmt/ArrayTools.jl#master`
  [77e5a97a] AsyPlots v0.2.1
  [c52e3926] Atom v0.12.30
  [b6338580] BPFnative v0.1.0 `https://github.com/jpsamaroo/BPFnative.jl.git#master`
  [6e4b80f9] BenchmarkTools v0.5.0
  [7cffe744] BetterExp v0.1.0 `https://github.com/oscardssmith/BetterExp.jl#master`
  [7f725544] BinaryBuilderBase v0.5.0 `https://github.com/JuliaPackaging/BinaryBuilderBase.jl.git#master`
  [ad839575] Blink v0.12.4
  [336ed68f] CSV v0.8.3
  [159f3aea] Cairo v1.0.5
  [49dc2e85] Calculus v0.5.1
  [710b7bb7] ClearStacktrace v0.2.2
  [aaaa29a8] Clustering v0.14.2
  [da1fd8a2] CodeTracking v1.0.5
  [944b1d66] CodecZlib v0.7.0
  [35d6a980] ColorSchemes v3.10.2
  [5ae59095] Colors v0.12.6
  [8f4d0f93] Conda v1.5.0
  [1b08a953] Dash v0.1.3
  [03207cf0] DashBase v0.1.0 `https://github.com/plotly/DashBase.jl.git#master`
  [1b08a953] DashCoreComponents v1.12.0
  [1b08a953] DashHtmlComponents v1.1.1
  [1b08a953] DashTable v4.10.1
  [9024f26f] Dashboards v0.2.8
  [a93c6f00] DataFrames v0.22.5 `https://github.com/JuliaData/DataFrames.jl.git#main`
  [1313f7d8] DataFramesMeta v0.6.0
  [864edb3b] DataStructures v0.17.20
  [55939f99] DecFP v1.1.0
  [85a47980] Dictionaries v0.3.7
  [a1bb12fb] Electron v3.1.1
  [d872a56f] ElectronDisplay v1.0.1-DEV `https://github.com/queryverse/ElectronDisplay.jl.git#master`
  [c04bee98] ExcelReaders v0.11.0
  [53c48c17] FixedPointNumbers v0.8.4
  [186bb1d3] Fontconfig v0.4.0
  [5752ebe1] GMT v0.29.0
  [28b8d3ca] GR v0.55.0
  [c91e804a] Gadfly v1.3.1
  [4b11ee91] Gaston v1.0.2 `https://github.com/mbaz/Gaston.jl.git#master`
  [c43c736e] Genie v1.15.1
  [bc5e4493] GitHub v5.4.0
  [dc211083] Gnuplot v1.3.0
  [4c0ca9eb] Gtk v1.1.6
  [cd3eb016] HTTP v0.8.19
  [60101457] ICOADSDict v0.1.0
  [7073ff75] IJulia v1.23.2
  [e8efc688] ImPlot v0.1.1
  [5903a43b] Infiltrator v0.3.0
  [1c8ee90f] IterableTables v1.0.0
  [82899510] IteratorInterfaceExtensions v1.0.0
  [4138dd39] JLD v0.12.1
  [682c06a0] JSON v0.21.1
  [2535ab7d] JSON2 v0.3.2
  [0f8b85d8] JSON3 v1.7.0
  [aa1ae85d] JuliaInterpreter v0.8.9
  [75827d1f] JuliaTutor v0.1.0 `https://github.com/caseykneale/JuliaTutor.jl#master`
  [e5e0dc1b] Juno v0.8.4
  [fc18253b] LazyJSON v0.2.2
  [194296ae] LibPQ v1.6.2
  [23992714] MAT v0.10.1
  [10e44e05] MATLAB v0.8.0
  [1914dd2f] MacroTools v0.5.6
  [739be429] MbedTLS v1.0.3
  [c03570c3] Memoize v0.4.4
  [13650a8a] NicePipes v0.1.3
  [47be7bcc] ORCA v0.5.0
  [5fb14364] OhMyREPL v0.5.8 `~/.julia/dev/OhMyREPL`
  [3ad7be9e] OpenCV v0.1.1 `https://github.com/TakekazuKATO/OpenCV.jl#master`
  [bac558e1] OrderedCollections v1.4.0
  [ace2c81b] PETSc v0.1.0 `https://github.com/JuliaParallel/PETSc.jl#master`
  [06a5ddd6] PGPlot v0.1.0 `https://github.com/emmt/PGPlot.jl#master`
  [9b87118b] PackageCompiler v1.2.5
  [eadc2687] Pandas v1.4.0 `https://github.com/PallHaraldsson/Pandas.jl#master`
  [f5117550] PandasLite v0.1.7
  [69de0a69] Parsers v1.0.15
  [32113eaa] PkgBenchmark v0.2.10
  [f3e62ec7] PkgCleanup v0.1.0 `https://github.com/giordano/PkgCleanup.jl#main`
  [58dd65bb] Plotly v0.3.0
  [a03496cd] PlotlyBase v0.3.1
  [f0f68f2c] PlotlyJS v0.13.1
  [91a5bcdd] Plots v1.10.6
  [8e5c59b7] PlusPlus v0.1.0
  [c3e4b0f8] Pluto v0.12.21
  [c46f51b8] ProfileView v0.6.9
  [438e738f] PyCall v1.92.2 `https://github.com/JuliaPy/PyCall.jl.git#master`
  [d330b81b] PyPlot v2.9.0 `~/.julia/dev/PyPlot`
  [6099a3de] Python v0.1.0 `https://github.com/cjdoris/Python.jl#master`
  [ce6b1742] RDatasets v0.7.4
  [295af30f] Revise v3.1.12
  [22415677] RobotOS v0.7.2
  [5e6a0ad9] SafeREPL v0.0.1 `https://github.com/rfourquet/SafeREPL.jl#master`
  [88634af6] SaferIntegers v2.5.1
  [1277b4bf] ShiftedArrays v1.0.0
  [01919df6] SimpleTypePrint v0.1.2
  [aa65fe97] SnoopCompile v2.5.2
  [90137ffa] StaticArrays v0.12.5
  [4acbeb90] Stipple v0.6.0
  [30ddb3f0] StippleCharts v0.4.1
  [ec984513] StipplePlotly v0.1.0 `https://github.com/GenieFramework/StipplePlotly.jl#main`
  [a3c5d34a] StippleUI v0.2.4 `https://github.com/GenieFramework/StippleUI.jl.git#master`
  [88034a9c] StringDistances v0.8.0
  [69024149] StringEncodings v0.3.3
  [b9c42661] SwapLiterals v0.1.0 `https://github.com/rfourquet/SafeREPL.jl:SwapLiterals#master`
  [382cd787] TableTraitsUtils v1.0.1
  [05f542c5] TestGtkMegaJLL v0.1.0 `~/.julia/dev/TestGtkMegaJLL`
  [db00978d] TightBinding v0.1.3
  [37f6aa50] TikzPictures v3.3.1 `https://github.com/JuliaTeX/TikzPictures.jl.git#master`
  [30578b45] URIParser v0.4.1
  [112f6efa] VegaLite v2.4.1
  [81def892] VersionParsing v1.2.0
  [5e69872b] Xcb v0.2.0 `https://gitlab.com/poposca/Xcb.jl.git#master`
  [77ec8976] GTK3_jll v3.24.11+5
  [2e76f6c2] HarfBuzz_jll v2.6.1+10
  [9c32591e] Poppler_jll v0.87.0+2
  [ab825dc5] SDL2_jll v2.0.12+2
  [502467ad] UBPF_jll v0.0.1+0
  [3f19e933] p7zip_jll

Announce on discourse

Make docs less PythonCall-centric

i.e. put juliacall on the same footing.

• readme title
• doc home title

Remove lazy functionality from PyObject

It is basically unused, increases memory usage and (for some reason I don't understand) slows down pyptr even on objects that are not lazy.

The only instances of it are _none, _true and _false.

Investigate Python.jl for diffeqpy

Hey! Found this repo, looks cool! I'm looking into it as a possible new backing for https://github.com/SciML/diffeqpy . Specifically the issue is whether overloads in Julia for ModelingToolkit.jl could work well enough from Python. They do in diffeqr via JuliaCall, so it seems like it's just a wrapper issue with pyjulia. SciML/diffeqpy#57 (comment) in particular highlights such an issue. SciML/diffeqpy#78 is the same thing. If there's a good answer here, then I think got a new lead!

Serialising traceback objects

Python traceback objects are not picklable, and hence PyException cannot always be serialised.

As a fix, serialise the b field of a PyException as None. Or possibly always serialize traceback to None? Or to Unpicklable()?

Optional extra: if we detect that tblib is installed, allow pickling of traceback.

Document this behaviour in the compat section.

Preload C API pointers

The current behaviour lazily loads C API function/object pointers, requiring a check each time one is accessed.

Instead, load them all at init.

Release on conda

Use ArrayInterface.jl?

It might make handling conversion of arrays between Julia and Python easier - or at least provide a consistent interface to override to control the behaviour.

PythonCall/juliacall doesn't work in conda environment

I first noticed this because I tried to import juliacall from a plain Python script, running in a conda environment. I then got this error:

> python -c "import juliacall"
fatal: error thrown and no exception handler available.
InitError(mod=:Sys, error=ErrorException("could not load library "libpcre2-8"
The specified module could not be found. "))
jl_errorf at /cygdrive/c/buildbot/worker/package_win64/build/src\rtutils.c:77
jl_load_dynamic_library at /cygdrive/c/buildbot/worker/package_win64/build/src\dlload.c:277
jl_get_library_ at /cygdrive/c/buildbot/worker/package_win64/build/src\runtime_ccall.cpp:52
jl_get_library_ at /cygdrive/c/buildbot/worker/package_win64/build/src\runtime_ccall.cpp:41 [inlined]
jl_load_and_lookup at /cygdrive/c/buildbot/worker/package_win64/build/src\runtime_ccall.cpp:63
jlplt_pcre2_compile_8_32718.clone_1 at c:\users\nardi\appdata\local\programs\julia-1.6.1\lib\julia\sys.dll (unknown line)
compile at .\pcre.jl:124
compile at .\regex.jl:79
match at .\regex.jl:297
match at .\regex.jl:297 [inlined]
match at .\regex.jl:316 [inlined]
splitdrive at .\path.jl:38
joinpath at .\path.jl:256
abspath at .\path.jl:410 [inlined]
__init_build at .\sysinfo.jl:125
__init__ at .\sysinfo.jl:117
jfptr___init___38133.clone_1 at c:\users\nardi\appdata\local\programs\julia-1.6.1\lib\julia\sys.dll (unknown line)
jl_apply at /cygdrive/c/buildbot/worker/package_win64/build/src\julia.h:1703 [inlined]
jl_module_run_initializer at /cygdrive/c/buildbot/worker/package_win64/build/src\toplevel.c:72
_julia_init at /cygdrive/c/buildbot/worker/package_win64/build/src\init.c:794
jl_init_with_image__threading at /cygdrive/c/buildbot/worker/package_win64/build/src\jlapi.c:74 [inlined]
jl_init_with_image__threading at /cygdrive/c/buildbot/worker/package_win64/build/src\jlapi.c:63 [inlined]
jl_init__threading at /cygdrive/c/buildbot/worker/package_win64/build/src\jlapi.c:90
DllCanUnloadNow at C:\tools\miniconda3\envs\hgo_3.7\DLLs\_ctypes.pyd (unknown line)
DllCanUnloadNow at C:\tools\miniconda3\envs\hgo_3.7\DLLs\_ctypes.pyd (unknown line)
DllCanUnloadNow at C:\tools\miniconda3\envs\hgo_3.7\DLLs\_ctypes.pyd (unknown line)
DllCanUnloadNow at C:\tools\miniconda3\envs\hgo_3.7\DLLs\_ctypes.pyd (unknown line)
unknown function (ip: 00007ffaa94e6de9)
PyObject_FastCallKeywords at C:\tools\miniconda3\envs\hgo_3.7\python37.dll (unknown line)
PyMethodDef_RawFastCallKeywords at C:\tools\miniconda3\envs\hgo_3.7\python37.dll (unknown line)
PyEval_EvalFrameDefault at C:\tools\miniconda3\envs\hgo_3.7\python37.dll (unknown line)
PyMethodDef_RawFastCallKeywords at C:\tools\miniconda3\envs\hgo_3.7\python37.dll (unknown line)
PyEval_EvalFrameDefault at C:\tools\miniconda3\envs\hgo_3.7\python37.dll (unknown line)
PyEval_EvalCodeWithName at C:\tools\miniconda3\envs\hgo_3.7\python37.dll (unknown line)
PyImport_Import at C:\tools\miniconda3\envs\hgo_3.7\python37.dll (unknown line)
PyImport_Import at C:\tools\miniconda3\envs\hgo_3.7\python37.dll (unknown line)
PyMethodDef_RawFastCallDict at C:\tools\miniconda3\envs\hgo_3.7\python37.dll (unknown line)
PySlice_New at C:\tools\miniconda3\envs\hgo_3.7\python37.dll (unknown line)
PyEval_EvalFrameDefault at C:\tools\miniconda3\envs\hgo_3.7\python37.dll (unknown line)
PyEval_EvalCodeWithName at C:\tools\miniconda3\envs\hgo_3.7\python37.dll (unknown line)
PyMethodDef_RawFastCallKeywords at C:\tools\miniconda3\envs\hgo_3.7\python37.dll (unknown line)
PyEval_EvalFrameDefault at C:\tools\miniconda3\envs\hgo_3.7\python37.dll (unknown line)
PyMethodDef_RawFastCallKeywords at C:\tools\miniconda3\envs\hgo_3.7\python37.dll (unknown line)
PyEval_EvalFrameDefault at C:\tools\miniconda3\envs\hgo_3.7\python37.dll (unknown line)
PyMethodDef_RawFastCallKeywords at C:\tools\miniconda3\envs\hgo_3.7\python37.dll (unknown line)
PyEval_EvalFrameDefault at C:\tools\miniconda3\envs\hgo_3.7\python37.dll (unknown line)
PyMethodDef_RawFastCallKeywords at C:\tools\miniconda3\envs\hgo_3.7\python37.dll (unknown line)
PyEval_EvalFrameDefault at C:\tools\miniconda3\envs\hgo_3.7\python37.dll (unknown line)
PyFunction_FastCallDict at C:\tools\miniconda3\envs\hgo_3.7\python37.dll (unknown line)
PyObject_CallMethodIdObjArgs at C:\tools\miniconda3\envs\hgo_3.7\python37.dll (unknown line)
PyObject_CallMethodIdObjArgs at C:\tools\miniconda3\envs\hgo_3.7\python37.dll (unknown line)
PyMember_SetOne at C:\tools\miniconda3\envs\hgo_3.7\python37.dll (unknown line)
PyImport_ImportModuleLevelObject at C:\tools\miniconda3\envs\hgo_3.7\python37.dll (unknown line)
PyObject_LengthHint at C:\tools\miniconda3\envs\hgo_3.7\python37.dll (unknown line)
PyEval_EvalFrameDefault at C:\tools\miniconda3\envs\hgo_3.7\python37.dll (unknown line)
PyEval_EvalCodeWithName at C:\tools\miniconda3\envs\hgo_3.7\python37.dll (unknown line)
PyEval_EvalCodeEx at C:\tools\miniconda3\envs\hgo_3.7\python37.dll (unknown line)
PyEval_EvalCode at C:\tools\miniconda3\envs\hgo_3.7\python37.dll (unknown line)
PyArena_Free at C:\tools\miniconda3\envs\hgo_3.7\python37.dll (unknown line)
PyRun_StringFlags at C:\tools\miniconda3\envs\hgo_3.7\python37.dll (unknown line)
PyRun_SimpleStringFlags at C:\tools\miniconda3\envs\hgo_3.7\python37.dll (unknown line)
Py_Main at C:\tools\miniconda3\envs\hgo_3.7\python37.dll (unknown line)
Py_Main at C:\tools\miniconda3\envs\hgo_3.7\python37.dll (unknown line)
PyBytes_AsString at C:\tools\miniconda3\envs\hgo_3.7\python37.dll (unknown line)
Py_Main at C:\tools\miniconda3\envs\hgo_3.7\python37.dll (unknown line)
unknown function (ip: 00007ff6b69e1267)
BaseThreadInitThunk at C:\WINDOWS\System32\KERNEL32.DLL (unknown line)
RtlUserThreadStart at C:\WINDOWS\SYSTEM32\ntdll.dll (unknown line)

It looks like it can't load libpcre, which seems weird, because the conda environment should only add to the paths/environment variables, right?
On the other hand, using PythonCall from Julia was working fine, and it was using the Python executable from the environment as well. However, when I switch it to the conda environment explicitly using ENV["JULIA_PYTHONCALL_EXE"] = "CONDA:..." it also crashes:

julia> @pyv `1+1`
ERROR: UndefRefError: access to undefined reference
Stacktrace:
 [1] Py_CompileString(code::String, filename::String, mode::Int64)
   @ PythonCall.CPython C:\Users\Nardi\Programming\PythonCall.jl\src\cpython\fundamentals.jl:64
 [2] pyptr
   @ C:\Users\Nardi\Programming\PythonCall.jl\src\PyCode.jl:27 [inlined]
 [3] eval_impl(::Type{PyObject}, code::PyCode, globals::PyDict{String, PyObject}, locals::Nothing, extrakeys::Tuple{}, extravals::Tuple{})
   @ PythonCall C:\Users\Nardi\Programming\PythonCall.jl\src\eval.jl:3
 [4] top-level scope
   @ REPL[14]:1

I am running on Windows 10, and the conda environment has Python 3.7 installed. However, running julia when the environment is activated works fine, and regexes (so libpcre) also work. The only thing I can think of is that there is a mismatch between the libpcre version inside and outside the conda environment. But I'm not still not sure why that only shows up when setting up the Julia runtime from "inside" Python.

PyCall compat mode

Support serialising/pickling recursive types

This can be done by having a custom Pickler and Unpickler classes which take a Julia Serializer on construction which is used to serialize any Julia objects it sees in persistent_id. These classes are used when serializing a Python object.

Similarly we can have a custom Serializer type which takes a Python Pickler and Unpickler on construction which is used to (de-)serialize any Python objects it sees. This class is used when pickling/unpickling a Julia object.

How to pass a numpy array into a Julia function?

Consider a dummy function inside my custom package EOS:

function dummy(x::AbstractVector)
    @show typeof(x)
end

From Python, I call it as follows

jl.EOS.dummy(np.array([1.1, 2.3]))

and the following error occurs

TypeError: Julia: MethodError: no method matching dummy(::PythonCall.PyIterable{PythonCall.Py})
Closest candidates are:
  dummy(!Matched::AbstractVector{T} where T) at /home/shuhua/Work/opt/EOS/src/EOS.jl:26

According to Conversion to Julia, I expect that a numpy array is converted to a Julia array automatically. Did I miss anything?

Serialise NULL to Nothing

Serialisation does not support NULL Py objects. Support it by serialising the value to nothing instead of pickling the object (and check for nothing when deserialising).

Example python package using juliacall

Error when getting columns on PyPandasDataFrame

When running the following example code:

using PythonCall
using Tables

@pyg `
import numpy as np
import pandas as pd

$x = pd.DataFrame(np.array((100, 100)))`

# Make sure the columns are strings, or it will fail in any case.
@pyg `$x.columns = map(str, $x.columns)`

y = PyPandasDataFrame(x)
Tables.columns(y)

I get the error:

ERROR: LoadError: map is not defined on sets
Stacktrace:
 [1] error(s::String)
   @ Base .\error.jl:33
 [2] map(f::Type, #unused#::Base.KeySet{String, PyPandasDataFrame})
   @ Base .\abstractarray.jl:2326
 [3] columns(x::PyPandasDataFrame)
   @ PythonCall [,,,]\.julia\packages\PythonCall\MBkDb\src\PyPandasDataFrame.jl:155

This could be fixed by changing the map call on line 155 to map(Symbol, collect(keys(x))) instead.

Check all getptr calls

Ensure the parent object is preserved

Doesn't work on 1.7.0-beta2 or 1.8

This works:

(@v1.8) pkg> add PythonCall#master  # with or without master, while I get an error for 1.7, likely bad installation of Julia.

[Do you have any idea what might be happening? And since works in 1.6, an issue for Julia not your package?]

But doesn't work in 1.8 nor beta2 (and likely not in beta3 either):

julia> @time using PythonCall
ERROR: InitError: type DataType has no field mutable
Stacktrace:
  [1] getproperty
    @ ./Base.jl:37 [inlined]
  [2] pointer_from_obj(o::Vector{String})
    @ PythonCall ~/.julia/packages/PythonCall/r2eLx/src/utils.jl:236
  [3] PyJuliaValue_SetValue(__o::Ptr{PythonCall.CPython.PyObject}, v::Vector{String})
    @ PythonCall.CPython ~/.julia/packages/PythonCall/r2eLx/src/cpython/juliabase.jl:70
  [4] PyJuliaValue_New(t::Ptr{PythonCall.CPython.PyObject}, v::Vector{String})
    @ PythonCall.CPython ~/.julia/packages/PythonCall/r2eLx/src/cpython/juliabase.jl:89
  [5] PyJuliaVectorValue_New
    @ ~/.julia/packages/PythonCall/r2eLx/src/cpython/juliavector.jl:244 [inlined]
  [6] PyJuliaValue_From
    @ ~/.julia/packages/PythonCall/r2eLx/src/cpython/juliavector.jl:245 [inlined]
  [7] PyObject_From
    @ ~/.julia/packages/PythonCall/r2eLx/src/cpython/object.jl:82 [inlined]
  [8] exec_impl(f::PythonCall.var"#245#263", code::PythonCall.PyCode, globals::PythonCall.PyDict{String, PythonCall.PyObject}, locals::Nothing, extrakeys::Tuple{PythonCall.PyInternedString, PythonCall.PyInternedString}, extravals::Tuple{Vector{String}, Bool})
    @ PythonCall ~/.julia/packages/PythonCall/r2eLx/src/eval.jl:38
  [9] macro expansion
    @ ~/.julia/packages/PythonCall/r2eLx/src

┌ Error: Pkg.precompile error
│   exception =
│    SystemError: mktemp: Permission denied
│    Stacktrace:
│      [1] systemerror(p::Symbol, errno::Int32; extrainfo::Nothing)
│        @ Base ./error.jl:174
│      [2] #systemerror#68
│        @ ./error.jl:173 [inlined]
│      [3] systemerror
│        @ ./error.jl:173 [inlined]
│      [4] mktemp(parent::String; cleanup::Bool)


[ Info: Precompiling PythonCall [6099a3de-0909-46bc-b1f4-468b9a2dfc0d]
ERROR: InitError: type DataType has no field mutable
Stacktrace:
  [1] getproperty
    @ ./Base.jl:37 [inlined]
  [2] pointer_from_obj(o::Vector{String})
    @ PythonCall ~/.julia/packages/PythonCall/r2eLx/src/utils.jl:236
  [3] PyJuliaValue_SetValue(__o::Ptr{PythonCall.CPython.PyObject}, v::Vector{String})
    @ PythonCall.CPython ~/.julia/packages/PythonCall/r2eLx/src/cpython/juliabase.jl:70
  [4] PyJuliaValue_New(t::Ptr{PythonCall.CPython.PyObject}, v::Vector{String})
    @ PythonCall.CPython ~/.julia/packages/PythonCall/r2eLx/src/cpython/juliabase.jl:89
  [5] PyJuliaVectorValue_New
    @ ~/.julia/packages/PythonCall/r2eLx/src/cpython/juliavector.jl:244 [inlined]
  [6] PyJuliaValue_From
    @ ~/.julia/packages/PythonCall/r2eLx/src/cpython/juliavector.jl:245 [inlined]
  [7] PyObject_From
    @ ~/.julia/packages/PythonCall/r2eLx/src/cpython/object.jl:82 [inlined]
  [8] exec_impl(f::PythonCall.var"#245#263", code::PythonCall.PyCode, globals::PythonCall.PyDict{String, PythonCall.PyObject}, locals::Nothing, extrakeys::Tuple{PythonCall.PyInternedString, PythonCall.PyInternedString}, extravals::Tuple{Vector{String}, Bool})
    @ PythonCall ~/.julia/packages/PythonCall/r2eLx/src/eval.jl:38
  [9] macro expansion
    @ ~/.julia/packages/PythonCall/r2eLx/src/eval.jl:156 [inlined]
 [10] (::PythonCall.var"#243#261")()
    @ PythonCall ~/.julia/packages/PythonCall/r2eLx/src/init.jl:203
 [11] with_gil(f::PythonCall.var"#243#261", c::Bool)
    @ PythonCall ~/.julia/packages/PythonCall/r2eLx/src/gil.jl:10
 [12] with_gil(f::Function)
    @ PythonCall ~/.julia/packages/PythonCall/r2eLx/src/gil.jl:9
 [13] init()
    @ PythonCall ~/.julia/packages/PythonCall/r2eLx/src/init.jl:195
 [14] (::PythonCall.var"#275#276")()
    @ PythonCall ~/.julia/packages/Requires/035xH/src/init.jl:11
 [15] __init__()
    @ PythonCall ~/.julia/packages/Requires/035xH/src/init.jl:18
 [16] _include_from_serialized(path::String, depmods::Vector{Any})
    @ Base ./loading.jl:768
 [17] _require_from_serialized(path::String)
    @ Base ./loading.jl:821
 [18] _require(pkg::Base.PkgId)
    @ Base ./loading.jl:1130
 [19] require(uuidkey::Base.PkgId)
    @ Base ./loading.jl:1013
 [20] require(into::Module, mod::Symbol)
    @ Base ./loading.jl:997
 [21] top-level scope
    @ timing.jl:210
during initialization of module PythonCall

TagBot trigger issue

This issue is used to trigger TagBot; feel free to unsubscribe.

If you haven't already, you should update your TagBot.yml to include issue comment triggers.
Please see this post on Discourse for instructions and more details.

If you'd like for me to do this for you, comment TagBot fix on this issue.
I'll open a PR within a few hours, please be patient!

Rely less on PyRef

Originally PyRef was intended to be the single place to safely hold a reference to a Python object, so that other wrapper types (PyObject, PyList, etc.) could each simply have a PyRef field.

It would actually be simpler to just have these types hold a PyPtr and have a generic finalizer that decrefs whatever is in the ptr field (i.e. re-use pyref_finalize!).

unsafe_pynext, pyisnull, pynext, pyprint, ...

Using PythonCall within Julia modules

First of all, thanks for working on this, I really appreciate these improvements from PyCall.

Seems like calling pyimport from the top level of a module in precompiled code would create issues.

In PyCall, the following approach is recommended:

const scipy_opt = PyNULL()

function __init__()
    copy!(scipy_opt, pyimport_conda("scipy.optimize", "scipy"))
end

end

I think the equivalent in PythonCall would be

const scipy_opt = pynew()

function __init__()
   pycopy!(scipy_opt, pyimport("scipy.optimize"))
end

It would be worth documenting this somewhere. Obviously a lot of folks are going to be coming from PyCall (and will know to be mindful of this) but it would be better to make this fact explicit, or indicate that this isn't required.