pyxb stubs contain implicit relative imports on python 3

[pabigot]

pyxb stubs contain implicit relative imports on python 3

[pabigot]

PyXB does not use module-relative imports. It only looks that way because (I'm guessing, you still haven't provided the exact steps you're using to produce the final import path) you didn't tell PyXB that you wanted the generated bindings to have a parent when you import them. If you want:

import my.schema.top

to work, you tell PyXB that with --module-prefix=my.schema and it will ensure that when top needs _other it will use:

import my.schema._other

which is not module-relative. Try it.

If you want:

import my.schema.top

to work without adding the directory that contains my into your Python path I have no idea how you'd get that to work. If you do have that working, using --module-prefix will fix your perceived relative import problem.

But if you don't want to tell PyXB the path you intend to use to import the generated bindings, then yes, you will have to manually edit the bindings to make them relative to wherever you install them, because I'm not going to change the existing absolute imports to be relative via from somewhere import _other.

[pabigot]

I've never encountered the problem. If you can provide the command line you used to generate the bindings, ideally with enough detail that I can replicate it in a standalone system, that would help.

[novocaine]

/cygdrive/c/Python27/python.exe "c:/Python27/Scripts/pyxbgen" -W file:///c:/xplanbase/version/9.99.999/wsdl/pyxb/postcoder/ProvideAddressesV1.wsdl --module=ProvideAddressesV1 --binding-root="c:\xplanbase\version\9.99.999\lib\py\pyxb_stubs\postcoder" --no-write-for-customization

then

/cygdrive/c/xplanbase/version/9.99.999/lib/py/pyxb_stubs/postcoder> ls

__init__.py  _ds.py  _eshead.py  _esreq.py  _esresp.py  _pareq.py  _paresp.py  ProvideAddressesV1.py

Looks good. However, _esreq.py contains the following implicit relative imports, which are not valid in python 3:

# Import bindings for namespaces imported into schema
import _pareq as _ImportedBinding__pareq
import _ds as _ImportedBinding__ds

WSDL files are at https://www.dropbox.com/s/nx400jgyzrdwmny/wsdl.zip?dl=0

[pabigot]

I'm not duplicating the problem:

llc[77]$ ls
ESProvideAddressesRequestV1.xsd   ProvideAddressesResponseV1.xsd
ESProvideAddressesResponseV1.xsd  ProvideAddressesV1.wsdl
ExchangeServicesHeaderV1.xsd      __pycache__
ProvideAddressesRequestV1.xsd     xmldsig-core-schema.xsd
llc[78]$ rm -rf __pycache__
llc[79]$ pyxbgen -W ProvideAddressesV1.wsdl --module-prefix=irss -m es
Retrieving WSDL from /tmp/salter/ProvideAddressesV1.wsdl
WARNING:pyxb.binding.basis:Unable to convert DOM node {http://www.w3.org/2001/XMLSchema}schema at ProvideAddressesV1.wsdl[30:2] to binding
PS urn:uuid:fd7ff7d0-1419-11e5-80dd-00e04c779b7b
Python for http://www.exchange.co.uk/schema/ExchangeServices/v1 requires 7 modules
llc[80]$ grep import irss/_es
_eshead.py  _esreq.py   _esresp.py  
llc[80]$ grep import irss/_esreq.py 
from __future__ import unicode_literals
import pyxb
import pyxb.binding
import pyxb.binding.saxer
import io
import pyxb.utils.utility
import pyxb.utils.domutils
import sys
import pyxb.utils.six as _six
# Import bindings for namespaces imported into schema
import irss._pareq as _ImportedBinding_irss__pareq
import irss._ds as _ImportedBinding_irss__ds
import pyxb.binding.datatypes
    import pyxb.utils.fac as fac
    import pyxb.utils.fac as fac
    import pyxb.utils.fac as fac
llc[81]$ python
Python 3.4.3 (default, Jun 16 2015, 05:55:30) 
[GCC 4.8.2] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import irss._esreq
>>> 

I.e. importing into python a generated binding that has the problematic import statements works.

How does this fail for you, and specifically what do you do to make it fail?

[pabigot]

Removing the module prefix doesn't change that. BTW: Generating the bindings with this command produces a much cleaner module layout, especially since it appears your entrypoint doesn't define anything in the service namespace:

pyxbgen \
  --module-prefix=irss \
  -W ProvideAddressesV1.wsdl -m es \
  -m eshead -u ExchangeServicesHeaderV1.xsd \
  -m esreq -u ESProvideAddressesRequestV1.xsd \
  -m esresp -u ESProvideAddressesResponseV1.xsd \
  -m pareq -u ProvideAddressesRequestV1.xsd \
  -m paresp -u ProvideAddressesResponseV1.xsd

I also should point out that it appears the copyright on those schemas disallows distribution. Just FYI.

[novocaine]

Thank you Peter.

In the output you have given, your --module-prefix flag is being prepended
to the submodules which appears to rectify the problem.

What do you expect those import statements to look like when do you not
specify a --module-prefix (as I have not done)?

[pabigot]

They look exactly the same, and behave exactly the same, at least with 3.4.3.

llc[115]$ pyxbgen -W ProvideAddressesV1.wsdl -m es
Retrieving WSDL from /tmp/salter/ProvideAddressesV1.wsdl
WARNING:pyxb.binding.basis:Unable to convert DOM node {http://www.w3.org/2001/XMLSchema}schema at ProvideAddressesV1.wsdl[30:2] to binding
PS urn:uuid:1d32667c-141e-11e5-ab96-00e04c779b7b
Python for http://www.exchange.co.uk/schema/ExchangeServices/v1 requires 7 modules
llc[116]$ grep ' as _ImportedBinding' _esreq.py 
import _pareq as _ImportedBinding__pareq
import _ds as _ImportedBinding__ds
llc[117]$ python
Python 3.4.3 (default, Jun 16 2015, 05:55:30) 
[GCC 4.8.2] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import _esreq
>>> 

[novocaine]

That apparently only works if you are in the same directory.

Python 3.4.2 (v3.4.2:ab2c023a9432, Oct  6 2014, 22:15:05) [MSC v.1600 32 bit (Intel)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> import postcoder._esreq
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "C:\xplanbase\version\30.0.0\lib\py\pyxb_stubs\postcoder\_esreq.py", line 27, in <module>
    import _ds as _ImportedBinding__ds
ImportError: No module named '_ds'

[pabigot]

This should provide a clue. The problem has nothing to do with Python 3; fix your PYTHONPATH.

llc[31]$ pwd
/tmp/postcode
llc[32]$ pyxbgen -W ProvideAddressesV1.wsdl -m es
Retrieving WSDL from /tmp/postcode/ProvideAddressesV1.wsdl
WARNING:pyxb.binding.basis:Unable to convert DOM node {http://www.w3.org/2001/XMLSchema}schema at ProvideAddressesV1.wsdl[30:2] to binding
PS urn:uuid:fc576368-1426-11e5-bc0f-00e04c779b7b
Python for http://www.exchange.co.uk/schema/ExchangeServices/v1 requires 7 modules
llc[33]$ env PYTHONPATH=/opt/pyxb python
Python 3.4.3 (default, Jun 16 2015, 05:55:30) 
[GCC 4.8.2] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import _esreq
>>> 
llc[34]$ cd ..
llc[35]$ env PYTHONPATH=/opt/pyxb python
Python 3.4.3 (default, Jun 16 2015, 05:55:30) 
[GCC 4.8.2] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import postcode._esreq
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/tmp/postcode/_esreq.py", line 30, in <module>
    import _pareq as _ImportedBinding__pareq
ImportError: No module named '_pareq'
>>> 
llc[36]$ env PYTHONPATH=/tmp/postcode:/opt/pyxb python
Python 3.4.3 (default, Jun 16 2015, 05:55:30) 
[GCC 4.8.2] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import postcode._esreq
>>> 

If you use --module-prefix=postcode then you have to have the directory that contains postcode in your PYTHONPATH.

[novocaine]

postcode just is a package inside my application as you can see - I am not
going to be putting it in my PYTHONPATH.

The stubs are trying to execute relative implicit imports, which work in
python 2 - and not python 3. They should try to execute relative implicit
imports in the form 'from . import x'. Then, nobody needs to pollute their
PYTHONPATH with the location of pyxb stubs.

I guess I will just edit the stubs to get the correct behaviour as you
refuse to recognise the issue - sorry to have bothered you.

On Tue, Jun 16, 2015 at 1:58 PM, Peter A. Bigot [email protected]
wrote:

This should provide a clue. The problem has nothing to do with Python 3;
fix your PYTHONPATH.

llc[31]$ pwd
/tmp/postcode
llc[32]$ pyxbgen -W ProvideAddressesV1.wsdl -m es
Retrieving WSDL from /tmp/postcode/ProvideAddressesV1.wsdl
WARNING:pyxb.binding.basis:Unable to convert DOM node {http://www.w3.org/2001/XMLSchema}schema at ProvideAddressesV1.wsdl[30:2] to binding
PS urn:uuid:fc576368-1426-11e5-bc0f-00e04c779b7b
Python for http://www.exchange.co.uk/schema/ExchangeServices/v1 requires 7 modules
llc[33]$ env PYTHONPATH=/opt/pyxb python
Python 3.4.3 (default, Jun 16 2015, 05:55:30)
[GCC 4.8.2] on linux
Type "help", "copyright", "credits" or "license" for more information.

import _esreq

llc[34]$ cd ..
llc[35]$ env PYTHONPATH=/opt/pyxb python
Python 3.4.3 (default, Jun 16 2015, 05:55:30)
[GCC 4.8.2] on linux
Type "help", "copyright", "credits" or "license" for more information.
import postcode._esreq
Traceback (most recent call last):
File "", line 1, in
File "/tmp/postcode/_esreq.py", line 30, in
import _pareq as _ImportedBinding__pareq
ImportError: No module named '_pareq'

llc[36]$ env PYTHONPATH=/tmp/postcode:/opt/pyxb python
Python 3.4.3 (default, Jun 16 2015, 05:55:30)
[GCC 4.8.2] on linux
Type "help", "copyright", "credits" or "license" for more information.
import postcode._esreq

If you use --module-prefix=postcode then you have to have the directory
that contains postcode in your PYTHONPATH.

—
Reply to this email directly or view it on GitHub
#37 (comment).

[pabigot]

Understand, it's not that I refuse to recognize the issue. It's that you haven't provided a command sequence that allows me to replicate the problem. If you can take the example I showed, which works, and show how you're using it that doesn't work, I'll be happy to continue to provide support.

If pyxb_stubs is not in your pythonpath, how exactly is it that importing something that's in pyxb_stubs works in your application?

[novocaine]

My PYTHONPATH contains lib/py, and then you can import from subdirs in
lib/py as e.g. import pyxb_stubs.postcoder.ProvideAddressesV1.

I should not have to modify my PYTHONPATH to get the stubs to work. This is
very simply rectified by switching those imports to explicit relative.

On Tue, Jun 16, 2015 at 2:12 PM, Peter A. Bigot [email protected]
wrote:

Reopened #37 #37.

—
Reply to this email directly or view it on GitHub
#37 (comment).

[pabigot]

The only way I can see your example working is if . is in your PYTHONPATH and you're running your application from within pyxb_stubs. In that case, you could make things work by simply generating the schema from within pyxb_stubs using --moduleprefix=postcoder. In that case do this:

llc[51]$ pyxbgen -W ProvideAddressesV1.wsdl -m es --module-prefix=postcoder
Retrieving WSDL from /tmp/salter/ProvideAddressesV1.wsdl
WARNING:pyxb.binding.basis:Unable to convert DOM node {http://www.w3.org/2001/XMLSchema}schema at ProvideAddressesV1.wsdl[30:2] to binding
PS urn:uuid:31bc8f94-142a-11e5-b3a3-00e04c779b7b
Python for http://www.exchange.co.uk/schema/ExchangeServices/v1 requires 7 modules
llc[52]$ env PYTHONPATH=/opt/pyxb python
Python 3.4.3 (default, Jun 16 2015, 05:55:30) 
[GCC 4.8.2] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import postcoder._esreq
>>> 
llc[53]$ 

[novocaine]

Sorry - I gave that example as a demonstration that the import does not work if you're not in the same directory - my situation is as outlined in the previous message. I don't think this is a particularly unusual situation in medium to large apps, and I don't think pyxb stubs only working when the dir they live in is in the PYTHONPATH is a good solution

[pabigot]

My PYTHONPATH contains lib/py, and then you can import from subdirs in lib/py as e.g. import pyxb_stubs.postcoder.ProvideAddressesV1.

Yes, exactly. But your example didn't import pyxb_stubs.postcoder.ProvideAddressV1. It bypassed pyxb_stubs and imported from postcoder directly. I really can't see how that could work, whether it's Python3 or Python2.

Please provide a complete sequence that I can run that explicitly shows where you are and what's in your PYTHONPATH, similar to the sequences I provided to show what works for me.

[novocaine]

It's the same as your example, except without hacking the PYTHONPATH. Close the issue if you think it's fine that pyxb stubs have to always be in PYTHONPATH (only on python 3). I'll just edit my stub to use the correct import technique.

[pabigot]

It's the same as your example, except without hacking the PYTHONPATH. Close the issue if you think it's fine that pyxb stubs have to always be in PYTHONPATH (only on python 3). I'll just edit my stub to use the correct import technique.

You keep giving vague instructions like this, and when I try to reproduce what I think you want me to do either works fine or it fails in a way that's clearly not PyXB's fault. The only way I can replicate the behavior you've described is by executing steps that you haven't provided. However, I've finally managed to do that, and the solution to your problem is below.

In short, PyXB does not generate relative import statements, i.e. things like:

import .submodule

If you give it a module prefix, it will include the prefix in the import statement. If you do not give it a module prefix, it will assume that the generated bindings are directly in PYTHONPATH.

If you generate bindings without a prefix, then move them into a subdirectory and use a prefix to import the top level one, then yes that will work with Python 2, but that's by chance not by design, and it's not what you're expected to do.

Python 2 doing it the wrong way: generate the bindings without a module path and manually constructing a surrounding module:

llc[54]$ cd /tmp
llc[55]$ cd salter/
llc[56]$ ls
ESProvideAddressesRequestV1.xsd   ProvideAddressesResponseV1.xsd
ESProvideAddressesResponseV1.xsd  ProvideAddressesV1.wsdl
ExchangeServicesHeaderV1.xsd      genbind
ProvideAddressesRequestV1.xsd     xmldsig-core-schema.xsd
llc[57]$ pyxbgen -W ProvideAddressesV1.wsdl -m es
Retrieving WSDL from /tmp/salter/ProvideAddressesV1.wsdl
WARNING:pyxb.binding.basis:Unable to convert DOM node {http://www.w3.org/2001/XMLSchema}schema at ProvideAddressesV1.wsdl[30:2] to binding
PS urn:uuid:cb6972c0-1438-11e5-813f-00e04c779b7b
Python for http://www.exchange.co.uk/schema/ExchangeServices/v1 requires 7 modules
llc[58]$ mkdir postcoder
llc[59]$ mv *.py postcoder
llc[60]$ rm -rf /tmp/lib/py ; mkdir -p /tmp/lib/py
llc[61]$ mv postcoder /tmp/lib/py
llc[62]$ ls
ESProvideAddressesRequestV1.xsd   ProvideAddressesResponseV1.xsd
ESProvideAddressesResponseV1.xsd  ProvideAddressesV1.wsdl
ExchangeServicesHeaderV1.xsd      genbind
ProvideAddressesRequestV1.xsd     xmldsig-core-schema.xsd
llc[63]$ env PYTHONPATH=/tmp/lib/py:/opt/pyxb python 
Python 2.7.6 (default, Mar 22 2014, 22:59:56) 
[GCC 4.8.2] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import postcoder._esreq
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ImportError: No module named postcoder._esreq
>>> 
llc[64]$ touch /tmp/lib/py/postcoder/__init__.py
llc[65]$ env PYTHONPATH=/tmp/lib/py:/opt/pyxb python 
Python 2.7.6 (default, Mar 22 2014, 22:59:56) 
[GCC 4.8.2] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import postcoder._esreq
>>> 
llc[66]$ 

Yes, that happens to work. When repeated in Python 3 it doesn't:

llc[106]$ pwd
/tmp/salter
llc[107]$ ls
ESProvideAddressesRequestV1.xsd   ProvideAddressesRequestV1.xsd   genbind
ESProvideAddressesResponseV1.xsd  ProvideAddressesResponseV1.xsd  xmldsig-core-schema.xsd
ExchangeServicesHeaderV1.xsd      ProvideAddressesV1.wsdl
llc[108]$ pyxbgen -W ProvideAddressesV1.wsdl -m es
Retrieving WSDL from /tmp/salter/ProvideAddressesV1.wsdl
WARNING:pyxb.binding.basis:Unable to convert DOM node {http://www.w3.org/2001/XMLSchema}schema at ProvideAddressesV1.wsdl[30:2] to binding
PS urn:uuid:3341dfb8-1439-11e5-b1cf-00e04c779b7b
Python for http://www.exchange.co.uk/schema/ExchangeServices/v1 requires 7 modules
llc[109]$ mkdir postcoder
llc[110]$ mv *.py postcoder
llc[111]$ touch postcoder/__init__.py
llc[112]$ rm -rf /tmp/lib/py ; mkdir -p /tmp/lib/py
llc[113]$ mv postcoder /tmp/lib/py
llc[114]$ ls
ESProvideAddressesRequestV1.xsd   ProvideAddressesRequestV1.xsd   genbind
ESProvideAddressesResponseV1.xsd  ProvideAddressesResponseV1.xsd  xmldsig-core-schema.xsd
ExchangeServicesHeaderV1.xsd      ProvideAddressesV1.wsdl
llc[115]$ env PYTHONPATH=/tmp/lib/py:/opt/pyxb python
Python 3.4.3 (default, Jun 16 2015, 05:55:30) 
[GCC 4.8.2] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import postcoder._esreq
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/tmp/lib/py/postcoder/_esreq.py", line 30, in <module>
    import _pareq as _ImportedBinding__pareq
ImportError: No module named '_pareq'
>>> 
llc[117]$ 

But it was never intended to work. You do it correctly by using --module-prefix to tell PyXB how you intend to import the modules:

llc[117]$ pwd
/tmp/salter
llc[118]$ ls
ESProvideAddressesRequestV1.xsd   ProvideAddressesRequestV1.xsd   genbind
ESProvideAddressesResponseV1.xsd  ProvideAddressesResponseV1.xsd  xmldsig-core-schema.xsd
ExchangeServicesHeaderV1.xsd      ProvideAddressesV1.wsdl
llc[119]$ pyxbgen -W ProvideAddressesV1.wsdl -m es --module-prefix=postcoder
Retrieving WSDL from /tmp/salter/ProvideAddressesV1.wsdl
WARNING:pyxb.binding.basis:Unable to convert DOM node {http://www.w3.org/2001/XMLSchema}schema at ProvideAddressesV1.wsdl[30:2] to binding
PS urn:uuid:7f470794-1439-11e5-8eed-00e04c779b7b
Python for http://www.exchange.co.uk/schema/ExchangeServices/v1 requires 7 modules
llc[120]$ rm -rf /tmp/lib/py ; mkdir -p /tmp/lib/py
llc[121]$ mv postcoder/ /tmp/lib/py
llc[122]$ env PYTHONPATH=/tmp/lib/py:/opt/pyxb python
Python 3.4.3 (default, Jun 16 2015, 05:55:30) 
[GCC 4.8.2] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import postcoder._esreq
>>> 

So yes, I'm fine with this: it's behaving exactly as I would expect.

Does this make sense?

[novocaine]

Supplying a --module-prefix does not alter the fact that you will need to put the root of the pyxb stubs in your PYTHONPATH in order to import them. I am not going to do that; my application has a clean PYTHONPATH with a minimal number of entries. Meanwhile, the issue is cleanly resolved if the stub contains an explicit relative import rather than an implicit one.

Put another way. I perceive your suggestion that I just add the pyxb stubs to PYTHONPATH to make the implicit relative imports work as about as reasonable as just adding every subdirectory in my app to PYTHONPATH to keep any implicit relatives working after upgrading to python 3. That's not the right answer; the right answer is to fix the imports to use explicit relative imports. For example, 2to3.py does this automatically - I'm just asking for pyxb to do it out of the box.

[novocaine]

There is lots of info around on what's wrong with implicit relative imports, e.g. http://programmers.stackexchange.com/questions/159503/whats-wrong-with-relative-imports-in-python and https://docs.python.org/2/whatsnew/2.5.html#pep-328

[novocaine]

I see - yes if you want to tell pyxb what its parent module is, that is a solution, but then pyxb build needs to be aware of the structure of your project and the files can't be moved relative to the rest of the project without being edited or regenerated. If it used explicit relative imports instead I don't think the flag would need to exist, and the case where the user has not specified a --prefix (on which pyxb is not especially clear about the necessity of) would just work. But you're not going to change it, okay thanks.

[pabigot]

I don't think it's as easy as you suggest given the need to reference schema generated in separate invocations of PyXB. In six years of PyXB use by hundreds/thousands (?) of people with fairly complicated schema nobody's needed this feature. So yes, I'm not going to change it.

[novocaine]

Fair enough. Thanks, I do appreciate your time today and your work on the
library.

On Tue, Jun 16, 2015 at 5:54 PM, Peter A. Bigot [email protected]
wrote:

I don't think it's as easy as you suggest given the need to reference
schema generated in separate invocations of PyXB. In six years of PyXB use
by hundreds/thousands (?) of people with fairly complicated schema nobody's
needed this feature. So yes, I'm not going to change it.

—
Reply to this email directly or view it on GitHub
#37 (comment).