Incompatible with Python 3.11

[facebookresearch]

Incompatible with Python 3.11

[talgalili]

Thanks @tomwagstaff-opml

The good news is that we continued to discuss this internally and we might have a direction of how to monkey-patch this problem (thanks to the wonderful @stevemandala ).
It's still being tested. We'll update here once we find something that seems to work. (Hopefully today but it might only get to happen next week).

T

[talgalili]

@tomwagstaff-opml great news:
@stevemandala just landed a patch to make balance support 3.11.
Could you please test to see if it works for you by installing it via:

python -m pip install git+https://github.com/facebookresearch/balance.git

And let us know how it goes?

[talgalili]

Sorry @tomwagstaff-opml
I honestly didn't expact @stevemandala to find a solution that fast :D

[talgalili]

Fascinating. Especially given that it runs fine on gh actions under 3.11.
What I think we might do is later today release balance 0.3.0 (which should support 3.11). Once it gets to pypi I'll ping here again and you can try installing it through pypi.

Thanks for helping us debug this.

[talgalili]

Ok, so it's not clear to me what's broken then :(

I suggest we leave this issue closed, and if you come across more issues - feel free to open new issues for them.

[talgalili]

Hey @tomwagstaff-opml

This is an issue because of the dependancy of the IPW implementation on glmnet (which is getting to be old at this point).
We plan to move to sklearn by June, but that will take some time.

In the meantime, would using the package only with CBPS be good enough for your needs?
(Because if so, there is a hack we can discuss)

[tomwagstaff-opml]

Hi @talgalili !

Thanks for your speedy response.

Understand the constraint, and looking around it looks like the easiest solution is probably for me to roll back to Python 3.9 or something, so I'm working on that...

Having said that - my main interest in the package was road-testing CBPS (although I wanted to try all the methods it offers). So if there's an (easy) hack I'd be all ears :-)

[talgalili]

The hack (which I'm not sure how easy), is to fork the repo, manually change the setup.py to not restrict you to scipy 1.18, and then install the package from the fork.
You might also need to remove the dependancy on glmnet.

I haven't tried it, but I think it's worth a shot.

If you try it, could you please share how it went?

(P.s., whenever invoking sample.adjust you'll need to use method="cbps", since the default of "ipw" will not work)

[talgalili]

For our own reference, solving this issue depends on solving: #16

[stevemandala]

Adding to @talgalili's summary, the main reason we pinned the scipy dependency is that glmnet_python relies on several scipy libraries that were deprecated & moved in 1.9.0 (full release notes here). Top of mind, glmnet_python relies on
scipy.random at several code pointers

While resolving #16 would be the ideal mitigation, another route would be to migrate glmnet_python source to also support scipy 1.9.0+, e.g. migrating scipy.random to numpy.random. I submitted an issue at bbalasub1/glmnet_python#64 for tracking this.

[tomwagstaff-opml]

Thanks both - this all makes sense, and it doesn't sound quick or easy to address from your side (although I guess it's something you'll have to grapple with sooner or later). I'm happy to get hold of another version of Python to play with the package

[tomwagstaff-opml]

Oh gosh - I just had IT roll back my Python installation! I'll see if I can create a 3.11 environment to test it - but either way, thanks very much for your efforts @stevemandala

[tomwagstaff-opml]

I've tried installing and I'm afraid it refused to install in Python 3.11 :-(

Just to recap what I did - after the rollback I got miniconda installed and created a Python 3.11 environment to test the package. I conda installed git, and then tried to pip install balance as per your command. It fails to install, with this really weird character encoding error, so who knows if that's real or a red herring...

The good news is I created a Python 3.10 environment and successfully pip installed balance there - just from PyPi. I ran into the same error as below when I tried to install from GitHub, so maybe that's a clue as to where the problem is.

Cloning https://github.com/facebookresearch/balance.git to c:\users\twagstaff\appdata\local\temp\pip-req-build-feduampl
  Running command git clone --filter=blob:none --quiet https://github.com/facebookresearch/balance.git 'C:\Users\twagstaff\AppData\Local\Temp\pip-req-build-feduampl'
  Resolved https://github.com/facebookresearch/balance.git to commit c6f056f91a238b70cddccc97e685d6a654a91a63
  Preparing metadata (setup.py) ... error
  error: subprocess-exited-with-error

  × python setup.py egg_info did not run successfully.
  │ exit code: 1
  ╰─> [12 lines of output]
      Traceback (most recent call last):
        File "<string>", line 2, in <module>
        File "<pip-setuptools-caller>", line 34, in <module>
        File "C:\Users\twagstaff\AppData\Local\Temp\pip-req-build-feduampl\setup.py", line 90, in <module>
          setup_package()
        File "C:\Users\twagstaff\AppData\Local\Temp\pip-req-build-feduampl\setup.py", line 55, in setup_package
          long_description = fh.read()
                             ^^^^^^^^^
        File "C:\Users\twagstaff\AppData\Local\miniconda3\envs\balance_311\Lib\encodings\cp1252.py", line 23, in decode
          return codecs.charmap_decode(input,self.errors,decoding_table)[0]
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
      UnicodeDecodeError: 'charmap' codec can't decode byte 0x9d in position 1494: character maps to <undefined>
      [end of output]

  note: This error originates from a subprocess, and is likely not a problem with pip.
error: metadata-generation-failed

× Encountered error while generating package metadata.
╰─> See above for output.

[talgalili]

Thanks @tomwagstaff-opml

Just to double check, you used this code:

python -m pip install git+https://github.com/facebookresearch/balance.git

?

[tomwagstaff-opml]

Yes indeed - well actually just the pip install... bit as I don't think I need to explicitly call python again once I'm in the environment

[stevemandala]

@tomwagstaff-opml As @talgalili mentioned, this should be resolved for PyPi installs with the new release. I also have a quick fix for the UnicodeDecodeError error as well, which was due to errors when decoding the utf-8 files on Windows: #28

This should fix the issue with builds from source directly

(Also want to echo Tal's thank you for your help raising and debugging these issues!)

[talgalili]

@tomwagstaff-opml - update: we've now landed a fix that should make installing from github on windows possible again.
Would you please try to reinstall balance on 3.11 from your windows machine using the github repo?

[tomwagstaff-opml]

I'm afraid that failed again, but I don't think it's on you.

The problem now is the dependency on pandas. If I try to conda install it, I'm told it's not available for Python 3.11 - I think that must be a specific issue with the conda repos because it's listed as compatible on PyPi.

If I try to pip install balance - and let pip take care of the pandas dependency - I'm prompted to install C++ first, which I'm afraid is an IT ticket too far for me 😅

[talgalili]

Oh my.
@tomwagstaff-opml :
We currently require pandas<=1.4.3. Is the issue that we need to make sure we support pandas 1.5.0 or some other version?

(Please let us know - I'd be happy to open a ticket for this as well)

[tomwagstaff-opml]

Ahh, I think it might be - I'm using pandas 1.4.3. in the Python 3.10 environment...