ivanlen/pipreqsnb

pipreqs with jupyter notebook support

pipreqsnb's Issues

pipreqsnb with magic cells

Analyzing a python notebook with a magic cell breaks pipreqsnb.

Is there a way to use it with such notebooks where there are some cells not written in python ?

See the example given from https://discourse.jupyter.org/t/d3-in-jupyterlab/11367/6

File permission denied error when using utf8

Hi,
installed pipreqsnb version 0.2.3 today with pip.

When running pipreqsnb --encoding=utf8 on a jupyter notebook in a folder where my current user has full rights, I get permission denied error in following error trace:

pipreqs --encoding utf8 --savepath /requirements.txt .//__temp_pipreqsnb_folder/
Traceback (most recent call last):
File "c:\anaconda3\lib\runpy.py", line 193, in _run_module_as_main
"main", mod_spec)
File "c:\anaconda3\lib\runpy.py", line 85, in run_code
exec(code, run_globals)
File "C:\Anaconda3\Scripts\pipreqs.exe_main.py", line 7, in
File "c:\anaconda3\lib\site-packages\pipreqs\pipreqs.py", line 470, in main
init(args)
File "c:\anaconda3\lib\site-packages\pipreqs\pipreqs.py", line 460, in init
generate_requirements_file(path, imports)
File "c:\anaconda3\lib\site-packages\pipreqs\pipreqs.py", line 165, in generate_requirements_file
with _open(path, "w") as out_file:
File "c:\anaconda3\lib\contextlib.py", line 112, in enter
return next(self.gen)
File "c:\anaconda3\lib\site-packages\pipreqs\pipreqs.py", line 88, in _open
file = open(filename, mode)
PermissionError: [Errno 13] Permission denied: '/requirements.txt'

This permission denied error does not occur if I run without the encoding option. However, I need the encoding option as otherwise I get a charmp error.

jupyter version: 1.0.0
python version: 3.7.10
pip version: 21.0.1
conda version: 4.10.1

cannot work on "from ... import (...)"

just like:

`from sklearn.preprocessing import (
StandardScaler,
QuantileTransformer,
Normalizer,
MinMaxScaler,
PolynomialFeatures,
)

from sklearn.inspection import permutation_importance as impt`

will SyntaxError:
ERROR: Failed on file: ./__temp_pipreqsnb_folder/_pipreqsnb_temp_file.py Traceback (most recent call last): File "/home/myf/mambaforge/envs/skorch/bin/pipreqs", line 8, in <module> sys.exit(main()) ^^^^^^ File "/home/myf/mambaforge/envs/skorch/lib/python3.11/site-packages/pipreqs/pipreqs.py", line 528, in main init(args) File "/home/myf/mambaforge/envs/skorch/lib/python3.11/site-packages/pipreqs/pipreqs.py", line 455, in init candidates = get_all_imports(input_path, ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/myf/mambaforge/envs/skorch/lib/python3.11/site-packages/pipreqs/pipreqs.py", line 131, in get_all_imports raise exc File "/home/myf/mambaforge/envs/skorch/lib/python3.11/site-packages/pipreqs/pipreqs.py", line 117, in get_all_imports tree = ast.parse(contents) ^^^^^^^^^^^^^^^^^^^ File "/home/myf/mambaforge/envs/skorch/lib/python3.11/ast.py", line 50, in parse return compile(source, filename, mode, flags, ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "<unknown>", line 12 from sklearn.inspection import permutation_importance as impt ^^^^ SyntaxError: invalid syntax

Fails when there is a space in the path to the project directory

I have a project whose path contains a space. The project uses a venv installed in the project home. I installed pipreqsnb in that venv and then tried to run. This should be enough to reproduce. (Sorry I don't have time to debug and submit a PR!)

% cd "Machine Learning"
% cd ProjectName
% python -m venv venv
% source venv/bin/activate
% pwd
/Users/john/Machine Learning/ProjectName
% pipreqsnb .
zsh: /Users/john/Machine Learning/ProjectName/venv/bin/pipreqsnb: bad interpreter: "/Users/jmwagner/Machine: no such file or directory

Key: cells is missing

(base) Yours-Mac-Pro:plaidml jpyrett$ pipreqsnb --ignore ./bazel-plaidml/ .
Traceback (most recent call last):
File "/Library/Frameworks/Python.framework/Versions/3.8/bin/pipreqsnb", line 11, in
load_entry_point('pipreqsnb==0.1.2', 'console_scripts', 'pipreqsnb')()
File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/pipreqsnb/pipreqsnb.py", line 53, inmain
for cell in nb['cells']:
KeyError: 'cells'

Export the requirements for a single notebook

Hi there!
Sometime I have multiple notebooks that are not related to each other, or for used different tasks.
It could be nice if I could run pipreqsnb for a single notebook, e.g.:
pipreqsnb -nb mynb.ipynb

Thanks a lot!

--ignore does not work

I am seeing all kinds of problems with this. Must have the ignore dir ending in '/' or it does not work. Line 53: when it reads in the file, the path of: . is not working. There is just something fatal with this. Recursive = True is a bad choice. We should be able to select the notebook to analyze , not every .ipynb at once. It is totally broken

pipreqsnb failing

pipreqsnb                                                     main         10:11:08  
Traceback (most recent call last):
  File "/Users/paulaceccon/Documents/Projects/ml-helper-scripts/ml-helper/lib/python3.7/site-packages/pipreqsnb/pipreqsnb.py", line 122, in main
    for n_cell, cell in enumerate(nb['cells']):
KeyError: 'cells'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/paulaceccon/Documents/Projects/ml-helper-scripts/ml-helper/bin/pipreqsnb", line 11, in <module>
    load_entry_point('pipreqsnb==0.2.3', 'console_scripts', 'pipreqsnb')()
  File "/Users/paulaceccon/Documents/Projects/ml-helper-scripts/ml-helper/lib/python3.7/site-packages/pipreqsnb/pipreqsnb.py", line 128, in main
    print("Exception occurred while working on file {}, cell {}/{}".format(nb_file, n_cell + 1, len(nb['cells'])))

Cannot find a method to run pipreqsnb in anaconda

Assigned python notebook to variable:

notebook = PureWindowsPath("C:\Users\Henry\Anaconda3\envs\fastai2\varmint_identifier.ipynb")
correct_path = Path(notebook)
print(correct_path)
Output: C:\Users\Henry\Anaconda3\envs\fastai2\varmint_identifier.ipynb

Attempted to create reqs.txt file

pipreqsnb (correct_path --encoding=utf8 --savepath=reqs.txt)

Received an invalid syntax error

File "C:\Users\Henry\AppData\Local\Temp/ipykernel_10728/2599089689.py", line 1
pipreqsnb (correct_path --encoding=utf8 --savepath=reqs.txt)

SyntaxError: invalid syntax (at savepath '='

possible pipreqs bug but fixeable from here

def gaussian_kernel_offset(x1,x2,sig,delta):
    from scipy.spatial.distance import cdist
    k = np.exp((-cdist(x1,x2, 'euclidean')**2/ 2*sig**2))
    offset = np.full((np.shape(x1)[0],np.shape(x2)[0]),delta)
    
    return k + offset

will generate the following error.

ERROR: Failed on file: ./__temp_pipreqsnb_folder/_pipreqsnb_temp_file.py
Traceback (most recent call last):
  File "/home/ivan/coding_projects/env_tf/bin/pipreqs", line 8, in <module>
    sys.exit(main())
  File "/home/ivan/coding_projects/env_tf/lib/python3.7/site-packages/pipreqs/pipreqs.py", line 470, in main
    init(args)
  File "/home/ivan/coding_projects/env_tf/lib/python3.7/site-packages/pipreqs/pipreqs.py", line 409, in init
    follow_links=follow_links)
  File "/home/ivan/coding_projects/env_tf/lib/python3.7/site-packages/pipreqs/pipreqs.py", line 138, in get_all_imports
    raise exc
  File "/home/ivan/coding_projects/env_tf/lib/python3.7/site-packages/pipreqs/pipreqs.py", line 124, in get_all_imports
    tree = ast.parse(contents)
  File "/home/ivan/coding_projects/env_tf/lib/python3.7/ast.py", line 35, in parse
    return compile(source, filename, mode, PyCF_ONLY_AST)
  File "<unknown>", line 111
    from scipy.spatial.distance import cdist
    ^
IndentationError: unexpected indent

We can solve this for notebooks removing all indetation from imports

Parses non .ipynb python files

There are some valid .py files this is breaking on when it tries to parse the file.
Such as one with plaidml. https://github.com/plaidml/plaidml/blob/master/bzl/version.tpl.py

ERROR: Failed on file: ./bazel-plaidml/bzl/version.tpl.py
Traceback (most recent call last):
File "/Library/Frameworks/Python.framework/Versions/3.8/bin/pipreqs", line 8, in
sys.exit(main())
File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/pipreqs/pipreqs.py", line 470, in main
init(args)
File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/pipreqs/pipreqs.py", line 406, in init
candidates = get_all_imports(input_path,
File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/pipreqs/pipreqs.py", line 138, in get_all_imports
raise exc
File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/pipreqs/pipreqs.py", line 124, in get_all_imports
tree = ast.parse(contents)
File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/ast.py", line 47, in parse
return compile(source, filename, mode, flags,
File "", line 3
{PREFIX}_VERSION = "{VERSION}"
^
SyntaxError: invalid syntax

SyntaxError for multiline imports

I encountered error when I have multiline imports in my notebook (typical when using e.g. black as I'm using). The example notebook Imports.ipynb has only one cell:

from sklearn.model_selection import (
    GridSearchCV,
    RandomizedSearchCV,
    StratifiedShuffleSplit,
)
from sklearn.neighbors import KNeighborsRegressor

and when I run pipreqsnb Imports.ipynb I get

pipreqs  --savepath requirements.txt /Users/jakubpernis/notebooks/__temp_pipreqsnb_folder
ERROR: Failed on file: /Users/jakubpernis/notebooks/__temp_pipreqsnb_folder/_pipreqsnb_temp_file.py
Traceback (most recent call last):
  File "/Library/Frameworks/Python.framework/Versions/3.9/bin/pipreqs", line 8, in <module>
    sys.exit(main())
  File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages/pipreqs/pipreqs.py", line 488, in main
    init(args)
  File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages/pipreqs/pipreqs.py", line 415, in init
    candidates = get_all_imports(input_path,
  File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages/pipreqs/pipreqs.py", line 131, in get_all_imports
    raise exc
  File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages/pipreqs/pipreqs.py", line 117, in get_all_imports
    tree = ast.parse(contents)
  File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/ast.py", line 50, in parse
    return compile(source, filename, mode, flags,
  File "<unknown>", line 2
    from sklearn.neighbors import KNeighborsRegressor
    ^
SyntaxError: invalid syntax

However when I don't use the formatter and keep the import on one line it works just fine:

pipreqsnb Imports.ipynb
pipreqs  --savepath requirements.txt /Users/jakubpernis/notebooks/__temp_pipreqsnb_folder
INFO: Successfully saved requirements file in requirements.txt

I'm using pipreqsnb 0.2.4.

pipreqsnb with ipyparallel

I have a notebook using ipythons parallel features.

this includes a parallel import of packages like this

with rc[:].sync_imports():
    import pandas as pd
    import cx_Oracle
    import csv
    import datetime
    from calendar import monthrange
    import glob
    from io import StringIO
    from sqlalchemy import create_engine
    import psycopg2

Now pipreqsnb complains about the indention of the import statements, which are totally correct and working in this case.

  File "<unknown>", line 3
    import pandas as pd
    ^
IndentationError: unexpected indent

Any workaround for this?

Syntax Error arising when using "from x import y"

I've run pipreqsnb on a few different directories and it seems to consistently break when it runs into a file with "from" import in a notebook, or it has something to do with the "unknown" file, though I'm not sure what that means given all my ipynb files are named.

The console log doesn't show exactly what file it's breaking in so it's hard to debug as well.

Example (i've edited the paths for privacy):

pipreqs /home/<my_folder_path>
ERROR: Failed on file: /home/<my_folder_path>/__temp_pipreqsnb_folder/_pipreqsnb_temp_file.py
Traceback (most recent call last):
File "/home/.local/bin/pipreqs", line 8, in
sys.exit(main())
File "/home/.local/lib/python3.8/site-packages/pipreqs/pipreqs.py", line 488, in main
init(args)
File "/home/.local/lib/python3.8/site-packages/pipreqs/pipreqs.py", line 415, in init
candidates = get_all_imports(input_path,
File "/home/.local/lib/python3.8/site-packages/pipreqs/pipreqs.py", line 131, in get_all_imports
raise exc
File "/home/.local/lib/python3.8/site-packages/pipreqs/pipreqs.py", line 117, in get_all_imports
tree = ast.parse(contents)
File "/usr/lib/python3.8/ast.py", line 47, in parse
return compile(source, filename, mode, flags,
File "", line 268
from tools.preprocessing.pipeline_utils.feature_generator import wkts_to_wkt
^
SyntaxError: invalid syntax

SyntaxError for '!pip install ...' with leading tab

pipreqsnb fails when encountering a line with !pip install ... and a tab in front of it and throws a SyntaxError, complaining about the exclamation mark.

Minimal example:

if 'google.colab' in str(get_ipython()):
    !pip install prophet
import prophet

Output:

  File "<unknown>", line 2
    !pip install prophet
    ^
SyntaxError: invalid syntax

--no-pin doesn't work

$ pipreqsnb --no-pin .
pipreqs  .
INFO: Successfully saved requirements file in ./requirements.txt

It looks like pipreqsnb is not passing pipreqs the --no-pin option, but only the folder path .. As a matter of fact, the resulting requirements.txt does pin versions:

$ cat requirements.txt 
pandas==1.2.4
matplotlib==3.4.2
numpy==1.20.3

Can you fix? Thanks. I'm using pipreqsnb 0.2.3 System info:

$ cat /etc/os-release
NAME="Ubuntu"
VERSION="20.04.1 LTS (Focal Fossa)"
ID=ubuntu
ID_LIKE=debian
PRETTY_NAME="Ubuntu 20.04.1 LTS"
VERSION_ID="20.04"
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
VERSION_CODENAME=focal
UBUNTU_CODENAME=focal

Syntax Error

Hiya, I'm getting the following syntax error:

Traceback (most recent call last):
  File "/home/arj39/anaconda3/bin/pipreqsnb", line 8, in <module>
    sys.exit(main())
  File "/home/arj39/anaconda3/lib/python3.7/site-packages/pipreqsnb/pipreqsnb.p
y", line 85, in main
    imports += get_import_string_from_source(source)
  File "/home/arj39/anaconda3/lib/python3.7/site-packages/pipreqsnb/pipreqsnb.py", line 23, in get_import_string_from_source
    tree = ast.parse(source)
  File "/home/arj39/anaconda3/lib/python3.7/ast.py", line 35, in parse
    return compile(source, filename, mode, PyCF_ONLY_AST)
  File "<unknown>", line 18
    !mkdir $output_dir
    ^
SyntaxError: invalid syntax

pipreqsnb with pyfunctional

Since pyfunctional is imported with, for example, from functional import seq, pipreqsnb mistakenly adds functional instead of pyfunctional to the requirements.txt file.