madhup/libify

Import Databricks notebooks as libraries/modules

libify's Issues

Parse Exception

Hi Team,

I am facing a parse exception on libify.importer(globals(), config={}. In config I am passing key and file path through a json. See below screen shot:

I also tried to pass the key and file in this way (libify.importer(globals(), config={"key"= "afdasdfsadfsf" , "file" : "/abc/abc"}))

Can someone please help me here to understand the issue and its resolution?

Thanks,

About importing function

Code that I run:
!pip install libify
import libify
io = libify.importer(globals(), '/Users/[email protected]/src/input_output/test_notebook')
function = io.cast_days()

Error that I get:
AttributeError: module 'libified_' has no attribute 'cast_days'

AttributeError Traceback (most recent call last)
in
----> 1 function = io.cast_days()

AttributeError: module 'libified_' has no attribute 'cast_days

Code in the notebook:
import pandas as pd
import libify

def cast_days(days):
if not isinstance(days, pd.DatetimeIndex):
return pd.DatetimeIndex(list(days))
return days

libify.exporter(globals())

libified functions can't be pickled

Hey,

Thanks a lot for the lib, very useful! :) There is however one annoying issue with it at the moment.

Whenever we have the need to use a function that we import using libify within a dataframe foreach/foreachPartition we receive these errors:

ModuleNotFoundError: No module named 'libified_'
or
Can't pickle <class 'libified_.BW'>: it's not found as libified_.BW

My use-case is the following. I want to create a database connection for each partition to do some sort of batch export from a spark dataframe using https://github.com/SAP/PyHDB

If I create directly a pyhdb connection using pyhdb.connect(...) within my foreachPartition it works fine but if I am using a library that I imported with libify that would do a

def get_connection():
  return pyhdb.connect(...)

it does not work with the errors mentioned above

Not able to access the output of libify.exporter(globals())

Currently it doesn't seem possible to access the json output of libify.exporter(globals()). The only way to do so is by copying it from the console. Is there a work-around?

Libify speed & debugging

Hi,

this is more of a question than a bug report. I just implement the lib to organize code for my somewhat complex databricks project. However, I am experiencing significant drops in loading speeds when compared to standard %run './utils', that is from seconds to minutes.

Is there something I could do to improve the loading speed?

Also, how do you debug such a code without reporting the actual error messages in the importing notebook?

Thanks.

Br,

MF

Missing the dependencies setting in setup.py

Summary

setup.py in this package seems to have no install_requires. This causes error "ModuleNotFoundError: No module named 'cryptography'" when importing libify because pip doesn't know dependencies of the package.

Version

v0.78

Steps to Reproduce

$ pip install libify
Collecting libify
  Downloading libify-0.78-py3-none-any.whl (4.3 kB)
Installing collected packages: libify
Successfully installed libify-0.78

$ python -c "import libify"
Traceback (most recent call last):    
  File "<string>", line 1, in <module>
  File "/.../python3.8/site-packages/libify/__init__.py", line 3, in <module>
    from cryptography.fernet import Fernet
ModuleNotFoundError: No module named 'cryptography'

Possible Solution

By writing dependencies in setup.py, pip should find information for the dependencies correctly.

--- a/setup.py
+++ b/setup.py
@@ -20,4 +20,7 @@ setup(name='libify',
         'Operating System :: OS Independent',
     ],
     python_requires='>=3.5',
+    install_requires=[
+      'cryptography>=37.0.2',
+    ],
     zip_safe=False)```