curts0 / pytabular Goto Github PK
View Code? Open in Web Editor NEWConnect to Tabular Models via Python
Home Page: https://curts0.github.io/PyTabular/
License: MIT License
Connect to Tabular Models via Python
Home Page: https://curts0.github.io/PyTabular/
License: MIT License
Describe the bug
Running the provided test suite produces an error with importing clr
with Python 3.10 and Manjaro Linux. If we try importing the package from an interactive Python shell, the error is more specific about the clr
package not having an AddReference
attribute. Logs attached
To Reproduce
pytest
Expected behavior
Tests load correctly and start running.
Screenshots
(None)
Desktop (please complete the following information):
Additional context
Seeing that the module comes bundled with .dll
files led me to believe that this package was never meant to run under anything other than Microsoft Windows even though any automation of data processes would likely be carried out by data engineers, developed and tested under Linux or the Windows Subsystem for Linux (WSL), and deployed to some cloud service such as functions, app services, containers or virtual machines. All of those predominantly run Linux. The current package should either try to support and test on such systems or explicitly state that they are unsupported in the README file.
Pytest log:
=================================================================================== test session starts ====================================================================================
platform linux -- Python 3.10.6, pytest-7.1.1, pluggy-1.0.0 -- /usr/bin/python
cachedir: .pytest_cache
rootdir: /home/radi/Code/PyTabular, configfile: pyproject.toml
plugins: anyio-3.6.1
collected 0 items / 1 error
========================================================================================== ERRORS ==========================================================================================
__________________________________________________________________________ ERROR collecting test/test_tabular.py ___________________________________________________________________________
ImportError while importing test module '/home/radi/Code/PyTabular/test/test_tabular.py'.
Hint: make sure your test modules/packages have valid Python names.
Traceback:
../../.local/lib/python3.10/site-packages/_pytest/python.py:608: in _importtestmodule
mod = import_path(self.path, mode=importmode, root=self.config.rootpath)
../../.local/lib/python3.10/site-packages/_pytest/pathlib.py:533: in import_path
importlib.import_module(module_name)
/usr/lib/python3.10/importlib/__init__.py:126: in import_module
return _bootstrap._gcd_import(name[level:], package, level)
<frozen importlib._bootstrap>:1050: in _gcd_import
???
<frozen importlib._bootstrap>:1027: in _find_and_load
???
<frozen importlib._bootstrap>:1006: in _find_and_load_unlocked
???
<frozen importlib._bootstrap>:688: in _load_unlocked
???
../../.local/lib/python3.10/site-packages/_pytest/assertion/rewrite.py:168: in exec_module
exec(co, module.__dict__)
test/test_tabular.py:1: in <module>
import pytabular
pytabular/__init__.py:32: in <module>
import clr
E ModuleNotFoundError: No module named 'clr'
================================================================================= short test summary info ==================================================================================
ERROR test/test_tabular.py
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! Interrupted: 1 error during collection !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
===================================================================================== 1 error in 0.08s =====================================================================================
Import log:
Python 3.10.6 (main, Aug 3 2022, 17:39:45) [GCC 12.1.1 20220730] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import pytabular
22/10/01 10:06:51 +0300::__init__::<module>::INFO::Logging configured...
22/10/01 10:06:51 +0300::__init__::<module>::INFO::To update PyTabular logger...
22/10/01 10:06:51 +0300::__init__::<module>::INFO::>>> import logging
22/10/01 10:06:51 +0300::__init__::<module>::INFO::>>> pytabular.logger.setLevel(level=logging.INFO)
22/10/01 10:06:51 +0300::__init__::<module>::INFO::See https://docs.python.org/3/library/logging.html#logging-levels
22/10/01 10:06:51 +0300::__init__::<module>::DEBUG::Python Version::3.10.6 (main, Aug 3 2022, 17:39:45) [GCC 12.1.1 20220730]
22/10/01 10:06:51 +0300::__init__::<module>::DEBUG::Python Location::/home/radi/Code/PyTabular/venv
22/10/01 10:06:51 +0300::__init__::<module>::DEBUG::Package Location::/home/radi/Code/PyTabular/pytabular/__init__.py
22/10/01 10:06:51 +0300::__init__::<module>::DEBUG::Working Directory::/home/radi/Code/PyTabular
22/10/01 10:06:51 +0300::__init__::<module>::DEBUG::Platform::linux-5.19.7-1-MANJARO
22/10/01 10:06:51 +0300::__init__::<module>::DEBUG::Beginning CLR references...
22/10/01 10:06:51 +0300::__init__::<module>::DEBUG::Adding Reference Microsoft.AnalysisServices.AdomdClient
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/home/radi/Code/PyTabular/pytabular/__init__.py", line 35, in <module>
clr.AddReference("Microsoft.AnalysisServices.AdomdClient")
AttributeError: module 'clr' has no attribute 'AddReference'
Skimming through the code, I see that we are violating quite a lot of standard Python conventions both in the object names, project structure, and imports. Considering that the project is considered in early alpha, would it be OK to take steps to make backwards-incompatible changes at this point in time and apply as many conventions as possible?
Most notably, I'd like to focus on the following:
function_name
, method_name
, global_variable_name
ClassName
, ExceptionName
GLOBAL_CONSTANT
from pytabular.sub_module import whatever
(You seem to have fixed that recently?)tests
folder (plural)Implementing the above would probably also remove some flake8 exceptions the project has identified.
Hi,
I'm getting a Recursion after merging some changes from the master branch. Did anything change? I'm testing it on a production model, so maybe the adventure works model doesn't have enough translations to generate a recursion error.
self.Cultures = PyCultures(
[
PyCulture(culture, self)
for culture in self.Model.Cultures.GetEnumerator()
]
)
PyTabular/pytabular/pytabular.py
Line 119 in 18210eb
Then can run pytest in github workflow
I've been looking into Open Metadata and seeing if it's possible to add lineage from SQL Server to Power BI datasets.
So I'll be trying to add that to PyTabular or at the very least leverage PyTabular.
@Curts0; How likely is Linux support?
Hi,
I was looking into the PyTabular package, and first of thanks, finally a worthy implementation! On top of this package, I started "building" a Documenter for our models, because we use Docusaurus with generated markdown files to create our documentation site, which users can use to see how things are calculated for instance.
It seems a bit "strange" to build it on top of the PyTabular package, instead of including it in the package. Is this something you have envisioned for PyTabular? Otherwise, I'll continue building it on top of PyTabular.
Simple Example
Thanks!
I've added a ModelDocumenter
class which enables generating markdown files based on the Docusaurus notation for rendering the pages. It's not neat yet, but I will address that in the coming week. Is it easier to put the documentation in a submodule/subfolder?
@Curts0 What is easier, keep everything in the root folder of pytabular or create a documentation subfolder with all specific doc generation functions and classes?
import pytabular
import logging
logger = logging.getLogger("PyTabular")
model = pytabular.Tabular(f"{SERVER};Catalog={INITIAL_CATALOG}")
docs = pytabular.ModelDocumenter(model)
docs.save_documentation()
To-do:
## 'Date'[Date] {#date_date}
Hi,
Is there a way to first process Data Only and the Calculate?
import pytabular
model = pytabular.Tabular(CONNECTION_STR)
tables = model.Tables
if len(tables) > 0:
tables.refresh(RefreshType.DataOnly)
model.refresh(RefreshType.Calculate)
I am using python-tabular to refresh partitions in SSAS model through python script. While deploying the code in Azure Functions, I tried to trigger the azure functions, getting the import error in the import statement of this particular library.
Azure Function:
Operating System : Linux
Runtime version: 4.25.2.2
Language used : Python
Python version: 3.10
Full Exception :
Exception while executing function
/Functions.httpTrigger ---> Microsoft.Azure.WebJobs.Script.Workers.Rpc.RpcException
/Result
/Failure Exception
/RuntimeError
/Could not find libmono
/
'/azure-functions-host/workers/python/3.10/LINUX/X64/azure_functions_worker/dispatcher.py', line 380, in _handle__function_load_request func = loader.load_function(
'/azure-functions-host/workers/python/3.10/LINUX/X64/azure_functions_worker/utils/wrappers.py', line 44, in call return func(*args, **kwargs)
'/azure-functions-host/workers/python/3.10/LINUX/X64/azure_functions_worker/loader.py', line 132, in load_function mod = importlib.import_module(fullmodname)
'/usr/local/lib/python3.10/importlib/__init__.py', line 126, in import_module return _bootstrap._gcd_import(name[level:], package, level)
'<frozen importlib._bootstrap>', line 1050, in _gcd_import
'<frozen importlib._bootstrap>', line 1027, in _find_and_load
'<frozen importlib._bootstrap>', line 1006, in _find_and_load_unlocked
'<frozen importlib._bootstrap>', line 688, in _load_unlocked
'<frozen importlib._bootstrap_external>', line 883, in exec_module
'<frozen importlib._bootstrap>', line 241, in _call_with_frames_removed
'/home/site/wwwroot/httpTrigger/__init__.py', line 9, in <module> from .refresh_automation import main_func
'/home/site/wwwroot/httpTrigger/refresh_automation.py', line 6, in <module> import pytabular
'/home/site/wwwroot/.python_packages/lib/site-packages/pytabular/__init__.py', line 45, in <module> import clr
'/home/site/wwwroot/.python_packages/lib/site-packages/clr.py', line 6, in <module> load()
'/home/site/wwwroot/.python_packages/lib/site-packages/pythonnet/__init__.py', line 36, in load set_default_runtime()
'/home/site/wwwroot/.python_packages/lib/site-packages/pythonnet/__init__.py', line 22, in set_default_runtime set_runtime(clr_loader.get_mono())
'/home/site/wwwroot/.python_packages/lib/site-packages/clr_loader/__init__.py', line 21, in get_mono libmono = find_libmono(sgen)
'/home/site/wwwroot/.python_packages/lib/site-packages/clr_loader/util/find.py', line 67, in find_libmono raise RuntimeError('Could not find libmono')
System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
async Microsoft.Azure.WebJobs.Script.Description.WorkerFunctionInvoker.InvokeCore(Object[] parameters,FunctionInvocationContext context)
/src/azure-functions-host/src/WebJobs.Script/Description/Workers/WorkerFunctionInvoker.cs
/101
System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
async Microsoft.Azure.WebJobs.Script.Description.FunctionInvokerBase.Invoke(Object[] parameters)
/src/azure-functions-host/src/WebJobs.Script/Description/FunctionInvokerBase.cs
/82
System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
async Microsoft.Azure.WebJobs.Script.Description.FunctionGenerator.Coerce[T](Task`1 src)
/src/azure-functions-host/src/WebJobs.Script/Description/FunctionGenerator.cs
/225
System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
async Microsoft.Azure.WebJobs.Host.Executors.FunctionInvoker`2.InvokeAsync[TReflected,TReturnValue](Object instance,Object[] arguments)
D:\a\_work\1\s\src\Microsoft.Azure.WebJobs.Host\Executors\FunctionInvoker.cs
/52
System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
async Microsoft.Azure.WebJobs.Host.Executors.FunctionExecutor.InvokeWithTimeoutAsync(IFunctionInvoker invoker,ParameterHelper parameterHelper,CancellationTokenSource timeoutTokenSource,CancellationTokenSource functionCancellationTokenSource,Boolean throwOnTimeout,TimeSpan timerInterval,IFunctionInstance instance)
D:\a\_work\1\s\src\Microsoft.Azure.WebJobs.Host\Executors\FunctionExecutor.cs
/581
System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
async Microsoft.Azure.WebJobs.Host.Executors.FunctionExecutor.ExecuteWithWatchersAsync(IFunctionInstanceEx instance,ParameterHelper parameterHelper,ILogger logger,CancellationTokenSource functionCancellationTokenSource)
D:\a\_work\1\s\src\Microsoft.Azure.WebJobs.Host\Executors\FunctionExecutor.cs
/527
System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
async Microsoft.Azure.WebJobs.Host.Executors.FunctionExecutor.ExecuteWithLoggingAsync(IFunctionInstanceEx instance,FunctionStartedMessage message,FunctionInstanceLogEntry instanceLogEntry,ParameterHelper parameterHelper,ILogger logger,CancellationToken cancellationToken)
D:\a\_work\1\s\src\Microsoft.Azure.WebJobs.Host\Executors\FunctionExecutor.cs
/306 End of inner exception
System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
async Microsoft.Azure.WebJobs.Host.Executors.FunctionExecutor.ExecuteWithLoggingAsync(IFunctionInstanceEx instance,FunctionStartedMessage message,FunctionInstanceLogEntry instanceLogEntry,ParameterHelper parameterHelper,ILogger logger,CancellationToken cancellationToken)
D:\a\_work\1\s\src\Microsoft.Azure.WebJobs.Host\Executors\FunctionExecutor.cs
/352
System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
async Microsoft.Azure.WebJobs.Host.Executors.FunctionExecutor.TryExecuteAsync(IFunctionInstance functionInstance,CancellationToken cancellationToken)
D:\a\_work\1\s\src\Microsoft.Azure.WebJobs.Host\Executors\FunctionExecutor.cs
/108
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.