looker-open-source / pylookml Goto Github PK
View Code? Open in Web Editor NEWA pythonic api for automatic lookml
Home Page: https://pylookml.readthedocs.io/en/latest/introduction.html
License: MIT License
A pythonic api for automatic lookml
Home Page: https://pylookml.readthedocs.io/en/latest/introduction.html
License: MIT License
Having an invalid looker_project_name may not give errors, but it means the deploy webhook might fail silently. Ideally we catch the response from the webhook and notify that nothing was updated
proj = lookml.Project(
repo= "organization/customer-repo",
access_token="abc123",
#Optional args for the deploy URL (for deploying directly to prodcution mode)
looker_host="https://customer.looker.com/",
looker_project_name="wrong_project_name"
)
for f in files:
proj.put(f)
proj.deploy()
Here's the code
import lookml
order_items = lookml.View('order_items')
order_items.setSqlTableName(sql_table_name='events',schema='public')
order_items + 'id' + 'sale_price'
order_items.addSum(order_items.sale_price)
print(order_items)
Here's the output, notice the extra new line between id, sale_price and sale_price and total_sale_price
view: order_items {
sql_table_name: `public`.`events` ;;
dimension: id {
type: string
sql: ${TABLE}.`id` ;;
}
dimension: sale_price {
type: string
sql: ${TABLE}.`sale_price` ;;
}
measure: total_sale_price {
type: sum
sql: ${sale_price} ;;
}
}
Output Format of new dimension made by addDimension method is incorrect indent
・Code
view = lookml.VIew('pylookml')
view.addDimension('id')
view.id.sql = '${TABLE}.id'
file = lookml.File('lookml_file')
file.put(view)
・Expected Output
dimension: id {
type: string
sql: ${TABLE}.id ;;
}
・Result Output
dimension: id { type: string
sql: ${TABLE}.id ;; }
This writes it in working directory
order_items.setFolder('new')
order_items.write(overWriteExisting=True)
full path also writes in working directory
order_items.setFolder('~/Documents/GitHub/schema_webhook/new')
order_items.write(overWriteExisting=True)
When going to https://pylookml.readthedocs.io/en/latest/reference.html , I see nothing:
What I've tried so far:
when a model file has 1 or more includes - for explore files, these files are not processed and added to the explores dict
e.g.
include: "/explores/customers.explore"
include: "/explores/orders.explore"
i can see that includes is enumerated - but it appears that they need to be parsed seperately.
I am not familiar enough with the code to change myself - may try to attempt that
API Endpoint for aggregate table LookML for a dashboard [Link]
Traceback (most recent call last):
File "/Users/timgoldstein/working/tools_lookml/tool1_lookml.py", line 20, in
myFile = proj.file('order_items.view.lkml')
File "/usr/local/lib/python3.7/site-packages/lookml/lookml.py", line 400, in file
return File(self.gitControllerSession.absoluteOutputPath + '/' + path)
File "/usr/local/lib/python3.7/site-packages/lookml/lookml.py", line 632, in init
self.vws = self.view_collection(self.json_data['views'])
File "/usr/local/lib/python3.7/site-packages/lookml/lookml.py", line 497, in init
self.add(view)
File "/usr/local/lib/python3.7/site-packages/lookml/lookml.py", line 507, in add
v = View(v)
File "/usr/local/lib/python3.7/site-packages/lookml/lookml.py", line 891, in init
super(View, self).init(input)
File "/usr/local/lib/python3.7/site-packages/lookml/lookml.py", line 773, in init
self._bind_lkml(input)
File "/usr/local/lib/python3.7/site-packages/lookml/lookml.py", line 924, in _bind_lkml
self + Dimension(field)
File "/usr/local/lib/python3.7/site-packages/lookml/lookml.py", line 982, in add
return self.addField(other)
File "/usr/local/lib/python3.7/site-packages/lookml/lookml.py", line 1205, in addField
self.setPrimaryKey(field.identifier)
File "/usr/local/lib/python3.7/site-packages/lookml/lookml.py", line 1261, in setPrimaryKey
self.unSetPrimaryKey()
File "/usr/local/lib/python3.7/site-packages/lookml/lookml.py", line 1307, in unSetPrimaryKey
pk.unSetPrimaryKey()
File "/usr/local/lib/python3.7/site-packages/lookml/lookml.py", line 2139, in unSetPrimaryKey
self.unSetProperty('primary_key')
File "/usr/local/lib/python3.7/site-packages/lookml/lookml.py", line 829, in unSetProperty
self.properties.del(name)
AttributeError: 'Properties' object has no attribute 'del'
I am currently looking on views creation by pylookml and I have successfully created .view files with pylookml. In order to expose these views as explores I am not getting any documentation of working with looker model via pylookml. Please suggest
Adding the .add() method to ProjectSSH files allows users to stage the changes. See pull request.
Keeping this separate from the .commit() method allows users to stage and commit in batches, if desired.
the project controller assumes that the "project name" is the same as the ssh repo name which becomes a folder under .tmp when the project is initialized
You should be able to include an explore file to a model file. Possibly related to this #8
This currently doesn't work:
import lookml
vw = lookml.View('new-view')
ex = lookml.Explore(vw)
md = lookml.Model('new-model')
md.include(vw)
md.include(ex)
(v) 15er:tracks bryanweber$ python testing.py
Traceback (most recent call last):
File "testing.py", line 8, in <module>
print(md)
File "/Users/bryanweber/python/lookml/lookml/lookml.py", line 712, in __str__
'\n'.join([str(p) for p in self.properties.getProperties()]),
File "/Users/bryanweber/python/lookml/lookml/lookml.py", line 712, in <listcomp>
'\n'.join([str(p) for p in self.properties.getProperties()]),
File "/Users/bryanweber/python/lookml/lookml/lookml.py", line 821, in getProperties
yield Property(k, n)
File "/Users/bryanweber/python/lookml/lookml/lookml.py", line 775, in __init__
raise Exception('not a dict, list or string')
Exception: not a dict, list or string
current validation does not allow one character length names for views
Add an example of working with a local file
import lookml was working fine with 3.6.12 until Jan 08.
Started failing with below error on Jan 09
ModuleNotFoundError: No module named 'dataclasses'
I understand there were some changes in https://github.com/joshtemple/lkml on Jan 9th that might be related.
Just wondering why there is no new release with these changes and are all part of lookml 2.0.0?
explore + join
doesn't add the join to the explore
workaround:
explore.addJoin(join)
in lookml.py, it's hardcoded decode by 'ascii'.
#Parse Step: Github content is returned base64 encoded
data = base64.b64decode(f.content).decode('ascii')
proj = lookml.Project(
repo=GIT_REPO,
access_token=GIT_ACCESS_TOKEN,
looker_host='https://example.looker.com/',
looker_project_name = LOOKER_PROJECT,
branch = GIT_BRANCH
)
sb = proj._git_connection.get_branch('master')
proj._git_connection.create_git_ref(ref='refs/heads/' + GIT_BRANCH, sha=sb.commit.sha)
wrap those last two lines in a project method
The cookbook shows how to connect to a lookml github project.
Would it be possible to connect to to a gitlab project?
Add support for command line interface
Parse and generate YAML LookML dashboards
foo.bar.lkml should work, although currently typing is derived from the naming convention
I noticed that when one of the views includes a field name name
, the field gets dropped after the view is parsed.
Here's a simplified script that shows the problem:
view_lookml = """
view: dim_test {
sql_table_name: dim_test;;
dimension: name {
type: string
sql: ${TABLE}."NAME" ;;
}
}
"""
view = lookml.View(view_lookml)
print(view)
And the output of this script is the following:
view: dim_test {
sql_table_name: dim_test ;;
}
Hello all,
Upon performing lookml.File("file_path").write()
it will remove some key values such as fields_hidden_by_default: yes
on version 3.0.3
To recreate:
the file:
view: view_name {
fields_hidden_by_default: yes
dimension: pk {
description: "some pk"
type: string
primary_key: yes
sql: ${TABLE}."pk" ;;
}
dimension: name_of_cats {
description: "names of cats"
type: string
sql: ${TABLE}."name_of_cats" ;;
}
}
the code:
import lookml
file_path = "path/to/yaml.view.lookml"
view_file = lookml.File(file_path)
view_file.write()
It would be great if you could initialise a Project
from a local directory, or have the option to use the cached version saved in .tmp
instead of loading from git each time. This could speed up the development process
have a method like proj.map() which then has a exposes a variable deps with a full dependency map
new default filters syntax requires change in the underlying lkml library for parsing and needs to be written as default syntax by pyLookML's renderer
filters: {
field:identifier
value:"string"
}
filters: {
field:identifier2
value:"string2"
}
becomes
filters: [
identifier: "string",
identifier2:"string2"
]
view_name = lookml.View('test_n')
lookml.File(view_name)
Traceback (most recent call last):
File "", line 1, in
File python3.8\3.8.10\lib\site-packages\lookml\lib\project.py", line 47, in init
self.name = self.path.split('/')[-1]
TypeError: 'NoneType' object is not callable
something like
lookml.dense = False
# TODO: Ancestor functions?
# TODO: Child function support renaming across all properties (html, links, etc)
# TODO: Multi-generation dependency tracing (ancestor / decendangt)
# TODO: cross file / whole project?
currently
myView = lookml.View('''view: foo { sql_table_name: public.foo ;; }''')
if 'sql_table_name' in myView:
print("success")
works, but ideally this would as well:
if my_view.sql_table_name:
print("success")
currently the boolean check always returns true. Investigate how difficult this would be to address.
Currently there does not exist an ability to create a new model file from scratch, as experienced by Hugo.
You should be able to write a file to
explore_name.explore.lkml
This is failing currently
import lookml
vw = lookml.View('new-view')
ex = lookml.Explore(vw)
print(ex)
ex.write()
Traceback (most recent call last):
File "testing.py", line 6, in <module>
ex.write()
TypeError: 'dict' object is not callable
when using lookml.File(file).str() , the output file doesn't have the comments (line starting with #).
any idea how to keep them ?
setProperty is missing for Join
import lookml
join = lookml.Join('kewl')
join.setProperty('sql','1=1')
current workaround is join.properties.addProperty('sql','1=1')
import lookml
lookml.DB_FIELD_DELIMITER_START = ''
lookml.DB_FIELD_DELIMITER_END = ''
order_items = lookml.View('order_items')
order_items.setSqlTableName(sql_table_name='events',schema='public')
order_items + 'id' + 'sale_price'
order_items.addSum(order_items.sale_price)
print(order_items)
does this >>
view: order_items {
sql_table_name: `public`.`events` ;;
dimension: id {
type: string
sql: ${TABLE}.`id` ;;
}
dimension: sale_price {
type: string
sql: ${TABLE}.`sale_price` ;;
}
measure: total_sale_price {
type: sum
sql: ${sale_price} ;;
}
}
sql property still has backticks 💥
I'm looking for a way to stage all my files into a branch, preferably in one commit,
I'm writing many LookML files (>100) as part of an automation flow. At the end of file creation, I have an array of File classes that I want to push to production.
# files is an array of lookml.File class
for f in files:
f.put()
This will commit and push one file to master and use the looker deploy webhook, which means looping through we're doing this 100+ times. In an ideal world there is an example or functionality of having a branch to write to, preferably each file update on the same commit, and one push to production and one deploy webhook
since the iterator behavior is based off the index, index_whole the index is empty.
It would be great to support arbitrary file extensions that end .lkml
rather than throwing an exception (lookml.py line 629). I have worked on projects where people use all sorts of file types for organisation, like foo.tests.lkml
or bar.datagroups.lkml
Hello, simple question, some of my database columns have numbers in their names and I've noticed lookml doesn't like them. Is this by design?
This works fine:
lookml.Dimension('dimension: street {}')
This gives an error:
lookml.Dimension('dimension: street1 {}')
Exception: dimension: street1 {} does not match a view: name {} or lookml name pattern
In exists() function,
self.repo.get_contents(f0)
should have branch parameter like
self.repo.get_contents(f0, ref=self.branch)
Add parameters to allow better whitespace control
line break before final }
in most stanzas
dense lookml settings
call out unnecessary parameters etc
right now the f and f_github class instances are not allowed outside of their projects, foreign File instances can be put to a project however
Create a PR
check for the presence of a file in the index
project.exists()
'foo.view.lkml' in project.files()?
Running into an issue while running autotune through terminal "local variable 'repo' referenced before assignment". Has anyone seen this error before?
Thanks!
Field level references (ancestor, children, 1st or 2nd order of both)
View explore relationships (i.e. find an orphan)
Project level, ideally also a project imports level....
Refinements?
Have project mode in github which commits all code at once rather than at file.write()
Note: commits are already batched in SSH mode
Manifest files
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.