looker-open-source/pylookml

A pythonic api for automatic lookml

pylookml's Issues

Whitespace umbrella issue

Add parameters to allow better whitespace control

line break before final } in most stanzas

dense lookml settings

call out unnecessary parameters etc

changing field delimiter to blank doesn't remove backticks

import lookml

lookml.DB_FIELD_DELIMITER_START = ''

lookml.DB_FIELD_DELIMITER_END = ''

order_items = lookml.View('order_items')
order_items.setSqlTableName(sql_table_name='events',schema='public')
order_items + 'id' + 'sale_price'
order_items.addSum(order_items.sale_price)
print(order_items)

does this >>

view: order_items {
  sql_table_name: `public`.`events` ;;

  dimension: id {
    type: string
    sql: ${TABLE}.`id` ;;
  }
  
  dimension: sale_price {
    type: string
    sql: ${TABLE}.`sale_price` ;;
  }
  
  measure: total_sale_price {
    type: sum
    sql: ${sale_price} ;;
  }
}

sql property still has backticks 💥

allow arbitrary file names

foo.bar.lkml should work, although currently typing is derived from the naming convention

output new dimension isn't readable

Output Format of new dimension made by addDimension method is incorrect indent

・Code

view = lookml.VIew('pylookml')
view.addDimension('id')
view.id.sql = '${TABLE}.id'
file = lookml.File('lookml_file')
file.put(view)

・Expected Output

  dimension: id {
     type: string
     sql: ${TABLE}.id ;;
  }

・Result Output

    dimension: id {    type: string
    sql: ${TABLE}.id ;; }

explore + join doesn't add join

explore + join doesn't add the join to the explore

workaround:
explore.addJoin(join)

After creating a new file in a branch, reading or updating the file always gets error

In exists() function,
self.repo.get_contents(f0)
should have branch parameter like
self.repo.get_contents(f0, ref=self.branch)

View.setFolder doesn't write in that folder, only in working directory

This writes it in working directory

order_items.setFolder('new')
order_items.write(overWriteExisting=True)

full path also writes in working directory

order_items.setFolder('~/Documents/GitHub/schema_webhook/new')
order_items.write(overWriteExisting=True)

ProjectSSH project files aren't being staged before committing

Adding the .add() method to ProjectSSH files allows users to stage the changes. See pull request.

Keeping this separate from the .commit() method allows users to stage and commit in batches, if desired.

one char name length

current validation does not allow one character length names for views

Crash with 2 PKs & duplicate dim name

Traceback (most recent call last):
File "/Users/timgoldstein/working/tools_lookml/tool1_lookml.py", line 20, in
myFile = proj.file('order_items.view.lkml')
File "/usr/local/lib/python3.7/site-packages/lookml/lookml.py", line 400, in file
return File(self.gitControllerSession.absoluteOutputPath + '/' + path)
File "/usr/local/lib/python3.7/site-packages/lookml/lookml.py", line 632, in init
self.vws = self.view_collection(self.json_data['views'])
File "/usr/local/lib/python3.7/site-packages/lookml/lookml.py", line 497, in init
self.add(view)
File "/usr/local/lib/python3.7/site-packages/lookml/lookml.py", line 507, in add
v = View(v)
File "/usr/local/lib/python3.7/site-packages/lookml/lookml.py", line 891, in init
super(View, self).init(input)
File "/usr/local/lib/python3.7/site-packages/lookml/lookml.py", line 773, in init
self._bind_lkml(input)
File "/usr/local/lib/python3.7/site-packages/lookml/lookml.py", line 924, in _bind_lkml
self + Dimension(field)
File "/usr/local/lib/python3.7/site-packages/lookml/lookml.py", line 982, in add
return self.addField(other)
File "/usr/local/lib/python3.7/site-packages/lookml/lookml.py", line 1205, in addField
self.setPrimaryKey(field.identifier)
File "/usr/local/lib/python3.7/site-packages/lookml/lookml.py", line 1261, in setPrimaryKey
self.unSetPrimaryKey()
File "/usr/local/lib/python3.7/site-packages/lookml/lookml.py", line 1307, in unSetPrimaryKey
pk.unSetPrimaryKey()
File "/usr/local/lib/python3.7/site-packages/lookml/lookml.py", line 2139, in unSetPrimaryKey
self.unSetProperty('primary_key')
File "/usr/local/lib/python3.7/site-packages/lookml/lookml.py", line 829, in unSetProperty
self.properties.del(name)
AttributeError: 'Properties' object has no attribute 'del'

lookml.Explore.write() support

You should be able to write a file to
explore_name.explore.lkml

This is failing currently

import lookml
vw = lookml.View('new-view')
ex = lookml.Explore(vw)

print(ex)
ex.write()

Traceback (most recent call last):
  File "testing.py", line 6, in <module>
    ex.write()
TypeError: 'dict' object is not callable

example of working with local file

Add an example of working with a local file

Not able to create a file from view

view_name = lookml.View('test_n')
lookml.File(view_name)
Traceback (most recent call last):
File "", line 1, in
File python3.8\3.8.10\lib\site-packages\lookml\lib\project.py", line 47, in init
self.name = self.path.split('/')[-1]
TypeError: 'NoneType' object is not callable

Traverse whole project dependencies

have a method like proj.map() which then has a exposes a variable deps with a full dependency map

`fields_hidden_by_default: yes` is removed

Hello all,

Upon performing lookml.File("file_path").write() it will remove some key values such as fields_hidden_by_default: yes on version 3.0.3

To recreate:

the file:

view: view_name {
  fields_hidden_by_default: yes

  dimension: pk {
    description: "some pk"
    type: string
    primary_key: yes
    sql: ${TABLE}."pk" ;;
  }

  dimension: name_of_cats {
    description: "names of cats"
    type: string
    sql: ${TABLE}."name_of_cats" ;;
  }
}

the code:

import lookml

file_path = "path/to/yaml.view.lookml"
view_file = lookml.File(file_path)
view_file.write()

batch commit github

Have project mode in github which commits all code at once rather than at file.write()

Note: commits are already batched in SSH mode

property check

currently
myView = lookml.View('''view: foo { sql_table_name: public.foo ;; }''')
if 'sql_table_name' in myView:
print("success")

works, but ideally this would as well:

if my_view.sql_table_name:
print("success")

currently the boolean check always returns true. Investigate how difficult this would be to address.

Support arbitrary .lkml file types

It would be great to support arbitrary file extensions that end .lkml rather than throwing an exception (lookml.py line 629). I have worked on projects where people use all sorts of file types for organisation, like foo.tests.lkml or bar.datagroups.lkml

rename repo to pylookml

ssh / shell based project file path

the project controller assumes that the "project name" is the same as the ssh repo name which becomes a folder under .tmp when the project is initialized

deletion of comments when using __str()__

when using lookml.File(file).str() , the output file doesn't have the comments (line starting with #).

any idea how to keep them ?

Dependency Tracing Enhancements

Dependency Graphing:

# TODO: Ancestor functions? 
# TODO: Child function support renaming across all properties (html, links, etc)
# TODO: Multi-generation dependency tracing (ancestor / decendangt)
# TODO: cross file / whole project?

Feature Request: Stage Commits and Deploy to master once

I'm looking for a way to stage all my files into a branch, preferably in one commit,

I'm writing many LookML files (>100) as part of an automation flow. At the end of file creation, I have an array of File classes that I want to push to production.

# files is an array of lookml.File class
for f in files:
  f.put()

This will commit and push one file to master and use the looker deploy webhook, which means looping through we're doing this 100+ times. In an ideal world there is an example or functionality of having a branch to write to, preferably each file update on the same commit, and one push to production and one deploy webhook

Add support for manifest files

Manifest files

New filters syntax

new default filters syntax requires change in the underlying lkml library for parsing and needs to be written as default syntax by pyLookML's renderer

filters: { 
      field:identifier 
      value:"string" 
    } 
filters: { 
     field:identifier2 
     value:"string2" 
   }

becomes

filters: [
     identifier: "string", 
     identifier2:"string2"
    ]

Check presence of file more easily

check for the presence of a file in the index
project.exists()
'foo.view.lkml' in project.files()?

.files() iterator behavior of projectGihub when index_whole turned off

since the iterator behavior is based off the index, index_whole the index is empty.

lookml.Model.include(explore)

You should be able to include an explore file to a model file. Possibly related to this #8

This currently doesn't work:

import lookml
vw = lookml.View('new-view')
ex = lookml.Explore(vw)
md = lookml.Model('new-model')
md.include(vw)
md.include(ex)

(v) 15er:tracks bryanweber$ python testing.py 
Traceback (most recent call last):
  File "testing.py", line 8, in <module>
    print(md)
  File "/Users/bryanweber/python/lookml/lookml/lookml.py", line 712, in __str__
    '\n'.join([str(p) for p in self.properties.getProperties()]), 
  File "/Users/bryanweber/python/lookml/lookml/lookml.py", line 712, in <listcomp>
    '\n'.join([str(p) for p in self.properties.getProperties()]), 
  File "/Users/bryanweber/python/lookml/lookml/lookml.py", line 821, in getProperties
    yield Property(k, n)
  File "/Users/bryanweber/python/lookml/lookml/lookml.py", line 775, in __init__
    raise Exception('not a dict, list or string')
Exception: not a dict, list or string

Fields named `name` are dropped after the file is parsed

I noticed that when one of the views includes a field name name, the field gets dropped after the view is parsed.

Here's a simplified script that shows the problem:

view_lookml = """
view: dim_test {

  sql_table_name: dim_test;;

  dimension: name {
    type: string
    sql: ${TABLE}."NAME" ;;
  }

}
"""
view = lookml.View(view_lookml)
print(view)

And the output of this script is the following:

view: dim_test {
  sql_table_name: dim_test ;;
}

Full API reference is blank on readthedocs.io

When going to https://pylookml.readthedocs.io/en/latest/reference.html , I see nothing:

What I've tried so far:

• Tried with Firefox and Chrome
• Tried going back to stable version

LookML Dashboards

Parse and generate YAML LookML dashboards

Ability to search for dependencies / relationships at a project and metaproject level

Field level references (ancestor, children, 1st or 2nd order of both)
View explore relationships (i.e. find an orphan)

Project level, ideally also a project imports level....

Refinements?

Incorporate agg lookml for a dashboard (looker will generate it from this API endpoint)

API Endpoint for aggregate table LookML for a dashboard [Link]

function to create a new branch on origin

proj = lookml.Project(
        repo=GIT_REPO,
        access_token=GIT_ACCESS_TOKEN,
        looker_host='https://example.looker.com/',
        looker_project_name = LOOKER_PROJECT,
        branch = GIT_BRANCH
      )
sb = proj._git_connection.get_branch('master')
proj._git_connection.create_git_ref(ref='refs/heads/' + GIT_BRANCH, sha=sb.commit.sha)

wrap those last two lines in a project method

Support loading from cached version of project

It would be great if you could initialise a Project from a local directory, or have the option to use the cached version saved in .tmp instead of loading from git each time. This could speed up the development process

Allow files from one project to be added to another seamlessly

right now the f and f_github class instances are not allowed outside of their projects, foreign File instances can be put to a project however

PDF Dependency Graph

Create a PR

Model file parsing by pylookml

I am currently looking on views creation by pylookml and I have successfully created .view files with pylookml. In order to expose these views as explores I am not getting any documentation of working with looker model via pylookml. Please suggest

local variable 'repo' referenced before assignment

Running into an issue while running autotune through terminal "local variable 'repo' referenced before assignment". Has anyone seen this error before?

Thanks!

model includes for explore files not processed

when a model file has 1 or more includes - for explore files, these files are not processed and added to the explores dict
e.g.

include: "/explores/customers.explore"
include: "/explores/orders.explore"

i can see that includes is enumerated - but it appears that they need to be parsed seperately.

I am not familiar enough with the code to change myself - may try to attempt that

Error when dimension name includes numbers

Hello, simple question, some of my database columns have numbers in their names and I've noticed lookml doesn't like them. Is this by design?

This works fine:

lookml.Dimension('dimension: street {}')

This gives an error:

lookml.Dimension('dimension: street1 {}')

Exception: dimension: street1 {} does not match a view: name {} or lookml name pattern

Option to turn off dense lookml

something like
lookml.dense = False

adding new dimensions and measures adds extra new line

Here's the code

import lookml

order_items = lookml.View('order_items')
order_items.setSqlTableName(sql_table_name='events',schema='public')
order_items + 'id' + 'sale_price'
order_items.addSum(order_items.sale_price)
print(order_items)

Here's the output, notice the extra new line between id, sale_price and sale_price and total_sale_price

view: order_items {
sql_table_name: `public`.`events` ;;

dimension: id {
    type: string
    sql: ${TABLE}.`id` ;;
}

    
dimension: sale_price {
    type: string
    sql: ${TABLE}.`sale_price` ;;
}

    
measure: total_sale_price {
    type: sum
    sql: ${sale_price} ;;
}

}

Cannot get lookml files with double-byte characters

in lookml.py, it's hardcoded decode by 'ascii'.

#Parse Step: Github content is returned base64 encoded
data = base64.b64decode(f.content).decode('ascii')

Join class missing setProperty

setProperty is missing for Join

import lookml
join = lookml.Join('kewl')
join.setProperty('sql','1=1')

current workaround is join.properties.addProperty('sql','1=1')

Method to Create a new model file

Currently there does not exist an ability to create a new model file from scratch, as experienced by Hugo.

Catch project.deploy() not updating

Having an invalid looker_project_name may not give errors, but it means the deploy webhook might fail silently. Ideally we catch the response from the webhook and notify that nothing was updated

proj = lookml.Project(
  repo= "organization/customer-repo",
  access_token="abc123",
  #Optional args for the deploy URL (for deploying directly to prodcution mode)
  looker_host="https://customer.looker.com/",
  looker_project_name="wrong_project_name"
)

for f in files:
  proj.put(f)

proj.deploy()

ModuleNotFoundError: No module named 'dataclasses'

import lookml was working fine with 3.6.12 until Jan 08.
Started failing with below error on Jan 09
ModuleNotFoundError: No module named 'dataclasses'

I understand there were some changes in https://github.com/joshtemple/lkml on Jan 9th that might be related.
Just wondering why there is no new release with these changes and are all part of lookml 2.0.0?

gitlab integration?

The cookbook shows how to connect to a lookml github project.
Would it be possible to connect to to a gitlab project?

Command Line Interface

Add support for command line interface