titan-systems/titan

Snowflake infrastructure-as-code. Provision environments, automate deploys, CI/CD. Manage RBAC, users, roles, and data access. Declarative Python Resource API. Change Management tool for the Snowflake data warehouse.

titan's Issues

[Nitpick] - Python support for Terraform in the README

Really small, but should the README be updated to mention that Terraform actually supports "Python-based definitions" (as well as typescript, go and more) via CDKTF?

Incorrect behaviour in Jupyter notebooks due to caching

I've been experimenting with Titan in a Jupyter notebook, was seeing some strange behaviour where roles were attempting to be created again.
Realised that SHOW ROLES LIKE .... etc were not actually running and found the @cache attributes.

If the cache is cleared at the start of each plan(), I figure it will all work fine in Jupyter.

Support Other Dialects

Currently all the resources are for Snowflake. Are their plans to write resources for other backends?

KeyError during Blueprint.plan

I'm currently receiving a KeyError when fetch_session is called when running Blueprint.plan, specifically when tags are being retrieved:

https://github.com/teej/titan/blob/371364c8d10eb596da2b323deec0632c3d37c08c/titan/data_provider.py#L175

Just looking at the Snowflake docs, the column names appear to be database_name and schema_name.

I'm using the new 0.1.0 release also:

Schemas named PUBLIC under new databases are added to the plan for creation

Blueprint planner should handle the case where the resource doesn't yet exist, but will be created automatically during apply()

UnboundLocalError in Install script

When following the install instructions and executing the below an exception is thrown.:

EXECUTE IMMEDIATE
  FROM @titan.public.titan_aws/install;

Response from Snowflake is shown below.
Currently the project seems to not to be installable.

Python Interpreter Error:
Traceback (most recent call last):
  File "/home/udf/501152416749/titan-0.1.1.zip/titan/spi.py", line 72, in install
    session_ctx = dp.fetch_session(conn)
  File "/home/udf/501152416749/titan-0.1.1.zip/titan/data_provider.py", line 160, in fetch_session
    session_obj = execute(
  File "/home/udf/501152416749/titan-0.1.1.zip/titan/client.py", line 72, in execute
    return _execute(session, sql, use_role)
  File "/home/udf/501152416749/titan-0.1.1.zip/titan/client.py", line 51, in _execute
    print(f"[{session.user}:{session.role}] >", sql_text, end="")
UnboundLocalError: local variable 'session' referenced before assignment
 in function INSTALL with handler titan.spi.install

Renaming resources isn't ordered correctly

Renaming resources in a plan that changes other

ALTER USER aaa RENAME TO bbb;
ALTER USER aaa SET LOGIN_NAME = 'bbb'; # fails

Either the rename should propagate to the other alters (hard) or it should be sequenced last (easier).

Using Grant resource returns a TypeError

I have the following code:

import os

import snowflake.connector
from titan import Blueprint
from titan.resources import Database, Grant, Role, RoleGrant, Warehouse

connection_params = {
    "account": os.environ["SNOWFLAKE_ACCOUNT"],
    "user": os.environ["SNOWFLAKE_USER"],
    "password": os.environ["SNOWFLAKE_PASSWORD"],
}

DEFAULT_WAREHOUSE_OPTIONS = {
    "warehouse_size": "XSMALL",
    "auto_suspend": 60,
    "min_cluster_count": 1,
    "max_cluster_count": 1,
}


def dbt():

    prod_db = Database(name="ANALYTICS")

    transforming_wh = Warehouse(
        name="TRANSFORMING_DPG_WH",
        comment="Warehouse used for transformation workloads.",
        **DEFAULT_WAREHOUSE_OPTIONS,
    )

    transformer_role = Role(
        name="TRANSFORMER_ROLE",
        comment=f"Able to read from and write to {prod_db.name} database.",
    )

    grants = [
        RoleGrant(role=transformer_role, to_role="SYSADMIN"),
        Grant(priv="USAGE", on=prod_db, to=transformer_role),
    ]

    return (
        prod_db,
        transforming_wh,
        transformer_role,
        *grants,
    )


if __name__ == "__main__":
    bp = Blueprint(name="setup-dbt", account=os.environ["SNOWFLAKE_ACCOUNT"])
    bp.add(*dbt())
    session = snowflake.connector.connect(**connection_params)
    plan = bp.plan(session)

    # Update snowflake to match blueprint
    bp.apply(session, plan)

When I run this script, I get back the following error:

  File "/Users/dpg/personal/snowflake-dbt-demo-new/.venv/lib/python3.9/site-packages/titan/identifiers.py", line 38, in <listcomp>
    params = "?" + "&".join([f"{k.lower()}={v}" for k, v in self.params.items()]) if self.params else ""
TypeError: __format__ must return a str, not FQN

When I remove the lone Grant resource within my grants variable, I don't have any issues. I may also just be doing something wrong, so apologize in advance if that's the case. Appreciate any help!