yu-iskw/dbt-artifacts-parser

A dbt artifacts parser in python

dbt-artifacts-parser's Issues

none is not an allowed value: sources -> source.default.xyz -> metadata -> type

I get the above error for dbt version 1.4.1 using snowflake.

The error is related to this line: https://github.com/yu-iskw/dbt-artifacts-parser/blob/main/dbt_artifacts_parser/parsers/catalog/catalog_v1.py#L33

The obvious fix here is to adapt the type to be Optional[str] - but if I understand this is auto-generated based on this line. And this is based on the dbt-core definition here.

What's the standard process in this case?

ManifestV10 parsing errors on full 1.6 release

"metrics" are not passing validation for V10. Example:

13:10:44 ERROR    51203 validation errors for ManifestV10                                                                                                     
                  metrics -> metric.package.metric_name -> type_params -> input_measures                                
                    extra fields not permitted (type=value_error.extra)           

Looks like the schema was updated on Aug. 7th https://schemas.getdbt.com/dbt/manifest/v10.json:

        "dbt_version": {
          "type": "string",
          "default": "1.6.0"
        },
        "generated_at": {
          "type": "string",
          "format": "date-time",
          "default": "2023-08-07T20:10:03.381822Z"
        },

Add support for semantic manifest files

It would be great if this library supported semantic_manifest parsing.

Support pydantic 2

As of this writing, we support only pydantic v1. It would be good to support pydantic v2 as well.

Using local copies of artifact schemas for validation causes breakage

Recently the package dbt-common, which is an dependency of dbt-core has updated from 1.5.0 to 1.6.0. This introduced schema changes in mainfest.json which in turn caused this package to break, since it's using a local copy of the V12 schema which has meanwhile updated to include the new schema changes (see here).

At first I've actually opened an issue with them (please see here ), but apparently their policy is, that changes to the schemas are allowed to happen in between versions, which means your package might fail at any time (when any dbt-core dependancies update etc), even if no new artifact schema version was released.

add support for `manifest_v9.json`

dbt-core version 1.5 will start using version 9 of the manifest schema

Can we add support in this package as well? Thanks!

Show a list of contributors in README

Motivation

I would like to respect contributors of the repository. So, I will make it more visible to to a list of the contributors in README.

resource_type not defined correctly for manifest v11

In the manifest v11, resource_type parameter for ModelNode is defined as Any here. Due to this the actual value from manifest.json maps as a str type in pydantic model.

In the previous manifest v10,v9 this used to be mapped as an enum in python. Example here and here

This is causing dtype mismatching while processing the v11 manifest.

Solution would be to convert the v11 resource_type attribute to enum instead of any

@yu-iskw

Add an option to be more permissive with extra fields for manifest parsing

DBT Cloud released a versionless deployment mode which rollout update on Manifest it seems. It adds new properties.

You can check schema here: https://schemas.getdbt.com/dbt/manifest/v12.json

I think we should be more permissive with it, allowing users to specify if extra fields should be blocking or not. I guess in vast majority of cases (lineage parsing tools like https://github.com/open-metadata/OpenMetadata) it should not be blocking.

Security Policy violation SECURITY.md

This issue was automatically created by Allstar.

Security Policy Violation
Security policy not enabled.
A SECURITY.md file can give users information about what constitutes a vulnerability and how to report one securely so that information about a bug is not publicly visible. Examples of secure reporting methods include using an issue tracker with private issue support, or encrypted email with a published key.

To fix this, add a SECURITY.md file that explains how to handle vulnerabilities found in your repository. Go to https://github.com/yu-iskw/dbt-artifacts-parser/security/policy to enable.

For more information, see https://docs.github.com/en/code-security/getting-started/adding-a-security-policy-to-your-repository.

---

This issue will auto resolve when the policy is in compliance.

Issue created by Allstar. See https://github.com/ossf/allstar/ for more information. For questions specific to the repository, please contact the owner or maintainer.

Validation errors for dbt > 1.4.1

FYI if you're noticing manifests not parsing for dbt > 1.4.1

dbt-labs/dbt-core#7119

Support v8 manifest

Hey all,

I noticed that there's a new manifest version. Happy to help with this if no one is working on it yet

https://schemas.getdbt.com/dbt/manifest/v8.json

Base Classes

I'm not really sure how this would work, however, it might be nice to be able to have base classes for the various artifacts which would allow for better type hinting than BaseModel.

I'm open to ideas and open to helping.

Support dbt manifest.json version 6

https://schemas.getdbt.com/dbt/manifest/v6/index.html

[Feature request] Support for ManifestV7

V7 was introduced with dbt 1.3.