fivetran/dbt_zendesk_source

Fivetran's Zendesk Support source dbt package

dbt_zendesk_source's Issues

[Feature] update tmp models to use dbt_utils.star

Is there an existing feature request for this?

• I have searched the existing issues

Describe the Feature

Similar to issue fivetran/dbt_hubspot_source#116, we will also want to update this package's tmp models to use select {{ dbt_utils.star(...) }} instead of select *.

Describe alternatives you've considered

No response

Are you interested in contributing this feature?

• Yes.
• Yes, but I will need assistance and will schedule time during your office hours for guidance.
• No.

Anything else?

No response

[Feature] Update collect_freshness config in src_zendesk.yml to account for dbt macro logic

Is there an existing feature request for this?

• I have searched the existing issues

Describe the Feature

Our fivetran_utils collect_freshness macro was designed to override dbt's equivalent macro to allow for the enabling/disabling of variables when running dbt source snapshot-freshness. Thus if a source table does not exist, dbt will not run (and error on) a freshness test on the table.

dbt has since updated its macro in recent versions to basically provide the same functionality. So warnings are being thrown to upgrade the package logic.

We will want to modify all versions of the meta: is_enabled config in our src_zendesk.yml package (example here) to match the config: enabled dbt logic. Once those dependencies are either updated or removed, we can further explore updates to our own internal collect_freshness macro.

Describe alternatives you've considered

At the moment, we will be proceeding forward with updating dbt_fivetran_utils to match the logic of dbt's collect_freshness macro. That way customers will no longer experience that warning.

More details can be found in this ticket.

Are you interested in contributing this feature?

• Yes.
• Yes, but I will need assistance and will schedule time during your office hours for guidance.
• No.

Anything else?

Customers raising this issue can be found in this dbt slack thread https://getdbt.slack.com/archives/C03SAHKKG2Z/p1683127164993329

Pass-through columns for custom fields

Is there an existing feature request for this?

• I have searched the existing issues

Describe the Feature

I would like to be able to have custom fields pass through to the tickets table the same way the Asana dbt package does. I want the downstream tables to be able to leverage custom fields that we use routinely in our zendesk workflow.

Frankly, I would love it if this feature were enabled across all of the packages I use (zendesk, jira, etc).

Describe alternatives you've considered

I'm new to dbt so the only alternative I can imagine would be to do ANOTHER join after the enriched table is already built BACK to the original raw table where the custom fields exist. That seems antithetical to the dbt paradigm.

Are you interested in contributing this feature?

• Yes.
• Yes, but I will need assistance and will schedule time during your office hours for guidance.
• No.

Anything else?

No response

Update ERD link

Currently links to a deprecated version

BUG - order by causing resources exceeded error

Are you a current Fivetran customer?

Mark MacArdle, Bought by Many

Describe the bug

An order by in the stg_zendesk__ticket_field_history model is causing a resources exceeded error in bigquery

Steps to reproduce

1. Go to '...'
2. Click on '....'
3. Scroll down to '....'
4. See error

Expected behavior

Project variables configuration

copy dbt_project.yml here

Package Version

packages:
  - package: fishtown-analytics/dbt_utils
    version: 0.6.4  # dbt recommend pinning all packages, `*` doesn't work.

  # will also import fivetran/zendesk_source as a dependency
  - package: fivetran/zendesk
    version: 0.4.1

Warehouse

• BigQuery
• Redshift
• Snowflake
• Postgres
• Databricks
• Other (provide details below)

Additional context

Screenshots

Please indicate the level of urgency

Low

Are you interested in contributing to this package?

• Yes, I can do this and open a PR for your review.
• Possibly, but I'm not quite sure how to do this. I'd be happy to do a live coding session with someone to get this fixed.
• No, I'd prefer if someone else fixed this. I don't have the time and/or don't know what the root cause of the problem is.

BUG - Freshness checks fail for sources whose tables are excluded from sync

Are you a current Fivetran customer?
Jon Wu, Architect, Paper Culture

Describe the bug

When ticket form history isn't being used, freshness is still checked and subsequently causes an error. It should be ignored.

Steps to reproduce

1. In dbt_project.yml disable form history with using_ticket_form_history: False. In addition, don't select this table for sync in Fivetran so that it doesn't exist.
2. Run dbt source snapshot-freshness
3. You'll see an error like ERROR freshness of zendesk.ticket_form_history followed by info saying that the table doesn't exist. For example, something like this for BigQuery:

Runtime Error in source ticket_form_history (models/src_zendesk.yml)
  404 GET https://bigquery.googleapis.com/bigquery/v2/projects/[project]/queries/[query-id]?maxResults=0&location=US&prettyPrint=false: Not found: Table [project]:zendesk.ticket_form_history was not found in location US

Expected behavior

When you set using_ticket_form_history: False, freshness shouldn't be checked. Ideally the source isn't even defined so that it doesn't appear in the docs.

Project variables configuration

name: [project-name]
version: 1.0.0
config-version: 2

profile: default
source-paths: [models]
analysis-paths: [analysis]
test-paths: [tests]
data-paths: [data]
macro-paths: [macros]
snapshot-paths: [snapshots]
target-path: target
clean-targets:
  - target
  - dbt_modules

vars:
  zendesk_source:
    zendesk_database: [project]
  # Disable models for Zendesk features we aren't using or syncing
  using_ticket_form_history: False
  using_schedules: False

Package Version

packages:
  - package: fishtown-analytics/dbt_utils
    version: 0.6.6
  - package: fivetran/ad_reporting
    version: 0.2.2
  - package: fivetran/facebook_ads
    version: 0.2.1
  - package: fivetran/google_ads
    version: 0.2.1
  - package: fivetran/marketo
    version: 0.4.0
  - package: fivetran/microsoft_ads
    version: 0.2.1
  - package: fivetran/pinterest
    version: 0.3.2
  - package: fivetran/zendesk
    version: 0.5.1

Warehouse

• BigQuery
• Redshift
• Snowflake
• Postgres
• Databricks
• Other (provide details below)

Additional context
Similar to fivetran/dbt_hubspot#41

Screenshots

Please indicate the level of urgency

Testing for the first time, but given that this was fixed in another package, it seems like a fix is known.

Are you interested in contributing to this package?

• Yes, I can do this and open a PR for your review.
• Possibly, but I'm not quite sure how to do this. I'd be happy to do a live coding session with someone to get this fixed.
• No, I'd prefer if someone else fixed this. I don't have the time and/or don't know what the root cause of the problem is.

[Feature] Add source for `TICKET_CUSTOM_STATUS` table and accompanying staging models

Is there an existing feature request for this?

• I have searched the existing issues

Describe the Feature

Zendesk now supports custom statuses for tickets. The zendesk_source package should support this and have appropriate staging models for the same.

Describe alternatives you've considered

No response

Are you interested in contributing this feature?

• Yes.
• Yes, but I will need assistance and will schedule time during your office hours for guidance.
• No.

Anything else?

No response

disabling `using_<source_table>` doesn't remove table from sources

Hi team!

We added this package to our dbt project and disabled user_tags as we don't have this enabled on our account and this table doesn't exist in our warehouse (see configuration below)

However, it looks like in the src_zendesk.yml file, the user_tag table is still being counted when we run a dbt source snapshot-freshness command.

We were able to solve this issue by explicitly disabling user_tags in a source configuration block (see image below), but figured I'd bring this to your attention :)

[Feature] Add phone to stg_zendesk__user

Is there an existing feature request for this?

• I have searched the existing issues

Describe the Feature

Looks like the staging models are a bit out of date with what fields fivetran actually replicates from Zendesk. This is just one I've noticed that would be helpful, but there are others.

Describe alternatives you've considered

No response

Are you interested in contributing this feature?

• Yes.
• Yes, but I will need assistance and will schedule time during your office hours for guidance.
• No.

Anything else?

No response

Custom Field Definition Not Working

Is there an existing issue for this?

• I have searched the existing issues

Describe the issue

Question on this, say I have a custom field like so:

Field Name: Foo
Field ID: 12345

How should I structure it in vars?

I tried this and it failed:

##dbt_project.yml
vars:
zendesk__ticket_passthrough_columns: [Foo]
I got the message:
Unrecognized name: Foo at [415:9]
compiled Code at zendesk_source/models/stg_zendesk__ticket.sql

Relevant error log or model output

Unrecognized name: Foo at [415:9]
compiled Code at zendesk_source/models/stg_zendesk__ticket.sql

Expected behavior

The custom field to be picked up by dbt models

dbt Project configurations

Project name -- the same with Github Repo name

name: 'data_dbt'
version: '1.3.0'
config-version: 2

These configurations specify where dbt should look for different types of files.

model-paths: ["models"]
analysis-paths: ["analysis"]
test-paths: ["tests"]
seed-paths: ["seeds"]
macro-paths: ["macros"]
snapshot-paths: ["snapshots"]

target-path: "target" # directory which will store compiled SQL files
clean-targets: # directories to be removed by dbt clean

• "target"
• "dbt_packages"

vars:
quickbooks_schema: landing_quickbooks
stripe_schema: landing_stripe
stripe__using_credit_notes: True
zendesk_schema: landing_zendesk
using_domain_names: False
using_organization_tags: False
# zendesk__ticket_passthrough_columns: [Inquiry]

Package versions

packages:

• package: fivetran/quickbooks
  version: 0.11.0
• package: fivetran/stripe
  version: 0.10.1
• package: dbt-labs/dbt_utils
  version: 1.1.1

- package: fivetran/zendesk

version: 0.12.0

• git: https://github.com/fivetran/dbt_zendesk.git
  revision: bugfix/daylight-end-catch
  warn-unpinned: false

What database are you using dbt with?

bigquery

dbt Version

1.3.0

Additional Context

None

Are you willing to open a PR to help address this issue?

• Yes.
• Yes, but I will need assistance and will schedule time during our office hours for guidance
• No.

[Feature] Support setting database and schema with environment variables

Is there an existing feature request for this?

• I have searched the existing issues

Describe the Feature

Currently, source configurations for database and schema names can only be set through project variables, which are not very dynamic.

I would like to use the same dbt_project.yml across environments that have different source databases, but project vars are not Jinja rendered, like most other configuration sections. The only mechanism for overriding projects vars is with the --vars command line option. While this would work, I'd like to avoid it because it complicates calling dbt for developers and in our deployments. In addition, using env vars is already a pattern that's supported in setting the database value in sources, and so I'd like to be able to do the same with dbt_zendesk_source.

There's a lot of discussion about how to make vars more configurable on the dbt-core Githib, but nothing has been decided yet.

Describe alternatives you've considered

No response

Are you interested in contributing this feature?

• Yes.
• Yes, but I will need assistance and will schedule time during your office hours for guidance.
• No.

Anything else?

No response

An error occurs when the Destination is Snowflake.

I am using snowflake. I am using dbt packeage (zendesk).

I have a table here called "Group". The "Group" has to be enclosed in double-cotations. Because that's a Snowflake specification.
However, dbt automatically adds a single quote and I get an error.

This is the error.

Database Error in model stg_zendesk_group (models/stg_zendesk_group.sql)
001003 (42000): SQL compilation error:
syntax error line 5 at position 43 unexpected 'group'.
syntax error line 7 at position 0 unexpected ')'.
compiled SQL at target/run/zendesk_source/models/stg_zendesk_group.sql

Adding double quotes on the model side results in an error because when you "dbt run", dbt encloses it in a single coat.
I would like to know the specific code that can solve this problem.

update required dbt version to allow v0.19.0

Same as an issue I raised in the stripe package here.

If the requirements could be updated to allow dbt v0.19 that'd be great.

BUG - Freshness check fails for organization_tag

Are you a current Fivetran customer?
Jon Wu, Architect, Paper Culture

Describe the bug

zendesk.organization_tag fails freshness checks. It appears that it shouldn't be checked.

Steps to reproduce

1. Install this package and sync zendesk data
2. Run dbt source snapshot-freshness
3. You'll see an error like ERROR STALE freshness of zendesk.organization_tag

Expected behavior

No freshness error should occur if data sync is working. In my case, it appears that this data just doesn't get updated very often. I noticed that freshness: null is configured for zendesk.organization. Should that just be copied to organization_tag to ignore freshness here too?

Project variables configuration

name: [project-name]
version: 1.0.0
config-version: 2

profile: default
source-paths: [models]
analysis-paths: [analysis]
test-paths: [tests]
data-paths: [data]
macro-paths: [macros]
snapshot-paths: [snapshots]
target-path: target
clean-targets:
  - target
  - dbt_modules

vars:
  zendesk_source:
    zendesk_database: [project]
  # Disable models for Zendesk features we aren't using or syncing
  using_ticket_form_history: False
  using_schedules: False

Package Version

packages:
  - package: fishtown-analytics/dbt_utils
    version: 0.6.6
  - package: fivetran/ad_reporting
    version: 0.2.2
  - package: fivetran/facebook_ads
    version: 0.2.1
  - package: fivetran/google_ads
    version: 0.2.1
  - package: fivetran/marketo
    version: 0.4.0
  - package: fivetran/microsoft_ads
    version: 0.2.1
  - package: fivetran/pinterest
    version: 0.3.2
  - package: fivetran/zendesk
    version: 0.5.1

Warehouse

• BigQuery
• Redshift
• Snowflake
• Postgres
• Databricks
• Other (provide details below)

Additional context
Freshness is ignored for organization: https://github.com/fivetran/dbt_zendesk_source/blob/master/models/src_zendesk.yml#L130

Screenshots

Please indicate the level of urgency

Testing for the first time, but hopefully this is a simple fix.

Are you interested in contributing to this package?

• Yes, I can do this and open a PR for your review.
• Possibly, but I'm not quite sure how to do this. I'd be happy to do a live coding session with someone to get this fixed.
• No, I'd prefer if someone else fixed this. I don't have the time and/or don't know what the root cause of the problem is.

[Bug] Incorrect "agent replies" and "agent comments" metrics in zendesk__ticket_metrics

Is there an existing issue for this?

• I have searched the existing issues

Describe the issue

The metrics 'total_agent_replies' and 'count_agent_comments' in the zendesk__ticket_metrics model are incorrect when compared with what Zendesk uses for their analytics. It looks like the metrics in the Fivetran model ignores "agent comments" from users who were agents at the time the comment was left on a ticket, but who no longer are agents (if they left the business).

In my view these should still be counted as 'agent comments', and this is how Zendesk treats them in their own analytics tool. Otherwise, this makes historical analysis incorrect as these metrics will be skewed based on whether a historical ticket was handled by someone who is still an agent today, or by someone who has left the business and therefore no longer an agent.

Relevant error log or model output

No response

Expected behavior

In my view the metrics should count comments made by ex-agents who were agents at the time of the comment. Zendesk is able to provide this detail through their Data API endpoints we received through our previous data movement platform so I assume there must be an audit trail somewhere in the schema to track when users changed roles.

dbt Project configurations

Running the Fivetran dbt transformations as they come

Package versions

What database are you using dbt with?

snowflake

dbt Version

Running on the latest dbt version 1.7

Additional Context

No response

Are you willing to open a PR to help address this issue?

• Yes.
• Yes, but I will need assistance and will schedule time during our office hours for guidance
• No.