jeremyyeo

jeremyyeo

Support Engineer @dbt-labs, ex-Sharesies, ex-Xero

Member Since 7 years ago

@dbt-labs , Wellington, New Zealand

Experience Points
11
follower
Lessons Completed
5
follow
Lessons Completed
45
stars
Best Reply Awards
53
repos

290 contributions in the last year

Pinned
Activity
May
16
1 week ago
Activity icon
issue

jeremyyeo issue comment dbt-labs/dbt-utils

jeremyyeo
jeremyyeo

Fix: Make `union_relations` `include/exclude` case insensitive

This is a:

  • documentation update
  • bug fix with no breaking changes
  • new functionality
  • a breaking change

All pull requests from community contributors should target the main branch (default).

Description & motivation

Fixes #578.

Checklist

  • I have verified that these changes work locally on the following warehouses (Note: it's okay if you do not have access to all warehouses, this helps us understand what has been covered)
    • BigQuery
    • Postgres
    • Redshift
    • Snowflake
  • I followed guidelines to ensure that my changes will work on "non-core" adapters by:
    • dispatching any new macro(s) so non-core adapters can also use them (e.g. the star() source)
    • using the limit_zero() macro in place of the literal string: limit 0
    • using dbt_utils.type_* macros instead of explicit datatypes (e.g. dbt_utils.type_timestamp() instead of TIMESTAMP
  • I have updated the README.md (if applicable)
  • I have added tests & descriptions to my models (and macros if applicable)
  • I have added an entry to CHANGELOG.md
jeremyyeo
jeremyyeo

@dbeatty10, added your suggested test but I tweaked it a little to actually include upper/lowercase in the exclude param... wasn't sure if the test previously would have had that or not 😁

push

jeremyyeo push dbt-labs/dbt-utils

jeremyyeo
jeremyyeo

Integration test when excluding columns from union_relations

jeremyyeo
jeremyyeo

commit sha: efee51fa2af8cb0df562cfbc1d68b03546aa7542

push time in 1 week ago
May
15
1 week ago
Activity icon
issue

jeremyyeo issue comment dbt-labs/dbt-core

jeremyyeo
jeremyyeo

[CT-648] [Feature] Distinguish what dbt command has been executed in Jinja context

Is this your first time opening an issue?

Describe the Feature

Hi colleagues, In multiple advanced cases like dbt hooks, operations and unit testing we would like to execute queries against the database and also run some state changing operations (e.g. agate table print to file) Currently dbt doesn't gives the opportunity to avoid such misbehavior

Suggested idea is to add executed_command to the Jinja context storing compile | run | test | docs etc from dbt command palette

So all macroses for hooks and operations can use it and flexibly configure when to run and what Same for unit testing use-cases:

  1. I can skip unit tests on dbt compile and dbt docs
  2. I can now mock current_timestamp function in dbt_utils to write advanced tests without changing the logic

Describe alternatives you've considered

I was trying to do workarounds using tags and relying on presence of attributes like run_result, schemas, graph nodes in the Jinja context Some of them are working, but all of them doesn't smell good

Who will this benefit?

Looks like multiple developers are facing the same issue So, users of hooks, operations and unit-testing will benefit

Are you interested in contributing this feature?

Yes, PR will come soon

Anything else?

Easily found several related cases https://github.com/dbt-labs/dbt-core/issues/4785 https://github.com/dbt-labs/dbt-core/issues/4445

jeremyyeo
jeremyyeo

Hey @SOVALINUX, have you taken a look at flags.WHICH (https://docs.getdbt.com/reference/dbt-jinja-functions/flags) by any chance?

2022-05-16 09 44 55

Kind of wondering if (flags.WHICH + execute) simply gives you what you need to do what you had wanted to with this proposed executed_command Jinja context?

May
13
1 week ago
Activity icon
issue

jeremyyeo issue comment dbt-labs/dbt-core

jeremyyeo
jeremyyeo

[CT-639] [Bug] DBT Cloud is using a very old version of the python connector to send queries to Snowflake periodically

Is there an existing issue for this?

  • I have searched the existing issues

Current Behavior

This has nothing to do with dbt-core itself, but I'm not sure where else to report this.

It seems that for any connection configured in dbt Cloud, queries will be sent periodically. I don't know why, but I assume there must be some reason. Maybe to check if the credentials are still valid? Anyway, that's not the end of the world, as those queries only consume cloud services (no active warehouse required).

However, the version of the python connector used to send those queries is really old (2.2.1). We're doing our quarterly review of Snowflake client drivers, and this one came up as seriously out of date, and no longer supported.

I assume this is not unique to Snowflake, but we're not using any other data warehouse to check whether or not if affects them as well.

image image

Expected Behavior

Python connector is kept up-to-date with pip (or dbt-core, or something more recent).

Steps To Reproduce

  • Create a connection to Snowflake in dbt Cloud
  • Create a corresponding environment, with a configured user & credentials
  • dbt Cloud will start sending queries periodically (using this old version of the python connector)

Relevant log output

N/A

Environment

dbt Cloud

dbt-core version does not matter.

What database are you using dbt with?

snowflake

Additional Context

No response

jeremyyeo
jeremyyeo

Thanks for raising @mroy-seedbox. We have raised this internally with the dbt Cloud team.

@jtcohen6 I tagged you in the internal thread.

Activity icon
created branch

jeremyyeo in jeremyyeo/my-dbt-project create branch new-feature

createdAt 1 week ago
Activity icon
delete
deleted time in 1 week ago
Activity icon
created branch
createdAt 1 week ago
Activity icon
created branch
createdAt 1 week ago
Activity icon
created repository
createdAt 1 week ago
May
12
1 week ago
pull request

jeremyyeo pull request dbt-labs/dbt-utils

jeremyyeo
jeremyyeo

[Fix] Make `union_relations` `include/exclude` case insensitive

This is a:

  • documentation update
  • bug fix with no breaking changes
  • new functionality
  • a breaking change

All pull requests from community contributors should target the main branch (default).

Description & motivation

Fixes #578.

Checklist

  • I have verified that these changes work locally on the following warehouses (Note: it's okay if you do not have access to all warehouses, this helps us understand what has been covered)
    • BigQuery
    • Postgres
    • Redshift
    • Snowflake
  • I followed guidelines to ensure that my changes will work on "non-core" adapters by:
    • dispatching any new macro(s) so non-core adapters can also use them (e.g. the star() source)
    • using the limit_zero() macro in place of the literal string: limit 0
    • using dbt_utils.type_* macros instead of explicit datatypes (e.g. dbt_utils.type_timestamp() instead of TIMESTAMP
  • I have updated the README.md (if applicable)
  • I have added tests & descriptions to my models (and macros if applicable)
  • I have added an entry to CHANGELOG.md
Activity icon
created branch

jeremyyeo in dbt-labs/dbt-utils create branch fix/578-make-union-relations-case-insensitive

createdAt 1 week ago
Activity icon
issue

jeremyyeo issue comment dbt-labs/dbt-utils

jeremyyeo
jeremyyeo

`union_relations` `exclude` param not actually excluding

Describe the bug

The exclude param in the union_relations function doesn't actually exclude the column from the final result.

Steps to reproduce

  1. Create 2 sources:
create or replace table development.dbt_jyeo.my_source_a as (
  select 1 as user_id, 'alice' as user_name, 'active' as status
);

create or replace table development.dbt_jyeo.my_source_b as (
  select 2 as user_id, 'bob' as user_name
);
  1. Add those sources to your project:
version: 2
sources:
  - name: dbt_jyeo
    tables:
      - name: my_source_a
      - name: my_source_b
  1. Add to dbt_utils to package.yml and then do dbt deps.

  2. Use union_relations in a model and specify exclude:

-- models/my_model.sql
{{ 
  dbt_utils.union_relations(
    relations=[
      source('dbt_jyeo', 'my_source_a'), 
      source('dbt_jyeo', 'my_source_b')
    ],
    exclude=['status']
  ) 
}}
  1. Run or compile the model above.

  2. Check logs or query the table to see that status column is not excluded as expected.

Expected results

Expected that the status column is not added to my_model table.

Actual results

status column shows up in my_model table.

Screenshots and log output

image

debug logs:

2022-05-09T22:29:37.055722Z: 22:29:37  Using snowflake connection "model.my_dbt_project.my_model"
2022-05-09T22:29:37.055847Z: 22:29:37  On model.my_dbt_project.my_model: /* {"app": "dbt", "dbt_version": "1.0.6", "profile_name": "user", "target_name": "default", "node_id": "model.my_dbt_project.my_model"} */


      create or replace transient table development.dbt_jyeo.my_model  as
      (

        (
            select

                cast('development.dbt_jyeo.my_source_a' as 
    varchar
) as _dbt_source_relation,
                
                    cast("USER_ID" as NUMBER(1,0)) as "USER_ID" ,
                    cast("USER_NAME" as character varying(5)) as "USER_NAME" ,
                    cast("STATUS" as character varying(6)) as "STATUS" 

            from development.dbt_jyeo.my_source_a
        )

        union all
        

        (
            select

                cast('development.dbt_jyeo.my_source_b' as 
    varchar
) as _dbt_source_relation,
                
                    cast("USER_ID" as NUMBER(1,0)) as "USER_ID" ,
                    cast("USER_NAME" as character varying(5)) as "USER_NAME" ,
                    cast(null as character varying(6)) as "STATUS" 

            from development.dbt_jyeo.my_source_b
        )

        
      );
2022-05-09T22:29:37.765446Z: 22:29:37  SQL status: SUCCESS 1 in 0.71 seconds

System information

The contents of your packages.yml file:

packages:
  - package: dbt-labs/dbt_utils
    version: 0.8.4

Which database are you using dbt with?

  • postgres
  • redshift
  • bigquery
  • snowflake
  • other (specify: ____________)

The output of dbt --version:

1.0.6 in Cloud

Additional context

Haven't tried to find out why this is happening yet - just reproducing.

Are you interested in contributing the fix?

Yes

jeremyyeo
jeremyyeo

Looks like this is a case sensitivity thing with Snowflake...

Previous