This is a:

• documentation update
• bug fix with no breaking changes
• new functionality
• a breaking change

All pull requests from community contributors should target the main branch (default).

Description & motivation

Fixes #578.

Checklist

• I have verified that these changes work locally on the following warehouses (Note: it's okay if you do not have access to all warehouses, this helps us understand what has been covered)
  • BigQuery
  • Postgres
  • Redshift
  • Snowflake
• I followed guidelines to ensure that my changes will work on "non-core" adapters by:
  • dispatching any new macro(s) so non-core adapters can also use them (e.g. the star() source)
  • using the limit_zero() macro in place of the literal string: limit 0
  • using dbt_utils.type_* macros instead of explicit datatypes (e.g. dbt_utils.type_timestamp() instead of TIMESTAMP
• I have updated the README.md (if applicable)
• I have added tests & descriptions to my models (and macros if applicable)
• I have added an entry to CHANGELOG.md

Is this your first time opening an issue?

• I have read the expectations for open source contributors

Describe the Feature

Hi colleagues, In multiple advanced cases like dbt hooks, operations and unit testing we would like to execute queries against the database and also run some state changing operations (e.g. agate table print to file) Currently dbt doesn't gives the opportunity to avoid such misbehavior

Suggested idea is to add executed_command to the Jinja context storing compile | run | test | docs etc from dbt command palette

So all macroses for hooks and operations can use it and flexibly configure when to run and what Same for unit testing use-cases:

1. I can skip unit tests on dbt compile and dbt docs
2. I can now mock current_timestamp function in dbt_utils to write advanced tests without changing the logic

Describe alternatives you've considered

I was trying to do workarounds using tags and relying on presence of attributes like run_result, schemas, graph nodes in the Jinja context Some of them are working, but all of them doesn't smell good

Who will this benefit?

Looks like multiple developers are facing the same issue So, users of hooks, operations and unit-testing will benefit

Are you interested in contributing this feature?

Yes, PR will come soon

Anything else?

Easily found several related cases https://github.com/dbt-labs/dbt-core/issues/4785 https://github.com/dbt-labs/dbt-core/issues/4445

Is there an existing issue for this?

• I have searched the existing issues

Current Behavior

This has nothing to do with dbt-core itself, but I'm not sure where else to report this.

It seems that for any connection configured in dbt Cloud, queries will be sent periodically. I don't know why, but I assume there must be some reason. Maybe to check if the credentials are still valid? Anyway, that's not the end of the world, as those queries only consume cloud services (no active warehouse required).

However, the version of the python connector used to send those queries is really old (2.2.1). We're doing our quarterly review of Snowflake client drivers, and this one came up as seriously out of date, and no longer supported.

I assume this is not unique to Snowflake, but we're not using any other data warehouse to check whether or not if affects them as well.

Expected Behavior

Python connector is kept up-to-date with pip (or dbt-core, or something more recent).

Steps To Reproduce

• Create a connection to Snowflake in dbt Cloud
• Create a corresponding environment, with a configured user & credentials
• dbt Cloud will start sending queries periodically (using this old version of the python connector)

Relevant log output

N/A

Environment

dbt Cloud

dbt-core version does not matter.

What database are you using dbt with?

snowflake

Additional Context

No response

Describe the bug

The exclude param in the union_relations function doesn't actually exclude the column from the final result.

Steps to reproduce

1. Create 2 sources:

create or replace table development.dbt_jyeo.my_source_a as (
  select 1 as user_id, 'alice' as user_name, 'active' as status
);

create or replace table development.dbt_jyeo.my_source_b as (
  select 2 as user_id, 'bob' as user_name
);

2. Add those sources to your project:

version: 2
sources:
  - name: dbt_jyeo
    tables:
      - name: my_source_a
      - name: my_source_b

3. Add to dbt_utils to package.yml and then do dbt deps.

4. Use union_relations in a model and specify exclude:

-- models/my_model.sql
{{ 
  dbt_utils.union_relations(
    relations=[
      source('dbt_jyeo', 'my_source_a'), 
      source('dbt_jyeo', 'my_source_b')
    ],
    exclude=['status']
  ) 
}}

6. Run or compile the model above.

7. Check logs or query the table to see that status column is not excluded as expected.

Expected results

Expected that the status column is not added to my_model table.

Actual results

status column shows up in my_model table.

Screenshots and log output

debug logs:

2022-05-09T22:29:37.055722Z: 22:29:37  Using snowflake connection "model.my_dbt_project.my_model"
2022-05-09T22:29:37.055847Z: 22:29:37  On model.my_dbt_project.my_model: /* {"app": "dbt", "dbt_version": "1.0.6", "profile_name": "user", "target_name": "default", "node_id": "model.my_dbt_project.my_model"} */


      create or replace transient table development.dbt_jyeo.my_model  as
      (

        (
            select

                cast('development.dbt_jyeo.my_source_a' as 
    varchar
) as _dbt_source_relation,
                
                    cast("USER_ID" as NUMBER(1,0)) as "USER_ID" ,
                    cast("USER_NAME" as character varying(5)) as "USER_NAME" ,
                    cast("STATUS" as character varying(6)) as "STATUS" 

            from development.dbt_jyeo.my_source_a
        )

        union all
        

        (
            select

                cast('development.dbt_jyeo.my_source_b' as 
    varchar
) as _dbt_source_relation,
                
                    cast("USER_ID" as NUMBER(1,0)) as "USER_ID" ,
                    cast("USER_NAME" as character varying(5)) as "USER_NAME" ,
                    cast(null as character varying(6)) as "STATUS" 

            from development.dbt_jyeo.my_source_b
        )

        
      );
2022-05-09T22:29:37.765446Z: 22:29:37  SQL status: SUCCESS 1 in 0.71 seconds

System information

The contents of your packages.yml file:

packages:
  - package: dbt-labs/dbt_utils
    version: 0.8.4

Which database are you using dbt with?

• postgres
• redshift
• bigquery
• snowflake
• other (specify: ____________)

The output of dbt --version:

1.0.6 in Cloud

Additional context

Haven't tried to find out why this is happening yet - just reproducing.

Are you interested in contributing the fix?

Yes