gshank

gshank

Member Since 13 years ago

Ithaca, NY

Experience Points
35
follower
Lessons Completed
13
follow
Lessons Completed
21
stars
Best Reply Awards
28
repos

469 contributions in the last year

Pinned
⚡ a Perl Moose HTML form handler
⚡ HTML::FormHandler::Model::DBIC
⚡ Perl Catalyst blog
⚡ Examples of using HTML::FormHandler
⚡ Simple login controller and template bundle for Catalyst
Activity
May
20
3 days ago
Activity icon
issue

gshank issue comment dbt-labs/dbt-core

gshank
gshank

[CT-662] [Feature] Allow the configuration of the timezone for timestamps in event logging

Is this your first time opening an issue?

Describe the Feature

Currently the timestamp in event logging is always output in utc. It would be helpful to be able to set the time zone via a configuration or parameter (either local or any other time zone).

Describe alternatives you've considered

I have not found any alternatives. One has to live with utc.

Who will this benefit?

This would make it easier to evaluate the logs and help to avoid confusion. Additionally the evaluation of dbt logs in combination with other logs using lokal or any other timezone would be easier.

Are you interested in contributing this feature?

No response

Anything else?

No response

gshank
gshank

This sounds like a reasonable request. We're going to be looking at logging updates later in this quarter. I'm not sure if it would get onto the schedule or not, but we can take a look.

Activity icon
issue

gshank issue comment dbt-labs/dbt-core

gshank
gshank

[CT-631] DBT suppresses warnings in other applications

I've found that dbt is suppressing import-time warnings here: https://github.com/dbt-labs/dbt-core/blob/72c17c446440afe7d4b6501ed364effe8e14c9f2/core/dbt/logger.py#L356

This starts a context manager which is never closed. The unclosed context manager is preventing warnings in my application from being displayed.

FYI, dbt is being imported in my application by sqlfluff, which I'm using to parse SQL queries.

A quick fix could be to use:

debug_warnings = DebugWarnings()
debug_warnings.start()

instead of

DebugWarnings().__enter__()

and then adding

debug_warnings.end()

to the bottom of logger.py.

Although, I'm not sure if this will provide the right behaviour, it does allow my application to issue warnings again.

If my solution is reasonable, I'd be happy to open a PR.


I'm using:

  • dbt-core 1.0.6
  • dbt-snowflake 1.0.1
  • sqlfluff 0.11.2
  • sqlfluff-templater-dbt 0.11.0
gshank
gshank

Sorry for the delay in responding. I don't see any indication of why we're actually doing this. We'd welcome a PR that removes the DebugWarnings thing entirely. If some warning is encountered we can deal with it in a more appropriate way.

push

gshank push dbt-labs/dbt-bigquery

gshank
gshank

Remove legacy docs_generate tests

commit sha: 9d49e7aa57bc960799dc411a523471554a7323d4

push time in 2 days ago
Activity icon
issue

gshank issue dbt-labs/dbt-redshift

gshank
gshank

[CT-603] Add docs_generate tests from the adapter zone and remove old tests.

The docs generate tests in dbt-core have been converted and put into the adapter zone. This implements Redshift versions of those tests.

push

gshank push dbt-labs/dbt-redshift

gshank
gshank

Ct-603 docs generate test conversion (#116)

  • Update test_basic with docs_generate test

  • Implement DocsGenerate and DocsGenReferences tests

  • Remove existing docs_generate_tests

  • Fix docs gen references test, and skip

commit sha: 50399f56d912c30a55d813ec61047fb3dc9da223

push time in 2 days ago
Activity icon
delete

gshank in dbt-labs/dbt-redshift delete branch ct-603-docs_generate_tests

deleted time in 2 days ago
pull request

gshank pull request dbt-labs/dbt-redshift

gshank
gshank

Ct-603 docs generate test conversion

resolves #115

Description

Use dbt-core adapter zone docs generate tests.

Checklist

  • I have signed the CLA
  • I have run this code in development and it appears to resolve the stated issue
  • This PR includes tests, or tests are not required/relevant for this PR
  • I have updated the CHANGELOG.md and added information about my change to the "dbt-redshift next" section.
push

gshank push dbt-labs/dbt-snowflake

gshank
gshank

rename dev_requirements -> dev-requirements to match dbt-core (#152)

  • rename dev_requirements -> dev-requirements to match dbt-core

  • changelog entry

gshank
gshank

Merge branch 'main' into ct-605-docs_generate_tests

commit sha: ccffd79ac1c57c2a3869a9b5a6d2f0c43373f951

push time in 2 days ago
push

gshank push dbt-labs/dbt-redshift

gshank
gshank

Merge branch 'main' into ct-603-docs_generate_tests

commit sha: b177725872465cbbb42866b979763f5d26a470f5

push time in 2 days ago
pull request

gshank merge to dbt-labs/dbt-core

gshank
gshank

[TEST] 1.1 networkx test

resolves #

Description

Checklist

pull request

gshank merge to dbt-labs/dbt-core

gshank
gshank

[1.0.latest] Lowering networkx dependency

The new release of networkx (2.8.1) is causing issues with model selection logic so lowering the version range to not include it.

pull request

gshank pull request dbt-labs/dbt-bigquery

gshank
gshank

Implement TestDocsGenerateBigQuery test

resolves #189

Description

Implement adapter zone docs generate tests

Checklist

  • I have signed the CLA
  • I have run this code in development and it appears to resolve the stated issue
  • This PR includes tests, or tests are not required/relevant for this PR
  • I have updated the CHANGELOG.md and added information about my change to the "dbt-bigquery next" section.
Activity icon
created branch

gshank in dbt-labs/dbt-bigquery create branch ct-675-docs_generate_tests

createdAt 2 days ago
Activity icon
issue

gshank issue dbt-labs/dbt-bigquery

gshank
gshank

Implement adapter zone docs generate tests

Implement adapter zone docs generate tests

pull request

gshank pull request dbt-labs/dbt-spark

gshank
gshank

Implement TestDocsGenerateSpark

resolves #361

Description

Implement adapter zone docs generate tests

Checklist

  • I have signed the CLA
  • I have run this code in development and it appears to resolve the stated issue
  • This PR includes tests, or tests are not required/relevant for this PR
  • I have updated the CHANGELOG.md and added information about my change to the "dbt-spark next" section.
Activity icon
created branch

gshank in dbt-labs/dbt-spark create branch ct-674-docs_generate_tests

createdAt 2 days ago
Activity icon
issue

gshank issue dbt-labs/dbt-spark

gshank
gshank

Implement adapter zone docs generate tests

Implement adapter zone docs generate tests

May
19
4 days ago
Activity icon
issue

gshank issue comment dbt-labs/dbt-core

gshank
gshank

[CT-530] [Bug] Partial parsing should handle volatile variables like `invocation_id` and `run_started_at`

Is there an existing issue for this?

  • I have searched the existing issues

Current Behavior

You can trick dbt into giving dynamic freshness check behaviour with something like this:

        freshness:
          error_after:
            count: "{{ 48 if run_started_at.strftime('%w') == '1' else 24 }}" #looser expectations on Monday
            period: hour

Community Slack thread.

@jtcohen6 pointed out that

Small word of warning — this won't be re-parsed by partial parsing, and so it's liable to get stale. Shouldn't be an issue in scheduled runs (which don't persist partial-parse state between runs), it might just be cause for surprise if you come back to it locally a few days from now.

https://docs.getdbt.com/reference/parsing#known-limitations

Expected Behavior

Again from jerco:

In the future, we should think about ways to always re-parse (invalidate cache) for any configs/properties/nodes/etc that depend on "volatile" variables like run_started_at or invocation_id

Steps To Reproduce

  1. Configure a source freshness check as described above, such that it will fail today and pass every other day of the week
  2. Run dbt source freshness, note that the test fails
  3. Wait a day (or change the code above to be dependent on the minute of the day or something)
  4. Run dbt source freshness again. Note that the test still fails, even though it's a different day of the week.
  5. Run dbt source freshness --no-partial-parse. Note that the test now passes, because it's re-parsed the project and re-evaluated run_started_at.

Relevant log output

No response

Environment

- OS: macOS
- Python: 3.8.x
- dbt: 1.0-rc1

What database are you using dbt with?

snowflake

Additional Context

Related: https://github.com/dbt-labs/dbt-core/issues/2330

gshank
gshank

In order to estimate this, we need to know which variables would be supported.

push

gshank push dbt-labs/dbt-core

gshank
gshank

Ct-581 grants as configs (#5230)

  • Handle 'grants' in NodeConfig, with correct merge behavior

  • Fix a bunch of tests

  • Add changie

  • Actually add the test

  • Change to default replace of grants with '+' extending them

  • Additional tests, fix config_call_dict handling

  • Tweak _add_config_call to remove unnecessary isinstance checks

commit sha: e50678c91424142aa1a4995ee1d358dac6a6fdf8

push time in 3 days ago
Activity icon
delete

gshank in dbt-labs/dbt-core delete branch ct-581-grants_as_configs

deleted time in 3 days ago
Activity icon
issue

gshank issue dbt-labs/dbt-core

gshank
gshank

[CT-581] [Feature] Grants as Node Configs

Description

This is the first in a two ticket series. #5263 is the second ticket. They will be merged sequentially, but both are required for this feature to be exposed to users.

Today users often configure a post_hook to grant permissions on models, seeds, and snapshots:

{{ config(post_hook = 'grant select on {{ this }} to role reporter') }}

These two tickets aim to make it easier to specify grants allowing it to be configured directly both in dbt_project.yml as well as in source files:

# dbt_project.yml
models:
  export:
    +grants:
      select: ['reporter', 'bi']
-- SQL
{{ config(grants = {'select': ['other_user']}) }}

These grant configs will not necessarily look the same for each and every warehouse. The logic to generate the sql from these configs can be overriden by adapters.

Implementation

  • a "grants" config should be added to model, seed, and snapshot nodes. The value of this config is a dictionary.
  • merge behavior for grants should be to override / clobber not append.
  • test that projects can define grants in 1) the dbt_project.yml, 2) models, seeds, and snapshots, and 3) both the dbt_project.yml and models, seeds, and snapshots with the correct merge behavior.
pull request

gshank pull request dbt-labs/dbt-core

gshank
gshank

Ct-581 grants as configs

resolves #5189

Description

Add "grants" attribute to NodeConfig. Fix up all the tests.

This also contains a fix for a partial parsing error that was encountered when writing the test.

Checklist

push

gshank push dbt-labs/dbt-core

gshank
gshank

Switch the Makefile to use the default installed Python version instead of py38 for local testing (#5269)

  • Setting up an env var to use to override the tox python variable used for local dev

  • Switch over to py-based tox envs instead of the py38 ones to be friendly to dbt-core devs who don't work at dbt Labs

  • changie

commit sha: ae62f5708cc630df6f36d8ab05f349fb9cea2517

push time in 3 days ago
pull request

gshank pull request dbt-labs/dbt-core

gshank
gshank

Switch the Makefile to use the default installed Python version instead of py38 for local testing

resolves #5257

This is a pretty simple string substitution PR, changing up all instances of py38 to py in the Makefile, to support running with the user's local default python version instead of always requiring Python 3.8.

Checklist

Activity icon
issue

gshank issue dbt-labs/dbt-core

gshank
gshank

[CT-653] [Feature] Allow running unit/integration tests locally with any supported Python version

Is this your first time opening an issue?

Describe the Feature

This is more of a rough edge for a non-professional dbt-core developer: the Makefile hard-codes py38 as the preferred python version in tox for running unit tests and integration tests; we use Python 3.9 at my day job, so when I need to do work on dbt-core I need to remember to swap out the version in the Makefile to py39 before I run dbt tests and then switch it back before I send a PR upstream.

Describe alternatives you've considered

I would love it if the Makefile was either a) configured to work with the Python version that I had installed locally as long as it was supported (i.e., if the default in the Makefile was py and py-integration instead of py38 and py38-integration), or, if that presents a problem for e.g. CI or some such thing, if it was easy (like via an environment variable that was documented in the CONTRIBUTING.md guide) for me to override the setting in the Makefile on my local machine.

Who will this benefit?

Developers who want to create PRs for dbt-core who do not work at dbt Labs.

Are you interested in contributing this feature?

Happy to!

Anything else?

Issue originally discussed in a Slack thread in dbt-core-development: https://getdbt.slack.com/archives/C50NEBJGG/p1652466095907019

pull request

gshank merge to dbt-labs/dbt-core

gshank
gshank

Switch the Makefile to use the default installed Python version instead of py38 for local testing

resolves #5257

This is a pretty simple string substitution PR, changing up all instances of py38 to py in the Makefile, to support running with the user's local default python version instead of always requiring Python 3.8.

Checklist

gshank
gshank

Looks good!

pull request

gshank merge to dbt-labs/dbt-core

gshank
gshank

Ct-581 grants as configs

resolves #5189

Description

Add "grants" attribute to NodeConfig. Fix up all the tests.

This also contains a fix for a partial parsing error that was encountered when writing the test.

Checklist

open pull request

gshank wants to merge dbt-labs/dbt-core

gshank
gshank

Ct-581 grants as configs

resolves #5189

Description

Add "grants" attribute to NodeConfig. Fix up all the tests.

This also contains a fix for a partial parsing error that was encountered when writing the test.

Checklist

gshank
gshank

I've removed the isinstance check. It ought to be unnecessary.

open pull request

gshank wants to merge dbt-labs/dbt-core

gshank
gshank

Ct-581 grants as configs

resolves #5189

Description

Add "grants" attribute to NodeConfig. Fix up all the tests.

This also contains a fix for a partial parsing error that was encountered when writing the test.

Checklist

gshank
gshank

I don't think it would cause any problems, but you're right that it's unnecessary. I've removed the isinstance check.

Previous