This sounds like a reasonable request. We're going to be looking at logging updates later in this quarter. I'm not sure if it would get onto the schedule or not, but we can take a look.
Pinned
Activity
gshank issue comment dbt-labs/dbt-core
[CT-631] DBT suppresses warnings in other applications
I've found that dbt
is suppressing import-time warnings here:
https://github.com/dbt-labs/dbt-core/blob/72c17c446440afe7d4b6501ed364effe8e14c9f2/core/dbt/logger.py#L356
This starts a context manager which is never closed. The unclosed context manager is preventing warnings in my application from being displayed.
FYI, dbt
is being imported in my application by sqlfluff
, which I'm using to parse SQL queries.
A quick fix could be to use:
debug_warnings = DebugWarnings()
debug_warnings.start()
instead of
DebugWarnings().__enter__()
and then adding
debug_warnings.end()
to the bottom of logger.py
.
Although, I'm not sure if this will provide the right behaviour, it does allow my application to issue warnings again.
If my solution is reasonable, I'd be happy to open a PR.
I'm using:
- dbt-core 1.0.6
- dbt-snowflake 1.0.1
- sqlfluff 0.11.2
- sqlfluff-templater-dbt 0.11.0
Sorry for the delay in responding. I don't see any indication of why we're actually doing this. We'd welcome a PR that removes the DebugWarnings thing entirely. If some warning is encountered we can deal with it in a more appropriate way.
gshank push dbt-labs/dbt-bigquery
commit sha: 9d49e7aa57bc960799dc411a523471554a7323d4
push time in 2 days agogshank issue dbt-labs/dbt-redshift
[CT-603] Add docs_generate tests from the adapter zone and remove old tests.
The docs generate tests in dbt-core have been converted and put into the adapter zone. This implements Redshift versions of those tests.
gshank push dbt-labs/dbt-redshift
commit sha: 50399f56d912c30a55d813ec61047fb3dc9da223
push time in 2 days agogshank in dbt-labs/dbt-redshift delete branch ct-603-docs_generate_tests
gshank pull request dbt-labs/dbt-redshift
Ct-603 docs generate test conversion
resolves #115
Description
Use dbt-core adapter zone docs generate tests.
Checklist
- I have signed the CLA
- I have run this code in development and it appears to resolve the stated issue
- This PR includes tests, or tests are not required/relevant for this PR
- I have updated the
CHANGELOG.md
and added information about my change to the "dbt-redshift next" section.
gshank push dbt-labs/dbt-snowflake
commit sha: 3f26874e8af0e4c4c0c6b10ebf7cb17255c71fef
push time in 2 days agogshank push dbt-labs/dbt-snowflake
commit sha: ccffd79ac1c57c2a3869a9b5a6d2f0c43373f951
push time in 2 days agogshank push dbt-labs/dbt-redshift
commit sha: b177725872465cbbb42866b979763f5d26a470f5
push time in 2 days agogshank merge to dbt-labs/dbt-core
[TEST] 1.1 networkx test
resolves #
Description
Checklist
- I have read the contributing guide and understand what's expected of me
- I have signed the CLA
- I have run this code in development and it appears to resolve the stated issue
- This PR includes tests, or tests are not required/relevant for this PR
- I have opened an issue to add/update docs, or docs changes are not required/relevant for this PR
- I have run
changie new
to create a changelog entry
gshank merge to dbt-labs/dbt-core
[1.0.latest] Lowering networkx dependency
The new release of networkx (2.8.1) is causing issues with model selection logic so lowering the version range to not include it.
gshank pull request dbt-labs/dbt-bigquery
Implement TestDocsGenerateBigQuery test
resolves #189
Description
Implement adapter zone docs generate tests
Checklist
- I have signed the CLA
- I have run this code in development and it appears to resolve the stated issue
- This PR includes tests, or tests are not required/relevant for this PR
- I have updated the
CHANGELOG.md
and added information about my change to the "dbt-bigquery next" section.
gshank in dbt-labs/dbt-bigquery create branch ct-675-docs_generate_tests
gshank issue dbt-labs/dbt-bigquery
Implement adapter zone docs generate tests
Implement adapter zone docs generate tests
gshank pull request dbt-labs/dbt-spark
Implement TestDocsGenerateSpark
resolves #361
Description
Implement adapter zone docs generate tests
Checklist
- I have signed the CLA
- I have run this code in development and it appears to resolve the stated issue
- This PR includes tests, or tests are not required/relevant for this PR
- I have updated the
CHANGELOG.md
and added information about my change to the "dbt-spark next" section.
gshank in dbt-labs/dbt-spark create branch ct-674-docs_generate_tests
gshank issue dbt-labs/dbt-spark
Implement adapter zone docs generate tests
Implement adapter zone docs generate tests
gshank issue comment dbt-labs/dbt-core
[CT-530] [Bug] Partial parsing should handle volatile variables like `invocation_id` and `run_started_at`
Is there an existing issue for this?
- I have searched the existing issues
Current Behavior
You can trick dbt into giving dynamic freshness check behaviour with something like this:
freshness:
error_after:
count: "{{ 48 if run_started_at.strftime('%w') == '1' else 24 }}" #looser expectations on Monday
period: hour
@jtcohen6 pointed out that
Small word of warning — this won't be re-parsed by partial parsing, and so it's liable to get stale. Shouldn't be an issue in scheduled runs (which don't persist partial-parse state between runs), it might just be cause for surprise if you come back to it locally a few days from now.
https://docs.getdbt.com/reference/parsing#known-limitations
Expected Behavior
Again from jerco:
In the future, we should think about ways to always re-parse (invalidate cache) for any configs/properties/nodes/etc that depend on "volatile" variables like run_started_at or invocation_id
Steps To Reproduce
- Configure a source freshness check as described above, such that it will fail today and pass every other day of the week
- Run
dbt source freshness
, note that the test fails - Wait a day (or change the code above to be dependent on the minute of the day or something)
- Run
dbt source freshness
again. Note that the test still fails, even though it's a different day of the week. - Run
dbt source freshness --no-partial-parse
. Note that the test now passes, because it's re-parsed the project and re-evaluatedrun_started_at
.
Relevant log output
No response
Environment
- OS: macOS
- Python: 3.8.x
- dbt: 1.0-rc1
What database are you using dbt with?
snowflake
Additional Context
In order to estimate this, we need to know which variables would be supported.
gshank push dbt-labs/dbt-core
commit sha: e50678c91424142aa1a4995ee1d358dac6a6fdf8
push time in 3 days agogshank in dbt-labs/dbt-core delete branch ct-581-grants_as_configs
gshank issue dbt-labs/dbt-core
[CT-581] [Feature] Grants as Node Configs
Description
This is the first in a two ticket series. #5263 is the second ticket. They will be merged sequentially, but both are required for this feature to be exposed to users.
Today users often configure a post_hook
to grant permissions on models, seeds, and snapshots:
{{ config(post_hook = 'grant select on {{ this }} to role reporter') }}
These two tickets aim to make it easier to specify grants allowing it to be configured directly both in dbt_project.yml as well as in source files:
# dbt_project.yml
models:
export:
+grants:
select: ['reporter', 'bi']
-- SQL
{{ config(grants = {'select': ['other_user']}) }}
These grant configs will not necessarily look the same for each and every warehouse. The logic to generate the sql from these configs can be overriden by adapters.
Implementation
- a "grants" config should be added to model, seed, and snapshot nodes. The value of this config is a dictionary.
- merge behavior for grants should be to override / clobber not append.
- test that projects can define grants in 1) the dbt_project.yml, 2) models, seeds, and snapshots, and 3) both the dbt_project.yml and models, seeds, and snapshots with the correct merge behavior.
gshank pull request dbt-labs/dbt-core
Ct-581 grants as configs
resolves #5189
Description
Add "grants" attribute to NodeConfig. Fix up all the tests.
This also contains a fix for a partial parsing error that was encountered when writing the test.
Checklist
- I have read the contributing guide and understand what's expected of me
- I have signed the CLA
- I have run this code in development and it appears to resolve the stated issue
- This PR includes tests, or tests are not required/relevant for this PR
- I have opened an issue to add/update docs, or docs changes are not required/relevant for this PR
- I have run
changie new
to create a changelog entry
gshank push dbt-labs/dbt-core
commit sha: ae62f5708cc630df6f36d8ab05f349fb9cea2517
push time in 3 days agogshank pull request dbt-labs/dbt-core
Switch the Makefile to use the default installed Python version instead of py38 for local testing
resolves #5257
This is a pretty simple string substitution PR, changing up all instances of py38
to py
in the Makefile
, to support running with the user's local default python version instead of always requiring Python 3.8.
Checklist
- I have read the contributing guide and understand what's expected of me
- I have signed the CLA
- I have run this code in development and it appears to resolve the stated issue
- This PR includes tests, or tests are not required/relevant for this PR
- I have opened an issue to add/update docs, or docs changes are not required/relevant for this PR
- I have run
changie new
to create a changelog entry
gshank issue dbt-labs/dbt-core
[CT-653] [Feature] Allow running unit/integration tests locally with any supported Python version
Is this your first time opening an issue?
- I have read the expectations for open source contributors
Describe the Feature
This is more of a rough edge for a non-professional dbt-core developer: the Makefile
hard-codes py38
as the preferred python version in tox for running unit tests and integration tests; we use Python 3.9 at my day job, so when I need to do work on dbt-core I need to remember to swap out the version in the Makefile to py39
before I run dbt tests and then switch it back before I send a PR upstream.
Describe alternatives you've considered
I would love it if the Makefile was either a) configured to work with the Python version that I had installed locally as long as it was supported (i.e., if the default in the Makefile was py
and py-integration
instead of py38
and py38-integration
), or, if that presents a problem for e.g. CI or some such thing, if it was easy (like via an environment variable that was documented in the CONTRIBUTING.md
guide) for me to override the setting in the Makefile
on my local machine.
Who will this benefit?
Developers who want to create PRs for dbt-core who do not work at dbt Labs.
Are you interested in contributing this feature?
Happy to!
Anything else?
Issue originally discussed in a Slack thread in dbt-core-development: https://getdbt.slack.com/archives/C50NEBJGG/p1652466095907019
gshank merge to dbt-labs/dbt-core
Switch the Makefile to use the default installed Python version instead of py38 for local testing
resolves #5257
This is a pretty simple string substitution PR, changing up all instances of py38
to py
in the Makefile
, to support running with the user's local default python version instead of always requiring Python 3.8.
Checklist
- I have read the contributing guide and understand what's expected of me
- I have signed the CLA
- I have run this code in development and it appears to resolve the stated issue
- This PR includes tests, or tests are not required/relevant for this PR
- I have opened an issue to add/update docs, or docs changes are not required/relevant for this PR
- I have run
changie new
to create a changelog entry
Looks good!
gshank merge to dbt-labs/dbt-core
Ct-581 grants as configs
resolves #5189
Description
Add "grants" attribute to NodeConfig. Fix up all the tests.
This also contains a fix for a partial parsing error that was encountered when writing the test.
Checklist
- I have read the contributing guide and understand what's expected of me
- I have signed the CLA
- I have run this code in development and it appears to resolve the stated issue
- This PR includes tests, or tests are not required/relevant for this PR
- I have opened an issue to add/update docs, or docs changes are not required/relevant for this PR
- I have run
changie new
to create a changelog entry
gshank wants to merge dbt-labs/dbt-core
Ct-581 grants as configs
resolves #5189
Description
Add "grants" attribute to NodeConfig. Fix up all the tests.
This also contains a fix for a partial parsing error that was encountered when writing the test.
Checklist
- I have read the contributing guide and understand what's expected of me
- I have signed the CLA
- I have run this code in development and it appears to resolve the stated issue
- This PR includes tests, or tests are not required/relevant for this PR
- I have opened an issue to add/update docs, or docs changes are not required/relevant for this PR
- I have run
changie new
to create a changelog entry
I've removed the isinstance check. It ought to be unnecessary.
gshank wants to merge dbt-labs/dbt-core
Ct-581 grants as configs
resolves #5189
Description
Add "grants" attribute to NodeConfig. Fix up all the tests.
This also contains a fix for a partial parsing error that was encountered when writing the test.
Checklist
- I have read the contributing guide and understand what's expected of me
- I have signed the CLA
- I have run this code in development and it appears to resolve the stated issue
- This PR includes tests, or tests are not required/relevant for this PR
- I have opened an issue to add/update docs, or docs changes are not required/relevant for this PR
- I have run
changie new
to create a changelog entry
I don't think it would cause any problems, but you're right that it's unnecessary. I've removed the isinstance check.
[CT-662] [Feature] Allow the configuration of the timezone for timestamps in event logging
Is this your first time opening an issue?
Describe the Feature
Currently the timestamp in event logging is always output in utc. It would be helpful to be able to set the time zone via a configuration or parameter (either local or any other time zone).
Describe alternatives you've considered
I have not found any alternatives. One has to live with utc.
Who will this benefit?
This would make it easier to evaluate the logs and help to avoid confusion. Additionally the evaluation of dbt logs in combination with other logs using lokal or any other timezone would be easier.
Are you interested in contributing this feature?
No response
Anything else?
No response