nathaniel-may

nathaniel-may

Sr Software Engineer. FP, Databases. ✨ Rust ✨ Haskell ✨ PureScript ✨

Member Since 7 years ago

@dbt-labs, Philadelphia, PA

Experience Points
10
follower
Lessons Completed
8
follow
Lessons Completed
22
stars
Best Reply Awards
26
repos

939 contributions in the last year

Pinned
⚡ Small compiled DSL for defining and applying a schema to filenames.
⚡ Client-side sudoku generator that supports the wordoku and colorku variants.
⚡ GHC-style stream fusion for Scala
⚡ Effect minimizing zipper type
Activity
May
17
6 days ago
Activity icon
issue

nathaniel-may issue comment dbt-labs/dbt-core

nathaniel-may
nathaniel-may

[CT-581] [Feature] Grants as Node Configs

Description

This is the first in a two ticket series. #5263 is the second ticket. They will be merged sequentially, but both are required for this feature to be exposed to users.

Today users often configure a post_hook to grant permissions on models, seeds, and snapshots:

{{ config(post_hook = 'grant select on {{ this }} to role reporter') }}

These two tickets aim to make it easier to specify grants allowing it to be configured directly both in dbt_project.yml as well as in source files:

# dbt_project.yml
models:
  export:
    +grants:
      select: ['reporter', 'bi']
-- SQL
{{ config(grants = {'select': ['other_user']}) }}

These grant configs will not necessarily look the same for each and every warehouse. The logic to generate the sql from these configs can be overriden by adapters.

Implementation

  • a "grants" config should be added to model, seed, and snapshot nodes. The value of this config is a dictionary.
  • merge behavior for grants should be to override / clobber not append.
  • test that projects can define grants in 1) the dbt_project.yml, 2) models, seeds, and snapshots, and 3) both the dbt_project.yml and models, seeds, and snapshots with the correct merge behavior.
nathaniel-may
nathaniel-may

Moved the second half of this description to #5263

Activity icon
issue

nathaniel-may issue dbt-labs/dbt-core

nathaniel-may
nathaniel-may

[Feature] <title>

Is this your first time opening an issue?

Describe the Feature

Description

This is the second in a two ticket series. #5189 is the first ticket. They will be merged sequentially, but both are required for this feature to be exposed to users.

Today users often configure a post_hook to grant permissions on models, seeds, and snapshots:

{{ config(post_hook = 'grant select on {{ this }} to role reporter') }}

These two tickets aim to make it easier to specify grants allowing it to be configured directly both in dbt_project.yml as well as in source files:

# dbt_project.yml
models:
  export:
    +grants:
      select: ['reporter', 'bi']
-- SQL
{{ config(grants = {'select': ['other_user']}) }}

These grant configs will not necessarily look the same for each and every warehouse. The logic to generate the sql from these configs can be overriden by adapters.

Implementation

  • create a new macro in the global project: get_grant_sql(grant_config: dict) -> str that creates the warehouse-specific sql from grant portion of the config. Since it will be common, the default implementation should return grant <privilege> on <object> to <recipient>. Warehouses that deviate from this can override. Use dispatch pattern (dbt docs) with a default__ to enable this. The macro signature here is designed to accept different shapes of grant configs for each adapter to use whatever best fits the warehouse's permissions system.
  • create a new macro in the global project: apply_grants which calls get_grant_sql to apply the grants (see persist_docs for an example of similar implementation). This should be overridable by adapters so use the dispatch pattern (dbt docs) with a default__ to enable this.
  • add a call to apply_grants in all materializations
  • test that postgres projects can define grants in 1) the dbt_project.yml, 2) models, seeds, and snapshots, and 3) both the dbt_project.yml and models, seeds, and snapshots and those permissions can be read back directly from the postgres warehouse after a run.

In order for this to be usable, the adapters must each override get_grant_sql. Because that does not prevent a merge into main for core, adapter work will be tracked in separate tickets.

Describe alternatives you've considered

No response

Who will this benefit?

No response

Are you interested in contributing this feature?

No response

Anything else?

No response

May
13
1 week ago
Activity icon
issue

nathaniel-may issue dbt-labs/dbt-core

nathaniel-may
nathaniel-may

[Draft] Streamline Incremental Strategies

Is this your first time opening an issue?

Describe the Feature

When defining an incremental model, there are several incremental strategies that could be used to execute the incremental model, (namely: "append," "delete+insert," "merge," and "insert_overwrite.") but not all of them are supported by every warehouse. Working with these incremental strategies as an adapter maintainer and as an advanced user is difficult. This ticket aims to improve the experience for both of these groups of people.

Functional Requirements

  1. Adapter maintainers need to be able to specify which incremental strategies are supported by the warehouse and be able to supply the correct sql statement to execute if the default is not sufficient.
  2. Advanced users that wish to specify a custom incremental strategy must be able to do so.

Behavior Today

  1. To specify which incremental strategies are supported by the warehouse, the adapter maintainer must write a macro that returns a list of strings that name the supported strategies, and include a boiler plate jinja macro that takes parameters such as the name of the incremental strategy along with more data such as "unique_key" and "partition" and returns the materialization sql. This macro is often copy-pasted from core, and is error prone when editing.
  2. Advanced dbt users who wish to specify a custom incremental strategy must override the same boiler plate jinja macro by copy pasting it into their dbt project.
  3. When an unsupported incremental strategy is specified, the user is notified via an exception at during the first dbt run.

Desired Behavior

  1. To specify which incremental strategies are supported by the warehouse, the adapter maintainer must write a macro that returns a list of strings that name the supported strategies. If the warehouse only supports a subset of the default incremental strategies and the default implementation in the global project produces sql that works on the target warehouse, no additional work needs to be done. If any of the default behavior needs to be overridden, it can be done by writing a macro that conforms to the naming convention "get_incremental_NAME_sql" where "NAME" is the string used to configure the incremental strategy (e.g. "get incremental_append_sql"). If the warehouse has an additional incremental strategy, defining a new macro using the same "get_incremental_NAME_sql" convention will make it available to users of the adapter.
  2. Advanced dbt users who wish to specify a custom incremental strategy will only need to create a macro that conforms to the naming convention "get_incremental_NAME_sql" that produces the correct sql for the target warehouse.
  3. When an unsupported incremental strategy is designated, raise an error to users before runtime.
  4. Existing projects will experience no breaking changes when they upgrade to the version of dbt that includes this new behavior.

Implementation

  • [] For backwards compatibility, create new macros in core/dbt/include/global_project/macros/materializations/models/incremental that follow the naming convention for the existing incremental strategies and pass through to the existing current macros. The new macros will take a dictionary, and pass the key-value pairs explicitly to the wrapped macro call. Concretely:
    • [] create a new macro get_incremental_append_sql that wraps get_insert_into_sql
    • [] create a new macro get_incremental_delete+insert_sql that wraps get_delete_insert_merge_sql
    • [] create a new macro get_incremental_merge_sql that wraps get_merge_sql
    • [] create a new macro get_incremental_insert_overwrite_sql that wraps get_insert_overwrite_sql
  • [] Write a new function in the config parser that looks for macros of the naming convention "get_incremental_NAME_sql"
  • [] TODO MORE THINGS

Describe alternatives you've considered

No response

Who will this benefit?

No response

Are you interested in contributing this feature?

No response

Anything else?

No response

May
10
1 week ago
pull request

nathaniel-may merge to dbt-labs/dbt-core

nathaniel-may
nathaniel-may

Creating ADR for versioning and branching strategy

Description

Creating an initial ADR to establish how we approach versioning and branching. I wanted to write this as a base to build the release process off of and make sure we have a shared understanding of how we are doing this and why.

Checklist

  • I have signed the CLA
  • I have run this code in development and it appears to resolve the stated issue
  • This PR includes tests, or tests are not required/relevant for this PR
  • I have added information about my change to be included in the CHANGELOG.
open pull request

nathaniel-may wants to merge dbt-labs/dbt-core

nathaniel-may
nathaniel-may

Creating ADR for versioning and branching strategy

Description

Creating an initial ADR to establish how we approach versioning and branching. I wanted to write this as a base to build the release process off of and make sure we have a shared understanding of how we are doing this and why.

Checklist

  • I have signed the CLA
  • I have run this code in development and it appears to resolve the stated issue
  • This PR includes tests, or tests are not required/relevant for this PR
  • I have added information about my change to be included in the CHANGELOG.
nathaniel-may
nathaniel-may

the specifics make it much clearer. thanks!

pull request

nathaniel-may merge to dbt-labs/dbt-core

nathaniel-may
nathaniel-may

Creating ADR for versioning and branching strategy

Description

Creating an initial ADR to establish how we approach versioning and branching. I wanted to write this as a base to build the release process off of and make sure we have a shared understanding of how we are doing this and why.

Checklist

  • I have signed the CLA
  • I have run this code in development and it appears to resolve the stated issue
  • This PR includes tests, or tests are not required/relevant for this PR
  • I have added information about my change to be included in the CHANGELOG.
Activity icon
issue

nathaniel-may issue comment dbt-labs/dbt-core

nathaniel-may
nathaniel-may

[CT-597] [Bug] Access denied when setting log-path: None

Is there an existing issue for this?

  • I have searched the existing issues

Current Behavior

In the dbt_project.yml I set the log-path: None. When I run it in a FS in which it has permissions to write it'll run just fine: without creating the logs/dbt.log file. But when I go into a FS in which the user has read-only access it fails due to not having permissions to write in said None dir.

Expected Behavior

When the log-path value is None skip the permissions check

Steps To Reproduce

Run dbt with RO permissions setting the log-path to None.

Relevant log output

[Errno 30] Read-only file system: 'None'

Environment

- OS: Alpine 3.15
- Python: 3.8.12
- dbt: 1.0.6

What database are you using dbt with?

redshift

Additional Context

Setting the log-path value to None was the only way I found to avoid the creation of the dbt.log file. Perhaps this should be a feature request for that: dbt run without creating a dbt.log.

nathaniel-may
nathaniel-may

@ignacioreyna I don't believe there is a way to disable logging to a file today, so I'll mark this as a feature request. I attempted to use log-path: "/dev/null" as workaround, but that doesn't work either: "[Errno 20] Not a directory: '/dev/null/dbt.log'"

Can you tell me a little bit more about your use case? In particular, what advantages would you have if you could disable file logging? dbt also writes files to the target directory, so disabling file logging wouldn't stop dbt from writing to files all together.

May
9
2 weeks ago
Activity icon
issue

nathaniel-may issue dbt-labs/dbt-core

nathaniel-may
nathaniel-may

[CT-121] Turn Mypy on Everywhere

Is there an existing feature request for this?

  • I have searched the existing issues

Describe the Feature

Right now, Mypy is not on everywhere and it is not a particularly high level of scrutiny. This has let some bugs slip through to users that could have been caught in devlopment. I would like to turn mypy on everywhere with --strict and maybe a few other settings. The places where it would be non-trivial to switch over we should manually disable mypy for, but this way all new files will be held to this new higher standard of Python.

Describe alternatives you've considered

Not doing this. Strict Mypy is a subset of Python, which isn't as smooth of a development experience especially for experienced Python devs who are not used to Mypy.

Who will this benefit?

Developers and users because some classes of bugs will not be able to make it into releases.

Are you interested in contributing this feature?

No response

Anything else?

related: #3203

Activity icon
issue

nathaniel-may issue comment dbt-labs/dbt-core

nathaniel-may
nathaniel-may

[CT-121] Turn Mypy on Everywhere

Is there an existing feature request for this?

  • I have searched the existing issues

Describe the Feature

Right now, Mypy is not on everywhere and it is not a particularly high level of scrutiny. This has let some bugs slip through to users that could have been caught in devlopment. I would like to turn mypy on everywhere with --strict and maybe a few other settings. The places where it would be non-trivial to switch over we should manually disable mypy for, but this way all new files will be held to this new higher standard of Python.

Describe alternatives you've considered

Not doing this. Strict Mypy is a subset of Python, which isn't as smooth of a development experience especially for experienced Python devs who are not used to Mypy.

Who will this benefit?

Developers and users because some classes of bugs will not be able to make it into releases.

Are you interested in contributing this feature?

No response

Anything else?

related: #3203

nathaniel-may
nathaniel-may
Activity icon
issue

nathaniel-may issue dbt-labs/dbt-core

nathaniel-may
nathaniel-may

Turn on mypy strict flag

Is this your first time opening an issue?

Describe the Feature

#5171 upgraded mypy to the latest version and changed it to run it on the entire codebase. This ticket is to turn on the --strict flag which dramatically increases the enforced level of code quality.

This will take a large effort to convert the existing code to comply with the strict flag, and should be done iteratively where possible. One strategy would be to run mypy in CI twice: once on as we currently do, and once with the strict flag on a small subset of the codebase that can be expanded with subsequent PRs. Following Imports may make this strategy limiting.

Describe alternatives you've considered

No response

Who will this benefit?

No response

Are you interested in contributing this feature?

No response

Anything else?

No response

Activity icon
issue

nathaniel-may issue comment dbt-labs/dbt-core

nathaniel-may
nathaniel-may

upgrade mypy to latest

Description

This PR Is stuck. I don't know what the path forward is.

From our version to this version we gain:

  • M1 wheels for developers
  • TypeGuard from 3.10 (and typing extensions) which can be used to replace some uses of isinstance for type narrowing
  • type narrowing with type() not just isinstance
  • significant performance improvements
  • adds the --exclude flag for specifying modules
  • official 3.9 support
  • many bug fixes

The two big issues with getting this to work:

  1. mypy now traverses projects differently, and needs changes to find all our source code.
  2. the mechanism our unit tests use to import core code is unknown and breaks when we fix issue number one.

Fix for 1: add empty __init__.py files to the following directories:

  • core/dbt/
  • core/dbt/adapters/
  • core/dbt/context/
  • core/dbt/deps/

Fix for 2: remove the __init__.py files from the following directories:

  • core/dbt/
  • core/dbt/adapters/

This means we have one of two things breaking at any given time. Either mypy fails because we it detects an error with the way we've named our modules because it's not traversing them all, or tests fail to import the postgres adapter from dbt.adapters.postgres. With 1 fixed, and 2 failing, dbt run still works after install, so it's likely tests are failing because of test setup when the only change that has been made is adding init files to the context and deps packages.

The reason the import mechanism is unknown is because the postgres adapter lives in the plugin directory. When you run pip install dbt-postgres, it installs dbt-core, then finds the postgres adapter in the plugins directory and copies those modules to the path dbt/core/adapters/postgres so they can be imported like from dbt.adapters.postgres import PostgresAdapter. When you manually inspect the virtual environment after doing a fresh pip install -r dev-requirements.txt -r editable-requirements.txt, which is what both the makefile and tox are doing, the adapter modules have not been moved to that location yet the tests still run and import the adapter just fine.

If we cannot find a path forward to make both mypy and unit tests pass, we may have to abandon this upgrade.

Checklist

  • I have signed the CLA
  • I have run this code in development and it appears to resolve the stated issue
  • This PR includes tests, or tests are not required/relevant for this PR
  • I have updated the CHANGELOG.md and added information about my change
nathaniel-may
nathaniel-may
pull request

nathaniel-may pull request dbt-labs/dbt-core

nathaniel-may
nathaniel-may

upgrade mypy to latest

Description

This PR Is stuck. I don't know what the path forward is.

From our version to this version we gain:

  • M1 wheels for developers
  • TypeGuard from 3.10 (and typing extensions) which can be used to replace some uses of isinstance for type narrowing
  • type narrowing with type() not just isinstance
  • significant performance improvements
  • adds the --exclude flag for specifying modules
  • official 3.9 support
  • many bug fixes

The two big issues with getting this to work:

  1. mypy now traverses projects differently, and needs changes to find all our source code.
  2. the mechanism our unit tests use to import core code is unknown and breaks when we fix issue number one.

Fix for 1: add empty __init__.py files to the following directories:

  • core/dbt/
  • core/dbt/adapters/
  • core/dbt/context/
  • core/dbt/deps/

Fix for 2: remove the __init__.py files from the following directories:

  • core/dbt/
  • core/dbt/adapters/

This means we have one of two things breaking at any given time. Either mypy fails because we it detects an error with the way we've named our modules because it's not traversing them all, or tests fail to import the postgres adapter from dbt.adapters.postgres. With 1 fixed, and 2 failing, dbt run still works after install, so it's likely tests are failing because of test setup when the only change that has been made is adding init files to the context and deps packages.

The reason the import mechanism is unknown is because the postgres adapter lives in the plugin directory. When you run pip install dbt-postgres, it installs dbt-core, then finds the postgres adapter in the plugins directory and copies those modules to the path dbt/core/adapters/postgres so they can be imported like from dbt.adapters.postgres import PostgresAdapter. When you manually inspect the virtual environment after doing a fresh pip install -r dev-requirements.txt -r editable-requirements.txt, which is what both the makefile and tox are doing, the adapter modules have not been moved to that location yet the tests still run and import the adapter just fine.

If we cannot find a path forward to make both mypy and unit tests pass, we may have to abandon this upgrade.

Checklist

  • I have signed the CLA
  • I have run this code in development and it appears to resolve the stated issue
  • This PR includes tests, or tests are not required/relevant for this PR
  • I have updated the CHANGELOG.md and added information about my change
pull request

nathaniel-may merge to dbt-labs/dbt-core

nathaniel-may
nathaniel-may

Creating ADR for versioning and branching strategy

Description

Creating an initial ADR to establish how we approach versioning and branching. I wanted to write this as a base to build the release process off of and make sure we have a shared understanding of how we are doing this and why.

Checklist

  • I have signed the CLA
  • I have run this code in development and it appears to resolve the stated issue
  • This PR includes tests, or tests are not required/relevant for this PR
  • I have added information about my change to be included in the CHANGELOG.
nathaniel-may
nathaniel-may

Left a few minor comments. Also all the number bullets are "1." instead of increasing in value.

open pull request

nathaniel-may wants to merge dbt-labs/dbt-core

nathaniel-may
nathaniel-may

Creating ADR for versioning and branching strategy

Description

Creating an initial ADR to establish how we approach versioning and branching. I wanted to write this as a base to build the release process off of and make sure we have a shared understanding of how we are doing this and why.

Checklist

  • I have signed the CLA
  • I have run this code in development and it appears to resolve the stated issue
  • This PR includes tests, or tests are not required/relevant for this PR
  • I have added information about my change to be included in the CHANGELOG.
nathaniel-may
nathaniel-may

This is a good point. "a new version of dbt is considered released when ....." would be a good definition to have in here.

open pull request

nathaniel-may wants to merge dbt-labs/dbt-core

nathaniel-may
nathaniel-may

Creating ADR for versioning and branching strategy

Description

Creating an initial ADR to establish how we approach versioning and branching. I wanted to write this as a base to build the release process off of and make sure we have a shared understanding of how we are doing this and why.

Checklist

  • I have signed the CLA
  • I have run this code in development and it appears to resolve the stated issue
  • This PR includes tests, or tests are not required/relevant for this PR
  • I have added information about my change to be included in the CHANGELOG.
nathaniel-may
nathaniel-may

I had to type in the version we're releasing to one of the workflows so there is room for human error if we type the wrong version.

open pull request

nathaniel-may wants to merge dbt-labs/dbt-core

nathaniel-may
nathaniel-may

Creating ADR for versioning and branching strategy

Description

Creating an initial ADR to establish how we approach versioning and branching. I wanted to write this as a base to build the release process off of and make sure we have a shared understanding of how we are doing this and why.

Checklist

  • I have signed the CLA
  • I have run this code in development and it appears to resolve the stated issue
  • This PR includes tests, or tests are not required/relevant for this PR
  • I have added information about my change to be included in the CHANGELOG.
nathaniel-may
nathaniel-may
open pull request

nathaniel-may wants to merge dbt-labs/dbt-core

nathaniel-may
nathaniel-may

Creating ADR for versioning and branching strategy

Description

Creating an initial ADR to establish how we approach versioning and branching. I wanted to write this as a base to build the release process off of and make sure we have a shared understanding of how we are doing this and why.

Checklist

  • I have signed the CLA
  • I have run this code in development and it appears to resolve the stated issue
  • This PR includes tests, or tests are not required/relevant for this PR
  • I have added information about my change to be included in the CHANGELOG.
nathaniel-may
nathaniel-may
open pull request

nathaniel-may wants to merge dbt-labs/dbt-core

nathaniel-may
nathaniel-may

Creating ADR for versioning and branching strategy

Description

Creating an initial ADR to establish how we approach versioning and branching. I wanted to write this as a base to build the release process off of and make sure we have a shared understanding of how we are doing this and why.

Checklist

  • I have signed the CLA
  • I have run this code in development and it appears to resolve the stated issue
  • This PR includes tests, or tests are not required/relevant for this PR
  • I have added information about my change to be included in the CHANGELOG.
open pull request

nathaniel-may wants to merge dbt-labs/dbt-core

nathaniel-may
nathaniel-may

Creating ADR for versioning and branching strategy

Description

Creating an initial ADR to establish how we approach versioning and branching. I wanted to write this as a base to build the release process off of and make sure we have a shared understanding of how we are doing this and why.

Checklist

  • I have signed the CLA
  • I have run this code in development and it appears to resolve the stated issue
  • This PR includes tests, or tests are not required/relevant for this PR
  • I have added information about my change to be included in the CHANGELOG.
pull request

nathaniel-may merge to dbt-labs/dbt-core

nathaniel-may
nathaniel-may

Creating ADR for versioning and branching strategy

Description

Creating an initial ADR to establish how we approach versioning and branching. I wanted to write this as a base to build the release process off of and make sure we have a shared understanding of how we are doing this and why.

Checklist

  • I have signed the CLA
  • I have run this code in development and it appears to resolve the stated issue
  • This PR includes tests, or tests are not required/relevant for this PR
  • I have added information about my change to be included in the CHANGELOG.
nathaniel-may
nathaniel-may

Left a few minor comments. Also all the number bullets are "1." instead of increasing in value.

open pull request

nathaniel-may wants to merge dbt-labs/dbt-core

nathaniel-may
nathaniel-may

Creating ADR for versioning and branching strategy

Description

Creating an initial ADR to establish how we approach versioning and branching. I wanted to write this as a base to build the release process off of and make sure we have a shared understanding of how we are doing this and why.

Checklist

  • I have signed the CLA
  • I have run this code in development and it appears to resolve the stated issue
  • This PR includes tests, or tests are not required/relevant for this PR
  • I have added information about my change to be included in the CHANGELOG.
nathaniel-may
nathaniel-may

we should probably use the term "release candidate" here even though it's implied.

open pull request

nathaniel-may wants to merge dbt-labs/dbt-core

nathaniel-may
nathaniel-may

Creating ADR for versioning and branching strategy

Description

Creating an initial ADR to establish how we approach versioning and branching. I wanted to write this as a base to build the release process off of and make sure we have a shared understanding of how we are doing this and why.

Checklist

  • I have signed the CLA
  • I have run this code in development and it appears to resolve the stated issue
  • This PR includes tests, or tests are not required/relevant for this PR
  • I have added information about my change to be included in the CHANGELOG.
nathaniel-may
nathaniel-may

I think this should be more like "alternatives we considered" since we're continuing with our current strategy after talking about it.

open pull request

nathaniel-may wants to merge dbt-labs/dbt-core

nathaniel-may
nathaniel-may

Creating ADR for versioning and branching strategy

Description

Creating an initial ADR to establish how we approach versioning and branching. I wanted to write this as a base to build the release process off of and make sure we have a shared understanding of how we are doing this and why.

Checklist

  • I have signed the CLA
  • I have run this code in development and it appears to resolve the stated issue
  • This PR includes tests, or tests are not required/relevant for this PR
  • I have added information about my change to be included in the CHANGELOG.
nathaniel-may
nathaniel-may

and actually stops us from releasing the wrong version.

I think I know what you're getting at, but the wording is confusing to me here. Maybe it should be a separate sentence.

Apr
29
3 weeks ago
Activity icon
issue

nathaniel-may issue comment dbt-labs/dbt-core

nathaniel-may
nathaniel-may

[CT-581] [Feature] Grants as Node Configs

Description

Today users often configure a post_hook to grant permissions on models, seeds, and snapshots:

{{ config(post_hook = 'grant select on {{ this }} to role reporter') }}

This ticket aims to make it easier to specify grants allowing it to be configured directly both in dbt_project.yml as well as in source files:

# dbt_project.yml
models:
  export:
    +grants:
      select: ['reporter', 'bi']
-- SQL
{{ config(grants = {'select': ['other_user']}) }}

These grant configs will not necessarily look the same for each and every warehouse. The logic to generate the sql from these configs can be overriden by adapters.

Implementation

The implementation can be split up into two testable parts. Although it is possible to merge them separately (sequentially), both are required for this feature to be exposed to users.

Adding the Configuration

  • a "grants" config should be added to model, seed, and snapshot nodes. The value of this config is a dictionary.
  • merge behavior for grants should be to override / clobber not append.
  • test that projects can define grants in 1) the dbt_project.yml, 2) models, seeds, and snapshots, and 3) both the dbt_project.yml and models, seeds, and snapshots with the correct merge behavior.

Issuing the grants

  • create a new macro: get_grant_sql(grant_config: dict) -> str that creates the warehouse-specific sql from grant portion of the config. Since it will be common, the default implementation should return grant <privilege> on <object> to <recipient>. Warehouses that deviate from this can override. The macro signature here is designed to accept different shapes of grant configs for each adapter to use whatever best fits the warehouse's permissions system.
  • create a new macro: apply_grants which calls get_grant_sql to apply the grants (see persist_docs for an example of similar implementation)
  • add a call to apply_grants in all materializations
  • test that postgres projects can define grants in 1) the dbt_project.yml, 2) models, seeds, and snapshots, and 3) both the dbt_project.yml and models, seeds, and snapshots and those permissions can be read back directly from the postgres warehouse after a run.

In order for this to be usable, the adapters must each override get_grant_sql. Because that does not prevent a merge into main for core, adapter work will be tracked in separate tickets.

nathaniel-may
nathaniel-may

✅ default implementation - edited.

  1. Since we don't control warehouse naming conventions I figured those would be completely overridable. For an extreme example, dbt_project.yml for FunkyDB could read something like:
grants:
  dance_moves: [1, 15, 79]
  dancers: "engineering_all"
  max_allowed_intensity: 9000

and the adapter override would have the freedom to map that directly to the correct SQL grant statement for FunkyDB.

  1. I don't think I totally understand. Since our merge strategy is to clobber here, I feel like it shouldn't matter if it applies to multiple recipients. I'm fine with requiring duplication if we need to, but I'd like to understand better.
pull request

nathaniel-may pull request dbt-labs/dbt-core

nathaniel-may
nathaniel-may

Bump jinja2 from 2.11.3 to 3.1.2 in /core

Bumps jinja2 from 2.11.3 to 3.1.2.

Release notes

Sourced from jinja2's releases.

3.1.2

This is a fix release for the 3.1.0 feature release.

3.1.1

3.1.0

This is a feature release, which includes new features and removes previously deprecated features. The 3.1.x branch is now the supported bugfix branch, the 3.0.x branch has become a tag marking the end of support for that branch. We encourage everyone to upgrade, and to use a tool such as pip-tools to pin all dependencies and control upgrades. We also encourage upgrading to MarkupSafe 2.1.1, the latest version at this time.

3.0.3

3.0.2

3.0.1

3.0.0

New major versions of all the core Pallets libraries, including Jinja 3.0, have been released! :tada:

This represents a significant amount of work, and there are quite a few changes. Be sure to carefully read the changelog, and use tools such as pip-compile and Dependabot to pin your dependencies and control your updates.

3.0.0rc2

Fixes an issue with the deprecated Markup subclass, #1401.

3.0.0rc1

Changelog

Sourced from jinja2's changelog.

Version 3.1.2

Released 2022-04-28

  • Add parameters to Environment.overlay to match __init__. :issue:1645
  • Handle race condition in FileSystemBytecodeCache. :issue:1654

Version 3.1.1

Released 2022-03-25

  • The template filename on Windows uses the primary path separator. :issue:1637

Version 3.1.0

Released 2022-03-24

  • Drop support for Python 3.6. :pr:1534

  • Remove previously deprecated code. :pr:1544

    • WithExtension and AutoEscapeExtension are built-in now.
    • contextfilter and contextfunction are replaced by pass_context. evalcontextfilter and evalcontextfunction are replaced by pass_eval_context. environmentfilter and environmentfunction are replaced by pass_environment.
    • Markup and escape should be imported from MarkupSafe.
    • Compiled templates from very old Jinja versions may need to be recompiled.
    • Legacy resolve mode for Context subclasses is no longer supported. Override resolve_or_missing instead of resolve.
    • unicode_urlencode is renamed to url_quote.
  • Add support for native types in macros. :issue:1510

  • The {% trans %} tag can use pgettext and npgettext by passing a context string as the first token in the tag, like {% trans "title" %}. :issue:1430

  • Update valid identifier characters from Python 3.6 to 3.7. :pr:1571

  • Filters and tests decorated with @async_variant are pickleable. :pr:1612

  • Add items filter. :issue:1561

... (truncated)

Commits
  • b08cd4b Merge pull request #1660 from pallets/release-3.1.2
  • 1e68ba8 release version 3.1.2
  • 8efee35 pre-commit updates latest release branch
  • a24df26 ignore new mypy finding
  • 9faee28 update requirements
  • b802b5a Merge pull request #1655 from dvitek/dvitek/issue1654
  • 746bb95 Fix race conditions in FileSystemBytecodeCache
  • 466a200 update requirements
  • 990602f Merge pull request #1647 from Tom-Brouwer/202204/add-missing-overlay-options
  • 5d3d241 fix flake8-bugbear finding
  • Additional commits viewable in compare view

Dependabot compatibility score

You can trigger a rebase of this PR by commenting @dependabot rebase.


Dependabot commands and options

You can trigger Dependabot actions by commenting on this PR:

  • @dependabot rebase will rebase this PR
  • @dependabot recreate will recreate this PR, overwriting any edits that have been made to it
  • @dependabot merge will merge this PR after your CI passes on it
  • @dependabot squash and merge will squash and merge this PR after your CI passes on it
  • @dependabot cancel merge will cancel a previously requested merge and block automerging
  • @dependabot reopen will reopen this PR if it is closed
  • @dependabot close will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually
  • @dependabot ignore this major version will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself)
  • @dependabot ignore this minor version will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself)
  • @dependabot ignore this dependency will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)
Activity icon
issue

nathaniel-may issue comment dbt-labs/dbt-core

nathaniel-may
nathaniel-may

Bump jinja2 from 2.11.3 to 3.1.2 in /core

Bumps jinja2 from 2.11.3 to 3.1.2.

Release notes

Sourced from jinja2's releases.

3.1.2

This is a fix release for the 3.1.0 feature release.

3.1.1

3.1.0

This is a feature release, which includes new features and removes previously deprecated features. The 3.1.x branch is now the supported bugfix branch, the 3.0.x branch has become a tag marking the end of support for that branch. We encourage everyone to upgrade, and to use a tool such as pip-tools to pin all dependencies and control upgrades. We also encourage upgrading to MarkupSafe 2.1.1, the latest version at this time.

3.0.3

3.0.2

3.0.1

3.0.0

New major versions of all the core Pallets libraries, including Jinja 3.0, have been released! :tada:

This represents a significant amount of work, and there are quite a few changes. Be sure to carefully read the changelog, and use tools such as pip-compile and Dependabot to pin your dependencies and control your updates.

3.0.0rc2

Fixes an issue with the deprecated Markup subclass, #1401.

3.0.0rc1

Changelog

Sourced from jinja2's changelog.

Version 3.1.2

Released 2022-04-28

  • Add parameters to Environment.overlay to match __init__. :issue:1645
  • Handle race condition in FileSystemBytecodeCache. :issue:1654

Version 3.1.1

Released 2022-03-25

  • The template filename on Windows uses the primary path separator. :issue:1637

Version 3.1.0

Released 2022-03-24

  • Drop support for Python 3.6. :pr:1534

  • Remove previously deprecated code. :pr:1544

    • WithExtension and AutoEscapeExtension are built-in now.
    • contextfilter and contextfunction are replaced by pass_context. evalcontextfilter and evalcontextfunction are replaced by pass_eval_context. environmentfilter and environmentfunction are replaced by pass_environment.
    • Markup and escape should be imported from MarkupSafe.
    • Compiled templates from very old Jinja versions may need to be recompiled.
    • Legacy resolve mode for Context subclasses is no longer supported. Override resolve_or_missing instead of resolve.
    • unicode_urlencode is renamed to url_quote.
  • Add support for native types in macros. :issue:1510

  • The {% trans %} tag can use pgettext and npgettext by passing a context string as the first token in the tag, like {% trans "title" %}. :issue:1430

  • Update valid identifier characters from Python 3.6 to 3.7. :pr:1571

  • Filters and tests decorated with @async_variant are pickleable. :pr:1612

  • Add items filter. :issue:1561

... (truncated)

Commits
  • b08cd4b Merge pull request #1660 from pallets/release-3.1.2
  • 1e68ba8 release version 3.1.2
  • 8efee35 pre-commit updates latest release branch
  • a24df26 ignore new mypy finding
  • 9faee28 update requirements
  • b802b5a Merge pull request #1655 from dvitek/dvitek/issue1654
  • 746bb95 Fix race conditions in FileSystemBytecodeCache
  • 466a200 update requirements
  • 990602f Merge pull request #1647 from Tom-Brouwer/202204/add-missing-overlay-options
  • 5d3d241 fix flake8-bugbear finding
  • Additional commits viewable in compare view

Dependabot compatibility score

You can trigger a rebase of this PR by commenting @dependabot rebase.


Dependabot commands and options

You can trigger Dependabot actions by commenting on this PR:

  • @dependabot rebase will rebase this PR
  • @dependabot recreate will recreate this PR, overwriting any edits that have been made to it
  • @dependabot merge will merge this PR after your CI passes on it
  • @dependabot squash and merge will squash and merge this PR after your CI passes on it
  • @dependabot cancel merge will cancel a previously requested merge and block automerging
  • @dependabot reopen will reopen this PR if it is closed
  • @dependabot close will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually
  • @dependabot ignore this major version will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself)
  • @dependabot ignore this minor version will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself)
  • @dependabot ignore this dependency will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)
nathaniel-may
nathaniel-may
Apr
28
3 weeks ago
Activity icon
issue

nathaniel-may issue dbt-labs/dbt-core

nathaniel-may
nathaniel-may

[Feature] Grants as Node Configs

Is this your first time opening an issue?

Describe the Feature

Description

Today users often configure a post_hook to grant permissions on models, seeds, and snapshots:

{{ config(post_hook = 'grant select on {{ this }} to role reporter') }}

This ticket aims to make it easier to specify grants allowing it to be configured directly both in dbt_project.yml as well as in source files:

# dbt_project.yml
models:
  export:
    +grants:
      select: ['reporter', 'bi']
-- SQL
{{ config(grants = {'select': ['other_user']}) }}

These grant configs will not necessarily look the same for each and every warehouse. The logic to generate the sql from these configs can be overriden by adapters.

Implementation

The implementation can be split up into two testable parts: adding the configuration and issuing the grants.

Adding the Configuration:

  • a "grants" config should be added to model, seed, and snapshot nodes. The value of this config is a dictionary.
  • merge behavior for grants should be to override / clobber not append.
  • test that projects can define grants in 1) the dbt_project.yml, 2) models, seeds, and snapshots, and 3) both the dbt_project.yml and models, seeds, and snapshots with the correct merge behavior.

Issuing the grants:

  • create a new macro: get_grant_sql(grant_config: dict) -> str that creates the warehouse-specific sql from grant portion of the config. The default implementation should throw or return an empty string because it is expected to be overridden by adapters. The macro signature here is designed to accept different shapes of grant configs for each adapter to use whatever best fits the warehouse's permissions system.
  • create a new macro: apply_grants which calls get_grant_sql to apply the grants (see persist_docs for an example of similar implementation)
  • add a call to apply_grants in all materializations
  • test that postgres projects can define grants in 1) the dbt_project.yml, 2) models, seeds, and snapshots, and 3) both the dbt_project.yml and models, seeds, and snapshots and those permissions can be read back directly from the postgres warehouse after a run.

In order for this to be usable, the adapters must each override get_grant_sql. Because that does not prevent a merge into main for core, adapter work will be tracked in separate tickets.

Describe alternatives you've considered

No response

Who will this benefit?

No response

Are you interested in contributing this feature?

No response

Anything else?

No response

open pull request

nathaniel-may wants to merge dbt-labs/dbt-core

nathaniel-may
nathaniel-may

Creating ADR for versioning and branching strategy

Description

Creating an initial ADR to establish how we approach versioning and branching. I wanted to write this as a base to build the release process off of and make sure we have a shared understanding of how we are doing this and why.

Checklist

  • I have signed the CLA
  • I have run this code in development and it appears to resolve the stated issue
  • This PR includes tests, or tests are not required/relevant for this PR
  • I have added information about my change to be included in the CHANGELOG.
nathaniel-may
nathaniel-may
pull request

nathaniel-may merge to dbt-labs/dbt-core

nathaniel-may
nathaniel-may

Creating ADR for versioning and branching strategy

Description

Creating an initial ADR to establish how we approach versioning and branching. I wanted to write this as a base to build the release process off of and make sure we have a shared understanding of how we are doing this and why.

Checklist

  • I have signed the CLA
  • I have run this code in development and it appears to resolve the stated issue
  • This PR includes tests, or tests are not required/relevant for this PR
  • I have added information about my change to be included in the CHANGELOG.
pull request

nathaniel-may merge to dbt-labs/dbt-core

nathaniel-may
nathaniel-may

Update mypy to latest and turn on everywhere

resolves #4805, 4089

Description

  • Mypy was upgraded to 0.942
  • Code changes were made so mypy can be run on all code in the repo (excepting the postgres plugin) using the --strict option

Checklist

Previous