ptitzler

ptitzler

Member Since 6 years ago

Center for Open-Source Data & AI Technologies, San Jose, CA, USA

Experience Points
35
follower
Lessons Completed
2
follow
Lessons Completed
38
stars
Best Reply Awards
84
repos

982 contributions in the last year

ptitzler Most Used Languages
ptitzler GitHub Stats

1 Pinned

⚡ Information and links you might find useful. Or not.
Sep
16
1 day ago
Activity icon
issue

ptitzler issue comment elyra-ai/elyra

ptitzler
ptitzler

elyra-metadata install component-registries doesn't handle incorrectly formatted array inputs well

Describe the issue

elyra-metadata install component-registries requires certain parameters to be formatted as an "array" (no specifics provided), yet no meaningful help is provided if the user enters an invalid format. The example below is a stack trace for such an invalid input.

Traceback (most recent call last):
  File "/opt/anaconda3/envs/upstream/bin/elyra-metadata", line 8, in <module>
    sys.exit(MetadataApp.main())
  File "/opt/anaconda3/envs/upstream/lib/python3.7/site-packages/elyra/metadata/metadata_app.py", line 330, in main
    elyra_metadata.start()
  File "/opt/anaconda3/envs/upstream/lib/python3.7/site-packages/elyra/metadata/metadata_app.py", line 342, in start
    return subinstance.start()
  File "/opt/anaconda3/envs/upstream/lib/python3.7/site-packages/elyra/metadata/metadata_app.py", line 275, in start
    return subinstance.start()
  File "/opt/anaconda3/envs/upstream/lib/python3.7/site-packages/elyra/metadata/metadata_app.py", line 197, in start
    super().start()  # process options
  File "/opt/anaconda3/envs/upstream/lib/python3.7/site-packages/elyra/metadata/metadata_app.py", line 51, in start
    self.process_cli_options(self.options)
  File "/opt/anaconda3/envs/upstream/lib/python3.7/site-packages/elyra/metadata/metadata_app_utils.py", line 300, in process_cli_options
    self.process_cli_option(option)
  File "/opt/anaconda3/envs/upstream/lib/python3.7/site-packages/elyra/metadata/metadata_app_utils.py", line 270, in process_cli_option
    cli_option.set_value(self.argv_mappings.get(option))
  File "/opt/anaconda3/envs/upstream/lib/python3.7/site-packages/elyra/metadata/metadata_app_utils.py", line 50, in set_value
    self.value = ast.literal_eval(value)
  File "/opt/anaconda3/envs/upstream/lib/python3.7/ast.py", line 46, in literal_eval
    node_or_string = parse(node_or_string, mode='eval')
  File "/opt/anaconda3/envs/upstream/lib/python3.7/ast.py", line 35, in parse
    return compile(source, filename, mode, PyCF_ONLY_AST)
  File "<unknown>", line 1
    https://raw.githubusercontent.com/elyra-ai/elyra/master/etc/config/components/kfp/run_notebook_using_papermill.yaml

There are several issues:

  • the stack trace is not useful to the user; it's display is likely a result of missing error handling
  • an error should be raised indicating which provided parameter is invalid
  • information should be provided what the valid format is, e.g. ['val1','val2'].

Deployment information Describe what you've deployed and how:

  • Elyra version: 3.1.1
ptitzler
ptitzler

https://github.com/elyra-ai/elyra/pull/2104 attempts to mitigate the problem somewhat but is of no help if the user does not read the documentation. In general it would be useful if the help would provide examples for array-type inputs, e.g.

$ elyra-metadata install  component-registries
'--display_name' is a required parameter.

Install a metadata instance into namespace 'component-registries'.

Options
-------

...
--paths=<array>
	A list of absolute paths to individual component specification files or to repositories containing multiple component files (e.g., a directory in the file system). Example: `['/path/to/file','/path/to/another/file']`

Activity icon
issue

ptitzler issue comment elyra-ai/elyra

ptitzler
ptitzler

Add manage pipeline components content to user guide

Closes #1873 Requires #2083

What changes were proposed in this pull request?

How was this pull request tested?

Reviewed output of make docs

Developer's Certificate of Origin 1.1

   By making a contribution to this project, I certify that:

   (a) The contribution was created in whole or in part by me and I
       have the right to submit it under the Apache License 2.0; or

   (b) The contribution is based upon previous work that, to the best
       of my knowledge, is covered under an appropriate open source
       license and I have the right under that license to submit that
       work with modifications, whether created in whole or in part
       by me, under the same open source license (unless I am
       permitted to submit under a different license), as indicated
       in the file; or

   (c) The contribution was provided directly to me by some other
       person who certified (a), (b) or (c) and I have not modified
       it.

   (d) I understand and agree that this project and the contribution
       are public and that a record of the contribution (including all
       personal information I submit with it, including my sign-off) is
       maintained indefinitely and may be redistributed consistent with
       this project or the open source license(s) involved.
ptitzler
ptitzler

@kiersten-stokes is it possible for a user to create filename and directory registry entries without having to rebuild elyra?

push

ptitzler push ptitzler/elyra

ptitzler
ptitzler

Add UI and CLI instructions and properties reference

commit sha: e7c32e2307658a1765964d60d0440affe00b6911

push time in 20 hours ago
Activity icon
issue

ptitzler issue elyra-ai/elyra

ptitzler
ptitzler

elyra-metadata install component-registries doesn't seem to work for URL entries?

Describe the issue

Not sure why it is not, but the following should be a valid invocation

$ elyra-metadata install component-registries --display_name="data load components" --description="Load data from external data sources" --runtime=kfp --location_type=URL --paths="https://raw.githubusercontent.com/elyra-ai/elyra/master/etc/config/components/kfp/run_notebook_using_papermill.yaml"

but fails with

Traceback (most recent call last):
  File "/opt/anaconda3/envs/upstream/bin/elyra-metadata", line 8, in <module>
    sys.exit(MetadataApp.main())
  File "/opt/anaconda3/envs/upstream/lib/python3.7/site-packages/elyra/metadata/metadata_app.py", line 330, in main
    elyra_metadata.start()
  File "/opt/anaconda3/envs/upstream/lib/python3.7/site-packages/elyra/metadata/metadata_app.py", line 342, in start
    return subinstance.start()
  File "/opt/anaconda3/envs/upstream/lib/python3.7/site-packages/elyra/metadata/metadata_app.py", line 275, in start
    return subinstance.start()
  File "/opt/anaconda3/envs/upstream/lib/python3.7/site-packages/elyra/metadata/metadata_app.py", line 197, in start
    super().start()  # process options
  File "/opt/anaconda3/envs/upstream/lib/python3.7/site-packages/elyra/metadata/metadata_app.py", line 51, in start
    self.process_cli_options(self.options)
  File "/opt/anaconda3/envs/upstream/lib/python3.7/site-packages/elyra/metadata/metadata_app_utils.py", line 300, in process_cli_options
    self.process_cli_option(option)
  File "/opt/anaconda3/envs/upstream/lib/python3.7/site-packages/elyra/metadata/metadata_app_utils.py", line 270, in process_cli_option
    cli_option.set_value(self.argv_mappings.get(option))
  File "/opt/anaconda3/envs/upstream/lib/python3.7/site-packages/elyra/metadata/metadata_app_utils.py", line 50, in set_value
    self.value = ast.literal_eval(value)
  File "/opt/anaconda3/envs/upstream/lib/python3.7/ast.py", line 46, in literal_eval
    node_or_string = parse(node_or_string, mode='eval')
  File "/opt/anaconda3/envs/upstream/lib/python3.7/ast.py", line 35, in parse
    return compile(source, filename, mode, PyCF_ONLY_AST)
  File "<unknown>", line 1
    https://raw.githubusercontent.com/elyra-ai/elyra/master/etc/config/components/kfp/run_notebook_using_papermill.yaml

The problematic parameter is the path parameter value because the error is not triggered if it is omitted. Since the provided is a valid URL, I'm not sure what I'm missing.

To Reproduce See above

Deployment information Describe what you've deployed and how:

  • Elyra version: 3.1.1
Activity icon
issue

ptitzler issue elyra-ai/elyra

ptitzler
ptitzler

elyra-metadata install component-registries ... does not handle input errors gracefully

Describe the issue

If invalid input property values (as seen in the example below) are specified, elyra-metadata install component-registries doesn't provide a user-friendly message and instead dumps a stack trace.

$ elyra-metadata install component-registries --display_name=abc --description "Load data from external data sources" --runtime=ttt --location_type=www --paths=ppp
Traceback (most recent call last):
  File "/opt/anaconda3/envs/upstream/bin/elyra-metadata", line 8, in <module>
    sys.exit(MetadataApp.main())
  File "/opt/anaconda3/envs/upstream/lib/python3.7/site-packages/elyra/metadata/metadata_app.py", line 330, in main
    elyra_metadata.start()
  File "/opt/anaconda3/envs/upstream/lib/python3.7/site-packages/elyra/metadata/metadata_app.py", line 342, in start
    return subinstance.start()
  File "/opt/anaconda3/envs/upstream/lib/python3.7/site-packages/elyra/metadata/metadata_app.py", line 275, in start
    return subinstance.start()
  File "/opt/anaconda3/envs/upstream/lib/python3.7/site-packages/elyra/metadata/metadata_app.py", line 197, in start
    super().start()  # process options
  File "/opt/anaconda3/envs/upstream/lib/python3.7/site-packages/elyra/metadata/metadata_app.py", line 51, in start
    self.process_cli_options(self.options)
  File "/opt/anaconda3/envs/upstream/lib/python3.7/site-packages/elyra/metadata/metadata_app_utils.py", line 300, in process_cli_options
    self.process_cli_option(option)
  File "/opt/anaconda3/envs/upstream/lib/python3.7/site-packages/elyra/metadata/metadata_app_utils.py", line 270, in process_cli_option
    cli_option.set_value(self.argv_mappings.get(option))
  File "/opt/anaconda3/envs/upstream/lib/python3.7/site-packages/elyra/metadata/metadata_app_utils.py", line 50, in set_value
    self.value = ast.literal_eval(value)
  File "/opt/anaconda3/envs/upstream/lib/python3.7/ast.py", line 91, in literal_eval
    return _convert(node_or_string)
  File "/opt/anaconda3/envs/upstream/lib/python3.7/ast.py", line 90, in _convert
    return _convert_signed_num(node)
  File "/opt/anaconda3/envs/upstream/lib/python3.7/ast.py", line 63, in _convert_signed_num
    return _convert_num(node)
  File "/opt/anaconda3/envs/upstream/lib/python3.7/ast.py", line 55, in _convert_num
    raise ValueError('malformed node or string: ' + repr(node))
ValueError: malformed node or string: <_ast.Name object at 0x7fae2d718fd0>

The provided information is not helpful to the user at all because it doesn't identify input that is invalid. Invalid input is likely to occur because the command help does not include any examples or guidance what valid values are.

To Reproduce See example above.

Expected behavior A clear and concise description of what you expected to happen.

The command identifies the invalid property value and (ideally) provides guidance on what valid values are.

Deployment information Describe what you've deployed and how:

  • Elyra version: 3.1+
Activity icon
issue

ptitzler issue elyra-ai/elyra

ptitzler
ptitzler

VPE/palette doesn't auto-refresh after component registry entry was deleted

Describe the issue A clear and concise description of what the issue is.

The VPE component palette displays deleted entries after the have been removed from the registry. Open/close of the VPE does not resolve the issue.

image

It appears that the cache is not invalidated when a component registry entry is deleted.

To Reproduce Steps to reproduce the behavior:

  1. Create a component registry entry
  2. Open VPE - the entry is displayed as expected
  3. Delete the component registry entry
  4. Open VPE - the entry is still displayed

Expected behavior A clear and concise description of what you expected to happen. The palette should reflect the current state of the registry or provide the means to manually trigger a re-load.

Deployment information Describe what you've deployed and how:

  • Elyra version: 3.1+
Activity icon
issue

ptitzler issue elyra-ai/elyra

ptitzler
ptitzler

VPE: component palette panel should be resizable

Is your feature request related to a problem? Please describe.

Component category names don't necessarily fit into the fixed space provided by the palette, leading to truncation:

image

Currently one has to hover over a category name to read the complete text.

Describe the solution you'd like Mirror the behavior of the properties panel, which can be resized.

Activity icon
issue

ptitzler issue comment elyra-ai/elyra

ptitzler
ptitzler

Can't run introduction-to-generic-pipelines, got invalidPipeline error

Error message

{
  "severity": 1,
  "source": "Elyra Pipeline Validation Service",
  "type": "invalidPipeline",
  "message": "Primary pipeline version field has an invalid value.",
  "data": {
    "supported_version": 4,
    "detected_version": 5
  }
}
ptitzler
ptitzler

@eromoe if you upgrade to version 3.1.1 things should work as expected.

pull request

ptitzler merge to elyra-ai/elyra

ptitzler
ptitzler

Update documentation for Troubleshooting and Contribution sections

Developer's Certificate of Origin 1.1

   By making a contribution to this project, I certify that:

   (a) The contribution was created in whole or in part by me and I
       have the right to submit it under the Apache License 2.0; or

   (b) The contribution is based upon previous work that, to the best
       of my knowledge, is covered under an appropriate open source
       license and I have the right under that license to submit that
       work with modifications, whether created in whole or in part
       by me, under the same open source license (unless I am
       permitted to submit under a different license), as indicated
       in the file; or

   (c) The contribution was provided directly to me by some other
       person who certified (a), (b) or (c) and I have not modified
       it.

   (d) I understand and agree that this project and the contribution
       are public and that a record of the contribution (including all
       personal information I submit with it, including my sign-off) is
       maintained indefinitely and may be redistributed consistent with
       this project or the open source license(s) involved.
open pull request

ptitzler wants to merge elyra-ai/elyra

ptitzler
ptitzler

Update documentation for Troubleshooting and Contribution sections

Developer's Certificate of Origin 1.1

   By making a contribution to this project, I certify that:

   (a) The contribution was created in whole or in part by me and I
       have the right to submit it under the Apache License 2.0; or

   (b) The contribution is based upon previous work that, to the best
       of my knowledge, is covered under an appropriate open source
       license and I have the right under that license to submit that
       work with modifications, whether created in whole or in part
       by me, under the same open source license (unless I am
       permitted to submit under a different license), as indicated
       in the file; or

   (c) The contribution was provided directly to me by some other
       person who certified (a), (b) or (c) and I have not modified
       it.

   (d) I understand and agree that this project and the contribution
       are public and that a record of the contribution (including all
       personal information I submit with it, including my sign-off) is
       maintained indefinitely and may be redistributed consistent with
       this project or the open source license(s) involved.
ptitzler
ptitzler
   - Place new or updated images in the `/docs/source/images` directory. For example, images for the _pipeline components_ topic in the _user guide_ are stored in `docs/source/images/user_guide/pipeline-components/`. `PNG` is the recommended format.
open pull request

ptitzler wants to merge elyra-ai/elyra

ptitzler
ptitzler

Update documentation for Troubleshooting and Contribution sections

Developer's Certificate of Origin 1.1

   By making a contribution to this project, I certify that:

   (a) The contribution was created in whole or in part by me and I
       have the right to submit it under the Apache License 2.0; or

   (b) The contribution is based upon previous work that, to the best
       of my knowledge, is covered under an appropriate open source
       license and I have the right under that license to submit that
       work with modifications, whether created in whole or in part
       by me, under the same open source license (unless I am
       permitted to submit under a different license), as indicated
       in the file; or

   (c) The contribution was provided directly to me by some other
       person who certified (a), (b) or (c) and I have not modified
       it.

   (d) I understand and agree that this project and the contribution
       are public and that a record of the contribution (including all
       personal information I submit with it, including my sign-off) is
       maintained indefinitely and may be redistributed consistent with
       this project or the open source license(s) involved.
ptitzler
ptitzler
The Elyra documentation's source is stored in the [Elyra repository](https://github.com/elyra-ai/elyra/tree/master/docs) and hosted at [Elyra's Read the Docs](https://elyra.readthedocs.io/). The documentation is written in [Markdown](https://www.sphinx-doc.org/en/master/usage/markdown.html) and built using [Sphinx](https://www.sphinx-doc.org/en/master/).
pull request

ptitzler merge to elyra-ai/elyra

ptitzler
ptitzler

Update documentation for Troubleshooting and Contribution sections

Developer's Certificate of Origin 1.1

   By making a contribution to this project, I certify that:

   (a) The contribution was created in whole or in part by me and I
       have the right to submit it under the Apache License 2.0; or

   (b) The contribution is based upon previous work that, to the best
       of my knowledge, is covered under an appropriate open source
       license and I have the right under that license to submit that
       work with modifications, whether created in whole or in part
       by me, under the same open source license (unless I am
       permitted to submit under a different license), as indicated
       in the file; or

   (c) The contribution was provided directly to me by some other
       person who certified (a), (b) or (c) and I have not modified
       it.

   (d) I understand and agree that this project and the contribution
       are public and that a record of the contribution (including all
       personal information I submit with it, including my sign-off) is
       maintained indefinitely and may be redistributed consistent with
       this project or the open source license(s) involved.
pull request

ptitzler merge to elyra-ai/elyra

ptitzler
ptitzler

Update documentation for Troubleshooting and Contribution sections

Developer's Certificate of Origin 1.1

   By making a contribution to this project, I certify that:

   (a) The contribution was created in whole or in part by me and I
       have the right to submit it under the Apache License 2.0; or

   (b) The contribution is based upon previous work that, to the best
       of my knowledge, is covered under an appropriate open source
       license and I have the right under that license to submit that
       work with modifications, whether created in whole or in part
       by me, under the same open source license (unless I am
       permitted to submit under a different license), as indicated
       in the file; or

   (c) The contribution was provided directly to me by some other
       person who certified (a), (b) or (c) and I have not modified
       it.

   (d) I understand and agree that this project and the contribution
       are public and that a record of the contribution (including all
       personal information I submit with it, including my sign-off) is
       maintained indefinitely and may be redistributed consistent with
       this project or the open source license(s) involved.
open pull request

ptitzler wants to merge elyra-ai/elyra

ptitzler
ptitzler

Update documentation for Troubleshooting and Contribution sections

Developer's Certificate of Origin 1.1

   By making a contribution to this project, I certify that:

   (a) The contribution was created in whole or in part by me and I
       have the right to submit it under the Apache License 2.0; or

   (b) The contribution is based upon previous work that, to the best
       of my knowledge, is covered under an appropriate open source
       license and I have the right under that license to submit that
       work with modifications, whether created in whole or in part
       by me, under the same open source license (unless I am
       permitted to submit under a different license), as indicated
       in the file; or

   (c) The contribution was provided directly to me by some other
       person who certified (a), (b) or (c) and I have not modified
       it.

   (d) I understand and agree that this project and the contribution
       are public and that a record of the contribution (including all
       personal information I submit with it, including my sign-off) is
       maintained indefinitely and may be redistributed consistent with
       this project or the open source license(s) involved.
Activity icon
issue

ptitzler issue comment elyra-ai/elyra

ptitzler
ptitzler

Support data exchange between custom components

Update the component registry response with additional type information for kfp specific components

Testing Instructions:

  1. Get this PR/branch : https://github.com/elyra-ai/pipeline-editor/pull/161 and run `make clean dev-link install" in the root of pipeline-editor directory
  2. Pull THIS PR and run `make clean dev-link install"
  3. Use the sample kfp pipeline under data_exchange directory or create your own.
  • ids have been made unique, to avoid collisions between identical component I/O names When parsing component parameters to determine the type, ui_hints``parameter_info``data``format can be reused instead of creating a new type field. The value of of this field will largely remain as-as except for the following:
  • inputPath- to indicate that it will be a reference to a parent outputPath
  • outputPath - to indicate that this field will provide this output for child nodes. Any component parameters with fields set to this will also be made read-only and provide an informational reference to what this component produces.
  • FoobarControl controller ID placeholder for inputPath fields, until the controller is complete and given a proper name

UI

  • Uniqueness of node labels will be handled by the front end to avoid potential conflicts.

  • Custom controller to handle inputPath / nested enum dropdown for parent output selection

  • Component Outputs will be displayed in read-only since the user cannot make any changes to them outside of the component definition. There is an issue with the description text not wrapping. image

  • InputPath fields like the text field in the filter-text component in a properly configured pipeline payload, should return a tuple consisting of two keys, a node_id for the component its linked to, and a output_key for the specific output key in that component. Any output parameters like output_filtered_text below should be removed from the payload as they are not necessary.

  • outputpath are being displayed with a control option e.g. dropdown, these should be displayed as read only image

  • AA - Node property types can double as both their original primitive/non primitive types as well as an xcom reference for parent outputs. Can we have a custom controller that will have the option to take in both these values but only in an either/or scenario? e.g. image

 "id":"1cf8c40a-b27c-4100-ab52-9c15bcd260b8",
               "type":"execution_node",
               "op":"filter-text",
               "app_data":{
                  "component_parameters":{
                     "text": {
                        node_id = "2d80d969-8c06-4e36-a861-0eef585d6fee"
                        output_key = "data"
                     }
                     "pattern":"line",
                     "output_filtered_text":""      < ---Remove
                  },
                  "label":"second filter",
                  "component_source":"kfp/filter_text_using_shell_and_grep.yaml",
                  "ui_data":{
                     "label":"second filter",
                     ....
                  }

Validation

  • Verify that I/O values are of the correct type for the field and that all required fields have values and are present
  • Update validation checks for component property types for kfp, currently disabled
  • AA - Type checking needs to be updated for all properties since they can double as xcom pulls as well as their original types

KFP Processor

  • We have a custom type file that we are using to read in the contents of a file into inputPath (papermill and filter text components) which is not in line with best practices. We should remove this functionality and treat it as a type and update the logic to instead take in the parent node output

What changes were proposed in this pull request?

How was this pull request tested?

Developer's Certificate of Origin 1.1

   By making a contribution to this project, I certify that:

   (a) The contribution was created in whole or in part by me and I
       have the right to submit it under the Apache License 2.0; or

   (b) The contribution is based upon previous work that, to the best
       of my knowledge, is covered under an appropriate open source
       license and I have the right under that license to submit that
       work with modifications, whether created in whole or in part
       by me, under the same open source license (unless I am
       permitted to submit under a different license), as indicated
       in the file; or

   (c) The contribution was provided directly to me by some other
       person who certified (a), (b) or (c) and I have not modified
       it.

   (d) I understand and agree that this project and the contribution
       are public and that a record of the contribution (including all
       personal information I submit with it, including my sign-off) is
       maintained indefinitely and may be redistributed consistent with
       this project or the open source license(s) involved.
ptitzler
ptitzler

Can we mark output properties more clearly? Currently they are rendered the same way that system properties are rendered, making it impossible to distinguish them.

push

ptitzler push ptitzler/elyra

ptitzler
ptitzler

Add notes about packaging changes (#2148)

ptitzler
ptitzler

Update Pipeline Editor to version 1.2.0 (#2143)

Update Pipeline Editor to version 1.2.0 and migrate existing test pipelines to version 5.

Co-authored-by: Alex Bozarth ajbozart@us.ibm.com

ptitzler
ptitzler

Update changelog for release 3.1.1

ptitzler
ptitzler

Prepare for next development iteration

ptitzler
ptitzler

Fix pipeline upload on Windows system (#2150)

ptitzler
ptitzler

Properly describe empty pipelines using Pipelines CLI (#2115)

commit sha: 2f25e33bc5efa9cd85e2bce9731c9bc27640306e

push time in 1 day ago
Sep
15
2 days ago
Activity icon
issue

ptitzler issue comment elyra-ai/elyra

ptitzler
ptitzler

'elyra-pipeline validate' command fails for generic pipelines

Describe the issue

$ elyra-pipeline validate generic.pipeline

────────────────────────────────────────────────────────────────
 Elyra Pipeline Validation
────────────────────────────────────────────────────────────────

Validating pipeline...
Traceback (most recent call last):
  File "/opt/anaconda3/envs/ga_31/bin/elyra-pipeline", line 8, in <module>
    sys.exit(pipeline())
  File "/opt/anaconda3/envs/ga_31/lib/python3.7/site-packages/click/core.py", line 829, in __call__
    return self.main(*args, **kwargs)
  File "/opt/anaconda3/envs/ga_31/lib/python3.7/site-packages/click/core.py", line 782, in main
    rv = self.invoke(ctx)
  File "/opt/anaconda3/envs/ga_31/lib/python3.7/site-packages/click/core.py", line 1259, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/opt/anaconda3/envs/ga_31/lib/python3.7/site-packages/click/core.py", line 1066, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/opt/anaconda3/envs/ga_31/lib/python3.7/site-packages/click/core.py", line 610, in invoke
    return callback(*args, **kwargs)
  File "/opt/anaconda3/envs/ga_31/lib/python3.7/site-packages/elyra/cli/pipeline_app.py", line 257, in validate
    _validate_pipeline_definition(pipeline_definition)
  File "/opt/anaconda3/envs/ga_31/lib/python3.7/site-packages/elyra/cli/pipeline_app.py", line 178, in _validate_pipeline_definition
    PipelineValidationManager.instance().validate(pipeline=pipeline_definition))
  File "/opt/anaconda3/envs/ga_31/lib/python3.7/asyncio/base_events.py", line 587, in run_until_complete
    return future.result()
  File "/opt/anaconda3/envs/ga_31/lib/python3.7/site-packages/elyra/pipeline/validation.py", line 152, in validate
    response=response)
  File "/opt/anaconda3/envs/ga_31/lib/python3.7/site-packages/elyra/pipeline/validation.py", line 255, in _validate_node_properties
    component_list = await PipelineProcessorManager.instance().get_components(pipeline_runtime)
  File "/opt/anaconda3/envs/ga_31/lib/python3.7/site-packages/elyra/pipeline/processor.py", line 98, in get_components
    processor = self._get_processor_for_runtime(processor_type)
  File "/opt/anaconda3/envs/ga_31/lib/python3.7/site-packages/elyra/pipeline/processor.py", line 91, in _get_processor_for_runtime
    processor = self._registry.get_processor(processor_type)
  File "/opt/anaconda3/envs/ga_31/lib/python3.7/site-packages/elyra/pipeline/processor.py", line 76, in get_processor
    raise RuntimeError('Could not find pipeline processor for [{}]'.format(processor_type))
RuntimeError: Could not find pipeline processor for [None]

To Reproduce Steps to reproduce the behavior:

  1. Create a notebook
  2. Create a generic pipeline
  3. Add notebook to pipeline
  4. Save pipeline
  5. Run elyra-pipeline validate ...

Deployment information Describe what you've deployed and how:

  • Elyra version: 3.1+
ptitzler
ptitzler

How did you create the pipeline? Is it missing the runtime field information?

See updated instructions - it's a plain vanilla pipeline. recreate.tar.gz

pull request

ptitzler pull request elyra-ai/elyra

ptitzler
ptitzler

Add notes about packaging change to installation documentation

Follow-up to https://github.com/elyra-ai/elyra/pull/2137

What changes were proposed in this pull request?

Add explicit note about the changed packaging process, which now requires extra targets to install optional dependencies.

How was this pull request tested?

Reviewed output of make docs

Developer's Certificate of Origin 1.1

   By making a contribution to this project, I certify that:

   (a) The contribution was created in whole or in part by me and I
       have the right to submit it under the Apache License 2.0; or

   (b) The contribution is based upon previous work that, to the best
       of my knowledge, is covered under an appropriate open source
       license and I have the right under that license to submit that
       work with modifications, whether created in whole or in part
       by me, under the same open source license (unless I am
       permitted to submit under a different license), as indicated
       in the file; or

   (c) The contribution was provided directly to me by some other
       person who certified (a), (b) or (c) and I have not modified
       it.

   (d) I understand and agree that this project and the contribution
       are public and that a record of the contribution (including all
       personal information I submit with it, including my sign-off) is
       maintained indefinitely and may be redistributed consistent with
       this project or the open source license(s) involved.
Activity icon
created branch

ptitzler in ptitzler/elyra create branch update-install-doc

createdAt 1 day ago
Activity icon
issue

ptitzler issue comment elyra-ai/elyra

ptitzler
ptitzler

KFP Processor should attempt to try Dex local auth

What changes were proposed in this pull request?

In the current implementation, the KFP processor auth module is assuming that the credentials passed are used for the redirect proxy. For use cases where an OIDC service is used, this is not applicable on cert-based authentications. In some instances, Dex can be used as it's own IdP and a local user can be created. This PR adds logic to attempt to try the Dex local auth prior to passing back a None value.

This references the pattern cited here: https://github.com/kubeflow/kfctl/issues/140#issuecomment-778647434

How was this pull request tested?

I tested the exact method on a setup that uses OIDC w/ cert-based authentication, and it correctly attempted local auth.

ptitzler
ptitzler

Similar changes are required for this template https://github.com/elyra-ai/elyra/blob/master/elyra/templates/kfp/kfp_template.jinja2#L83-L103 to support this type of authentication in the KFP export workflow:

  • export pipeline as KFP Python DSL in terminal window
  • run generated KFP Python DSL to create and run pipeline
open pull request

ptitzler wants to merge elyra-ai/elyra

ptitzler
ptitzler

KFP Processor should attempt to try Dex local auth

What changes were proposed in this pull request?

In the current implementation, the KFP processor auth module is assuming that the credentials passed are used for the redirect proxy. For use cases where an OIDC service is used, this is not applicable on cert-based authentications. In some instances, Dex can be used as it's own IdP and a local user can be created. This PR adds logic to attempt to try the Dex local auth prior to passing back a None value.

This references the pattern cited here: https://github.com/kubeflow/kfctl/issues/140#issuecomment-778647434

How was this pull request tested?

I tested the exact method on a setup that uses OIDC w/ cert-based authentication, and it correctly attempted local auth.

ptitzler
ptitzler

It would be better to not raise an exception (the calling method doesn't catch generic Exceptions) and instead utilize the existing error reporting approach https://github.com/elyra-ai/elyra/blob/12060dbf7b9273579da40ef9f15bf0d1e6efda14/elyra/pipeline/processor_kfp.py#L110-L111. One possible way to do that would be to extend the existing auth-info data structure with an optional property (e.g. authservice_response). The caller already logs the content of this data structure https://github.com/elyra-ai/elyra/blob/12060dbf7b9273579da40ef9f15bf0d1e6efda14/elyra/pipeline/processor_kfp.py#L115-L119

pull request

ptitzler merge to elyra-ai/elyra

ptitzler
ptitzler

KFP Processor should attempt to try Dex local auth

What changes were proposed in this pull request?

In the current implementation, the KFP processor auth module is assuming that the credentials passed are used for the redirect proxy. For use cases where an OIDC service is used, this is not applicable on cert-based authentications. In some instances, Dex can be used as it's own IdP and a local user can be created. This PR adds logic to attempt to try the Dex local auth prior to passing back a None value.

This references the pattern cited here: https://github.com/kubeflow/kfctl/issues/140#issuecomment-778647434

How was this pull request tested?

I tested the exact method on a setup that uses OIDC w/ cert-based authentication, and it correctly attempted local auth.

Activity icon
issue

ptitzler issue elyra-ai/elyra

ptitzler
ptitzler

'elyra-pipeline validate' command fails for generic pipelines

Describe the issue

$ elyra-pipeline validate generic.pipeline

────────────────────────────────────────────────────────────────
 Elyra Pipeline Validation
────────────────────────────────────────────────────────────────

Validating pipeline...
Traceback (most recent call last):
  File "/opt/anaconda3/envs/ga_31/bin/elyra-pipeline", line 8, in <module>
    sys.exit(pipeline())
  File "/opt/anaconda3/envs/ga_31/lib/python3.7/site-packages/click/core.py", line 829, in __call__
    return self.main(*args, **kwargs)
  File "/opt/anaconda3/envs/ga_31/lib/python3.7/site-packages/click/core.py", line 782, in main
    rv = self.invoke(ctx)
  File "/opt/anaconda3/envs/ga_31/lib/python3.7/site-packages/click/core.py", line 1259, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/opt/anaconda3/envs/ga_31/lib/python3.7/site-packages/click/core.py", line 1066, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/opt/anaconda3/envs/ga_31/lib/python3.7/site-packages/click/core.py", line 610, in invoke
    return callback(*args, **kwargs)
  File "/opt/anaconda3/envs/ga_31/lib/python3.7/site-packages/elyra/cli/pipeline_app.py", line 257, in validate
    _validate_pipeline_definition(pipeline_definition)
  File "/opt/anaconda3/envs/ga_31/lib/python3.7/site-packages/elyra/cli/pipeline_app.py", line 178, in _validate_pipeline_definition
    PipelineValidationManager.instance().validate(pipeline=pipeline_definition))
  File "/opt/anaconda3/envs/ga_31/lib/python3.7/asyncio/base_events.py", line 587, in run_until_complete
    return future.result()
  File "/opt/anaconda3/envs/ga_31/lib/python3.7/site-packages/elyra/pipeline/validation.py", line 152, in validate
    response=response)
  File "/opt/anaconda3/envs/ga_31/lib/python3.7/site-packages/elyra/pipeline/validation.py", line 255, in _validate_node_properties
    component_list = await PipelineProcessorManager.instance().get_components(pipeline_runtime)
  File "/opt/anaconda3/envs/ga_31/lib/python3.7/site-packages/elyra/pipeline/processor.py", line 98, in get_components
    processor = self._get_processor_for_runtime(processor_type)
  File "/opt/anaconda3/envs/ga_31/lib/python3.7/site-packages/elyra/pipeline/processor.py", line 91, in _get_processor_for_runtime
    processor = self._registry.get_processor(processor_type)
  File "/opt/anaconda3/envs/ga_31/lib/python3.7/site-packages/elyra/pipeline/processor.py", line 76, in get_processor
    raise RuntimeError('Could not find pipeline processor for [{}]'.format(processor_type))
RuntimeError: Could not find pipeline processor for [None]

To Reproduce Steps to reproduce the behavior:

  1. Create a valid generic pipeline
  2. Run elyra-pipeline validate ...

Deployment information Describe what you've deployed and how:

  • Elyra version: 3.1+
Activity icon
issue

ptitzler issue elyra-ai/elyra

ptitzler
ptitzler

Validation doesn't properly handle pipeline version problems

Describe the issue

See https://github.com/elyra-ai/elyra/issues/2145 for context

To Reproduce Steps to reproduce the behavior:

  1. Create a pipeline file
  2. Copy the pipeline file to file2
  3. In file2: Manually update the pipeline version identifier to be lower than the current version
  4. Copy the pipeline file to file3
  5. In file3: Manually update the pipeline version identifier to be higher than the current version
  6. Copy the pipeline file to file4
  7. In file4: Manually update the pipeline version identifier to be not numeric
  8. Open the modified pipeline files in the VPE and observe the behavior
  9. Run the modified pipeline files using the elyra-pipeline CLI

Deployment information Describe what you've deployed and how:

  • Elyra version: 3.1+/master
push

ptitzler push ptitzler/elyra

ptitzler
ptitzler

Treat doc build warnings as errors (#2099)

ptitzler
ptitzler

Add DAG repo to airflow submission response messages

ptitzler
ptitzler

Add support for private container image registries (#2092)

ptitzler
ptitzler

Fix path and capture problem determination data during execution (#2077)

Co-authored-by: Kiersten Stokes kiersten.stokes@ibm.com Co-authored-by: Kevin Bates kevin.bates@us.ibm.com

ptitzler
ptitzler

Pipeline definition abstraction (#2082)

ptitzler
ptitzler

Add support for validating pipelines via CLI (#2112)

ptitzler
ptitzler

Refactor: Improve Kubeflow docker image (#2114)

ptitzler
ptitzler

Migrate component registry to the metadata service (#2083)

Co-authored-by: Kevin Bates kevin.bates@us.ibm.com Co-authored-by: Luciano Resende lresende@us.ibm.com Co-authored-by: Martha Cryan martha.cryan@ibm.com Co-authored-by: Nick Bourdakos nicholas.bourdakos@ibm.com

ptitzler
ptitzler

Update elyra docker image to install all extra pkgs

Fixes #2061

ptitzler
ptitzler

Update changelog for release 3.1.0

ptitzler
ptitzler

Prepare for next development iteration

ptitzler
ptitzler

Update correct filename on publish goal of release script (#2123)

ptitzler
ptitzler

Fix bootstrapper start_minio test fixture (#2122)

ptitzler
ptitzler

Fix instances of type attribute in processors (#2126)

ptitzler
ptitzler

Fix Copyright year/attribution

ptitzler
ptitzler

Small validation fixes for pipeline references and cleanup

ptitzler
ptitzler

Update/add fixtures to setup factory metadata instances

ptitzler
ptitzler

Bump elyra version in KF Dockerfile (#2132)

Co-authored-by: Alan Chin akchin@us.ibm.com

ptitzler
ptitzler

Update elyra installation documentation (#2137)

Update root readme with instructions to install Elyra with kfp-tekton by default e.g. the [all] option.

Fixes #2133

commit sha: d3ab46cf9b5508e7ed783806e9c386173aedb9f0

push time in 1 day ago
Activity icon
issue

ptitzler issue comment elyra-ai/elyra

ptitzler
ptitzler

Can't run introduction-to-generic-pipelines, got invalidPipeline error

Error message

{
  "severity": 1,
  "source": "Elyra Pipeline Validation Service",
  "type": "invalidPipeline",
  "message": "Primary pipeline version field has an invalid value.",
  "data": {
    "supported_version": 4,
    "detected_version": 5
  }
}
ptitzler
ptitzler

Although the issue is not caused by it, there is a bug in the validation logic that is exposed. In the reported scenario (supported_version < detected_version) the correct code path is never reached and an incorrect validation message produced.

Activity icon
issue

ptitzler issue comment elyra-ai/elyra

ptitzler
ptitzler

Can't run introduction-to-generic-pipelines, got invalidPipeline error

Error message

{
  "severity": 1,
  "source": "Elyra Pipeline Validation Service",
  "type": "invalidPipeline",
  "message": "Primary pipeline version field has an invalid value.",
  "data": {
    "supported_version": 4,
    "detected_version": 5
  }
}
ptitzler
ptitzler

I reproduced this using 3.1.0/pip and local build from master

Activity icon
issue

ptitzler issue comment elyra-ai/examples

ptitzler
ptitzler

Can't run introduction-to-generic-pipelines, got invalidPipeline error

Error message

{
  "severity": 1,
  "source": "Elyra Pipeline Validation Service",
  "type": "invalidPipeline",
  "message": "Primary pipeline version field has an invalid value.",
  "data": {
    "supported_version": 4,
    "detected_version": 5
  }
}
ptitzler
ptitzler
  • Which version of Elyra are you using?
  • Could you please attach your .pipeline file?

The error message is indicating that the pipeline was created using a newer version of Elyra than the one you are using to run the pipeline.

Activity icon
issue

ptitzler issue elyra-ai/elyra

ptitzler
ptitzler

run local pipeline demo has error

os: win10 python:3.7.10 elyra:3.0.1

subprocess.CalledProcessError: Command '['python3', 'E:\Code\Elyra_examples/pipelines/setup_validation/python_script.py', '--PYTHONHOME', 'E:\Code\Elyra_examples/pipelines/setup_validation']' returned non-zero exit status 9009.

pls help me! thx!

Activity icon
issue

ptitzler issue comment elyra-ai/elyra

ptitzler
ptitzler

run local pipeline demo has error

os: win10 python:3.7.10 elyra:3.0.1

subprocess.CalledProcessError: Command '['python3', 'E:\Code\Elyra_examples/pipelines/setup_validation/python_script.py', '--PYTHONHOME', 'E:\Code\Elyra_examples/pipelines/setup_validation']' returned non-zero exit status 9009.

pls help me! thx!

ptitzler
ptitzler

Glad to hear! Closing this issue because all TODO's listed in https://github.com/elyra-ai/elyra/issues/2073#issuecomment-906218034 were addressed in https://github.com/elyra-ai/elyra/pull/2077 and delivered in version 3.1.0. Thanks again for helping us to improve our support for Windows!