UPSTREAM: <carry>: Rebase code to kfp 2.2.0 #78

rimolive · 2024-08-30T17:57:53Z

Description of your changes:

Checklist:

The title for your pull request (PR) should follow our title convention. Learn more about the pull request title convention used in this repository.

PiperOrigin-RevId: 591051675

PiperOrigin-RevId: 591075392

* add subprocess task handler * response to review feedback

…` when uploading models from `preview.llm.rlhf_pipeline` instead of hardcoding value as `text-bison@001` PiperOrigin-RevId: 591346782

…lm.rlhf_pipeline` run instead of reusing cached result PiperOrigin-RevId: 591365087

…f model checkpoint from `preview.llm.rlhf_pipeline` PiperOrigin-RevId: 591393116

…eflow#10325) * feat(sdk): add local execution output collection #localexecution * update ExecutorInput processing * address review feedback * fix exception handling test

PiperOrigin-RevId: 592033936

…10318) (kubeflow#10319) * feat: preserve querystring in pipeline root * refactor: create AppendToPipelineRoot Also apply to client.go * feat: remove query string from URIs (#1) * feat: remove query string from URIs * refactor(GenerateOutputURI): move and preserve comments

PiperOrigin-RevId: 592257929

…omponent (kubeflow#10315)

…ubeflow#10334)

…eflow#10330)

kubeflow#10333) * support Container Components * address review feedback

PiperOrigin-RevId: 592702825

PiperOrigin-RevId: 592769441

PiperOrigin-RevId: 594058915

PiperOrigin-RevId: 595174285

…ion (kubeflow#10353)

…logs (kubeflow#10354)

…ubeflow#10336)

…low#10355)

…low#10338)

…r component execution #localexecution (kubeflow#10348) * feat(sdk): support Concat and IfPresent placeholder in local container component execution #localexecution * address review feedback * fix test

…localexecution (kubeflow#10339)

…to rlhf_preprocessor component PiperOrigin-RevId: 628282787

…amespace PiperOrigin-RevId: 628459625

PiperOrigin-RevId: 629198194

… rlhf_preprocessor component PiperOrigin-RevId: 629315370

Signed-off-by: Chen Sun <[email protected]>

openshift-ci · 2024-08-30T17:58:08Z

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
Once this PR has been reviewed and has the lgtm label, please ask for approval from rimolive. For more information see the Kubernetes Code Review Process.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

dsp-developers · 2024-08-30T17:58:57Z

Commit Checker results:

**NOTE**: These are the results of the commit checker scans. 
If these are not commits from upstream kfp, then please ensure
you adhere to the commit checker formatting

commitchecker verson unknown
Validating 0 commits between a8fbbd2020d280f4cab31245a9639a350207b3f9...d16ee62a91c735453e1133bf241f69bd55d63b15

dsp-developers · 2024-08-30T18:19:02Z

Commit Checker results:

**NOTE**: These are the results of the commit checker scans. 
If these are not commits from upstream kfp, then please ensure
you adhere to the commit checker formatting

commitchecker verson unknown
Validating 0 commits between a8fbbd2020d280f4cab31245a9639a350207b3f9...f671ccb289bda01505c39086cb7f1ea3feda1c87

dsp-developers · 2024-08-30T18:56:41Z

Commit Checker results:

**NOTE**: These are the results of the commit checker scans. 
If these are not commits from upstream kfp, then please ensure
you adhere to the commit checker formatting

commitchecker verson unknown
Validating 0 commits between a8fbbd2020d280f4cab31245a9639a350207b3f9...c80bbeb1734ed0c2590d50bd301465b763933fbc

dsp-developers · 2024-08-30T19:05:50Z

Change to PR detected. A new PR build was completed.
A set of new images have been built to help with testing out this PR:
API Server: quay.io/opendatahub/ds-pipelines-api-server:pr-78
DSP DRIVER: quay.io/opendatahub/ds-pipelines-driver:pr-78
DSP LAUNCHER: quay.io/opendatahub/ds-pipelines-launcher:pr-78
Persistence Agent: quay.io/opendatahub/ds-pipelines-persistenceagent:pr-78
Scheduled Workflow Manager: quay.io/opendatahub/ds-pipelines-scheduledworkflow:pr-78
MLMD Server: quay.io/opendatahub/mlmd-grpc-server:latest
MLMD Envoy Proxy: registry.redhat.io/openshift-service-mesh/proxyv2-rhel8:2.3.9-2
UI: quay.io/opendatahub/ds-pipelines-frontend:pr-78

dsp-developers · 2024-09-03T16:41:32Z

Commit Checker results:

**NOTE**: These are the results of the commit checker scans. 
If these are not commits from upstream kfp, then please ensure
you adhere to the commit checker formatting

commitchecker verson unknown
Validating 0 commits between a8fbbd2020d280f4cab31245a9639a350207b3f9...186f1ca65986ce906f7f6406b8e8d2fe29f4f985

Kubeflow Pipelines 2.2.0 release Signed-off-by: Ricardo M. Oliveira <[email protected]>

dsp-developers · 2024-09-03T17:21:21Z

Commit Checker results:

**NOTE**: These are the results of the commit checker scans. 
If these are not commits from upstream kfp, then please ensure
you adhere to the commit checker formatting

commitchecker verson unknown
Validating 0 commits between a8fbbd2020d280f4cab31245a9639a350207b3f9...b9178ad594e4de7a6da372b42fc70bac2344cd13

dsp-developers · 2024-09-03T17:30:27Z

Change to PR detected. A new PR build was completed.
A set of new images have been built to help with testing out this PR:
API Server: quay.io/opendatahub/ds-pipelines-api-server:pr-78
DSP DRIVER: quay.io/opendatahub/ds-pipelines-driver:pr-78
DSP LAUNCHER: quay.io/opendatahub/ds-pipelines-launcher:pr-78
Persistence Agent: quay.io/opendatahub/ds-pipelines-persistenceagent:pr-78
Scheduled Workflow Manager: quay.io/opendatahub/ds-pipelines-scheduledworkflow:pr-78
MLMD Server: quay.io/opendatahub/mlmd-grpc-server:latest
MLMD Envoy Proxy: registry.redhat.io/openshift-service-mesh/proxyv2-rhel8:2.3.9-2
UI: quay.io/opendatahub/ds-pipelines-frontend:pr-78

dsp-developers · 2024-09-25T15:26:01Z

Commit Checker results:

**NOTE**: These are the results of the commit checker scans. 
If these are not commits from upstream kfp, then please ensure
you adhere to the commit checker formatting

commitchecker verson unknown
Validating 0 commits between 0f96e42b1bd0968e898f6912a7b23f0a1285c6db...86774871bde2083a4a590431d0a8a4510a9b3240

dsp-developers · 2024-09-25T15:35:14Z

Change to PR detected. A new PR build was completed.
A set of new images have been built to help with testing out this PR:
API Server: quay.io/opendatahub/ds-pipelines-api-server:pr-78
DSP DRIVER: quay.io/opendatahub/ds-pipelines-driver:pr-78
DSP LAUNCHER: quay.io/opendatahub/ds-pipelines-launcher:pr-78
Persistence Agent: quay.io/opendatahub/ds-pipelines-persistenceagent:pr-78
Scheduled Workflow Manager: quay.io/opendatahub/ds-pipelines-scheduledworkflow:pr-78
MLMD Server: quay.io/opendatahub/mlmd-grpc-server:latest
MLMD Envoy Proxy: registry.redhat.io/openshift-service-mesh/proxyv2-rhel8:2.3.9-2
UI: quay.io/opendatahub/ds-pipelines-frontend:pr-78

Signed-off-by: Ricardo M. Oliveira <[email protected]>

openshift-ci · 2024-09-27T12:49:45Z

The following users are mentioned in OWNERS file(s) but are untrusted for the following reasons. One way to make the user trusted is to add them as members of the opendatahub-io org. You can then trigger verification by writing /verify-owners in a comment.

Tomcli
- User is not a member of the org. User is not a collaborator. Satisfy at least one of these conditions to make the user trusted.
- backend/OWNERS

dsp-developers · 2024-09-27T12:50:30Z

Commit Checker results:

**NOTE**: These are the results of the commit checker scans. 
If these are not commits from upstream kfp, then please ensure
you adhere to the commit checker formatting

commitchecker verson unknown
Validating 0 commits between 0f96e42b1bd0968e898f6912a7b23f0a1285c6db...5cc93ff1ef534a8cf716caa66fa940ce6adf7e03

dsp-developers · 2024-09-27T12:59:22Z

Change to PR detected. A new PR build was completed.
A set of new images have been built to help with testing out this PR:
API Server: quay.io/opendatahub/ds-pipelines-api-server:pr-78
DSP DRIVER: quay.io/opendatahub/ds-pipelines-driver:pr-78
DSP LAUNCHER: quay.io/opendatahub/ds-pipelines-launcher:pr-78
Persistence Agent: quay.io/opendatahub/ds-pipelines-persistenceagent:pr-78
Scheduled Workflow Manager: quay.io/opendatahub/ds-pipelines-scheduledworkflow:pr-78
MLMD Server: quay.io/opendatahub/mlmd-grpc-server:latest
MLMD Envoy Proxy: registry.redhat.io/openshift-service-mesh/proxyv2-rhel8:2.3.9-2
UI: quay.io/opendatahub/ds-pipelines-frontend:pr-78

HumairAK · 2024-10-04T19:40:36Z

backend/api/v1beta1/go_client/auth.pb.go

+// 	protoc-gen-go v1.33.0
 // 	protoc        v3.17.3


we need to run protoc again, because some proto files are not in upstream yet and they are still on 1.26:

https://github.com/rimolive/data-science-pipelines/blob/rebase-2.2.0/backend/api/v2beta1/go_client/artifacts.pb.go#L17-L18

here's the make command: https://github.com/rimolive/data-science-pipelines/blob/rebase-2.2.0/backend/api/Makefile#L26

export API_VERSION=v2beta1 make generate

HumairAK · 2024-10-04T19:59:26Z

backend/src/apiserver/resource/resource_manager.go

+	// assign OwnerReference to scheduledworkflow
+	if run.RecurringRunId != "" {
+		job, err := r.jobStore.GetJob(run.RecurringRunId)
+		if err != nil {
+			return nil, util.NewInternalServerError(util.NewInvalidInputError("RecurringRunId doesn't exist: %s", run.RecurringRunId), "Failed to create a run due to invalid recurring run id")
+		}
+		swf, err := r.swfClient.ScheduledWorkflow(job.Namespace).Get(ctx, job.K8SName, v1.GetOptions{})
+		if err != nil {
+			return nil, util.NewInternalServerError(util.NewInvalidInputError("ScheduledWorkflow doesn't exist: %s", job.K8SName), "Failed to create a run due to invalid name")
+		}
+		executionSpec.SetOwnerReferences(swf)
+	}


this looks like it will definitely conflict with:

https://github.com/opendatahub-io/data-science-pipelines/pull/34/files

@hbelmiro can you take a look at this, do we need to undo the changes your commit introduced?

TheMichaelHu and others added 30 commits December 14, 2023 14:25

feat(components): Add better docstrings for AutoSxS

9f8495d

PiperOrigin-RevId: 591051675

test(components): INTERNAL

75ffbf9

PiperOrigin-RevId: 591075392

feat(sdk): add subprocess task handler #localexecution (kubeflow#10302)

21f8e9c

* add subprocess task handler * response to review feedback

fix(components): Use large_model_reference as `model_reference_name…

f51a930

…` when uploading models from `preview.llm.rlhf_pipeline` instead of hardcoding value as `text-bison@001` PiperOrigin-RevId: 591346782

test(sdk): add kfp-kubernetes execution tests (kubeflow#10304)

2009dfb

fix(components): Resolve unique model display name on each `preview.l…

075d58f

…lm.rlhf_pipeline` run instead of reusing cached result PiperOrigin-RevId: 591365087

fix(components): Upload the tuned adapter to Model Registry instead o…

2e2ba9e

…f model checkpoint from `preview.llm.rlhf_pipeline` PiperOrigin-RevId: 591393116

feat(sdk): add local execution output collection #localexecution (kub…

76aad8b

…eflow#10325) * feat(sdk): add local execution output collection #localexecution * update ExecutorInput processing * address review feedback * fix exception handling test

chore(sdk): fix use of invalid escape sequence in tests (kubeflow#10310)

d0da0ad

feat(sdk): add local execution logging #localexecution (kubeflow#10326)

7849272

chore(components): Fix argument description

8eddc27

PiperOrigin-RevId: 592033936

feat(components): change output format to allow possible post eval

44f9992

PiperOrigin-RevId: 592257929

fix(frontend): Add disableParsingRawHTML option for markdown-to-jsx c…

c6acac9

…omponent (kubeflow#10315)

feat(sdk): add DockerRunner #localexecution (kubeflow#10328)

adc5b3b

feat(sdk): add special dsl.OutputPath read logic #localexecution (k…

654bbde

…ubeflow#10334)

chore(sdk): write local execution logs to stdout #localexecution (kub…

f52ba56

…eflow#10330)

feat(sdk): support local Container Component execution #localexecution (

846f887

kubeflow#10333) * support Container Components * address review feedback

No public description

14de087

PiperOrigin-RevId: 592702825

feat(components): Output errors as a separate table from Arbiter

a66c599

PiperOrigin-RevId: 592769441

docs(components): Fix AutoSxS docstring formatting

216d3f8

PiperOrigin-RevId: 594058915

chore(components): add json_escape placeholder util

973610b

PiperOrigin-RevId: 595174285

fix(sdk): fix presentation of strings in local execution #localexecut…

89d4234

…ion (kubeflow#10353)

fix(sdk): remove redundant newline character in local DockerRunner …

86b7e23

…logs (kubeflow#10354)

test(sdk): improve KFP SDK local runner test safety #localexecution (k…

dcab051

…ubeflow#10336)

feat(backend): Enable logging for KFP components (kubeflow#10288)

5399585

feat(sdk): remove local execution feature flag #localexecution (kubef…

8a5a17e

…low#10355)

fix(sdk): permit empty local execution outputs #localexecution (kubef…

64d46df

…low#10338)

feat(sdk): support Concat and IfPresent placeholder in local containe…

2897a10

…r component execution #localexecution (kubeflow#10348) * feat(sdk): support Concat and IfPresent placeholder in local container component execution #localexecution * address review feedback * fix test

test(sdk): add test for local execution of None default parameter #…

ca5f97d

…localexecution (kubeflow#10339)

Googler and others added 6 commits April 25, 2024 21:10

feat(components): migrate function_based convert_to_delimited_string …

efefe34

…to rlhf_preprocessor component PiperOrigin-RevId: 628282787

feat(components): Move ModelImportEvaluationOp component to preview n…

33db128

…amespace PiperOrigin-RevId: 628459625

chore(components): Update AutoSxS and RLHF image tags

401aac7

PiperOrigin-RevId: 629198194

feat(components): migrate function_based resolve_regional_endpoint to…

f175c71

… rlhf_preprocessor component PiperOrigin-RevId: 629315370

chore(backend): update kfp launcher and driver images (kubeflow#10733)

8d32f0c

Signed-off-by: Chen Sun <[email protected]>

chore(release): bumped version to 2.2.0

dd59f48

openshift-ci bot requested review from gregsheremeta and HumairAK August 30, 2024 17:58

openshift-ci bot added the do-not-merge/invalid-owners-file label Aug 30, 2024

rimolive force-pushed the rebase-2.2.0 branch from d16ee62 to f671ccb Compare August 30, 2024 18:18

rimolive force-pushed the rebase-2.2.0 branch from f671ccb to c80bbeb Compare August 30, 2024 18:55

rimolive force-pushed the rebase-2.2.0 branch from c80bbeb to 186f1ca Compare September 3, 2024 16:40

Merge tag '2.2.0' into test-rebase

b9178ad

Kubeflow Pipelines 2.2.0 release Signed-off-by: Ricardo M. Oliveira <[email protected]>

rimolive force-pushed the rebase-2.2.0 branch from 186f1ca to b9178ad Compare September 3, 2024 17:20

Merge remote-tracking branch 'opendatahub/master' into rebase-2.2.0

5cc93ff

Signed-off-by: Ricardo M. Oliveira <[email protected]>

rimolive force-pushed the rebase-2.2.0 branch from 8677487 to 5cc93ff Compare September 27, 2024 12:49

HumairAK reviewed Oct 4, 2024

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

UPSTREAM: <carry>: Rebase code to kfp 2.2.0 #78

UPSTREAM: <carry>: Rebase code to kfp 2.2.0 #78

rimolive commented Aug 30, 2024

openshift-ci bot commented Aug 30, 2024

dsp-developers commented Aug 30, 2024

dsp-developers commented Aug 30, 2024

dsp-developers commented Aug 30, 2024

dsp-developers commented Aug 30, 2024

dsp-developers commented Sep 3, 2024

dsp-developers commented Sep 3, 2024

dsp-developers commented Sep 3, 2024

dsp-developers commented Sep 25, 2024

dsp-developers commented Sep 25, 2024

openshift-ci bot commented Sep 27, 2024

dsp-developers commented Sep 27, 2024

dsp-developers commented Sep 27, 2024

HumairAK Oct 4, 2024

HumairAK Oct 4, 2024

UPSTREAM: <carry>: Rebase code to kfp 2.2.0 #78

Are you sure you want to change the base?

UPSTREAM: <carry>: Rebase code to kfp 2.2.0 #78

Conversation

rimolive commented Aug 30, 2024

openshift-ci bot commented Aug 30, 2024

dsp-developers commented Aug 30, 2024

Commit Checker results:

dsp-developers commented Aug 30, 2024

Commit Checker results:

dsp-developers commented Aug 30, 2024

Commit Checker results:

dsp-developers commented Aug 30, 2024

dsp-developers commented Sep 3, 2024

Commit Checker results:

dsp-developers commented Sep 3, 2024

Commit Checker results:

dsp-developers commented Sep 3, 2024

dsp-developers commented Sep 25, 2024

Commit Checker results:

dsp-developers commented Sep 25, 2024

openshift-ci bot commented Sep 27, 2024

dsp-developers commented Sep 27, 2024

Commit Checker results:

dsp-developers commented Sep 27, 2024

HumairAK Oct 4, 2024

Choose a reason for hiding this comment

HumairAK Oct 4, 2024

Choose a reason for hiding this comment