Enhancement Description

• One-line enhancement description (can be used as a release note): Introduce the concept of a cluster ID to allow unique identification of clusters within a ClusterSet.
• Kubernetes Enhancement Proposal: https://github.com/kubernetes/enhancements/pull/2150
• Discussion Link: https://docs.google.com/document/d/1S0u6xzP2gcJKPipA6tBNDNuid76nVKeGhTk7PrCIuQY
• Primary contact (assignee): @lauralorenz
• Responsible SIGs: sig-multicluster
• Enhancement target (which target equals to which milestone):
  • Alpha release target (x.y): TK
  • Beta release target (x.y):
  • Stable release target (x.y):
• Alpha
  • KEP (k/enhancements) update PR(s): TK
  • Code (k/k) update PR(s):
  • Docs (k/website) update PR(s):

Please keep this description up to date. This will help the Enhancement Team to track the evolution of the enhancement efficiently.

Which jobs are flaking?

master-blocking/gce-cos-master-reboot master-blocking/gce-cos-master-default master-blocking/gce-cos-master-alpha-features master-blocking/gce-cos-master-scalability-100

Which tests are flaking?

All these tests seem to have similar starting test named "Overall" and "Pod" that are failing, ex ci-kubernetes-e2e-gci-gce-alpha-features.Pod and ci-kubernetes-e2e-gci-gce-alpha-features.Overall

Since when has it been flaking?

4/7, seems to reoccur daily or several times a day even with no change in test commit.

Testgrid link

See above links for each job

Reason for failure (if possible)

gce-cos-master-scalability-100's message is different from the others:

Init container clonerefs not ready: (state: waiting, reason: "PodInitializing", message: "") Init container initupload not ready: (state: waiting, reason: "PodInitializing", message: "") Init container place-entrypoint not ready: (state: waiting, reason: "PodInitializing", message: "")

All the others seem to be more similar (master-blocking/gce-cos-master-reboot, master-blocking/gce-cos-master-default, master-blocking/gce-cos-master-alpha-features) and emit either Job execution failed: Pod pending timeout. or Job execution failed: Pod got deleted unexpectedly; in the pod events the latest events I see are:

Warning | FailedCreatePodSandBox | 1h18m | kubelet | Failed to create pod sandbox: rpc error: code = Unknown desc = failed to reserve sandbox name "e62f5752-b9ae-11ec-afd0-aefa2f608b87_test-pods_da9a67b1-53d5-40ee-a40c-9bfec3582552_0": name "e62f5752-b9ae-11ec-afd0-aefa2f608b87_test-pods_da9a67b1-53d5-40ee-a40c-9bfec3582552_0" is reserved for "3a0532714cdaad3a07206f89fb21262b6e1d0bb8f252372aed0a9a52d70d90e6"

or

Warning | NodeNotReady | 1h52m | node-controller | Node is not ready

Anything else we need to know?

The issue with master-blocking/gce-cos-master-scalability-100 was previously observed by testing-ops on 4/8 (see slack thread) and was determined to resolve on its own then, but has reoccured after that.

Relevant SIG(s)

/sig testing

Hello,

Please review this first commit for the clusterproperty controller.

Included in the PR:

1. Make files to build and deploy the controller v1alpha version.
2. Create the CRD for clusterproperty scoped to cluster.
3. Create api group about.k8s.io.
4. kustomize patched for clusterproperty yaml.
5. Add the CRD to k8s.io apigroup as explained in this pull request. : https://github.com/kubernetes/enhancements/pull/1100

Thank you,

Which jobs are failing?

sig-release-master-informing#gce-cos-master-serial

Which tests are failing?

Kubernetes e2e suite.[sig-storage] CSI Volumes [Driver: csi-hostpath] [Testpattern: Dynamic PV (block volmode)] disruptive[Disruptive][LinuxOnly] Should test that pv used in a pod that is deleted while the kubelet is down cleans up when the kubelet returns. kubetest.Test ci-kubernetes-e2e-gci-gce-serial.Pod ci-kubernetes-e2e-gci-gce-serial.Overall

Since when has it been failing?

3/31 02:37 PDT

Testgrid link

https://testgrid.k8s.io/sig-release-master-informing#gce-cos-master-serial

Reason for failure (if possible)

Job execution failed: Pod pending timeout.

{ Failure test/e2e/storage/testsuites/disruptive.go:155
Mar 31 11:13:26.559: Encountered SSH error.
Unexpected error:
    <*errors.errorString | 0xc001202e50>: {
        s: "error getting SSH client to [email protected]:22: 'ssh: handshake failed: read tcp 10.34.99.13:46652->34.82.121.164:22: read: connection reset by peer'",
    }
    error getting SSH client to [email protected]:22: 'ssh: handshake failed: read tcp 10.34.99.13:46652->34.82.121.164:22: read: connection reset by peer'
occurred
test/e2e/storage/utils/utils.go:283}

Anything else we need to know?

No response

Relevant SIG(s)

/sig cloud-provider

Hello,

Please review this first commit for the clusterproperty controller.

Included in the PR:

1. Make files to build and deploy the controller v1alpha version.
2. Create the CRD for clusterproperty scoped to cluster.
3. Create api group about.k8s.io.
4. kustomize patched for clusterproperty yaml.
5. Add the CRD to k8s.io apigroup as explained in this pull request. : https://github.com/kubernetes/enhancements/pull/1100

Thank you,

Hello,

Please review this first commit for the clusterproperty controller.

Included in the PR:

1. Make files to build and deploy the controller v1alpha version.
2. Create the CRD for clusterproperty scoped to cluster.
3. Create api group about.k8s.io.
4. kustomize patched for clusterproperty yaml.
5. Add the CRD to k8s.io apigroup as explained in this pull request. : https://github.com/kubernetes/enhancements/pull/1100

Thank you,

Which jobs are failing?

sig-release-master-blocking#ci-kubernetes-unit

Which tests are failing?

k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server.TestPreShutdownHooks

Since when has it been failing?

17:42 PDT 3/29

Testgrid link

https://testgrid.k8s.io/sig-release-master-blocking#ci-kubernetes-unit

Reason for failure (if possible)

{Failed  === RUN   TestPreShutdownHooks
    genericapiserver_graceful_termination_test.go:560: Sending request - timeout: 100ms, url: https://127.0.0.1:40845/echo?message=attempt-1
I0330 02:05:41.123870   78579 tlsconfig.go:240] "Starting DynamicServingCertificateController"
I0330 02:05:41.124641   78579 secure_serving.go:210] Serving securely on [::]:40845
    genericapiserver_graceful_termination_test.go:671: [server] seen new connection: &net.TCPConn{conn:net.conn{fd:(*net.netFD)(0xc00052cf00)}}
    genericapiserver_graceful_termination_test.go:535: Still waiting for the server to start - err: <nil>
    genericapiserver_graceful_termination_test.go:653: [server] received a request, proto: HTTP/2.0, url: /echo?message=attempt-1
    genericapiserver_graceful_termination_test.go:560: Sending request - timeout: 100ms, url: https://127.0.0.1:40845/echo?message=attempt-2
    genericapiserver_graceful_termination_test.go:671: [server] seen new connection: &net.TCPConn{conn:net.conn{fd:(*net.netFD)(0xc0000f1f80)}}
    genericapiserver_graceful_termination_test.go:653: [server] received a request, proto: HTTP/2.0, url: /echo?message=attempt-2
    genericapiserver_graceful_termination_test.go:543: The API server has started
I0330 02:05:41.414509   78579 object_count_tracker.go:84] "StorageObjectCountTracker pruner is exiting"
    genericapiserver_graceful_termination_test.go:560: Sending request - timeout: 100ms, url: https://127.0.0.1:40845/echo?message=attempt-0
    genericapiserver_graceful_termination_test.go:671: [server] seen new connection: &net.TCPConn{conn:net.conn{fd:(*net.netFD)(0xc000436480)}}
    genericapiserver_graceful_termination_test.go:478: PreSHutdown hook can not access the API server - Get "https://127.0.0.1:40845/echo?message=attempt-0": context deadline exceeded
--- FAIL: TestPreShutdownHooks (0.51s)
}

Anything else we need to know?

Failed with the same error for last 4 runs. Beta release cut is scheduled for tomorrow and this is currently a blocker.

Relevant SIG(s)

/sig testing

Which jobs are failing?

sig-release-master-blocking#ci-kubernetes-unit

Which tests are failing?

k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server.TestPreShutdownHooks

Since when has it been failing?

17:42 PDT 3/29

Testgrid link

https://testgrid.k8s.io/sig-release-master-blocking#ci-kubernetes-unit

Reason for failure (if possible)

{Failed  === RUN   TestPreShutdownHooks
    genericapiserver_graceful_termination_test.go:560: Sending request - timeout: 100ms, url: https://127.0.0.1:40845/echo?message=attempt-1
I0330 02:05:41.123870   78579 tlsconfig.go:240] "Starting DynamicServingCertificateController"
I0330 02:05:41.124641   78579 secure_serving.go:210] Serving securely on [::]:40845
    genericapiserver_graceful_termination_test.go:671: [server] seen new connection: &net.TCPConn{conn:net.conn{fd:(*net.netFD)(0xc00052cf00)}}
    genericapiserver_graceful_termination_test.go:535: Still waiting for the server to start - err: <nil>
    genericapiserver_graceful_termination_test.go:653: [server] received a request, proto: HTTP/2.0, url: /echo?message=attempt-1
    genericapiserver_graceful_termination_test.go:560: Sending request - timeout: 100ms, url: https://127.0.0.1:40845/echo?message=attempt-2
    genericapiserver_graceful_termination_test.go:671: [server] seen new connection: &net.TCPConn{conn:net.conn{fd:(*net.netFD)(0xc0000f1f80)}}
    genericapiserver_graceful_termination_test.go:653: [server] received a request, proto: HTTP/2.0, url: /echo?message=attempt-2
    genericapiserver_graceful_termination_test.go:543: The API server has started
I0330 02:05:41.414509   78579 object_count_tracker.go:84] "StorageObjectCountTracker pruner is exiting"
    genericapiserver_graceful_termination_test.go:560: Sending request - timeout: 100ms, url: https://127.0.0.1:40845/echo?message=attempt-0
    genericapiserver_graceful_termination_test.go:671: [server] seen new connection: &net.TCPConn{conn:net.conn{fd:(*net.netFD)(0xc000436480)}}
    genericapiserver_graceful_termination_test.go:478: PreSHutdown hook can not access the API server - Get "https://127.0.0.1:40845/echo?message=attempt-0": context deadline exceeded
--- FAIL: TestPreShutdownHooks (0.51s)
}

Anything else we need to know?

Failed with the same error for last 4 runs. Beta release cut is scheduled for tomorrow and this is currently a blocker.

Relevant SIG(s)

/sig testing

Which jobs are failing?

sig-release-master-blocking#ci-kubernetes-unit

Which tests are failing?

k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server.TestPreShutdownHooks

Since when has it been failing?

17:42 PDT 3/29

Testgrid link

https://testgrid.k8s.io/sig-release-master-blocking#ci-kubernetes-unit

Reason for failure (if possible)

{Failed  === RUN   TestPreShutdownHooks
    genericapiserver_graceful_termination_test.go:560: Sending request - timeout: 100ms, url: https://127.0.0.1:40845/echo?message=attempt-1
I0330 02:05:41.123870   78579 tlsconfig.go:240] "Starting DynamicServingCertificateController"
I0330 02:05:41.124641   78579 secure_serving.go:210] Serving securely on [::]:40845
    genericapiserver_graceful_termination_test.go:671: [server] seen new connection: &net.TCPConn{conn:net.conn{fd:(*net.netFD)(0xc00052cf00)}}
    genericapiserver_graceful_termination_test.go:535: Still waiting for the server to start - err: <nil>
    genericapiserver_graceful_termination_test.go:653: [server] received a request, proto: HTTP/2.0, url: /echo?message=attempt-1
    genericapiserver_graceful_termination_test.go:560: Sending request - timeout: 100ms, url: https://127.0.0.1:40845/echo?message=attempt-2
    genericapiserver_graceful_termination_test.go:671: [server] seen new connection: &net.TCPConn{conn:net.conn{fd:(*net.netFD)(0xc0000f1f80)}}
    genericapiserver_graceful_termination_test.go:653: [server] received a request, proto: HTTP/2.0, url: /echo?message=attempt-2
    genericapiserver_graceful_termination_test.go:543: The API server has started
I0330 02:05:41.414509   78579 object_count_tracker.go:84] "StorageObjectCountTracker pruner is exiting"
    genericapiserver_graceful_termination_test.go:560: Sending request - timeout: 100ms, url: https://127.0.0.1:40845/echo?message=attempt-0
    genericapiserver_graceful_termination_test.go:671: [server] seen new connection: &net.TCPConn{conn:net.conn{fd:(*net.netFD)(0xc000436480)}}
    genericapiserver_graceful_termination_test.go:478: PreSHutdown hook can not access the API server - Get "https://127.0.0.1:40845/echo?message=attempt-0": context deadline exceeded
--- FAIL: TestPreShutdownHooks (0.51s)
}

Anything else we need to know?

Failed with the same error for last 4 runs. Beta release cut is scheduled for tomorrow and this is currently a blocker.

Relevant SIG(s)

/sig testing

Which jobs are failing?

sig-release-master-blocking#ci-kubernetes-unit

Which tests are failing?

k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server.TestPreShutdownHooks

Since when has it been failing?

17:42 PDT 3/29

Testgrid link

https://testgrid.k8s.io/sig-release-master-blocking#ci-kubernetes-unit

Reason for failure (if possible)

{Failed  === RUN   TestPreShutdownHooks
    genericapiserver_graceful_termination_test.go:560: Sending request - timeout: 100ms, url: https://127.0.0.1:40845/echo?message=attempt-1
I0330 02:05:41.123870   78579 tlsconfig.go:240] "Starting DynamicServingCertificateController"
I0330 02:05:41.124641   78579 secure_serving.go:210] Serving securely on [::]:40845
    genericapiserver_graceful_termination_test.go:671: [server] seen new connection: &net.TCPConn{conn:net.conn{fd:(*net.netFD)(0xc00052cf00)}}
    genericapiserver_graceful_termination_test.go:535: Still waiting for the server to start - err: <nil>
    genericapiserver_graceful_termination_test.go:653: [server] received a request, proto: HTTP/2.0, url: /echo?message=attempt-1
    genericapiserver_graceful_termination_test.go:560: Sending request - timeout: 100ms, url: https://127.0.0.1:40845/echo?message=attempt-2
    genericapiserver_graceful_termination_test.go:671: [server] seen new connection: &net.TCPConn{conn:net.conn{fd:(*net.netFD)(0xc0000f1f80)}}
    genericapiserver_graceful_termination_test.go:653: [server] received a request, proto: HTTP/2.0, url: /echo?message=attempt-2
    genericapiserver_graceful_termination_test.go:543: The API server has started
I0330 02:05:41.414509   78579 object_count_tracker.go:84] "StorageObjectCountTracker pruner is exiting"
    genericapiserver_graceful_termination_test.go:560: Sending request - timeout: 100ms, url: https://127.0.0.1:40845/echo?message=attempt-0
    genericapiserver_graceful_termination_test.go:671: [server] seen new connection: &net.TCPConn{conn:net.conn{fd:(*net.netFD)(0xc000436480)}}
    genericapiserver_graceful_termination_test.go:478: PreSHutdown hook can not access the API server - Get "https://127.0.0.1:40845/echo?message=attempt-0": context deadline exceeded
--- FAIL: TestPreShutdownHooks (0.51s)
}

Anything else we need to know?

Failed with the same error for last 4 runs. Beta release cut is scheduled for tomorrow and this is currently a blocker.

Relevant SIG(s)

/sig testing

Which jobs are failing?

sig-release-master-blocking#ci-kubernetes-unit

Which tests are failing?

k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server.TestPreShutdownHooks

Since when has it been failing?

17:42 PDT 3/29

Testgrid link

https://testgrid.k8s.io/sig-release-master-blocking#ci-kubernetes-unit

Reason for failure (if possible)

{Failed  === RUN   TestPreShutdownHooks
    genericapiserver_graceful_termination_test.go:560: Sending request - timeout: 100ms, url: https://127.0.0.1:40845/echo?message=attempt-1
I0330 02:05:41.123870   78579 tlsconfig.go:240] "Starting DynamicServingCertificateController"
I0330 02:05:41.124641   78579 secure_serving.go:210] Serving securely on [::]:40845
    genericapiserver_graceful_termination_test.go:671: [server] seen new connection: &net.TCPConn{conn:net.conn{fd:(*net.netFD)(0xc00052cf00)}}
    genericapiserver_graceful_termination_test.go:535: Still waiting for the server to start - err: <nil>
    genericapiserver_graceful_termination_test.go:653: [server] received a request, proto: HTTP/2.0, url: /echo?message=attempt-1
    genericapiserver_graceful_termination_test.go:560: Sending request - timeout: 100ms, url: https://127.0.0.1:40845/echo?message=attempt-2
    genericapiserver_graceful_termination_test.go:671: [server] seen new connection: &net.TCPConn{conn:net.conn{fd:(*net.netFD)(0xc0000f1f80)}}
    genericapiserver_graceful_termination_test.go:653: [server] received a request, proto: HTTP/2.0, url: /echo?message=attempt-2
    genericapiserver_graceful_termination_test.go:543: The API server has started
I0330 02:05:41.414509   78579 object_count_tracker.go:84] "StorageObjectCountTracker pruner is exiting"
    genericapiserver_graceful_termination_test.go:560: Sending request - timeout: 100ms, url: https://127.0.0.1:40845/echo?message=attempt-0
    genericapiserver_graceful_termination_test.go:671: [server] seen new connection: &net.TCPConn{conn:net.conn{fd:(*net.netFD)(0xc000436480)}}
    genericapiserver_graceful_termination_test.go:478: PreSHutdown hook can not access the API server - Get "https://127.0.0.1:40845/echo?message=attempt-0": context deadline exceeded
--- FAIL: TestPreShutdownHooks (0.51s)
}

Anything else we need to know?

Failed with the same error for last 4 runs. Beta release cut is scheduled for tomorrow and this is currently a blocker.

Relevant SIG(s)

/sig testing

Which jobs are failing?

capz-windows-containerd-master

Which tests are failing?

Kubernetes e2e suite.BeforeSuite

Since when has it been failing?

2022-03-20 21:01 IST

Testgrid link

https://testgrid.k8s.io/sig-release-master-informing#capz-windows-containerd-master

Reason for failure (if possible)

No response

Anything else we need to know?

No response

Relevant SIG(s)

not sure which sig is responsible here

/sig windows /kind failing-test

cc @kubernetes/ci-signal

Which jobs are flaking?

• gci-gce-ingress

Which tests are flaking?

• ci-kubernetes-e2e-gci-gce-ingress.Overall
• kubetest.diffResources

Since when has it been flaking?

History goes back to 2021-12-30

Testgrid link

https://testgrid.k8s.io/sig-release-master-blocking#gci-gce-ingress

Reason for failure (if possible)

1/7062 Tests Failed. | expand_less
-- | --
e2e.go: diffResources expand_less0sError: 2 leaked resources +NAME                              REGION    SESSION_AFFINITY  BACKUP  HEALTH_CHECKS +a1ab9e2204ff44d809dc9bceaaf3c8a7  us-west1  NONE                      a1ab9e2204ff44d809dc9bceaaf3c8a7 | e2e.go: diffResources expand_less | 0s | Error: 2 leaked resources +NAME                              REGION    SESSION_AFFINITY  BACKUP  HEALTH_CHECKS +a1ab9e2204ff44d809dc9bceaaf3c8a7  us-west1  NONE                      a1ab9e2204ff44d809dc9bceaaf3c8a7
e2e.go: diffResources expand_less | 0s
Error: 2 leaked resources +NAME                              REGION    SESSION_AFFINITY  BACKUP  HEALTH_CHECKS +a1ab9e2204ff44d809dc9bceaaf3c8a7  us-west1  NONE                      a1ab9e2204ff44d809dc9bceaaf3c8a7

Anything else we need to know?

No response

Relevant SIG(s)

/sig testing

Which jobs are failing?

• Conformance - GCE - master - kubetest2

Which tests are failing?

• ci-kubernetes-gce-conformance-latest-kubetest2.Overall

Since when has it been failing?

History goes back to 2022-03-23

Testgrid link

https://testgrid.k8s.io/sig-release-master-blocking#Conformance%20-%20GCE%20-%20master%20-%20kubetest2

Reason for failure (if possible)

+ go get sigs.k8s.io/[email protected]
go: go.mod file not found in current directory or any parent directory.
	'go get' is no longer supported outside a module.
	To build and install a command, use 'go install' with a version,
	like 'go install example.com/[email protected]'
	For more information, see https://golang.org/doc/go-get-install-deprecation
	or run 'go help get' or 'go help install'.
+ EXIT_VALUE=1

Anything else we need to know?

look like a simple replace of go get with go install is needed

Relevant SIG(s)

not sure which sig is responsible here

/sig cloud-provider /kind failing-test

• One-line PR description: Bump implementable now that API review and PRR review are both complete

• Issue link: #2149

• Other comments: N/A

Which jobs are failing?

release-master-blocking

gce-cos-master-scalability-100

Which tests are failing?

kubetest.ClusterLoaderV2 ci-kubernetes-e2e-gci-gce-scalability.Overall ClusterLoaderV2.load: [step: 19] Collecting pod startup latency ClusterLoaderV2.load overall (testing/load/config.yaml) (intermittently) ClusterLoaderV2.load: [step: 25] Waiting for 'delete objects' to be completed

Since when has it been failing?

3/15 6:47 PDT run; this run also happens to be the first at commit 1a5abe5d1

Testgrid link

https://testgrid.k8s.io/sig-release-master-blocking#gce-cos-master-scalability-100

Reason for failure (if possible)

[measurement call PodStartupLatency - PodStartupLatency error: pod startup: too high latency 99th percentile: got 10.515080076s expected: 5s]

Anything else we need to know?

No response

Relevant SIG(s)

/sig scalability