Enhancement Description

• One-line enhancement description (can be used as a release note): Track terminating endpoints so we can handle terminating endpoints gracefully.
• Kubernetes Enhancement Proposal: https://github.com/kubernetes/enhancements/tree/master/keps/sig-network/1672-tracking-terminating-endpoints
• Primary contact (assignee): @andrewsykim
• Responsible SIGs: sig-network
• Enhancement target (which target equals to which milestone):
  • Alpha release target: v1.20
  • Beta release: v1.22
  • Stable release target: v1.24

Please to keep this description up to date. This will help the Enhancement Team track efficiently the evolution of the enhancement

Enhancement Description

• One-line enhancement description (can be used as a release note): Track terminating endpoints so we can handle terminating endpoints gracefully.
• Kubernetes Enhancement Proposal: https://github.com/kubernetes/enhancements/tree/master/keps/sig-network/1672-tracking-terminating-endpoints
• Primary contact (assignee): @andrewsykim
• Responsible SIGs: sig-network
• Enhancement target (which target equals to which milestone):
  • Alpha release target: v1.20
  • Beta release: v1.22
  • Stable release target: v1.24

Please to keep this description up to date. This will help the Enhancement Team track efficiently the evolution of the enhancement

I observed this with cloud-provider-openstack, but I believe this is a framework bug not a provider bug. Specifically we had a permissions issue which caused this Patch to always fail: https://github.com/kubernetes/cloud-provider-openstack/blob/21ce6dd984e5599ba3c411c960acdbcf9b5dbf03/pkg/openstack/loadbalancer.go#L1783

However, completely deterministically this succeeds the second time it is called. There is no way for this reconcile to succeed without setting an annotation, but after the successful reconciliation we see that the annotation is not set. The only explanation I could muster for this is that we were called with the dirty object from the first reconciliation. Consequently the Patcher is based on a dirty object which already has the annotation set, even though we never succeeded to write that to the API, and therefore the Patcher believes second time round that it doesn't need to write the annotation and we erroneously succeed without having written the required annotation.

The obvious suspect was the cache used by the service controller, however for this specific call path we are always passing in a Service object which came from the informer.

...The informer code is hard to follow...

However, I believe our Service object is ultimately coming from a ThreadSafeStore. 2 things to note about this class:

• Its documentation says not to modify anything returned by it
• All its accessors return a shallow copy of the cached item

I believe the second thing is the immediate cause of the issue which we hit in cloud-provider-openstack. Annotations, being a slice, are not copied. Therefore the annotations in the second invocation are the same as the first invocation.

Given that:

• the documentation says not to modify these items
• AFAIK it's not a bug for a reconciler to add an annotation to the object

I believe we need to deep copy the object returned by the informer before passing it to the sync loop.

This is a Feature Request

What would you like to be added The kubelet image credential provider (KEP-2133) documentation does not have any references as to where users can actually obtain the exec plugins. There is in fact only a single link provided, to the AWS ECR product page.

After conversation with @andrewsykim on Slack it appears there are at least two plugins available in the out-of-tree cloud provider repos (I have not personally validated that these implement the documented API): AWS: https://github.com/kubernetes/cloud-provider-aws/tree/master/cmd/ecr-credential-provider GCP: https://github.com/kubernetes/cloud-provider-gcp/tree/master/cmd/auth-provider-gcp Azure: https://github.com/kubernetes-sigs/cloud-provider-azure/issues/289

Why is this needed Absent any more specific documentation, users are left trying existing image credential providers such as https://github.com/awslabs/amazon-ecr-credential-helper which are docker-specific and WILL NOT interoperate with the kubelet credential API.

Comments

I observed this with cloud-provider-openstack, but I believe this is a framework bug not a provider bug. Specifically we had a permissions issue which caused this Patch to always fail: https://github.com/kubernetes/cloud-provider-openstack/blob/21ce6dd984e5599ba3c411c960acdbcf9b5dbf03/pkg/openstack/loadbalancer.go#L1783

However, completely deterministically this succeeds the second time it is called. There is no way for this reconcile to succeed without setting an annotation, but after the successful reconciliation we see that the annotation is not set. The only explanation I could muster for this is that we were called with the dirty object from the first reconciliation. Consequently the Patcher is based on a dirty object which already has the annotation set, even though we never succeeded to write that to the API, and therefore the Patcher believes second time round that it doesn't need to write the annotation and we erroneously succeed without having written the required annotation.

The obvious suspect was the cache used by the service controller, however for this specific call path we are always passing in a Service object which came from the informer.

...The informer code is hard to follow...

However, I believe our Service object is ultimately coming from a ThreadSafeStore. 2 things to note about this class:

• Its documentation says not to modify anything returned by it
• All its accessors return a shallow copy of the cached item

I believe the second thing is the immediate cause of the issue which we hit in cloud-provider-openstack. Annotations, being a slice, are not copied. Therefore the annotations in the second invocation are the same as the first invocation.

Given that:

• the documentation says not to modify these items
• AFAIK it's not a bug for a reconciler to add an annotation to the object

I believe we need to deep copy the object returned by the informer before passing it to the sync loop.

I observed this with cloud-provider-openstack, but I believe this is a framework bug not a provider bug. Specifically we had a permissions issue which caused this Patch to always fail: https://github.com/kubernetes/cloud-provider-openstack/blob/21ce6dd984e5599ba3c411c960acdbcf9b5dbf03/pkg/openstack/loadbalancer.go#L1783

However, completely deterministically this succeeds the second time it is called. There is no way for this reconcile to succeed without setting an annotation, but after the successful reconciliation we see that the annotation is not set. The only explanation I could muster for this is that we were called with the dirty object from the first reconciliation. Consequently the Patcher is based on a dirty object which already has the annotation set, even though we never succeeded to write that to the API, and therefore the Patcher believes second time round that it doesn't need to write the annotation and we erroneously succeed without having written the required annotation.

The obvious suspect was the cache used by the service controller, however for this specific call path we are always passing in a Service object which came from the informer.

...The informer code is hard to follow...

However, I believe our Service object is ultimately coming from a ThreadSafeStore. 2 things to note about this class:

• Its documentation says not to modify anything returned by it
• All its accessors return a shallow copy of the cached item

I believe the second thing is the immediate cause of the issue which we hit in cloud-provider-openstack. Annotations, being a slice, are not copied. Therefore the annotations in the second invocation are the same as the first invocation.

Given that:

• the documentation says not to modify these items
• AFAIK it's not a bug for a reconciler to add an annotation to the object

I believe we need to deep copy the object returned by the informer before passing it to the sync loop.

We have resources spanning 2 subnets A. a production subnet that has public IPs B. an experimental subnet that is not publicly addressable, and that sits behind a NAT

We have deployed a k8s cluster with its master node on A - following the guidance found in https://kubernetes.io/docs/tasks/extend-kubernetes/setup-konnectivity/

With our current setup it is possible for the nodes in B to join the cluster also we can spawn and delete pods and containers on the B side as well

However there are at least 2 functions that are problematic:

• kubectl logs pod-on-the-B-side if that pod runs on the B side (from either A or B)
• kubectl exec -ti pod-on-the-B-side -- /bin/bash if that pod runs on the B side (from either A or B)

in both cases after a timeout we get a message like this one

kubectl exec -ti the-pod -- /bin/bash
Error from server: error dialing backend: dial tcp 192.168.3.1:10250: i/o timeout

is this the expected behaviour, or else what can we have done wrong ?

We have resources spanning 2 subnets A. a production subnet that has public IPs B. an experimental subnet that is not publicly addressable, and that sits behind a NAT

We have deployed a k8s cluster with its master node on A - following the guidance found in https://kubernetes.io/docs/tasks/extend-kubernetes/setup-konnectivity/

With our current setup it is possible for the nodes in B to join the cluster also we can spawn and delete pods and containers on the B side as well

However there are at least 2 functions that are problematic:

• kubectl logs pod-on-the-B-side if that pod runs on the B side (from either A or B)
• kubectl exec -ti pod-on-the-B-side -- /bin/bash if that pod runs on the B side (from either A or B)

in both cases after a timeout we get a message like this one

kubectl exec -ti the-pod -- /bin/bash
Error from server: error dialing backend: dial tcp 192.168.3.1:10250: i/o timeout

is this the expected behaviour, or else what can we have done wrong ?

We have resources spanning 2 subnets A. a production subnet that has public IPs B. an experimental subnet that is not publicly addressable, and that sits behind a NAT

We have deployed a k8s cluster with its master node on A - following the guidance found in https://kubernetes.io/docs/tasks/extend-kubernetes/setup-konnectivity/

With our current setup it is possible for the nodes in B to join the cluster also we can spawn and delete pods and containers on the B side as well

However there are at least 2 functions that are problematic:

• kubectl logs pod-on-the-B-side if that pod runs on the B side (from either A or B)
• kubectl exec -ti pod-on-the-B-side -- /bin/bash if that pod runs on the B side (from either A or B)

in both cases after a timeout we get a message like this one

kubectl exec -ti the-pod -- /bin/bash
Error from server: error dialing backend: dial tcp 192.168.3.1:10250: i/o timeout

is this the expected behaviour, or else what can we have done wrong ?

We use externalTrafficPolicy "local" for our service and want to setup our balancer with members from service endpoints, but we got all cluster nodes as argument to EnsureLoadBalancer.

We are able to perform health check with port from service healthCheckNodePort, but all cluster nodes in NLB members looks like huge overhead.

Proposed solution: In case externalTrafficPolicy == "local", use nodes with service endpoints in argument to EnsureLoadBalancer. Call EnsureLoadBalancer (trigger event) on each service endpoints update (pod scaling, migration etc.)

We have resources spanning 2 subnets A. a production subnet that has public IPs B. an experimental subnet that is not publicly addressable, and that sits behind a NAT

We have deployed a k8s cluster with its master node on A - following the guidance found in https://kubernetes.io/docs/tasks/extend-kubernetes/setup-konnectivity/

With our current setup it is possible for the nodes in B to join the cluster also we can spawn and delete pods and containers on the B side as well

However there are at least 2 functions that are problematic:

• kubectl logs pod-on-the-B-side if that pod runs on the B side (from either A or B)
• kubectl exec -ti pod-on-the-B-side -- /bin/bash if that pod runs on the B side (from either A or B)

in both cases after a timeout we get a message like this one

kubectl exec -ti the-pod -- /bin/bash
Error from server: error dialing backend: dial tcp 192.168.3.1:10250: i/o timeout

is this the expected behaviour, or else what can we have done wrong ?

ci-kubernetes-e2e-gci-gce-alpha-enabled-default has been 🔴 for a while: https://testgrid.k8s.io/google-gce#gci-gce-alpha-enabled-default

Looking at kubelet logs we see: Apr 17 23:39:21.263389 bootstrap-e2e-master kubelet[2039]: Error: failed to run Kubelet: failed to create kubelet: cloud provider "gce" was specified, but built-in cloud providers are disabled. Please set --cloud-provider=external and migrate to an external cloud provider

(from https://storage.googleapis.com/kubernetes-jenkins/logs/ci-kubernetes-e2e-gci-gce-alpha-enabled-default/1515836418243432448/artifacts/bootstrap-e2e-master/kubelet.log)

So we gotta set DisableCloudProviders explicitly to false

Signed-off-by: Davanum Srinivas [email protected]

ci-kubernetes-e2e-gci-gce-alpha-enabled-default has been 🔴 for a while: https://testgrid.k8s.io/google-gce#gci-gce-alpha-enabled-default

Looking at kubelet logs we see: Apr 17 23:39:21.263389 bootstrap-e2e-master kubelet[2039]: Error: failed to run Kubelet: failed to create kubelet: cloud provider "gce" was specified, but built-in cloud providers are disabled. Please set --cloud-provider=external and migrate to an external cloud provider

(from https://storage.googleapis.com/kubernetes-jenkins/logs/ci-kubernetes-e2e-gci-gce-alpha-enabled-default/1515836418243432448/artifacts/bootstrap-e2e-master/kubelet.log)

So we gotta set DisableCloudProviders explicitly to false

Signed-off-by: Davanum Srinivas [email protected]

ci-kubernetes-e2e-gci-gce-alpha-enabled-default has been 🔴 for a while: https://testgrid.k8s.io/google-gce#gci-gce-alpha-enabled-default

Looking at kubelet logs we see: Apr 17 23:39:21.263389 bootstrap-e2e-master kubelet[2039]: Error: failed to run Kubelet: failed to create kubelet: cloud provider "gce" was specified, but built-in cloud providers are disabled. Please set --cloud-provider=external and migrate to an external cloud provider

(from https://storage.googleapis.com/kubernetes-jenkins/logs/ci-kubernetes-e2e-gci-gce-alpha-enabled-default/1515836418243432448/artifacts/bootstrap-e2e-master/kubelet.log)

So we gotta set DisableCloudProviders explicitly to false

Signed-off-by: Davanum Srinivas [email protected]

ci-kubernetes-e2e-gci-gce-alpha-enabled-default has been 🔴 for a while: https://testgrid.k8s.io/google-gce#gci-gce-alpha-enabled-default

Looking at kubelet logs we see: Apr 17 23:39:21.263389 bootstrap-e2e-master kubelet[2039]: Error: failed to run Kubelet: failed to create kubelet: cloud provider "gce" was specified, but built-in cloud providers are disabled. Please set --cloud-provider=external and migrate to an external cloud provider

(from https://storage.googleapis.com/kubernetes-jenkins/logs/ci-kubernetes-e2e-gci-gce-alpha-enabled-default/1515836418243432448/artifacts/bootstrap-e2e-master/kubelet.log)

So we gotta set DisableCloudProviders explicitly to false

Signed-off-by: Davanum Srinivas [email protected]

ci-kubernetes-e2e-gci-gce-alpha-enabled-default has been 🔴 for a while: https://testgrid.k8s.io/google-gce#gci-gce-alpha-enabled-default

Looking at kubelet logs we see: Apr 17 23:39:21.263389 bootstrap-e2e-master kubelet[2039]: Error: failed to run Kubelet: failed to create kubelet: cloud provider "gce" was specified, but built-in cloud providers are disabled. Please set --cloud-provider=external and migrate to an external cloud provider

(from https://storage.googleapis.com/kubernetes-jenkins/logs/ci-kubernetes-e2e-gci-gce-alpha-enabled-default/1515836418243432448/artifacts/bootstrap-e2e-master/kubelet.log)

So we gotta set DisableCloudProviders explicitly to false

Signed-off-by: Davanum Srinivas [email protected]

While exploring more on issue #276, it was noticed that the idle connections are somehow stuck in a weird state where the connection is established i.e the kube-apiserver receives a DIAL_RSP from the proxy-server, but then thereafter there is complete inactivity from the apiserver i.e no packets of PacketType_DATA are sent.

In order to reproduce this, I added some debug statements to the konnectivity-client library which is vendored in Kubernetes and built a local cluster.

This is the diff in the kubernetes repo to build a local cluster

diff --git a/hack/local-up-cluster.sh b/hack/local-up-cluster.sh
index abe85bb29fc..49b526ef080 100755
--- a/hack/local-up-cluster.sh
+++ b/hack/local-up-cluster.sh
@@ -553,7 +553,20 @@ kind: EgressSelectorConfiguration
 egressSelections:
 - name: cluster
   connection:
-    proxyProtocol: Direct
+    # This controls the protocol between the API Server and the Konnectivity
+    # server. Supported values are "GRPC" and "HTTPConnect". There is no
+    # end user visible difference between the two modes. You need to set the
+    # Konnectivity server to work in the same mode.
+    proxyProtocol: GRPC
+    transport:
+      # This controls what transport the API Server uses to communicate with the
+      # Konnectivity server. UDS is recommended if the Konnectivity server
+      # locates on the same machine as the API Server. You need to configure the
+      # Konnectivity server to listen on the same UDS socket.
+      # The other supported transport is "tcp". You will need to set up TLS
+      # config to secure the TCP transport.
+      uds:
+        udsName: /tmp/uds-proxy
 - name: controlplane
   connection:
     proxyProtocol: Direct
@@ -617,6 +630,7 @@ EOF
       --requestheader-allowed-names=system:auth-proxy \
       --proxy-client-cert-file="${CERT_DIR}/client-auth-proxy.crt" \
       --proxy-client-key-file="${CERT_DIR}/client-auth-proxy.key" \
+      --api-audiences=https://kubernetes.default.svc,system:konnectivity-server \
       --cors-allowed-origins="${API_CORS_ALLOWED_ORIGINS}" >"${APISERVER_LOG}" 2>&1 &
     APISERVER_PID=$!

diff --git a/vendor/sigs.k8s.io/apiserver-network-proxy/konnectivity-client/pkg/client/client.go b/vendor/sigs.k8s.io/apiserver-network-proxy/konnectivity-client/pkg/client/client.go
index 31bf9e11fe5..82880b2f997 100644
--- a/vendor/sigs.k8s.io/apiserver-network-proxy/konnectivity-client/pkg/client/client.go
+++ b/vendor/sigs.k8s.io/apiserver-network-proxy/konnectivity-client/pkg/client/client.go
@@ -109,6 +109,7 @@ func (t *grpcTunnel) serve(c clientConn) {
 		switch pkt.Type {
 		case client.PacketType_DIAL_RSP:
 			resp := pkt.GetDialResponse()
+			fmt.Println("DEBUG - Receive PacketType_DIAL_RSP for connection ID=", resp.ConnectID)
 			t.pendingDialLock.RLock()
 			ch, ok := t.pendingDial[resp.Random]
 			t.pendingDialLock.RUnlock()
@@ -120,7 +121,7 @@ func (t *grpcTunnel) serve(c clientConn) {
 					err:    resp.Error,
 					connid: resp.ConnectID,
 				}
-				select  {
+				select {
 				case ch <- result:
 				default:
 					klog.ErrorS(fmt.Errorf("blocked pending channel"), "Received second dial response for connection request", "connectionID", resp.ConnectID, "dialID", resp.Random)
@@ -136,6 +137,7 @@ func (t *grpcTunnel) serve(c clientConn) {

 		case client.PacketType_DATA:
 			resp := pkt.GetData()
+			fmt.Println("DEBUG - Receive PacketType_DATA for connection ID=", resp.ConnectID)
 			// TODO: flow control
 			t.connsLock.RLock()
 			conn, ok := t.conns[resp.ConnectID]
@@ -155,6 +157,7 @@ func (t *grpcTunnel) serve(c clientConn) {
 			}
 		case client.PacketType_CLOSE_RSP:
 			resp := pkt.GetCloseResponse()
+			fmt.Println("DEBUG - Receive PacketType_CLOSE_RSP for connection ID=", resp.ConnectID)
 			t.connsLock.RLock()
 			conn, ok := t.conns[resp.ConnectID]
 			t.connsLock.RUnlock()
@@ -211,7 +214,7 @@ func (t *grpcTunnel) DialContext(ctx context.Context, protocol, address string)
 	klog.V(5).Infoln("DIAL_REQ sent to proxy server")

 	c := &conn{stream: t.stream, random: random}
-
+	fmt.Println("DEBUG - Send PacketType_DIAL_REQ")
 	select {
 	case res := <-resCh:
 		if res.err != "" {
diff --git a/vendor/sigs.k8s.io/apiserver-network-proxy/konnectivity-client/pkg/client/conn.go b/vendor/sigs.k8s.io/apiserver-network-proxy/konnectivity-client/pkg/client/conn.go
index cc6e66be237..acf0ee7ce61 100644
--- a/vendor/sigs.k8s.io/apiserver-network-proxy/konnectivity-client/pkg/client/conn.go
+++ b/vendor/sigs.k8s.io/apiserver-network-proxy/konnectivity-client/pkg/client/conn.go
@@ -18,6 +18,7 @@ package client

 import (
 	"errors"
+	"fmt"
 	"io"
 	"net"
 	"time"
@@ -61,6 +62,7 @@ func (c *conn) Write(data []byte) (n int, err error) {
 	if err != nil {
 		return 0, err
 	}
+	fmt.Println("DEBUG - Send PacketType_DATA for connection ID=", c.connID)
 	return len(data), err
 }

@@ -116,6 +118,7 @@ func (c *conn) Close() error {
 	klog.V(4).Infoln("closing connection")
 	var req *client.Packet
 	if c.connID != 0 {
+		fmt.Println("DEBUG - Send PacketType_CLOSE_REQ for connection ID=", c.connID)
 		req = &client.Packet{
 			Type: client.PacketType_CLOSE_REQ,
 			Payload: &client.Packet_CloseRequest{

Once the local cluster is up with ./hack/local-up-cluster.sh, you can start the proxy-server and agent as local processes in separate terminals with:

make clean && make certs && make gen && make build

./bin/proxy-server --server-port=0 --uds-name=/tmp/uds-proxy --cluster-ca-cert=certs/agent/issued/ca.crt --cluster-cert=certs/agent/issued/proxy-frontend.crt --cluster-key=certs/agent/private/proxy-frontend.key --warn-on-channel-limit --v 5 --keepalive-time 1m --frontend-keepalive-time 1m  --enable-profiling 2>&1 | tee server.log

./bin/proxy-agent --ca-cert=certs/agent/issued/ca.crt --agent-cert=certs/agent/issued/proxy-agent.crt --agent-key=certs/agent/private/proxy-agent.key

Stress the cluster, with concurrent kubectl log requests

Create some workloads

kubectl create deployment nginx --image nginx --replicas 20

get-logs.sh script:

#!/bin/bash

NAMESPACE=${NAMESPACE:-"default"}
pods=$(kubectl get pods -n ${NAMESPACE} --no-headers | awk '{print $1}')
FLOOR=1;
arr=(`echo ${pods}`);
CEILING=${#arr[@]};

for i in {1..3}
do
   RANGE=$(($CEILING-$FLOOR));
   random=$RANDOM;
   let "random %= $RANGE";
   random=$(($random+$FLOOR));
   kubectl logs -n ${NAMESPACE} ${arr[${random}]} > /dev/null 2> /dev/null
done

run.sh script:

#!/bin/bash

set -euo pipefail

for i in {1..500}
do
    ./get-logs.sh &
done

Execute the run.sh script after saving and giving execute permissions:

./run.sh

To verify the above hypothesis, I've created a small script, that prints those connection IDs which are idle i.e CLOSE_REQ has not been sent by the api-server and hence as a result never received a CLOSE_RSP from the proxy-server.

check.sh script:

#!/bin/bash

set -eou pipefail

total_connection_ids=$(rg "DEBUG - Send PacketType_DIAL_REQ" /tmp/kube-apiserver.log | wc -l)
for connection_id in $(seq 1 $total_connection_ids)
do
  close_req="DEBUG - Send PacketType_CLOSE_REQ for connection ID= ${connection_id}"
  close_rsp="DEBUG - Receive PacketType_CLOSE_RSP for connection ID= ${connection_id}"

  req_status=0
  set +e
  rg "$close_req\$" /tmp/kube-apiserver.log > /dev/null || req_status=1

  res_status=0
  rg "$close_rsp\$" /tmp/kube-apiserver.log > /dev/null || res_status=1

  set -e

  if [[ ( $req_status == 1 && $res_status == 1 ) ]]; then
    echo "Both missing for $connection_id"
  fi

  if [[ ( $req_status == 0 && $res_status == 1 ) ]]; then
    echo "CloseResponse missing for $connection_id"
  fi

  if [[ ( $req_status == 1 && $res_status == 0 ) ]]; then
    echo "Interesting case for $connection_id"
  fi
done

We are interested in the case of Both missing from $connection_id. All of the connection ids, that are printed for this case if cross-checked against the proxy-server logs have only the following logs snippet, example connectionID=505

I1215 16:36:06.122100   15067 server.go:714] "Received DIAL_RSP" dialID=4406796995779167332 agentID="f902ce27-b9b6-44a6-aef7-cfad5aff3611" connectionID=505
I1215 16:36:06.122132   15067 server.go:269] "Register frontend for agent" frontend=&{Mode:grpc Grpc:0xc000ee5f80 HTTP:<nil> CloseHTTP:<nil> connected:0xc000c58840 connectID:505 agentID:f902ce27-b9b6-44a6-aef7-cfad5aff3611 start:{wall:13863927826982877159 ext:149658304691 loc:0x212d6e0} backend:0xc000616108} agentID="f902ce27-b9b6-44a6-aef7-cfad5aff3611" connectionID=505

You can also verify if PacketType_DATA is sent by the apiserver for the same connection id

grep "DEBUG - Send PacketType_DATA for connection ID=" /tmp/kube-apiserver.log | grep 505

The problematic aspect of this is that in order to close/terminate these idle connections, we need connectionID which is only received when DIAL_RSP is sent by the agent and received by proxy-server and when PacketType_DATA is sent by apiserver to proxy-server.

/cc @iaguis @rata

What type of PR is this?

Uncomment only one /kind <> line, hit enter to put that in a new line, and remove leading whitespaces from that line:

/kind api-change /kind bug /kind cleanup /kind design /kind documentation /kind failing-test /kind feature /kind flake

What this PR does / why we need it: In ipvs mode, when service deleted but rs(pod) exist, related ipvs rule and address bound in ipvs0 will not be deleted. I think it's unreasonable that kubernetes-service deleted but the service ip is available(ipvs rule not deleted). When you create same service again, new virtual service(totally two ipvs virtual service) will proxy to the same rs, too. And the deleted service ip will exist on both ipvs rule and ipvs0 device.

Which issue(s) this PR fixes:

Fixes #

Special notes for your reviewer:

Does this PR introduce a user-facing change?:

Fix a bug in the IPVS proxier where virtual servers are not cleaned up even though the corresponding Service object was deleted.

Additional documentation e.g., KEPs (Kubernetes Enhancement Proposals), usage docs, etc.:

Signed-off-by: Andrew Sy Kim [email protected]

Our lint check in pre-submit is failing with the following error:

level=error msg="Running error: buildir: failed to load package goarch: could not load export data: cannot import \"internal/goarch\" (unknown iexport format version 2), export data is newer version - update tool"

e.g. https://prow.k8s.io/view/gs/kubernetes-jenkins/pr-logs/pull/kubernetes-sigs_apiserver-network-proxy/341/pull-apiserver-network-proxy-make-lint/1514261980213415936

I suspect this is because our prow jobs using the kubekins image was updated to use go v1.18 while we're still on go v1.17. Seems like we can fix this by eiher updating to go 1.18 as well or reverting the recent image updates.

Our lint check in pre-submit is failing with the following error:

level=error msg="Running error: buildir: failed to load package goarch: could not load export data: cannot import \"internal/goarch\" (unknown iexport format version 2), export data is newer version - update tool"

e.g. https://prow.k8s.io/view/gs/kubernetes-jenkins/pr-logs/pull/kubernetes-sigs_apiserver-network-proxy/341/pull-apiserver-network-proxy-make-lint/1514261980213415936

I suspect this is because our prow jobs using the kubekins image was updated to use go v1.18 while we're still on go v1.17. Seems like we can fix this by eiher updating to go 1.18 as well or reverting the recent image updates.

This is a follow-up PR to https://github.com/kubernetes-sigs/apiserver-network-proxy/pull/317, where we reduced error logs whenever reading from a closed network connection. This PR introduces a similar fix but also from the writer.

This is a follow-up PR to https://github.com/kubernetes-sigs/apiserver-network-proxy/pull/317, where we reduced error logs whenever reading from a closed network connection. This PR introduces a similar fix but also from the writer.

This is a follow-up PR to https://github.com/kubernetes-sigs/apiserver-network-proxy/pull/317, where we reduced error logs whenever reading from a closed network connection. This PR introduces a similar fix but also from the writer.

This is a follow-up PR to https://github.com/kubernetes-sigs/apiserver-network-proxy/pull/317, where we reduced error logs whenever reading from a closed network connection. This PR introduces a similar fix but also from the writer.