andrewsykim

andrewsykim

Member Since 6 years ago

Google, Canada

Experience Points
272
follower
Lessons Completed
3
follow
Lessons Completed
114
stars
Best Reply Awards
69
repos

699 contributions in the last year

Pinned
⚡ Production-Grade Container Scheduling and Management
⚡ Linux kernel source tree
Activity
Apr
28
3 weeks ago
Activity icon
issue

andrewsykim issue kubernetes/enhancements

andrewsykim
andrewsykim

Tracking Terminating Endpoints

Enhancement Description

Please to keep this description up to date. This will help the Enhancement Team track efficiently the evolution of the enhancement

Activity icon
issue

andrewsykim issue comment kubernetes/enhancements

andrewsykim
andrewsykim

Tracking Terminating Endpoints

Enhancement Description

Please to keep this description up to date. This will help the Enhancement Team track efficiently the evolution of the enhancement

andrewsykim
andrewsykim

oops sorry, I commented on the wrong KEP

Activity icon
issue

andrewsykim issue comment kubernetes/enhancements

andrewsykim
andrewsykim

Tracking Terminating Endpoints

Enhancement Description

Please to keep this description up to date. This will help the Enhancement Team track efficiently the evolution of the enhancement

andrewsykim
andrewsykim

Closing since this feature went GA in v1.24

/stage stable

Activity icon
issue

andrewsykim issue kubernetes/enhancements

andrewsykim
andrewsykim

Tracking Terminating Endpoints

Enhancement Description

Please to keep this description up to date. This will help the Enhancement Team track efficiently the evolution of the enhancement

Apr
25
4 weeks ago
Activity icon
issue

andrewsykim issue comment kubernetes/cloud-provider

andrewsykim
andrewsykim

Service Controller can call provider EnsureLoadBalancer with a dirty Service object

I observed this with cloud-provider-openstack, but I believe this is a framework bug not a provider bug. Specifically we had a permissions issue which caused this Patch to always fail: https://github.com/kubernetes/cloud-provider-openstack/blob/21ce6dd984e5599ba3c411c960acdbcf9b5dbf03/pkg/openstack/loadbalancer.go#L1783

However, completely deterministically this succeeds the second time it is called. There is no way for this reconcile to succeed without setting an annotation, but after the successful reconciliation we see that the annotation is not set. The only explanation I could muster for this is that we were called with the dirty object from the first reconciliation. Consequently the Patcher is based on a dirty object which already has the annotation set, even though we never succeeded to write that to the API, and therefore the Patcher believes second time round that it doesn't need to write the annotation and we erroneously succeed without having written the required annotation.

The obvious suspect was the cache used by the service controller, however for this specific call path we are always passing in a Service object which came from the informer.

...The informer code is hard to follow...

However, I believe our Service object is ultimately coming from a ThreadSafeStore. 2 things to note about this class:

I believe the second thing is the immediate cause of the issue which we hit in cloud-provider-openstack. Annotations, being a slice, are not copied. Therefore the annotations in the second invocation are the same as the first invocation.

Given that:

  • the documentation says not to modify these items
  • AFAIK it's not a bug for a reconciler to add an annotation to the object

I believe we need to deep copy the object returned by the informer before passing it to the sync loop.

andrewsykim
andrewsykim

I think up to this point it was implied that the Service object passed from the controller should be read-only but agreed that passing a deep copy is probably the safer thing to do since it's not always obvious

Apr
21
1 month ago
Activity icon
issue

andrewsykim issue comment kubernetes/website

andrewsykim
andrewsykim

Kubelet image credential provider docs should provide information on where to find exec plugin binaries

This is a Feature Request

What would you like to be added The kubelet image credential provider (KEP-2133) documentation does not have any references as to where users can actually obtain the exec plugins. There is in fact only a single link provided, to the AWS ECR product page.

After conversation with @andrewsykim on Slack it appears there are at least two plugins available in the out-of-tree cloud provider repos (I have not personally validated that these implement the documented API): AWS: https://github.com/kubernetes/cloud-provider-aws/tree/master/cmd/ecr-credential-provider GCP: https://github.com/kubernetes/cloud-provider-gcp/tree/master/cmd/auth-provider-gcp Azure: https://github.com/kubernetes-sigs/cloud-provider-azure/issues/289

Why is this needed Absent any more specific documentation, users are left trying existing image credential providers such as https://github.com/awslabs/amazon-ecr-credential-helper which are docker-specific and WILL NOT interoperate with the kubelet credential API.

Comments

andrewsykim
andrewsykim

A documented list of plugins sounds like a good idea, we can probably have a subsection in the existing kubelet credential provider docs as well

Activity icon
issue

andrewsykim issue comment kubernetes/cloud-provider

andrewsykim
andrewsykim

Service Controller can call provider EnsureLoadBalancer with a dirty Service object

I observed this with cloud-provider-openstack, but I believe this is a framework bug not a provider bug. Specifically we had a permissions issue which caused this Patch to always fail: https://github.com/kubernetes/cloud-provider-openstack/blob/21ce6dd984e5599ba3c411c960acdbcf9b5dbf03/pkg/openstack/loadbalancer.go#L1783

However, completely deterministically this succeeds the second time it is called. There is no way for this reconcile to succeed without setting an annotation, but after the successful reconciliation we see that the annotation is not set. The only explanation I could muster for this is that we were called with the dirty object from the first reconciliation. Consequently the Patcher is based on a dirty object which already has the annotation set, even though we never succeeded to write that to the API, and therefore the Patcher believes second time round that it doesn't need to write the annotation and we erroneously succeed without having written the required annotation.

The obvious suspect was the cache used by the service controller, however for this specific call path we are always passing in a Service object which came from the informer.

...The informer code is hard to follow...

However, I believe our Service object is ultimately coming from a ThreadSafeStore. 2 things to note about this class:

I believe the second thing is the immediate cause of the issue which we hit in cloud-provider-openstack. Annotations, being a slice, are not copied. Therefore the annotations in the second invocation are the same as the first invocation.

Given that:

  • the documentation says not to modify these items
  • AFAIK it's not a bug for a reconciler to add an annotation to the object

I believe we need to deep copy the object returned by the informer before passing it to the sync loop.

andrewsykim
andrewsykim

It's possible that bug fix would address your problem though? But there's a follow-up to be had where we remove the cache as you suggested

Activity icon
issue

andrewsykim issue comment kubernetes/cloud-provider

andrewsykim
andrewsykim

Service Controller can call provider EnsureLoadBalancer with a dirty Service object

I observed this with cloud-provider-openstack, but I believe this is a framework bug not a provider bug. Specifically we had a permissions issue which caused this Patch to always fail: https://github.com/kubernetes/cloud-provider-openstack/blob/21ce6dd984e5599ba3c411c960acdbcf9b5dbf03/pkg/openstack/loadbalancer.go#L1783

However, completely deterministically this succeeds the second time it is called. There is no way for this reconcile to succeed without setting an annotation, but after the successful reconciliation we see that the annotation is not set. The only explanation I could muster for this is that we were called with the dirty object from the first reconciliation. Consequently the Patcher is based on a dirty object which already has the annotation set, even though we never succeeded to write that to the API, and therefore the Patcher believes second time round that it doesn't need to write the annotation and we erroneously succeed without having written the required annotation.

The obvious suspect was the cache used by the service controller, however for this specific call path we are always passing in a Service object which came from the informer.

...The informer code is hard to follow...

However, I believe our Service object is ultimately coming from a ThreadSafeStore. 2 things to note about this class:

I believe the second thing is the immediate cause of the issue which we hit in cloud-provider-openstack. Annotations, being a slice, are not copied. Therefore the annotations in the second invocation are the same as the first invocation.

Given that:

  • the documentation says not to modify these items
  • AFAIK it's not a bug for a reconciler to add an annotation to the object

I believe we need to deep copy the object returned by the informer before passing it to the sync loop.

Activity icon
issue

andrewsykim issue comment kubernetes-sigs/apiserver-network-proxy

andrewsykim
andrewsykim

struggling with konnectivity

We have resources spanning 2 subnets A. a production subnet that has public IPs B. an experimental subnet that is not publicly addressable, and that sits behind a NAT

We have deployed a k8s cluster with its master node on A - following the guidance found in https://kubernetes.io/docs/tasks/extend-kubernetes/setup-konnectivity/

With our current setup it is possible for the nodes in B to join the cluster also we can spawn and delete pods and containers on the B side as well

However there are at least 2 functions that are problematic:

  • kubectl logs pod-on-the-B-side if that pod runs on the B side (from either A or B)
  • kubectl exec -ti pod-on-the-B-side -- /bin/bash if that pod runs on the B side (from either A or B)

in both cases after a timeout we get a message like this one

kubectl exec -ti the-pod -- /bin/bash
Error from server: error dialing backend: dial tcp 192.168.3.1:10250: i/o timeout

is this the expected behaviour, or else what can we have done wrong ?

andrewsykim
andrewsykim

So konnectivity doesn't actually overlap with CNI in anyway. So if you see FailedCreatePodSandBox errors on the kiada-fit01-pod pod, it seems unrelated to the network proxy and something to do with the CNI configuration of the cluster.

It seems like you do have pods that are working well though, is it possible that some nodes are misconfigured but others are not? Is kiada-fit01-pod on a different node than the other pods?

Activity icon
issue

andrewsykim issue comment kubernetes-sigs/apiserver-network-proxy

andrewsykim
andrewsykim

struggling with konnectivity

We have resources spanning 2 subnets A. a production subnet that has public IPs B. an experimental subnet that is not publicly addressable, and that sits behind a NAT

We have deployed a k8s cluster with its master node on A - following the guidance found in https://kubernetes.io/docs/tasks/extend-kubernetes/setup-konnectivity/

With our current setup it is possible for the nodes in B to join the cluster also we can spawn and delete pods and containers on the B side as well

However there are at least 2 functions that are problematic:

  • kubectl logs pod-on-the-B-side if that pod runs on the B side (from either A or B)
  • kubectl exec -ti pod-on-the-B-side -- /bin/bash if that pod runs on the B side (from either A or B)

in both cases after a timeout we get a message like this one

kubectl exec -ti the-pod -- /bin/bash
Error from server: error dialing backend: dial tcp 192.168.3.1:10250: i/o timeout

is this the expected behaviour, or else what can we have done wrong ?

andrewsykim
andrewsykim

Is it possible you have two CNIs configured in your cluster by accident?

Activity icon
issue

andrewsykim issue comment kubernetes-sigs/apiserver-network-proxy

andrewsykim
andrewsykim

struggling with konnectivity

We have resources spanning 2 subnets A. a production subnet that has public IPs B. an experimental subnet that is not publicly addressable, and that sits behind a NAT

We have deployed a k8s cluster with its master node on A - following the guidance found in https://kubernetes.io/docs/tasks/extend-kubernetes/setup-konnectivity/

With our current setup it is possible for the nodes in B to join the cluster also we can spawn and delete pods and containers on the B side as well

However there are at least 2 functions that are problematic:

  • kubectl logs pod-on-the-B-side if that pod runs on the B side (from either A or B)
  • kubectl exec -ti pod-on-the-B-side -- /bin/bash if that pod runs on the B side (from either A or B)

in both cases after a timeout we get a message like this one

kubectl exec -ti the-pod -- /bin/bash
Error from server: error dialing backend: dial tcp 192.168.3.1:10250: i/o timeout

is this the expected behaviour, or else what can we have done wrong ?

andrewsykim
andrewsykim

failed to delegate add: failed to set bridge addr: "cni0" already has an IP address different from 10.244.1.1/24

This seems like an issue with your CNI setup. What CNI are you using?

Apr
20
1 month ago
Activity icon
issue

andrewsykim issue comment kubernetes/cloud-provider

andrewsykim
andrewsykim

LoadBalancer controller: nodes listing with externalTrafficPolicy == "local"

We use externalTrafficPolicy "local" for our service and want to setup our balancer with members from service endpoints, but we got all cluster nodes as argument to EnsureLoadBalancer.

We are able to perform health check with port from service healthCheckNodePort, but all cluster nodes in NLB members looks like huge overhead.

Proposed solution: In case externalTrafficPolicy == "local", use nodes with service endpoints in argument to EnsureLoadBalancer. Call EnsureLoadBalancer (trigger event) on each service endpoints update (pod scaling, migration etc.)

andrewsykim
andrewsykim

We are able to perform health check with port from service healthCheckNodePort, but all cluster nodes in NLB members looks like huge overhead.

This is expected behavior -- as you mentioned, using the healthCheckNodePort ensures only nodes with the endpoint is being used by the LB

In case externalTrafficPolicy == "local", use nodes with service endpoints in argument to EnsureLoadBalancer. Call EnsureLoadBalancer (trigger event) on each service endpoints update (pod scaling, migration etc.)

My gut feeling is that this would be too costly. Services that have a lot of endpoints would churn a lot during a rolling update of the pods. Trying to add/remove backend nodes for an LB would make a lot of calls to the cloud provider and it's also possible that the cloud provider is not always able to add/remove backends as quickly as kube-proxy would. But we can make fairly safe assumptions that health check failures from LBs are responded to quickly as they are designed for this type of failure.

Activity icon
issue

andrewsykim issue comment kubernetes-sigs/apiserver-network-proxy

andrewsykim
andrewsykim

struggling with konnectivity

We have resources spanning 2 subnets A. a production subnet that has public IPs B. an experimental subnet that is not publicly addressable, and that sits behind a NAT

We have deployed a k8s cluster with its master node on A - following the guidance found in https://kubernetes.io/docs/tasks/extend-kubernetes/setup-konnectivity/

With our current setup it is possible for the nodes in B to join the cluster also we can spawn and delete pods and containers on the B side as well

However there are at least 2 functions that are problematic:

  • kubectl logs pod-on-the-B-side if that pod runs on the B side (from either A or B)
  • kubectl exec -ti pod-on-the-B-side -- /bin/bash if that pod runs on the B side (from either A or B)

in both cases after a timeout we get a message like this one

kubectl exec -ti the-pod -- /bin/bash
Error from server: error dialing backend: dial tcp 192.168.3.1:10250: i/o timeout

is this the expected behaviour, or else what can we have done wrong ?

andrewsykim
andrewsykim

Error from server: error dialing backend: dial tcp 192.168.3.1:10250: i/o timeout

From this error, it seems like the konnectivity agent on B is not able to connect to any of the nodes on B. Kubernetes' log functionality goes through kubelet which is 192.168.3.1:10250. Can you confirm that networking works between the pod and the node by running ping?

Apr
18
1 month ago
close pull request

andrewsykim wants to merge kubernetes/test-infra

andrewsykim
andrewsykim

specify DisableCloudProviders=false for gci-gce-alpha-enabled-default job

ci-kubernetes-e2e-gci-gce-alpha-enabled-default has been 🔴 for a while: https://testgrid.k8s.io/google-gce#gci-gce-alpha-enabled-default

Looking at kubelet logs we see: Apr 17 23:39:21.263389 bootstrap-e2e-master kubelet[2039]: Error: failed to run Kubelet: failed to create kubelet: cloud provider "gce" was specified, but built-in cloud providers are disabled. Please set --cloud-provider=external and migrate to an external cloud provider

(from https://storage.googleapis.com/kubernetes-jenkins/logs/ci-kubernetes-e2e-gci-gce-alpha-enabled-default/1515836418243432448/artifacts/bootstrap-e2e-master/kubelet.log)

So we gotta set DisableCloudProviders explicitly to false

Signed-off-by: Davanum Srinivas [email protected]

andrewsykim
andrewsykim

Only really an issue if this job runs tests pulling a private image

pull request

andrewsykim merge to kubernetes/test-infra

andrewsykim
andrewsykim

specify DisableCloudProviders=false for gci-gce-alpha-enabled-default job

ci-kubernetes-e2e-gci-gce-alpha-enabled-default has been 🔴 for a while: https://testgrid.k8s.io/google-gce#gci-gce-alpha-enabled-default

Looking at kubelet logs we see: Apr 17 23:39:21.263389 bootstrap-e2e-master kubelet[2039]: Error: failed to run Kubelet: failed to create kubelet: cloud provider "gce" was specified, but built-in cloud providers are disabled. Please set --cloud-provider=external and migrate to an external cloud provider

(from https://storage.googleapis.com/kubernetes-jenkins/logs/ci-kubernetes-e2e-gci-gce-alpha-enabled-default/1515836418243432448/artifacts/bootstrap-e2e-master/kubelet.log)

So we gotta set DisableCloudProviders explicitly to false

Signed-off-by: Davanum Srinivas [email protected]

close pull request

andrewsykim wants to merge kubernetes/test-infra

andrewsykim
andrewsykim

specify DisableCloudProviders=false for gci-gce-alpha-enabled-default job

ci-kubernetes-e2e-gci-gce-alpha-enabled-default has been 🔴 for a while: https://testgrid.k8s.io/google-gce#gci-gce-alpha-enabled-default

Looking at kubelet logs we see: Apr 17 23:39:21.263389 bootstrap-e2e-master kubelet[2039]: Error: failed to run Kubelet: failed to create kubelet: cloud provider "gce" was specified, but built-in cloud providers are disabled. Please set --cloud-provider=external and migrate to an external cloud provider

(from https://storage.googleapis.com/kubernetes-jenkins/logs/ci-kubernetes-e2e-gci-gce-alpha-enabled-default/1515836418243432448/artifacts/bootstrap-e2e-master/kubelet.log)

So we gotta set DisableCloudProviders explicitly to false

Signed-off-by: Davanum Srinivas [email protected]

andrewsykim
andrewsykim

We probably also want DisableKubeletCloudCredentialProviders=false in here until the prow jobs are configured w/ an external plugin

pull request

andrewsykim merge to kubernetes/test-infra

andrewsykim
andrewsykim

specify DisableCloudProviders=false for gci-gce-alpha-enabled-default job

ci-kubernetes-e2e-gci-gce-alpha-enabled-default has been 🔴 for a while: https://testgrid.k8s.io/google-gce#gci-gce-alpha-enabled-default

Looking at kubelet logs we see: Apr 17 23:39:21.263389 bootstrap-e2e-master kubelet[2039]: Error: failed to run Kubelet: failed to create kubelet: cloud provider "gce" was specified, but built-in cloud providers are disabled. Please set --cloud-provider=external and migrate to an external cloud provider

(from https://storage.googleapis.com/kubernetes-jenkins/logs/ci-kubernetes-e2e-gci-gce-alpha-enabled-default/1515836418243432448/artifacts/bootstrap-e2e-master/kubelet.log)

So we gotta set DisableCloudProviders explicitly to false

Signed-off-by: Davanum Srinivas [email protected]

pull request

andrewsykim merge to kubernetes/test-infra

andrewsykim
andrewsykim

specify DisableCloudProviders=false for gci-gce-alpha-enabled-default job

ci-kubernetes-e2e-gci-gce-alpha-enabled-default has been 🔴 for a while: https://testgrid.k8s.io/google-gce#gci-gce-alpha-enabled-default

Looking at kubelet logs we see: Apr 17 23:39:21.263389 bootstrap-e2e-master kubelet[2039]: Error: failed to run Kubelet: failed to create kubelet: cloud provider "gce" was specified, but built-in cloud providers are disabled. Please set --cloud-provider=external and migrate to an external cloud provider

(from https://storage.googleapis.com/kubernetes-jenkins/logs/ci-kubernetes-e2e-gci-gce-alpha-enabled-default/1515836418243432448/artifacts/bootstrap-e2e-master/kubelet.log)

So we gotta set DisableCloudProviders explicitly to false

Signed-off-by: Davanum Srinivas [email protected]

Activity icon
issue

andrewsykim issue comment kubernetes-sigs/apiserver-network-proxy

andrewsykim
andrewsykim

Idle connections are stuck after establishing connection. They never send PacketType_DATA

While exploring more on issue #276, it was noticed that the idle connections are somehow stuck in a weird state where the connection is established i.e the kube-apiserver receives a DIAL_RSP from the proxy-server, but then thereafter there is complete inactivity from the apiserver i.e no packets of PacketType_DATA are sent.

In order to reproduce this, I added some debug statements to the konnectivity-client library which is vendored in Kubernetes and built a local cluster.

This is the diff in the kubernetes repo to build a local cluster

diff --git a/hack/local-up-cluster.sh b/hack/local-up-cluster.sh
index abe85bb29fc..49b526ef080 100755
--- a/hack/local-up-cluster.sh
+++ b/hack/local-up-cluster.sh
@@ -553,7 +553,20 @@ kind: EgressSelectorConfiguration
 egressSelections:
 - name: cluster
   connection:
-    proxyProtocol: Direct
+    # This controls the protocol between the API Server and the Konnectivity
+    # server. Supported values are "GRPC" and "HTTPConnect". There is no
+    # end user visible difference between the two modes. You need to set the
+    # Konnectivity server to work in the same mode.
+    proxyProtocol: GRPC
+    transport:
+      # This controls what transport the API Server uses to communicate with the
+      # Konnectivity server. UDS is recommended if the Konnectivity server
+      # locates on the same machine as the API Server. You need to configure the
+      # Konnectivity server to listen on the same UDS socket.
+      # The other supported transport is "tcp". You will need to set up TLS
+      # config to secure the TCP transport.
+      uds:
+        udsName: /tmp/uds-proxy
 - name: controlplane
   connection:
     proxyProtocol: Direct
@@ -617,6 +630,7 @@ EOF
       --requestheader-allowed-names=system:auth-proxy \
       --proxy-client-cert-file="${CERT_DIR}/client-auth-proxy.crt" \
       --proxy-client-key-file="${CERT_DIR}/client-auth-proxy.key" \
+      --api-audiences=https://kubernetes.default.svc,system:konnectivity-server \
       --cors-allowed-origins="${API_CORS_ALLOWED_ORIGINS}" >"${APISERVER_LOG}" 2>&1 &
     APISERVER_PID=$!

diff --git a/vendor/sigs.k8s.io/apiserver-network-proxy/konnectivity-client/pkg/client/client.go b/vendor/sigs.k8s.io/apiserver-network-proxy/konnectivity-client/pkg/client/client.go
index 31bf9e11fe5..82880b2f997 100644
--- a/vendor/sigs.k8s.io/apiserver-network-proxy/konnectivity-client/pkg/client/client.go
+++ b/vendor/sigs.k8s.io/apiserver-network-proxy/konnectivity-client/pkg/client/client.go
@@ -109,6 +109,7 @@ func (t *grpcTunnel) serve(c clientConn) {
 		switch pkt.Type {
 		case client.PacketType_DIAL_RSP:
 			resp := pkt.GetDialResponse()
+			fmt.Println("DEBUG - Receive PacketType_DIAL_RSP for connection ID=", resp.ConnectID)
 			t.pendingDialLock.RLock()
 			ch, ok := t.pendingDial[resp.Random]
 			t.pendingDialLock.RUnlock()
@@ -120,7 +121,7 @@ func (t *grpcTunnel) serve(c clientConn) {
 					err:    resp.Error,
 					connid: resp.ConnectID,
 				}
-				select  {
+				select {
 				case ch <- result:
 				default:
 					klog.ErrorS(fmt.Errorf("blocked pending channel"), "Received second dial response for connection request", "connectionID", resp.ConnectID, "dialID", resp.Random)
@@ -136,6 +137,7 @@ func (t *grpcTunnel) serve(c clientConn) {

 		case client.PacketType_DATA:
 			resp := pkt.GetData()
+			fmt.Println("DEBUG - Receive PacketType_DATA for connection ID=", resp.ConnectID)
 			// TODO: flow control
 			t.connsLock.RLock()
 			conn, ok := t.conns[resp.ConnectID]
@@ -155,6 +157,7 @@ func (t *grpcTunnel) serve(c clientConn) {
 			}
 		case client.PacketType_CLOSE_RSP:
 			resp := pkt.GetCloseResponse()
+			fmt.Println("DEBUG - Receive PacketType_CLOSE_RSP for connection ID=", resp.ConnectID)
 			t.connsLock.RLock()
 			conn, ok := t.conns[resp.ConnectID]
 			t.connsLock.RUnlock()
@@ -211,7 +214,7 @@ func (t *grpcTunnel) DialContext(ctx context.Context, protocol, address string)
 	klog.V(5).Infoln("DIAL_REQ sent to proxy server")

 	c := &conn{stream: t.stream, random: random}
-
+	fmt.Println("DEBUG - Send PacketType_DIAL_REQ")
 	select {
 	case res := <-resCh:
 		if res.err != "" {
diff --git a/vendor/sigs.k8s.io/apiserver-network-proxy/konnectivity-client/pkg/client/conn.go b/vendor/sigs.k8s.io/apiserver-network-proxy/konnectivity-client/pkg/client/conn.go
index cc6e66be237..acf0ee7ce61 100644
--- a/vendor/sigs.k8s.io/apiserver-network-proxy/konnectivity-client/pkg/client/conn.go
+++ b/vendor/sigs.k8s.io/apiserver-network-proxy/konnectivity-client/pkg/client/conn.go
@@ -18,6 +18,7 @@ package client

 import (
 	"errors"
+	"fmt"
 	"io"
 	"net"
 	"time"
@@ -61,6 +62,7 @@ func (c *conn) Write(data []byte) (n int, err error) {
 	if err != nil {
 		return 0, err
 	}
+	fmt.Println("DEBUG - Send PacketType_DATA for connection ID=", c.connID)
 	return len(data), err
 }

@@ -116,6 +118,7 @@ func (c *conn) Close() error {
 	klog.V(4).Infoln("closing connection")
 	var req *client.Packet
 	if c.connID != 0 {
+		fmt.Println("DEBUG - Send PacketType_CLOSE_REQ for connection ID=", c.connID)
 		req = &client.Packet{
 			Type: client.PacketType_CLOSE_REQ,
 			Payload: &client.Packet_CloseRequest{

Once the local cluster is up with ./hack/local-up-cluster.sh, you can start the proxy-server and agent as local processes in separate terminals with:

make clean && make certs && make gen && make build

./bin/proxy-server --server-port=0 --uds-name=/tmp/uds-proxy --cluster-ca-cert=certs/agent/issued/ca.crt --cluster-cert=certs/agent/issued/proxy-frontend.crt --cluster-key=certs/agent/private/proxy-frontend.key --warn-on-channel-limit --v 5 --keepalive-time 1m --frontend-keepalive-time 1m  --enable-profiling 2>&1 | tee server.log

./bin/proxy-agent --ca-cert=certs/agent/issued/ca.crt --agent-cert=certs/agent/issued/proxy-agent.crt --agent-key=certs/agent/private/proxy-agent.key

Stress the cluster, with concurrent kubectl log requests

Create some workloads

kubectl create deployment nginx --image nginx --replicas 20

get-logs.sh script:

#!/bin/bash

NAMESPACE=${NAMESPACE:-"default"}
pods=$(kubectl get pods -n ${NAMESPACE} --no-headers | awk '{print $1}')
FLOOR=1;
arr=(`echo ${pods}`);
CEILING=${#arr[@]};

for i in {1..3}
do
   RANGE=$(($CEILING-$FLOOR));
   random=$RANDOM;
   let "random %= $RANGE";
   random=$(($random+$FLOOR));
   kubectl logs -n ${NAMESPACE} ${arr[${random}]} > /dev/null 2> /dev/null
done

run.sh script:

#!/bin/bash

set -euo pipefail

for i in {1..500}
do
    ./get-logs.sh &
done

Execute the run.sh script after saving and giving execute permissions:

./run.sh

To verify the above hypothesis, I've created a small script, that prints those connection IDs which are idle i.e CLOSE_REQ has not been sent by the api-server and hence as a result never received a CLOSE_RSP from the proxy-server.

check.sh script:

#!/bin/bash

set -eou pipefail

total_connection_ids=$(rg "DEBUG - Send PacketType_DIAL_REQ" /tmp/kube-apiserver.log | wc -l)
for connection_id in $(seq 1 $total_connection_ids)
do
  close_req="DEBUG - Send PacketType_CLOSE_REQ for connection ID= ${connection_id}"
  close_rsp="DEBUG - Receive PacketType_CLOSE_RSP for connection ID= ${connection_id}"

  req_status=0
  set +e
  rg "$close_req\$" /tmp/kube-apiserver.log > /dev/null || req_status=1

  res_status=0
  rg "$close_rsp\$" /tmp/kube-apiserver.log > /dev/null || res_status=1

  set -e

  if [[ ( $req_status == 1 && $res_status == 1 ) ]]; then
    echo "Both missing for $connection_id"
  fi

  if [[ ( $req_status == 0 && $res_status == 1 ) ]]; then
    echo "CloseResponse missing for $connection_id"
  fi

  if [[ ( $req_status == 1 && $res_status == 0 ) ]]; then
    echo "Interesting case for $connection_id"
  fi
done

We are interested in the case of Both missing from $connection_id. All of the connection ids, that are printed for this case if cross-checked against the proxy-server logs have only the following logs snippet, example connectionID=505

I1215 16:36:06.122100   15067 server.go:714] "Received DIAL_RSP" dialID=4406796995779167332 agentID="f902ce27-b9b6-44a6-aef7-cfad5aff3611" connectionID=505
I1215 16:36:06.122132   15067 server.go:269] "Register frontend for agent" frontend=&{Mode:grpc Grpc:0xc000ee5f80 HTTP:<nil> CloseHTTP:<nil> connected:0xc000c58840 connectID:505 agentID:f902ce27-b9b6-44a6-aef7-cfad5aff3611 start:{wall:13863927826982877159 ext:149658304691 loc:0x212d6e0} backend:0xc000616108} agentID="f902ce27-b9b6-44a6-aef7-cfad5aff3611" connectionID=505

You can also verify if PacketType_DATA is sent by the apiserver for the same connection id

grep "DEBUG - Send PacketType_DATA for connection ID=" /tmp/kube-apiserver.log | grep 505

The problematic aspect of this is that in order to close/terminate these idle connections, we need connectionID which is only received when DIAL_RSP is sent by the agent and received by proxy-server and when PacketType_DATA is sent by apiserver to proxy-server.

/cc @iaguis @rata

andrewsykim
andrewsykim

@ipochi any chance we can test this again with the recent fixes that merged?

Apr
14
1 month ago
Activity icon
issue

andrewsykim issue comment kubernetes/kubernetes

andrewsykim
andrewsykim

fix ipvs_svc deletion

What type of PR is this?

Uncomment only one /kind <> line, hit enter to put that in a new line, and remove leading whitespaces from that line:

/kind api-change /kind bug /kind cleanup /kind design /kind documentation /kind failing-test /kind feature /kind flake

What this PR does / why we need it: In ipvs mode, when service deleted but rs(pod) exist, related ipvs rule and address bound in ipvs0 will not be deleted. I think it's unreasonable that kubernetes-service deleted but the service ip is available(ipvs rule not deleted). When you create same service again, new virtual service(totally two ipvs virtual service) will proxy to the same rs, too. And the deleted service ip will exist on both ipvs rule and ipvs0 device.

Which issue(s) this PR fixes:

Fixes #

Special notes for your reviewer:

Does this PR introduce a user-facing change?:

Fix a bug in the IPVS proxier where virtual servers are not cleaned up even though the corresponding Service object was deleted.

Additional documentation e.g., KEPs (Kubernetes Enhancement Proposals), usage docs, etc.:


andrewsykim
andrewsykim

Please open a new issue with the details included and assign it to me and @uablrek

Activity icon
issue

andrewsykim issue comment kubernetes-sigs/apiserver-network-proxy

andrewsykim
andrewsykim

Lint is failing in pre-submits

Our lint check in pre-submit is failing with the following error:

level=error msg="Running error: buildir: failed to load package goarch: could not load export data: cannot import \"internal/goarch\" (unknown iexport format version 2), export data is newer version - update tool"

e.g. https://prow.k8s.io/view/gs/kubernetes-jenkins/pr-logs/pull/kubernetes-sigs_apiserver-network-proxy/341/pull-apiserver-network-proxy-make-lint/1514261980213415936

I suspect this is because our prow jobs using the kubekins image was updated to use go v1.18 while we're still on go v1.17. Seems like we can fix this by eiher updating to go 1.18 as well or reverting the recent image updates.

andrewsykim
andrewsykim

Opened https://github.com/kubernetes/test-infra/pull/25962 if we decide to switch prow jobs back to v1.17, but we can close it if we choose to upgrade to go 1.18 instead

pull request

andrewsykim pull request kubernetes/test-infra

andrewsykim
andrewsykim

apiserver-network-proxy: kubekins e2e image should use v1.23 image with go v1.17

Signed-off-by: Andrew Sy Kim [email protected]

Activity icon
created branch

andrewsykim in andrewsykim/test-infra create branch apiserver-network-proxy-1-17

createdAt 1 month ago
Activity icon
issue

andrewsykim issue comment kubernetes-sigs/apiserver-network-proxy

andrewsykim
andrewsykim

Lint is failing in pre-submits

Our lint check in pre-submit is failing with the following error:

level=error msg="Running error: buildir: failed to load package goarch: could not load export data: cannot import \"internal/goarch\" (unknown iexport format version 2), export data is newer version - update tool"

e.g. https://prow.k8s.io/view/gs/kubernetes-jenkins/pr-logs/pull/kubernetes-sigs_apiserver-network-proxy/341/pull-apiserver-network-proxy-make-lint/1514261980213415936

I suspect this is because our prow jobs using the kubekins image was updated to use go v1.18 while we're still on go v1.17. Seems like we can fix this by eiher updating to go 1.18 as well or reverting the recent image updates.

andrewsykim
andrewsykim

Seems like we have a choice between downgrading our prow runners to use go v1.17 again or updating the project to use v1.18.1.

open pull request

andrewsykim wants to merge kubernetes-sigs/apiserver-network-proxy

andrewsykim
andrewsykim

agent: reduce noisy error logs when trying to write to a closed network connection

This is a follow-up PR to https://github.com/kubernetes-sigs/apiserver-network-proxy/pull/317, where we reduced error logs whenever reading from a closed network connection. This PR introduces a similar fix but also from the writer.

andrewsykim
andrewsykim

Seems like we close the connection on the agent side before we actually delete connID from connManager, which makes me think this check is maybe not doing what we want:

https://github.com/kubernetes-sigs/apiserver-network-proxy/blob/6c2477cfa8a6e11752a27bc99b4a06cc9499557c/pkg/agent/client.go#L413-L421

I'm not sure if deleting connID from connManager before ctx.conn.Close() is safe though.

pull request

andrewsykim merge to kubernetes-sigs/apiserver-network-proxy

andrewsykim
andrewsykim

agent: reduce noisy error logs when trying to write to a closed network connection

This is a follow-up PR to https://github.com/kubernetes-sigs/apiserver-network-proxy/pull/317, where we reduced error logs whenever reading from a closed network connection. This PR introduces a similar fix but also from the writer.

pull request

andrewsykim merge to kubernetes-sigs/apiserver-network-proxy

andrewsykim
andrewsykim

agent: reduce noisy error logs when trying to write to a closed network connection

This is a follow-up PR to https://github.com/kubernetes-sigs/apiserver-network-proxy/pull/317, where we reduced error logs whenever reading from a closed network connection. This PR introduces a similar fix but also from the writer.

open pull request

andrewsykim wants to merge kubernetes-sigs/apiserver-network-proxy

andrewsykim
andrewsykim

agent: reduce noisy error logs when trying to write to a closed network connection

This is a follow-up PR to https://github.com/kubernetes-sigs/apiserver-network-proxy/pull/317, where we reduced error logs whenever reading from a closed network connection. This PR introduces a similar fix but also from the writer.

pull request

andrewsykim pull request kubernetes-sigs/apiserver-network-proxy

andrewsykim
andrewsykim

agent: reduce noisy error logs when trying to write to a closed network connection

This is a follow-up PR to https://github.com/kubernetes-sigs/apiserver-network-proxy/pull/317, where we reduced error logs whenever reading from a closed network connection. This PR introduces a similar fix but also from the writer.

Previous