Cfrgtkky

I’ve been learning about Kubernetes using Kelsey Hightower’s excellent kubernetes-the-hard-way-guide.

Using this guide I’ve installed v1.12 on GCE. Everything works perfectly apart from kubectl exec:

$ kubectl exec -it shell-demo – /bin/bash --kubeconfig=/root/certsconfigs/admin.kubeconfig

error: unable to upgrade connection: Forbidden (user=kubernetes, verb=create, resource=nodes, subresource=proxy)

Note that I have set KUBECONFIG=/root/certsconfigs/admin.kubeconfig.

Apart from exec all other kubectl functions work as expected with this admin.kubeconfig file, so from that I deduce it valid for use with my cluster.

I’m pretty sure I have made a beginners mistake somewhere, but if somebody could advise where I have gone away, I should be most grateful.

TIA

Shaun

I have double checked that no .kube/config file exists anywhere on my master controller:

root@controller-1:/root/deployment/kubernetes# kubectl get pods

NAME READY STATUS

shell-demo 1/1 Running 0 23m

Here is the output with -v8:

root@controller-1:/root/deployment/kubernetes# kubectl -v8 exec -it shell-demo – /bin/bash

I1118 15:18:16.898428 11117 loader.go:359] Config loaded from file /root/certsconfigs/admin.kubeconfig

I1118 15:18:16.899531 11117 loader.go:359] Config loaded from file /root/certsconfigs/admin.kubeconfig

I1118 15:18:16.900611 11117 loader.go:359] Config loaded from file /root/certsconfigs/admin.kubeconfig

I1118 15:18:16.902851 11117 round_trippers.go:383] GET ://127.0.0.1:6443/api/v1/namespaces/default/pods/shell-demo

I1118 15:18:16.902946 11117 round_trippers.go:390] Request Headers:

I1118 15:18:16.903016 11117 round_trippers.go:393] Accept: application/json, /

I1118 15:18:16.903091 11117 round_trippers.go:393] User-Agent: kubectl/v1.12.0 (linux/amd64) kubernetes/0ed3388

I1118 15:18:16.918699 11117 round_trippers.go:408] Response Status: 200 OK in 15 milliseconds

I1118 15:18:16.918833 11117 round_trippers.go:411] Response Headers:

I1118 15:18:16.918905 11117 round_trippers.go:414] Content-Type: application/json

I1118 15:18:16.918974 11117 round_trippers.go:414] Content-Length: 2176

I1118 15:18:16.919053 11117 round_trippers.go:414] Date: Sun, 18 Nov 2018 15:18:16 GMT

I1118 15:18:16.919218 11117 request.go:942] Response Body: {“kind”:“Pod”,“apiVersion”:“v1”,“metadata”:{“name”:“shell-demo”,“namespace”:“default”,“selfLink”:"/api/v1/namespaces/default/pods/shell-demo",“uid”:“99f320f8-eb42-11e8-a053-42010af0000b”,“resourceVersion”:“13213”,“creationTimestamp”:“2018-11-18T14:59:51Z”},“spec”:{“volumes”:[{“name”:“shared-data”,“emptyDir”:{}},{“name”:“default-token-djprb”,“secret”:{“secretName”:“default-token-djprb”,“defaultMode”:420}}],“containers”:[{“name”:“nginx”,“image”:“nginx”,“resources”:{},“volumeMounts”:[{“name”:“shared-data”,“mountPath”:"/usr/share/nginx/html"},{“name”:“default-token-djprb”,“readOnly”:true,“mountPath”:"/var/run/secrets/kubernetes.io/serviceaccount"}],“terminationMessagePath”:"/dev/termination-log",“terminationMessagePolicy”:“File”,“imagePullPolicy”:“Always”}],“restartPolicy”:“Always”,“terminationGracePeriodSeconds”:30,“dnsPolicy”:“ClusterFirst”,“serviceAccountName”:“default”,“serviceAccount”:“default”,“nodeName”:“worker-1”,“securityContext”:{},“schedulerName”:“default-scheduler”,“tolerations”:[{“key”:"node.kubernet [truncated 1152 chars]

I1118 15:18:16.925240 11117 round_trippers.go:383] POST …

error: unable to upgrade connection: Forbidden (user=kubernetes, verb=create, resource=nodes, subresource=proxy)

According to your logs,the connection between kubectl and the apiserver is fine, and is being authenticated correctly.

To satisfy an exec request, the apiserver contacts the kubelet running the pod, and that connection is what is being forbidden.

Your kubelet is configured to authenticate/authorize requests, and the apiserver credential is not authorized to make the exec request against the kubelet's API.

Based on the forbidden message, your apiserver is authenticating as the "kubernetes" user to the kubelet.

You can grant that user full permissions to the kubelet API with the following command:

kubectl create clusterrolebinding apiserver-kubelet-admin --user=kubernetes --clusterrole=system:kubelet-api-admin

See the following docs for more information

https://kubernetes.io/docs/reference/command-line-tools-reference/kubelet-authentication-authorization/#overview

https://kubernetes.io/docs/reference/access-authn-authz/rbac/#other-component-roles