Kubernetes: access the API insid

Sometimes you’d like to access the Kubernetes API inside a pod in order to retrieve some information, like for instance getting the IP addresses of some other pods without recurring to a headless service.

Note: The pod used to run commands inside the cluster for this post is created with the following command:

$ kubectl run -i --tty tools --image=[giantswarm/tiny-tools](https://github.com/giantswarm/tiny-tools) --restart=Never -- sh

On each container you have the required elements to access the API in the /var/run/secrets/kubernetes.io/serviceaccount directory:

/ # tree /var/run/secrets/
/var/run/secrets/
└── kubernetes.io
    └── serviceaccount
        ├── ca.crt -> ..data/ca.crt
        ├── namespace -> ..data/namespace
        └── token -> ..data/token

ca.crt contains the root certificate of the https certificate chain of the API, token contains the API token of the service account associated with the Pod and namespace contains the namespace of the pod.

You can access the API with the following command:

/ # curl -s -H "Authorization: Bearer $(cat /var/run/secrets/kubernetes.io/serviceaccount/token)" --cacert /var/run/secrets/kuber
netes.io/serviceaccount/ca.crt [https://kubernetes/api/v1/namespaces/$(cat](https://kubernetes/api/v1/namespaces/$(cat) /var/run/secrets/kubernetes.io/serviceaccount/namespace
)/pods

However, on a standard pod run on the default namespace, you will get the following response:

{
  "kind": "Status",
  "apiVersion": "v1",
  "metadata": {},
  "status": "Failure",
  "message": "pods is forbidden: User \"system:serviceaccount:default:default\" cannot list resource \"pods\" in API group \"\" in the namespace \"default\"",
  "reason": "Forbidden",
  "details": {
    "kind": "pods"
  },
  "code": 403
}

This is pretty explicit. You don’t have access to the API. At this point, you may consider running the Pod accessing the API under a service account with more privileges or giving read access to the default service account. Let’s do that. We assume that you are using RBAC (Role Based Access Control) on your cluster.

You first need to create a role with read access. Let’s create it with read access to pods, deployments and services:

$ kubectl create role pod-reader --verb=get --verb=list --verb=watch --resource=pods,services,deployments
role.rbac.authorization.k8s.io/pod-reader created

Then we need to bind this role to the default service account. This is done by creating a role binding:

$ kubectl create rolebinding default-pod-reader --role=pod-reader --serviceaccount=default:default --namespace=default
rolebinding.rbac.authorization.k8s.io/default-pod-reader created

You will get the same result with the following YAML file:

You can apply it with the following command:

$ kubectl apply -f [https://gist.githubusercontent.com/antoinemartin/1ef51645d9b55f0e445febb77973a9bf/raw/1490ac2515bc909b7dda0f3c475531cef333e7ca/pod-reader.yaml](https://gist.githubusercontent.com/antoinemartin/1ef51645d9b55f0e445febb77973a9bf/raw/1490ac2515bc909b7dda0f3c475531cef333e7ca/pod-reader.yaml)

If you go back to your pod and run the curl command again, you get a real result:

/ # curl -s -H "Authorization: Bearer $(cat /var/run/secrets/kubernetes.io/serviceaccount/token)" --cacert /var/run/secrets/kuber
netes.io/serviceaccount/ca.crt [https://kubernetes/api/v1/namespaces/$(cat](https://kubernetes/api/v1/namespaces/$(cat) /var/run/secrets/kubernetes.io/serviceaccount/namespace
)/pods
{
  "kind": "PodList",
  "apiVersion": "v1",
  "metadata": {
...

To ease following requests to the API, you can define a helper command:

/ # kapi() { curl -s -H "Authorization: Bearer $(cat /var/run/secrets/kubernetes.io/serviceaccount/token)" --cacert /var/run/secr
ets/kubernetes.io/serviceaccount/ca.crt "[https://kubernetes/api/v1/namespaces/$(cat](https://kubernetes/api/v1/namespaces/$(cat) /var/run/secrets/kubernetes.io/serviceaccount
/namespace)/$1" ;}

And then use it to retrieve information:

/ # kapi pods/`hostname` | jq -Mr '.status.podIP'
10.244.1.23

To get back to the initial requirement, if you have for instance a nginx deployment with two replicas (you can quickly deploy it with kubectl run nginx --image=nginx --replicas=2), you can retrieve the pods IP addresses with the following command:

/ # kapi pods | jq -Mr '.items[] | select(.metadata.labels.run=="nginx") | .status.podIP'
10.244.0.14
10.244.1.27

Same result is achieved with:

/ # kapi pods?labelSelector=run%3Dnginx | jq -Mr '.items[] | .status.podIP'
10.244.0.14
10.244.1.27

Note that instead of curling, you can use the [kubectl](https://kubernetes.io/docs/reference/generated/kubectl/kubectl-commands) command. It will use the credentials mentioned at the beginning of this post. You can install it with:

$ curl -s [https://storage.googleapis.com/kubernetes-release/release/v1.13.0/bin/linux/amd64/kubectl](https://storage.googleapis.com/kubernetes-release/release/v1.13.0/bin/linux/amd64/kubectl) -o /usr/bin/kubectl && chmod +x /usr/bin/kubectl

And then achieve the desired result without the need for the[jq](https://stedolan.github.io/jq/manual/) command with:

$ kubectl get pod -l run=nginx --template="{{range .items}}{{.status.podIP}}:{{end}}" | tr ':' '\n'
10.244.0.14
10.244.1.27

You can check that you only have read access by trying to delete your deployment:

# kubectl delete deployment/nginx
Error from server (Forbidden): deployments.extensions "nginx" is forbidden: User "system:serviceaccount:default:default" cannot delete resource "deployments" in API group "extensions" in the namespace "default"

That’s it. Having access to the API from pods can be useful in Job pods or on init containers. Giving read access to standard pods opens this option.

# give our webhook ap (as default:default) permissions to create workflows
---
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: argo-invocation
  namespace: default
rules:
- apiGroups:
  - "argoproj.io"
  resources:
  - "workflows"
  verbs:
  - get
  - list
  - watch
  - create
  - update
  - patch
  - delete
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: default-default-invocation
  namespace: default
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: Role
  name: argo-invocation
subjects:
- kind: ServiceAccount
  name: default
  namespace: default


# give workflows (as argo:default) permissions to run things
# see https://github.com/argoproj/argo/blob/master/docs/workflow-rbac.md
---
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: argo-workflow
  namespace: default
rules:
# pod get/watch is used to identify the container IDs of the current pod
# pod patch is used to annotate the step's outputs back to controller (e.g. artifact location)
- apiGroups:
  - ""
  resources:
  - pods
  verbs:
  - get
  - watch
  - patch
# logs get/watch are used to get the pods logs for script outputs, and for log archival
- apiGroups:
  - ""
  resources:
  - pods/log
  verbs:
  - get
  - watch
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: argo-default-workflow
  namespace: default
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: Role
  name: argo-workflow
subjects:
- kind: ServiceAccount
  name: default
  namespace: default