Kubernetes Build 总结以及问题

本文介绍Kubernetes编译的过程。
参考：
https://kubernetes.io/docs/setup/release/building-from-source/
https://github.com/kubernetes/kubernetes/tree/master/build/

1 下载源代码

https://github.com/kubernetes/kubernetes/releases/tag/v1.13.3

下载Source code(tar.gz) 到有Docker的环境，并解压缩。

image.png

2 编译

Key scripts
The following scripts are found in the build/ directory. Note that all scripts must be run from the Kubernetes root directory.

build/run.sh: Run a command in a build docker container. Common invocations:
build/run.sh make: Build just linux binaries in the container. Pass options and packages as necessary.
build/run.sh make cross: Build all binaries for all platforms
build/run.sh make kubectl KUBE_BUILD_PLATFORMS=darwin/amd64: Build the specific binary for the specific platform (kubectl and darwin/amd64 respectively in this example)
build/run.sh make test: Run all unit tests
build/run.sh make test-integration: Run integration test
build/run.sh make test-cmd: Run CLI tests
build/copy-output.sh: This will copy the contents of _output/dockerized/bin from the Docker container to the local _output/dockerized/bin. It will also copy out specific file patterns that are generated as part of the build process. This is run automatically as part of build/run.sh.
build/make-clean.sh: Clean out the contents of _output, remove any locally built container images and remove the data container.
build/shell.sh: Drop into a bash shell in a build container with a snapshot of the current repo code.

以编译Linux平台kubectl 为例：

[root@VM_0_4_centos kubernetes-1.13.3]# build/run.sh make kubectl 
+++ [0216 11:19:07] Verifying Prerequisites....
+++ [0216 11:19:07] Building Docker image kube-build:build-d7e9f7306e-5-v1.11.5-1
+++ Docker build command failed for kube-build:build-d7e9f7306e-5-v1.11.5-1

Sending build context to Docker daemon  8.704kB
Step 1/16 : FROM k8s.gcr.io/kube-cross:v1.11.5-1
Get https://k8s.gcr.io/v2/: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)

To retry manually, run:

docker build -t kube-build:build-d7e9f7306e-5-v1.11.5-1 --pull=false /root/Sandbox/kubernetes-1.13.3/_output/images/kube-build:build-d7e9f7306e-5-v1.11.5-1

!!! [0216 11:19:22] Call tree:
!!! [0216 11:19:22]  1: build/run.sh:31 kube::build::build_image(...)
!!! Error in build/../build/common.sh:461
  Error in build/../build/common.sh:461. '((i<3-1))' exited with status 1
Call stack:
  1: build/../build/common.sh:461 kube::build::build_image(...)
  2: build/run.sh:31 main(...)
Exiting with status 1

遇到上述问题的原因是：
k8s.gcr.io/kube-cross:v1.11.5-1在中国区被墙，无法下载。
解决办法是：
到DockerHub上找到googlecontainer/kube-cross，并下载指定Tag的image，下载后重新执行docker tag打标签。

kube-cross image kube-cross image tag list

[root@VM_0_4_centos kubernetes-1.13.3]# docker pull googlecontainer/kube-cross:v1.11.5-1
v1.11.5-1: Pulling from googlecontainer/kube-cross
ab1fc7e4bf91: Pull complete 
35fba333ff52: Pull complete 
f0cb1fa13079: Pull complete 
3d1dd648b5ad: Pull complete 
866ed2e2b1fe: Pull complete 
ddc4ad35a2f0: Pull complete 
ab8da8d8889d: Pull complete 
bc09b85f7ce1: Pull complete 
771b92c72d72: Pull complete 
a197f22cc074: Pull complete 
c09a2f634954: Pull complete 
e99d87f11b4a: Pull complete 
817e550ce839: Pull complete 
7489af17a433: Pull complete 
Digest: sha256:b4864400d58268b3abde3d9ded4ab23a9ddda6c85d63c80ed536cd2810d75042
Status: Downloaded newer image for googlecontainer/kube-cross:v1.11.5-1

重新打Tag：

[root@VM_0_4_centos kubernetes-1.13.3]# docker help tag

Usage:  docker tag SOURCE_IMAGE[:TAG] TARGET_IMAGE[:TAG]

Create a tag TARGET_IMAGE that refers to SOURCE_IMAGE

Options:

[root@VM_0_4_centos kubernetes-1.13.3]# docker tag googlecontainer/kube-cross:v1.11.5-1 k8s.gcr.io/kube-cross:v1.11.5-1
[root@VM_0_4_centos kubernetes-1.13.3]# docker images |grep kube-cross
googlecontainer/kube-cross                                 v1.11.5-1           b16987a9b305        3 weeks ago         1.75GB
k8s.gcr.io/kube-cross                                      v1.11.5-1           b16987a9b305        3 weeks ago         1.75GB

现在重新开始编译：

[root@VM_0_4_centos kubernetes-1.13.3]# build/run.sh make kubectl 
...
I0216 11:40:12.554451    3183 api_linter.go:109] validating API rules for type k8s.io/metrics/pkg/apis/metrics/v1beta1.valueToStringGenerated
I0216 11:40:12.554455    3183 api_linter.go:192] validating API rule names_match for type k8s.io/metrics/pkg/apis/metrics/v1beta1.valueToStringGenerated
I0216 11:40:12.554460    3183 api_linter.go:192] validating API rule omitempty_match_case for type k8s.io/metrics/pkg/apis/metrics/v1beta1.valueToStringGenerated
I0216 11:40:12.554636    3183 execute.go:67] Assembling file "/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/generated/openapi/zz_generated.openapi.go"
I0216 11:40:14.087868    3183 api_linter.go:44] Assembling file "_output/violations.report"
2019/02/16 11:40:14 Code for OpenAPI definitions generated
Go version: go version go1.11.5 linux/amd64
+++ [0216 11:40:14] Building go targets for linux/amd64:
    ./vendor/github.com/jteeuwen/go-bindata/go-bindata
Env for linux/amd64: GOOS=linux GOARCH=amd64 GOROOT=/usr/local/go CGO_ENABLED= CC=
Coverage is disabled.
+++ [0216 11:40:14] Placing binaries
Generated bindata file : test/e2e/generated/bindata.go has 9052 test/e2e/generated/bindata.go lines of lovely automated artifacts
No changes in generated bindata file: pkg/kubectl/generated/bindata.go
Go version: go version go1.11.5 linux/amd64
+++ [0216 11:40:15] Building go targets for linux/amd64:
    cmd/kubectl
Env for linux/amd64: GOOS=linux GOARCH=amd64 GOROOT=/usr/local/go CGO_ENABLED= CC=
Coverage is disabled.
+++ [0216 11:40:51] Placing binaries
+++ [0216 11:40:52] Syncing out of container
+++ [0216 11:40:52] Stopping any currently running rsyncd container
+++ [0216 11:40:52] Starting rsyncd container
+++ [0216 11:40:53] Running rsync
+++ [0216 11:40:54] Stopping any currently running rsyncd container
[root@VM_0_4_centos kubernetes-1.13.3]# ls

用于Build的环境基于如下Dockerfile构建：

[root@VM_0_4_centos kubernetes-1.13.3]# cat _output/images/kube-build:build-d7e9f7306e-5-v1.11.5-1/Dockerfile
# Copyright 2016 The Kubernetes Authors.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
#     http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.

# This file creates a standard build environment for building Kubernetes
FROM k8s.gcr.io/kube-cross:v1.11.5-1

# Mark this as a kube-build container
RUN touch /kube-build-image

# To run as non-root we sometimes need to rebuild go stdlib packages.
RUN chmod -R a+rwx /usr/local/go/pkg

# For running integration tests /var/run/kubernetes is required
# and should be writable by user
RUN mkdir /var/run/kubernetes && chmod a+rwx /var/run/kubernetes

# The kubernetes source is expected to be mounted here.  This will be the base
# of operations.
ENV HOME /go/src/k8s.io/kubernetes
WORKDIR ${HOME}

# Make output from the dockerized build go someplace else
ENV KUBE_OUTPUT_SUBPATH _output/dockerized

# Pick up version stuff here as we don't copy our .git over.
ENV KUBE_GIT_VERSION_FILE ${HOME}/.dockerized-kube-version-defs

# Add system-wide git user information
RUN git config --system user.email "nobody@k8s.io" \
  && git config --system user.name "kube-build-image"

# Fix permissions on gopath
RUN chmod -R a+rwx $GOPATH

# Make log messages use the right timezone
ADD localtime /etc/localtime
RUN chmod a+r /etc/localtime

# Set up rsyncd
ADD rsyncd.password /
RUN chmod a+r /rsyncd.password
ADD rsyncd.sh /
RUN chmod a+rx /rsyncd.sh

参照如下操作，将编译好的文件从容器copy到本地：

build/copy-output.sh: This will copy the contents of _output/dockerized/bin from the Docker container to the local _output/dockerized/bin. It will also copy out specific file patterns that are generated as part of the build process. This is run automatically as part of build/run.sh.

[root@VM_0_4_centos kubernetes-1.13.3]# build/copy-output.sh
+++ [0216 11:50:08] Verifying Prerequisites....
+++ [0216 11:50:08] Syncing out of container
+++ [0216 11:50:08] Stopping any currently running rsyncd container
+++ [0216 11:50:08] Starting rsyncd container
+++ [0216 11:50:08] Running rsync
+++ [0216 11:50:09] Stopping any currently running rsyncd container

[root@VM_0_4_centos kubernetes-1.13.3]# ll _output/dockerized/bin/linux/amd64/
total 71800
-rwxr-xr-x 1 root root  5800256 Feb 16 11:39 conversion-gen
-rwxr-xr-x 1 root root  5791968 Feb 16 11:38 deepcopy-gen
-rwxr-xr-x 1 root root  5783808 Feb 16 11:38 defaulter-gen
-rwxr-xr-x 1 root root  4460824 Feb 16 11:38 go2make
-rwxr-xr-x 1 root root  2001888 Feb 16 11:40 go-bindata
-rwxr-xr-x 1 root root 39185888 Feb 16 11:40 kubectl
-rwxr-xr-x 1 root root 10363744 Feb 16 11:40 openapi-gen

[root@VM_0_4_centos kubernetes-1.13.3]# _output/dockerized/bin/linux/amd64/kubectl  version
Client Version: version.Info{Major:"1", Minor:"13", GitVersion:"v1.13.3", GitCommit:"721bfa751924da8d1680787490c54b9179b1fed0", GitTreeState:"archive", BuildDate:"2019-02-16T03:40:15Z", GoVersion:"go1.11.5", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"10+", GitVersion:"v1.10.5-tke.3", GitCommit:"53e244be925234190938376fe8637189b6caf125", GitTreeState:"clean", BuildDate:"2018-12-04T04:10:15Z", GoVersion:"go1.9.3", Compiler:"gc", Platform:"linux/amd64"}

从执行kubectl version的结果可以看到kubectl可以工作，版本是v1.13.3。