1. Prerequisites
编译 ceph 源代码需要依赖于两个 repo，一个是 ceph，一个是 ceph-dev-docker，同时我们创建一个 ceph-ccache 目录用于存储 ceph compiler cache，通过 ccache 可以进行增量编译，大幅度减少编译时间。我们以编译 ceph pacific 为例。
$ git clone https://github.com/ceph/ceph
$ git clone https://github.com/microyahoo/ceph-dev-docker
$ mkdir ceph-ccache
# ceph 切换至 pacific 分支
🍺 /root/go/src/ceph ☞ git:(pacific) ✗ git checkout -b pacific upstream/pacific
🍺 /root/go/src ☞ ls
ceph-ccache/ ceph/ ceph-dev-docker/
2. Running a Docker Container
我们是以 [centos:stream](https://quay.io/repository/centos/centos?tab=tags&tag=stream) 作为基础镜像进行编译的，需要指定环境变量
NAME - 容器名
CCACHE - ccache 路径
CEPH - ceph repo 路径
VERSION - ceph version
🍺 /root/go/src ☞ cd ceph-dev-docker; NAME=ceph-dev CCACHE=/root/go/src/ceph-ccache CEPH=/root/go/src/ceph VERSION=pacific sh +x setup.sh
Error response from daemon: No such container: ceph-dev
Error: No such container: ceph-dev
[+] Building 1.0s (12/12) FINISHED
=> [internal] load build definition from pacific.Dockerfile 0.0s
=> => transferring dockerfile: 866B 0.0s
=> [internal] load .dockerignore 0.0s
=> => transferring context: 2B 0.0s
=> [internal] load metadata for quay.io/centos/centos:stream 0.9s
=> [internal] load build context 0.0s
=> => transferring context: 607B 0.0s
=> [1/7] FROM quay.io/centos/centos:stream@sha256:c9acf46f90fcb637eff59e269fbbebf5ec9e6b6215a07fbe2bbad7429aad6e7e 0.0s
=> CACHED [2/7] RUN yum install -y epel-release 0.0s
=> CACHED [3/7] RUN yum install -y gcc gcc-c++ python36 python3-devel wget iproute vim tmux git bash jq rpm-build zsh pyth 0.0s
=> CACHED [4/7] RUN yum install -y ccache 0.0s
=> CACHED [5/7] RUN dnf --enablerepo=powertools install -y protobuf-devel hwloc-devel xmlsec1-devel xmlsec1-openssl-devel 0.0s
=> CACHED [6/7] ADD /shared/docker/ /docker 0.0s
=> CACHED [7/7] RUN /docker/install-omz.sh 0.0s
=> exporting to image 0.0s
=> => exporting layers 0.0s
=> => writing image sha256:248c85c2d84fb5d469b72b09e3f1be50143e341c2d121146cf5e68e7fe0f4d7a 0.0s
=> => naming to docker.io/library/ceph-dev-docker-pacific 0.0s
328559b2abd723777ee23a6457b4649416f7c23b76a6416b2cd9caaf9ab5376a
╭─root@ceph-dev /
╰─# cd shared/bin
╭─root@ceph-dev /shared/bin
╰─# ls
create-dashboard-rgw-user.sh reload-cephadm.sh setup-cephadm.sh setup-nfs.sh start-ceph.sh
fetch-pr.sh setup-ceph.sh setup-modules.sh setup-proxy.sh stop-ceph.sh
╭─root@ceph-dev /shared/bin
╰─# ./setup-ceph.sh
脚本会在容器中进行源代码编译
🍺 /root/go/src ☞ docker ps -a
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
328559b2abd7 ceph-dev-docker-pacific "zsh" 52 minutes ago Up 52 minutes ceph-dev
脚本完成后会在 /ceph/build 目录生成 build 信息，包含相应的 Makefile。
(install-deps-python3) ╭─root@ceph-dev /shared/bin
╰─# ll /ceph/build
total 204K
-rw-r--r-- 1 root root 67K Dec 19 04:30 CMakeCache.txt
drwxr-xr-x 11 root root 4.0K Dec 19 04:30 CMakeFiles
-rw-r--r-- 1 root root 362 Dec 19 04:30 CTestTestfile.cmake
-rw-r--r-- 1 root root 105K Dec 19 04:30 Makefile
drwxr-xr-x 2 root root 75 Dec 19 04:30 bin
drwxr-xr-x 5 root root 43 Dec 19 04:30 boost
-rw-r--r-- 1 root root 49 Dec 19 04:30 ceph.conf
-rw-r--r-- 1 root root 2.1K Dec 19 04:30 cmake_install.cmake
drwxr-xr-x 4 root root 105 Dec 19 04:30 doc
drwxr-xr-x 3 root root 20 Dec 19 04:30 etc
drwxr-xr-x 2 root root 24 Dec 19 04:30 include
drwxr-xr-x 2 root root 6 Dec 19 04:30 lib
drwxr-xr-x 3 root root 94 Dec 19 04:30 man
drwxr-xr-x 3 root root 24 Dec 19 04:30 monitoring
drwxr-xr-x 3 root root 94 Dec 19 04:30 qa
drwxr-xr-x 47 root root 4.0K Dec 19 04:30 src
drwxr-xr-x 3 root root 4.0K Dec 19 04:30 systemd
-rw-r--r-- 1 root root 137 Dec 19 04:30 user-config.jam
执行cmake . -LH 查看下 ceph 有哪些编译选项
(install-deps-python3) ╭─root@ceph-dev /ceph/build ‹pacific›
╰─# cmake . -LH
我们也可以只编译我们需要的模块，可以通过 make help 进行查看
(install-deps-python3) ╭─root@ceph-dev /ceph/build ‹pacific›
╰─# pwd
/ceph/build
(install-deps-python3) ╭─root@ceph-dev /ceph/build ‹pacific›
╰─# make help
例如 make ceph-common rbd radosgw-admin 等等。
(install-deps-python3) ╭─root@ceph-dev /ceph ‹pacific›
╰─# ccache make rbd -j 2
(install-deps-python3) ╭─root@ceph-dev /ceph/build ‹pacific›
╰─# ccache make radosgw-admin -j 2
(install-deps-python3) ╭─root@ceph-dev /ceph/build ‹pacific›
╰─# ccache make ceph-common -j 2
等待 make 结束，这个过程会持续的比较久。结束后可以 install
make install
由于 make install 生成的二进制没有 stripped，所以二进制文件会很大，可以执行 make install/strip 代替 make install。
make install/strip
-
编译过程中问题处理
- pip 版本低
Collecting isodate>=0.5.0 (from python3-saml==1.4.1->-c constraints.txt (line 5))
Downloading https://files.pythonhosted.org/packages/b6/85/7882d311924cbcfc70b1890780763e36ff0b140c7e51c110fc59a532f087/isodate-0.6.1-py2.py3-none-any.whl (41kB)
100% |################################| 51kB 10.6MB/s
Saved ./wheelhouse-wip/isodate-0.6.1-py2.py3-none-any.whl
Collecting xmlsec>=0.6.0 (from python3-saml==1.4.1->-c constraints.txt (line 5))
Downloading https://files.pythonhosted.org/packages/37/9f/342d4562eac99178d0d515c780285e107c6828cefad37d02f05b7b7d8751/xmlsec-1.3.13.tar.gz (64kB)
100% |################################| 71kB 8.8MB/s
Complete output from command python setup.py egg_info:
Traceback (most recent call last):
File "<string>", line 1, in <module>
File "/tmp/pip-build-km5ufi4j/xmlsec/setup.py", line 542, in <module>
package_data={'xmlsec': ['py.typed', '*.pyi']},
File "/ceph/install-deps-python3/lib64/python3.6/site-packages/setuptools/__init__.py", line 129, in setup
return distutils.core.setup(**attrs)
File "/usr/lib64/python3.6/distutils/core.py", line 108, in setup
_setup_distribution = dist = klass(attrs)
File "/ceph/install-deps-python3/lib64/python3.6/site-packages/setuptools/dist.py", line 370, in __init__
k: v for k, v in attrs.items()
File "/usr/lib64/python3.6/distutils/dist.py", line 281, in __init__
self.finalize_options()
File "/ceph/install-deps-python3/lib64/python3.6/site-packages/setuptools/dist.py", line 529, in finalize_options
ep.load()(self, ep.name, value)
File "/ceph/install-deps-python3/lib64/python3.6/site-packages/pkg_resources/__init__.py", line 2318, in load
return self.resolve()
File "/ceph/install-deps-python3/lib64/python3.6/site-packages/pkg_resources/__init__.py", line 2324, in resolve
module = __import__(self.module_name, fromlist=['__name__'], level=0)
File "/tmp/pip-build-km5ufi4j/xmlsec/.eggs/setuptools_scm-7.1.0-py3.6.egg/setuptools_scm/__init__.py", line 5
from __future__ import annotations
^
SyntaxError: future feature annotations is not defined
----------------------------------------
Command "python setup.py egg_info" failed with error code 1 in /tmp/pip-build-km5ufi4j/xmlsec/
You are using pip version 9.0.3, however version 22.3.1 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.
可以升级 pip 版本（注意需要升级的是 venv 中的版本），然后重新执行脚本。
╭─root@ceph-dev /shared/bin
╰─# . /ceph/install-deps-python3/bin/activate 1 ↵
(install-deps-python3) ╭─root@ceph-dev /shared/bin
╰─# pip -V
/bin/sh: warning: setlocale: LC_ALL: cannot change locale (en_US.UTF-8)
pip 9.0.3 from /ceph/install-deps-python3/lib64/python3.6/site-packages (python 3.6)
(install-deps-python3) ╭─root@ceph-dev /shared/bin
╰─# pip install --upgrade pip
/bin/sh: warning: setlocale: LC_ALL: cannot change locale (en_US.UTF-8)
Collecting pip
Downloading https://files.pythonhosted.org/packages/a4/6d/6463d49a933f547439d6b5b98b46af8742cc03ae83543e4d7688c2420f8b/pip-21.3.1-py3-none-an
y.whl (1.7MB)
100% |################################| 1.7MB 779kB/s
Installing collected packages: pip
Found existing installation: pip 9.0.3
Uninstalling pip-9.0.3:
Successfully uninstalled pip-9.0.3
Successfully installed pip-21.3.1
You are using pip version 21.3.1, however version 22.3.1 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.
(install-deps-python3) ╭─root@ceph-dev /shared/bin
╰─# pip -V
pip 21.3.1 from /ceph/install-deps-python3/lib64/python3.6/site-packages/pip (python 3.6)
(install-deps-python3) ╭─root@ceph-dev /shared/bin
╰─# ./setup-ceph.sh
- 内存不足
c++: fatal error: Killed signal terminated program cc1plus
-
动态链接库找不到
rados: error while loading shared libraries: libradosstriper.so.1: cannot open shared object file: No such file or directory将
/usr/local/lib64/目录加到/etc/ld.so.conf，然后执行ldconfig。[root@ceph-dev build]# cat /etc/ld.so.conf /usr/local/lib64/ include ld.so.conf.d/*.conf [root@ceph-dev build]# ldconfig
网友评论