04-pySpark 番外（Linux 安装Python3，py

前期准备

虚拟机默认python版本2.7.5，因个人习惯，安装使用3+后进行测试

安装python3

建立软件下载目录

[mongodb@mongodb01 sbin]$ sudo mkdir -p /home/python3/software/
[mongodb@mongodb01 sbin]$ cd /home/python3/software/
[mongodb@mongodb01 software]$ 

下载安装包

[mongodb@mongodb01 software]$ sudo wget https://www.python.org/ftp/python/3.6.0/Python-3.6.0.tgz

[mongodb@mongodb01 software]$ ll
总用量 21736
-rw-r--r--. 1 root root 22256403 12月 23 2016 Python-3.6.0.tgz

解压

[mongodb@mongodb01 software]$ sudo tar -zxvf Python-3.6.0.tgz

[mongodb@mongodb01 software]$ ll
总用量 21740
drwxr--r--. 17  501 games     4096 12月 23 2016 Python-3.6.0
-rw-r--r--.  1 root root  22256403 12月 23 2016 Python-3.6.0.tgz

进入解压目录

[mongodb@mongodb01 software]$ cd ./Python-3.6.0
[mongodb@mongodb01 Python-3.6.0]$ 

建立安装目录

[mongodb@mongodb01 Python-3.6.0]$ sudo mkdir /usr/local/python3
[mongodb@mongodb01 Python-3.6.0]$ 

设置安装目录

注：当前目录依旧在下载解压目录中/home/python3/software/Python-3.6.0/

[mongodb@mongodb01 Python-3.6.0]$ sudo ./configure --prefix=/usr/local/python3

编译

[mongodb@mongodb01 Python-3.6.0]$ sudo make

安装

[mongodb@mongodb01 Python-3.6.0]$ sudo make install

建立软链接

[mongodb@mongodb01 Python-3.6.0]$ sudo ln -s /usr/local/python3/bin/python3 /usr/bin/python3

写入环境变量

[mongodb@mongodb01 Python-3.6.0]$ sudo vim ~/.bash_profile

PATH=$PATH:$HOME/.local/bin:$HOME/bin:/usr/local/python3/bin

测试

[mongodb@mongodb01 Python-3.6.0]$ python3
Python 3.6.0 (default, Nov 13 2019, 12:35:54) 
[GCC 4.8.5 20150623 (Red Hat 4.8.5-39)] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>>

修改 pySpark 启动 Python 版本

修改 spark-env.sh 文件

[mongodb@mongodb01 conf]$ cd /home/spark/software/spark-2.4.4-bin-hadoop2.7/conf

[mongodb@mongodb01 conf]$ sudo vim spark-env.sh

# 末尾添加
export PYSPARK_PYTHON=/usr/bin/python3

修改 pyspark 文件

[mongodb@mongodb01 conf]$ cd /home/spark/software/spark-2.4.4-bin-hadoop2.7/bin

[mongodb@mongodb01 bin]$ sudo vim pyspark

# PYSPARK_PYTHON=python 修改为 PYSPARK_PYTHON=python3
# Determine the Python executable to use for the executors:
if [[ -z "$PYSPARK_PYTHON" ]]; then
  if [[ $PYSPARK_DRIVER_PYTHON == *ipython* && ! $WORKS_WITH_IPYTHON ]]; then
    echo "IPython requires Python 2.7+; please install python2.7 or set PYSPARK_PYTHON" 1>&2
    exit 1
  else
    PYSPARK_PYTHON=python3
  fi
fi
export PYSPARK_PYTHON

测试

[mongodb@mongodb01 bin]$ ./pyspark
Python 3.6.0 (default, Nov 13 2019, 12:35:54) 
[GCC 4.8.5 20150623 (Red Hat 4.8.5-39)] on linux
Type "help", "copyright", "credits" or "license" for more information.
19/11/13 15:01:09 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Using Spark's default log4j profile: org/apache/spark/log4j-defaults.properties
Setting default log level to "WARN".
To adjust logging level use sc.setLogLevel(newLevel). For SparkR, use setLogLevel(newLevel).
Welcome to
      ____              __
     / __/__  ___ _____/ /__
    _\ \/ _ \/ _ `/ __/  '_/
   /__ / .__/\_,_/_/ /_/\_\   version 2.4.4
      /_/

Using Python version 3.6.0 (default, Nov 13 2019 12:35:54)
SparkSession available as 'spark'.
>>> print(1)
1

安装 Python3 可能遇到异常情况

异常零、cd 无法进入解压目录

[mongodb@mongodb01 software]$ cd ./Python-3.6.0
-bash: cd: ./Python-3.6.0: 权限不够

修复方法，修改目录权限后再进入

[mongodb@mongodb01 software]$ sudo chmod -R 777 ./Python-3.6.0
[mongodb@mongodb01 software]$ ll
总用量 21740
drwxrwxrwx. 17  501 games     4096 12月 23 2016 Python-3.6.0
-rw-r--r--.  1 root root  22256403 12月 23 2016 Python-3.6.0.tgz
[mongodb@mongodb01 software]$ cd ./Python-3.6.0
[mongodb@mongodb01 Python-3.6.0]$ 

异常一、./configure 无法配置

[mongodb@mongodb01 Python-3.6.0]$ sudo ./configure --prefix=/usr/local/python3
checking build system type... x86_64-unknown-linux-gnu
checking host system type... x86_64-unknown-linux-gnu
checking for python3.6... no
checking for python3... no
checking for python... python
checking for --enable-universalsdk... no
checking for --with-universal-archs... no
checking MACHDEP... linux
checking for --without-gcc... no
checking for --with-icc... no
checking for gcc... no
checking for cc... no
checking for cl.exe... no
configure: error: in `/home/python3/software/Python-3.6.0':
configure: error: no acceptable C compiler found in $PATH
See `config.log' for more details
[mongodb@mongodb01 Python-3.6.0]$ 

修复方法，安装GCC后再次设置

[mongodb@mongodb01 Python-3.6.0]$ sudo yum install gcc

[mongodb@mongodb01 Python-3.6.0]$ sudo ./configure --prefix=/usr/local/python3

异常二、make install 无法安装

Traceback (most recent call last):
  File "/home/python3/software/Python-3.6.0/Lib/runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "/home/python3/software/Python-3.6.0/Lib/runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "/home/python3/software/Python-3.6.0/Lib/ensurepip/__main__.py", line 4, in <module>
    ensurepip._main()
  File "/home/python3/software/Python-3.6.0/Lib/ensurepip/__init__.py", line 189, in _main
    default_pip=args.default_pip,
  File "/home/python3/software/Python-3.6.0/Lib/ensurepip/__init__.py", line 102, in bootstrap
    _run_pip(args + [p[0] for p in _PROJECTS], additional_paths)
  File "/home/python3/software/Python-3.6.0/Lib/ensurepip/__init__.py", line 27, in _run_pip
    import pip
zipimport.ZipImportError: can't decompress data; zlib not available
make: *** [install] 错误 1

修复方法，安装zlib，安装成功后再次执行安装命令make install

[mongodb@mongodb01 Python-3.6.0]$ sudo yum install zlib zlib-devel -y