Spark实战(三)本地连接远程Spark(Python环境)

一、Python环境准备

   远程服务器上Python版本要与本地想匹配，这里本地使用Anaconda来进行安装，远程服务器之间安装。

wget --no-check-certificate https://www.python.org/ftp/python/3.6.7/Python-3.6.7.tgz
tar -zxvf Python-3.6.7.tgz 
sudo ./configure --prefix=/usr/local/python3
make & make install
建立软连接
mv /usr/bin/python /usr/bin/python.bak
ln -s /usr/local/python3/bin/python3.6 /usr/bin/python

• 1
• 2
• 3
• 4
• 5
• 6
• 7

   本地之间下载Anaconda安装包，会自动安装相关环境。

二、配置PyCharm

   在file->setting->project interpreter中新增一个python环境，

   tool->Deployment->Configuration中新建一个SFTP

   Mappings中Local path路径正斜杠要修改一下改为反斜杠，否则可能会无法识别导致报错

    接下来需要在configuration中更改运行参数

   因为是远程环境，Environment Variables中需要指明服务器端python路径，spark路径，java路径。

   在file->setting->project interpreter下载安装py4j，pyspark等包，否则会报错

   以下是配置中遇到的一些常见错误：


   去上面搜索安装pyspark-stubs

    上面错误表示py4j版本不兼容，重新下载想匹配的版本

   spark FileNotFoundError: [WinError 2] 系统找不到指定的文件，需要查询edit configuration中python interpreter是否选择正确

   python3安装完成后默认没有安装pip，需要自己去安装，否则会出先pip未找到的错误

备份原来的源：输入命令sudo cp /etc/apt/sources.list /etc/apt/sources_init.list
输入命令sudo vi /etc/apt/sources.list
添加如下
deb http://mirrors.aliyun.com/ubuntu/ trusty main restricted universe multiverse  
deb http://mirrors.aliyun.com/ubuntu/ trusty-security main restricted universe multiverse  
deb http://mirrors.aliyun.com/ubuntu/ trusty-updates main restricted universe multiverse  
deb http://mirrors.aliyun.com/ubuntu/ trusty-proposed main restricted universe multiverse  
deb http://mirrors.aliyun.com/ubuntu/ trusty-backports main restricted universe multiverse  
deb-src http://mirrors.aliyun.com/ubuntu/ trusty main restricted universe multiverse  
deb-src http://mirrors.aliyun.com/ubuntu/ trusty-security main restricted universe multiverse  
deb-src http://mirrors.aliyun.com/ubuntu/ trusty-updates main restricted universe multiverse  
deb-src http://mirrors.aliyun.com/ubuntu/ trusty-proposed main restricted universe multiverse  
deb-src http://mirrors.aliyun.com/ubuntu/ trusty-backports main restricted universe multiverse 

sudo apt-get update
sudo apt-get upgrade
sudo apt-get install python3-pip
sudo apt-get install python-pip