centos yum安装zip centos yum安装python3.9

转载

mob64ca14133dc6 2024-05-26 16:11:45

文章标签 centos yum安装zip ide python Python 文章分类 架构后端开发

本人不懂爬虫技术。需要获取页面信息的时候，简单的需求直接用程序扒。复杂的需求己用chrome-mini向下渗透。前两天去参加了一个爬虫技术的聚会，发现这个领域非常有意思，所以回来搭建了个scrapy环境，打算学习学习。
由于本人是小白，所以搭建过程一步一个坎。这里记录下来，希望能帮助到其他新接触的同学。

一、安装python 3.7.9

centos7自带python2.7.5版本，我尝试在这个版本下直接安装scrapy2，搭建过程没有报错，但是运行程序的时候出问题了，说python2版本太低，所以只好安装python3了。

1、安装一次性开发工具，源码安装软件需要这个
yum -y groupinstall "Development tools"

2、安装各种依赖包
yum -y install zlib-devel bzip2-devel openssl-devel ncurses-devel sqlite-devel readline-devel tk-devel gdbm-devel db4-devel libpcap-devel xz-devel libffi-devel

3、下载python3.7.9安装包
wget "https://www.python.org/ftp/python/3.7.9/Python-3.7.9.tgz"

4、解压缩
tar -zxvf Python-3.7.9.tgz

5、进入根目录
cd Python-3.7.9

6、编译
./configure --prefix=/usr/local/python3

7、安装
make && make install

8、创建软链接
ln -s /usr/local/python3/bin/python3 /usr/bin/python3
ln -s /usr/local/python3/bin/pip3 /usr/bin/pip3

9、验证

[root@seeker-01 ~]# python
     Python 2.7.5 (default, Aug  7 2019, 00:51:29) 
     [GCC 4.8.5 20150623 (Red Hat 4.8.5-39)] on linux2
     Type "help", "copyright", "credits" or "license" for more information.
     >>>    [root@seeker-01 ~]# python3
     Python 3.7.9 (default, Aug 28 2020, 13:28:49) 
     [GCC 4.8.5 20150623 (Red Hat 4.8.5-39)] on linux
     Type "help", "copyright", "credits" or "license" for more information.
     >>>

二、安装scrapy2

1、安装scrapy2
pip3 install scrapy

2、在python3 shell中验证scrapy2

[root@seeker-01 ~]# python3
     Python 3.7.9 (default, Aug 28 2020, 13:28:49) 
     [GCC 4.8.5 20150623 (Red Hat 4.8.5-39)] on linux
     Type "help", "copyright", "credits" or "license" for more information.
     >>> import scrapy
     >>> scrapy.version_info
     (2, 3, 0)
     >>>

3、创建软scrapy链接
ln -s /usr/local/python3/bin/scrapy /usr/bin/scrapy

4、验证scrapy2

[root@seeker-01 ~]# scrapy
     Scrapy 2.3.0 - no active project    Usage:
       scrapy <command> [options] [args]    Available commands:
       bench         Run quick benchmark test
       commands      
       fetch         Fetch a URL using the Scrapy downloader
       genspider     Generate new spider using pre-defined templates
       runspider     Run a self-contained spider (without creating a project)
       settings      Get settings values
       shell         Interactive scraping console
       startproject  Create new project
       version       Print Scrapy version
       view          Open URL in browser, as seen by Scrapy      [ more ]      More commands available when run from project directory
    Use "scrapy <command> -h" to see more info about a command

三、创建一个scrapy项目

1、创建项目命令
scrapy startproject 项目名称(我的项目叫mytest01)

[root@seeker-01 ~]# scrapy startproject mytest01
     New Scrapy project 'my_test', using template directory '/usr/local/python3/lib/python3.7/site-packages/scrapy/templates/project', created in:
         /root/mytest01    You can start your first spider with:
         cd mytest01
         scrapy genspider example example.com

执行完成后，我的/root目录下就创建了一个mytest01目录，
/root/mytest01/mytest01/spiders就是放置.py文件的位置，我创建的文件名是test01.py

2、写一个scrapy功能
在/root/mytest01/mytest01/spiders目录下创建一个test01.py文件，写入下面内容：

import scrapy
class mingyan(scrapy.Spider): #需要继承scrapy.Spider类
    name = "mytest01" #蜘蛛名
     def start_requests(self): #由此方法通过下面链接爬取页面
         urls = [ #爬取的链接
             'http://lab.scrapyd.cn/page/1/',
             'http://lab.scrapyd.cn/page/2/',
         ]

for url in urls:
yield scrapy.Request(url=url, callback=self.parse) #爬取到的页面如何处理？提交给parse方法处理

def parse(self, response):

"""
start_requests已经爬取到页面，那如何提取我们想要的内容呢？那就可以在这个方法里面定义。
这里的话，并木有定义，只是简单的把页面做了一个保存，并没有涉及提取我们想要的数据，后面会慢慢说到
也就是用xpath、正则、或是css进行相应提取，这个例子就是让你看看scrapy运行的流程：
1、定义链接；
2、通过链接爬取（下载）页面；
3、定义规则，然后提取数据；
就是这么个流程，似不似很简单呀？

"""
         page = response.url.split("/")[-2]     #根据上面的链接提取分页,如：/page/1/，提取到的就是：1
         filename = 'mingyan-%s.html' % page    #拼接文件名，如果是第一页，最终文件名便是：mingyan-1.html
         with open(filename, 'wb') as f:        #python文件操作，不多说了；
             f.write(response.body)             #刚才下载的页面去哪里了？response.body就代表了刚才下载的页面！
         self.log('保存文件: %s' % filename)      # 打个日志

3、执行scrapy文件
scrapy crawl mytest01

2020-08-28 14:27:15 [scrapy.utils.log] INFO: Scrapy 2.3.0 started (bot: mingyan2)
2020-08-28 14:27:15 [scrapy.utils.log] INFO: Versions: lxml 4.5.2.0, libxml2 2.9.10, cssselect 1.1.0, parsel 1.6.0, w3lib 1.22.0, Twisted 20.3.0, Python 3.7.9 (default, Aug 28 2020, 13:28:49) - [GCC 4.8.5 20150623 (Red Hat 4.8.5-39)], pyOpenSSL 19.1.0 (OpenSSL 1.1.1g 21 Apr 2020), cryptography 3.1, Platform Linux-3.10.0-1062.12.1.el7.x86_64-x86_64-with-centos-7.6.1810-Core
2020-08-28 14:27:15 [scrapy.utils.log] DEBUG: Using reactor: twisted.internet.epollreactor.EPollReactor
2020-08-28 14:27:15 [scrapy.crawler] INFO: Overridden settings:
{'BOT_NAME': 'MyTest01',
'NEWSPIDER_MODULE': 'MyTest01.spiders',
'ROBOTSTXT_OBEY': True,
'SPIDER_MODULES': ['MyTest01.spiders']}
2020-08-28 14:27:15 [scrapy.extensions.telnet] INFO: Telnet Password: 168ea9bb811b4d38
2020-08-28 14:27:15 [scrapy.middleware] INFO: Enabled extensions:
['scrapy.extensions.corestats.CoreStats',
'scrapy.extensions.telnet.TelnetConsole',
'scrapy.extensions.memusage.MemoryUsage',
'scrapy.extensions.logstats.LogStats']
2020-08-28 14:27:15 [scrapy.middleware] INFO: Enabled downloader middlewares:
['scrapy.downloadermiddlewares.robotstxt.RobotsTxtMiddleware',
'scrapy.downloadermiddlewares.httpauth.HttpAuthMiddleware',
'scrapy.downloadermiddlewares.downloadtimeout.DownloadTimeoutMiddleware',
'scrapy.downloadermiddlewares.defaultheaders.DefaultHeadersMiddleware',
'scrapy.downloadermiddlewares.useragent.UserAgentMiddleware',
'scrapy.downloadermiddlewares.retry.RetryMiddleware',
'scrapy.downloadermiddlewares.redirect.MetaRefreshMiddleware',
'scrapy.downloadermiddlewares.httpcompression.HttpCompressionMiddleware',
'scrapy.downloadermiddlewares.redirect.RedirectMiddleware',
'scrapy.downloadermiddlewares.cookies.CookiesMiddleware',
'scrapy.downloadermiddlewares.httpproxy.HttpProxyMiddleware',
'scrapy.downloadermiddlewares.stats.DownloaderStats']
2020-08-28 14:27:15 [scrapy.middleware] INFO: Enabled spider middlewares:
['scrapy.spidermiddlewares.httperror.HttpErrorMiddleware',
'scrapy.spidermiddlewares.offsite.OffsiteMiddleware',
'scrapy.spidermiddlewares.referer.RefererMiddleware',
'scrapy.spidermiddlewares.urllength.UrlLengthMiddleware',
'scrapy.spidermiddlewares.depth.DepthMiddleware']
2020-08-28 14:27:15 [scrapy.middleware] INFO: Enabled item pipelines:
[]
2020-08-28 14:27:15 [scrapy.core.engine] INFO: Spider opened
2020-08-28 14:27:15 [scrapy.extensions.logstats] INFO: Crawled 0 pages (at 0 pages/min), scraped 0 items (at 0 items/min)
2020-08-28 14:27:15 [scrapy.extensions.telnet] INFO: Telnet console listening on 127.0.0.1:6023
2020-08-28 14:27:15 [scrapy.core.engine] DEBUG: Crawled (404) <GET http://lab.scrapyd.cn/robots.txt> (referer: None)
2020-08-28 14:27:15 [scrapy.core.engine] DEBUG: Crawled (200) <GET http://lab.scrapyd.cn/page/1/> (referer: None)
2020-08-28 14:27:15 [scrapy.core.engine] DEBUG: Crawled (200) <GET http://lab.scrapyd.cn/page/2/> (referer: None)
2020-08-28 14:27:15 [mingyan2] DEBUG: 保存文件: mingyan-1.html
2020-08-28 14:27:15 [mingyan2] DEBUG: 保存文件: mingyan-2.html
2020-08-28 14:27:15 [scrapy.core.engine] INFO: Closing spider (finished)
2020-08-28 14:27:15 [scrapy.statscollectors] INFO: Dumping Scrapy stats:
{'downloader/request_bytes': 666,
'downloader/request_count': 3,
'downloader/request_method_count/GET': 3,
'downloader/response_bytes': 6436,
'downloader/response_count': 3,
'downloader/response_status_count/200': 2,
'downloader/response_status_count/404': 1,
'elapsed_time_seconds': 0.344661,
'finish_reason': 'finished',
'finish_time': datetime.datetime(2020, 8, 28, 6, 27, 15, 851756),
'log_count/DEBUG': 5,
'log_count/INFO': 10,
'memusage/max': 47456256,
'memusage/startup': 47456256,
'response_received_count': 3,
'robotstxt/request_count': 1,
'robotstxt/response_count': 1,
'robotstxt/response_status_count/404': 1,
'scheduler/dequeued': 2,
'scheduler/dequeued/memory': 2,
'scheduler/enqueued': 2,
'scheduler/enqueued/memory': 2,
'start_time': datetime.datetime(2020, 8, 28, 6, 27, 15, 507095)}
2020-08-28 14:27:15 [scrapy.core.engine] INFO: Spider closed (finished)