学习Python开发工具库-爬虫
原创
©著作权归作者所有：来自51CTO博客作者六月暴雪飞梨花的原创作品，请联系作者获取转载授权，否则将追究法律责任
🏮1 爬虫
- 什么是爬虫？
简单来说，就是自动抓取互联网信息的程序。
- 爬虫提取网页数据流程
- lxml模块和Xpath
lxml是基于libxml2这一XML解析库的Python封装，是python的库。lxml支持XML和HTML的解析，也支持XPath的方式解析，解析效率也比较高。
🏮2 lxml模块
在lxml库的模块中，使用最多的要数lxml.etree ，其次就是 Element 对象。在解析数据的时候，大量的代码都是基于 Element 对象的 API 实现。
🎈2.1 安装
打开终端，输入安装命令pip install lxml，提示有Successfully则表示安装成功。
Aion.$ python -m pip install lxml
Collecting lxml
Downloading lxml-4.9.1.tar.gz (3.4 MB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 3.4/3.4 MB 23.8 kB/s eta 0:00:00
Preparing metadata (setup.py) ... done
Building wheels for collected packages: lxml
Building wheel for lxml (setup.py) ... done
Created wheel for lxml: filename=lxml-4.9.1-cp310-cp310-macosx_12_0_x86_64.whl size=1672209 sha256=5b6b5d05a4dc63d8a25a267b045a908dc6249f1e4bc0e847b0b8fbdf2d1a100c
Stored in directory: xxx/a4/ec/7b/8acde6da24b5aabeee049213d5bec12d1e9214d3cae276387b
Successfully built lxml
Installing collected packages: lxml
Successfully installed lxml-4.9.1
下载过程有点慢，需要等待几分钟。
🎈2.2 解析HTML字符串
>>>from lxml import etree
>>>text = '''
>>><html><body>
>>> <div class="key">
>>> <div class="name">小明</div>
>>> <div class="age">21</div>
>>> <div class="address">广东省广州市天河区白云路123号</div>
>>> </div>
>>></body></html>
>>>'''
# 开始初始化，传入一个html形式的字符串
>>>html = etree.HTML(text)
>>>print(html)
<Element html at 0x10af8cbc0>
>>>print(type)
<class 'type'>
# 将字符串序列化为html字符串
>>>result = etree.tostring(html).decode('utf-8')
>>>print(result)
<html><body>
<div class="key">
<div class="name">小明</div>
<div class="age">21</div>
<div class="address">广东省广州市天河区白云路123号</div>
</div>
</body></html>
>>>print(type(result))
<class 'str'>
🎈2.3 解析HTML网页文件
模拟实验从HTML文件解析，首先创建一个实验使用的html文件，命名为c17.html，内容还是2.2所示的字符串内容即可。
>>> from lxml import etree
>>> html = etree.parse('c17.html')
>>>
>>> result = etree.tostring(html).decode('utf-8')
>>>
>>> print(result)
<html>
<body>
<div class="key">
<div class="name">小明</div>
<div class="age">21</div>
<div class="address">广东省广州市天河区白云路123号</div>
</div>
</body>
</html>
>>>
>>> print(type(result))
<class 'str'>
>>>
>>> html = etree.HTML(result)
>>>
>>> print(html)
<Element html at 0x10d605880>
>>>
>>> print(type)
<class 'type'>
>>>
🏮3 XPath 详解
XPath 是一门在 XML 文档中查找信息的语言。XPath 可用来在 XML 文档中对元素和属性进行遍历。XPath 是 W3C XSLT 标准的主要元素，并且 XQuery 和 XPointer 都构建于 XPath 表达之上。
🎈3.1 创建一个HTML文件
我是在豆瓣电影主页任意抽取一段网页代码，然后拷贝到一个文件中，命名文件为c16.html：
<html>
<div class="billboard-bd">
<table>
<tbody><tr>
<td class="order">1</td>
<td class="title"><a onclick="moreurl(this, {from:'mv_rk'})" href="https://movie.douban.com/subject/35056243/">13条命</a></td>
</tr>
<tr>
<td class="order">2</td>
<td class="title"><a onclick="moreurl(this, {from:'mv_rk'})" href="https://movie.douban.com/subject/35338562/">全职</a></td>
</tr>
<tr>
<td class="order">3</td>
<td class="title"><a onclick="moreurl(this, {from:'mv_rk'})" href="https://movie.douban.com/subject/35073886/">分手的决心</a></td>
</tr>
<tr>
<td class="order">4</td>
<td class="title"><a onclick="moreurl(this, {from:'mv_rk'})" href="https://movie.douban.com/subject/35447590/">失踪的莱昂纳多</a></td>
</tr>
<tr>
<td class="order">5</td>
<td class="title"><a onclick="moreurl(this, {from:'mv_rk'})" href="https://movie.douban.com/subject/27124451/">洞</a></td>
</tr>
<tr>
<td class="order">6</td>
<td class="title"><a onclick="moreurl(this, {from:'mv_rk'})" href="https://movie.douban.com/subject/26305582/">猫王</a></td>
</tr>
<tr>
<td class="order">7</td>
<td class="title"><a onclick="moreurl(this, {from:'mv_rk'})" href="https://movie.douban.com/subject/35408460/">太他妈好相处了</a></td>
</tr>
<tr>
<td class="order">8</td>
<td class="title"><a onclick="moreurl(this, {from:'mv_rk'})" href="https://movie.douban.com/subject/26642033/">小黄人大眼萌：神偷奶爸前传</a></td>
</tr>
<tr>
<td class="order">9</td>
<td class="title"><a onclick="moreurl(this, {from:'mv_rk'})" href="https://movie.douban.com/subject/34940681/">小野田的丛林万夜</a></td>
</tr>
<tr>
<td class="order">10</td>
<td class="title"><a onclick="moreurl(this, {from:'mv_rk'})" href="https://movie.douban.com/subject/35490166/">同车异路</a></td>
</tr>
</tbody></table>
</div>
<html>
🎈3.2 获取所有标签
很是奇怪，我按照老师博文的步骤来执行，是报错的，错误如下：
>>> html_c16 = etree.parse("c16.html")
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "src/lxml/etree.pyx", line 3538, in lxml.etree.parse
File "src/lxml/parser.pxi", line 1876, in lxml.etree._parseDocument
File "src/lxml/parser.pxi", line 1902, in lxml.etree._parseDocumentFromURL
File "src/lxml/parser.pxi", line 1805, in lxml.etree._parseDocFromFile
File "src/lxml/parser.pxi", line 1177, in lxml.etree._BaseParser._parseDocFromFile
File "src/lxml/parser.pxi", line 615, in lxml.etree._ParserContext._handleParseResultDoc
File "src/lxml/parser.pxi", line 725, in lxml.etree._handleParseResult
File "src/lxml/parser.pxi", line 654, in lxml.etree._raiseParseError
File "c16.html", line 7
lxml.etree.XMLSyntaxError: Input is not proper UTF-8, indicate encoding !
Bytes: 0xCC 0xF5 0xC3 0xFC, line 7, column 138看下错误的原因lxml.etree.XMLSyntaxError: Input is not proper UTF-8, indicate encoding !，这才看明白，既然需要这个参数（其实是我的html文件不规范导致），那就增加参数来妥协吧。
>>> parser = etree.HTMLParser(encoding='utf-8')
>>> html_c16 = etree.parse('c16.html', parser=parser)
>>> html_tr = html_c16.xpath('//tr')
>>> print(html_tr)
[<Element tr at 0x10b915980>, <Element tr at 0x10b9159c0>, <Element tr at 0x10b915a00>, <Element tr at 0x10b915a40>, <Element tr at 0x10b915a80>, <Element tr at 0x10b915b00>, <Element tr at 0x10b915b40>, <Element tr at 0x10b915b80>, <Element tr at 0x10b915bc0>, <Element tr at 0x10b915ac0>]
>>>
🎈3.3 获取所有的class属性
>>> html_c16 = etree.parse('c16.html',parser=parser)
>>> HTML_td_class = html_c16.xpath('//td/@class')
>>>
>>> print(HTML_td_class)
['order', 'title', 'order', 'title', 'order', 'title', 'order', 'title', 'order', 'title', 'order', 'title', 'order', 'title', 'order', 'title', 'order', 'title', 'order', 'title']
>>>
🎈3.4 获取所有下的特定标签
例如，这里我想获取<td>标签下面超链接hre为https://movie.douban.com/subject/35408460/的a标签。
>>>
>>> html_a = html_c16.xpath('//td/a[@href="https://movie.douban.com/subject/35408460/"]')
>>>
>>> print(html_a)
[<Element a at 0x10b916740>]
🎈3.5 获取所有的下的全部
>>> html_a_list = html_c16.xpath('//td//a')
>>> print(html_a_list)
[<Element a at 0x10b916840>, <Element a at 0x10b916880>, <Element a at 0x10b9168c0>, <Element a at 0x10b916900>, <Element a at 0x10b916940>, <Element a at 0x10b9169c0>, <Element a at 0x10b916740>, <Element a at 0x10b916a00>, <Element a at 0x10b916a40>, <Element a at 0x10b916980>]
>>>
>>> html_a_method = html_c16.xpath('//td/a//@onclick')
>>> print(html_a_method)
["moreurl(this, {from:'mv_rk'})", "moreurl(this, {from:'mv_rk'})", "moreurl(this, {from:'mv_rk'})", "moreurl(this, {from:'mv_rk'})", "moreurl(this, {from:'mv_rk'})", "moreurl(this, {from:'mv_rk'})", "moreurl(this, {from:'mv_rk'})", "moreurl(this, {from:'mv_rk'})", "moreurl(this, {from:'mv_rk'})", "moreurl(this, {from:'mv_rk'})"]
>>>🎈3.7 获取所有class=order标签
>>> index_result = html_c16.xpath('//*[@class="order"]')
>>> print(index_result)
[<Element td at 0x10b915e00>, <Element td at 0x10b915f00>, <Element td at 0x10b916000>, <Element td at 0x10b916140>, <Element td at 0x10b916240>, <Element td at 0x10b916300>, <Element td at 0x10b916400>, <Element td at 0x10b916500>, <Element td at 0x10b916600>, <Element td at 0x10b916700>]
>>> print(index_result[0].tag)
td
>>>🏮4 总结
本文按照官方提供的示例实验操作，给自己学习增添不少便利。其实操作并不难，你如果发现了我的操作全都是使用的Python来实现的时候，你的重心就应该去放在这个类上面，里面所有的方法都有注释，看一遍就能学会。本篇文章就到这里了，希望能够给你带来帮助，也希望您能够多多关注我们的更多内容！
🏮5 参考
[1] 参考重要文档： https://lxml.de/[2] 项目开源地址在：https://github.com/lxml/lxml
