BeautifulSoup文档3-详细方法 | 如何对文档树进行遍历？

推荐原创

虫无涯N 2023-02-16 15:23:22 博主文章分类：测试工具 ©著作权

文章标签 python 子节点父节点 beautifulsoup 文章分类 软件测试

©著作权归作者所有：来自51CTO博客作者虫无涯N的原创作品，请联系作者获取转载授权，否则将追究法律责任

(3-详细方法 | 如何对文档树进行遍历？)

以下实例还是官网的例子：

html_doc = """
<html><head><title>The Dormouse's story</title></head>
    <body>
<p class="title"><b>The Dormouse's story</b></p>

<p class="story">Once upon a time there were three little sisters; and their names were
<a rel="nofollow" href="http://example.com/elsie" class="sister" id="link1">Elsie</a>,
<a rel="nofollow" href="http://example.com/lacie" class="sister" id="link2">Lacie</a> and
<a rel="nofollow" href="http://example.com/tillie" class="sister" id="link3">Tillie</a>;
and they lived at the bottom of a well.</p>

<p class="story">...</p>
"""

from bs4 import BeautifulSoup
soup = BeautifulSoup(html_doc, 'html.parser')

1 子节点

1.1 tag的名字

1.1.1 获取head标签

# 获取head
print(soup.head)

输出为：

<head><title>The Dormouse's story</title></head>

1.1.2 获取title

# 获取title
print(soup.title)

输出为：

<title>The Dormouse's story</title>

1.1.3 获取`body`标签中的第一个`b`标签

# 获取<body>标签中的第一个<b>标签
print(soup.body.b)

输出为：

<b>The Dormouse's story</b>

1.1.4 获得当前名字的第一个tag

# 获得当前名字的第一个tag
print(soup.a)

输出为：

<a rel="nofollow" class="sister" href="http://example.com/elsie" id="link1">Elsie</a>

1.1.5 获取所有a标签

# 获取所有a标签
print(soup.find_all('a'))

输出为：

[<a rel="nofollow" class="sister" href="http://example.com/elsie" id="link1">Elsie</a>, 
<a rel="nofollow" class="sister" href="http://example.com/lacie" id="link2">Lacie</a>, 
<a rel="nofollow" class="sister" href="http://example.com/tillie" id="link3">Tillie</a>]

1.2 .contents 和 .children

.contents 属性将tag子节点以列表的方式输出:

# .contents属性将`tag`子节点以列表的方式输出
head_tag = soup.head
print(head_tag)
print(head_tag.contents)
title_tag = head_tag.contents[0]
print(title_tag)
print(title_tag.contents)

输出为：

<head><title>The Dormouse's story</title></head>
[<title>The Dormouse's story</title>]
<title>The Dormouse's story</title>
["The Dormouse's story"]

.children 生成器,可以对tag的子节点进行循环:

# .children生成器,可以对tag的子节点进行循环
for child in title_tag.children:
    print(child)

输出为：

The Dormouse's story

1.3 .descendants

.descendants 属性对所有tag的子孙节点进行递归循环：

for child in head_tag.descendants:
    print(child)

输出为：

<title>The Dormouse's story</title>
The Dormouse's story

1.4 .string

如果tag只有一个 NavigableString 类型子节点,那么这个tag可以使用 .string 得到子节点：

# 如果tag只有一个 NavigableString 类型子节点,那么这个tag可以使用 .string 得到子节点:
print(title_tag.string)

输出为：

The Dormouse's story

1.5 .strings 和 stripped_strings

如果tag中包含多个字符串,可以使用 .strings 来循环获取:

for string in soup.strings:
    print(repr(string))

输出为：

'\n'
"The Dormouse's story"
'\n'
'\n'
"The Dormouse's story"
'\n'
'Once upon a time there were three little sisters; and their names were\n'
'Elsie'
',\n'
'Lacie'
' and\n'
'Tillie'
';\nand they lived at the bottom of a well.'
'\n'
'...'
'\n'

使用 .stripped_strings 可以去除多余空白内容:

# 使用 .stripped_strings 可以去除多余空白内容:
for string in soup.stripped_strings:
    print(repr(string))

输出为：

"The Dormouse's story"
"The Dormouse's story"
'Once upon a time there were three little sisters; and their names were'
'Elsie'
','
'Lacie'
'and'
'Tillie'
';\nand they lived at the bottom of a well.'
'...'

2 父节点

2.1 .parent

通过 .parent 属性来获取某个元素的父节点；
head标签是title标签的父节点:

#  通过 .parent 属性来获取某个元素的父节点，head标签是title标签的父节点:
title_tag = soup.title
print(title_tag)
print(title_tag.parent)

输出为：

<title>The Dormouse's story</title>
<head><title>The Dormouse's story</title></head>

2.2 .parents

通过元素的 .parents 属性可以递归得到元素的所有父辈节点：

link = soup.a
print(link)
for parent in link.parents:
    if parent is None:
        print(parent)
    else:
        print(parent.name)

输出为：

<a rel="nofollow" class="sister" href="http://example.com/elsie" id="link1">Elsie</a>
p
body
html
[document]

3 兄弟节点

两个标签是同一层，他们是同一个元素的子节点，则这两个标签是兄弟节点；
如下，b和c标签是兄弟节点：

sibling_soup = BeautifulSoup("<a><b>text1</b><c>text2</c></b></a>")
print(sibling_soup.prettify())

输出为：

<a>
 <b>
  text1
 </b>
 <c>
  text2
 </c>
</a>

3.1 .next_sibling 和 .previous_sibling

使用 .next_sibling 和 .previous_sibling 属性来查询兄弟节点:

# 使用 .next_sibling 和 .previous_sibling 属性来查询兄弟节点:
print(sibling_soup.b.next_sibling)
print(sibling_soup.c.previous_sibling)

输出为：

<c>text2</c>
<b>text1</b>

3.2 .next_siblings 和 .previous_siblings

通过 .next_siblings 和 .previous_siblings 属性可以对当前节点的兄弟节点迭代输出：

for sibling in soup.a.next_siblings:
    print(repr(sibling))
for sibling in soup.find(id="link3").previous_siblings:
    print(repr(sibling))

输出为：

',\n'
<a rel="nofollow" class="sister" href="http://example.com/lacie" id="link2">Lacie</a>
' and\n'
<a rel="nofollow" class="sister" href="http://example.com/tillie" id="link3">Tillie</a>
';\nand they lived at the bottom of a well.'
' and\n'
<a rel="nofollow" class="sister" href="http://example.com/lacie" id="link2">Lacie</a>
',\n'
<a rel="nofollow" class="sister" href="http://example.com/elsie" id="link1">Elsie</a>
'Once upon a time there were three little sisters; and their names were\n'

4 回退和前进

4.1 .next_element 和 .previous_element

.next_element 属性指向解析过程中下一个被解析的对象(字符串或tag)：

last_a_tag = soup.find("a", id="link3")
print(last_a_tag)
print(last_a_tag.next_sibling)

输出为：

<a rel="nofollow" class="sister" href="http://example.com/tillie" id="link3">Tillie</a>
;
and they lived at the bottom of a well.

.previous_element 属性刚好与 .next_element 相反,它指向当前被解析的对象的前一个解析对象：

print(last_a_tag.previous_element)
print(last_a_tag.previous_element.next_element)

输出为：

 and
<a rel="nofollow" class="sister" href="http://example.com/tillie" id="link3">Tillie</a>

4.2 .next_elements 和 .previous_elements

通过 .next_elements 和 .previous_elements 的迭代器就可以向前或向后访问文档的解析内容：

for element in last_a_tag.next_elements:
    print(repr(element))

输出为：

'Tillie'
';\nand they lived at the bottom of a well.'
'\n'
<p class="story">...</p>
'...'
'\n'

5 本为涉及的源码：

# -*- coding:utf-8 -*-
# 作者：NoamaNelson
# 日期：2023/2/16
# 文件名称：bs03.py
# 作用：BeautifulSoup的使用
# 博客：https://blog.csdn.net/NoamaNelson

from bs4 import BeautifulSoup

html_doc = """
<html><head><title>The Dormouse's story</title></head>
    <body>
<p class="title"><b>The Dormouse's story</b></p>

<p class="story">Once upon a time there were three little sisters; and their names were
<a rel="nofollow" href="http://example.com/elsie" class="sister" id="link1">Elsie</a>,
<a rel="nofollow" href="http://example.com/lacie" class="sister" id="link2">Lacie</a> and
<a rel="nofollow" href="http://example.com/tillie" class="sister" id="link3">Tillie</a>;
and they lived at the bottom of a well.</p>

<p class="story">...</p>
"""

soup = BeautifulSoup(html_doc, 'html.parser')

# ====== 子节点======
# 获取head
print(soup.head)

# 获取title
print(soup.title)

# 获取<body>标签中的第一个<b>标签
print(soup.body.b)

# 获得当前名字的第一个tag
print(soup.a)

# 获取所有a标签
print(soup.find_all('a'))

# .contents属性将`tag`子节点以列表的方式输出
head_tag = soup.head
print(head_tag)
print(head_tag.contents)
title_tag = head_tag.contents[0]
print(title_tag)
print(title_tag.contents)

# .children生成器,可以对tag的子节点进行循环
for child in title_tag.children:
    print(child)

# .descendants属性对所有tag的子孙节点进行递归循环
for child in head_tag.descendants:
    print(child)

# 如果tag只有一个 NavigableString 类型子节点,那么这个tag可以使用 .string 得到子节点:
print(title_tag.string)

# 如果tag中包含多个字符串,可以使用 .strings来循环获取
for string in soup.strings:
    print(repr(string))

# 使用 .stripped_strings 可以去除多余空白内容:
for string in soup.stripped_strings:
    print(repr(string))


# ====== 父节点======
#  通过 .parent 属性来获取某个元素的父节点，head标签是title标签的父节点:
title_tag = soup.title
print(title_tag)
print(title_tag.parent)

# 通过元素的 .parents 属性可以递归得到元素的所有父辈节点
link = soup.a
print(link)
for parent in link.parents:
    if parent is None:
        print(parent)
    else:
        print(parent.name)


# ====== 兄弟节点======
sibling_soup = BeautifulSoup("<a><b>text1</b><c>text2</c></b></a>", 'html.parser')
print(sibling_soup.prettify())

# 使用 .next_sibling 和 .previous_sibling 属性来查询兄弟节点:
print(sibling_soup.b.next_sibling)
print(sibling_soup.c.previous_sibling)

# 通过 .next_siblings 和 .previous_siblings 属性可以对当前节点的兄弟节点迭代输出
for sibling in soup.a.next_siblings:
    print(repr(sibling))
for sibling in soup.find(id="link3").previous_siblings:
    print(repr(sibling))


# ====== 回退和前进======
# .next_element 属性指向解析过程中下一个被解析的对象(字符串或tag)
last_a_tag = soup.find("a", id="link3")
print(last_a_tag)
print(last_a_tag.next_sibling)

# .previous_element 属性刚好与 .next_element 相反,它指向当前被解析的对象的前一个解析对象
print(last_a_tag.previous_element)
print(last_a_tag.previous_element.next_element)

# 通过 .next_elements 和 .previous_elements 的迭代器就可以向前或向后访问文档的解析内容
for element in last_a_tag.next_elements:
    print(repr(element))