python异步爬虫框架 python爬虫异步加载

前言

作者：努力努力再努力

爬取qq音乐歌手数据接口数据

https://y.qq.com/portal/singer_list.html这是歌手列表的网址

分析网页

• f12开发者选项 找到network 里面有异步加载的数据
• 刷新看找数据
  看他们的response

https://u.y.qq.com/cgi-bin/musicu.fcg?-=getUCGI20652690515538596&g_tk=944669952&loginUin=1638786338&hostUin=0&format=json&inCharset=utf8&outCharset=utf-8&notice=0&platform=yqq.json&needNewCode=0&data={"comm"%3A{"ct"%3A24%2C"cv"%3A0}%2C"singerList"%3A{"module"%3A"Music.SingerListServer"%2C"method"%3A"get_singer_list"%2C"param"%3A{"area"%3A-100%2C"sex"%3A-100%2C"genre"%3A-100%2C"index"%3A-100%2C"sin"%3A0%2C"cur_page"%3A1}}}会有一个网址的response是歌手列表就是上述网址 记住这个网址然后就是请求这个网址

• 先贴源码

import jsonpath
import json
import requests
import csv
import time
def writecsv(content):
	with open('spider2.csv','a+',encoding='utf-8',newline='') as filecsv:
		writer = csv.writer(filecsv)
		writer.writerow(content)
writecsv(['歌手名','歌手地区','歌手id','歌手图片'])
def getHtml(url):
	response=requests.get(url)
	response.encoding='utf-8'
	html=response.text
	html=json.loads(html)
	print(html)
	singname=jsonpath.jsonpath(html,'$..singerList..singer_name')
	singcountry=jsonpath.jsonpath(html,'$..singerList..country')
	singer_mid=jsonpath.jsonpath(html,'$..singerList..singer_mid')
	singer_pic=jsonpath.jsonpath(html,'$..singerList..singer_pic')
	print(singer_mid)
	for index,i in enumerate(singname):
		writecsv([i,singcountry[index],singer_mid[index],singer_pic[index]])
index=0
for i in range(0,801,80):
	index=index+1
	getHtml('https://u.y.qq.com/cgi-bin/musicu.fcg?-=getUCGI01616088836276819&g_tk=5381&loginUin=0&hostUin=0&format=json&inCharset=utf8&outCharset=utf-8¬ice=0&platform=yqq.json&needNewCode=0&data={"comm":{"ct":24,"cv":0},"singerList":{"module":"Music.SingerListServer","method":"get_singer_list","param":{"area":-100,"sex":-100,"genre":-100,"index":-100,"sin":%(no)d,"cur_page":%(page)d}}}'% {'no':i,'page':index})
	time.sleep(3)
# index=0
# for i in range(0,801,80):
# 	index=index+1
# print(i)
# print(index)

html=json.loads(html)这里是将网页数据保存为json格式然后通过jsonpath解析json文件jsonpath语法类似xpath可以看下教程

使用jsonpath.jsonpath()解析json格式的数据$是根节点不能丢

singname=jsonpath.jsonpath(html,'$..singerList..singer_name')

{
“singerList”: {
“data”: {
“area”: -100,
“genre”: -100,
“index”: -100,
“sex”: -100,
“singerlist”: [
{
“country”: “内地”,
“singer_id”: 5062,
“singer_mid”: “002J4UUk29y8BY”,
“singer_name”: “薛之谦”,
“singer_pic”: “http://y.gtimg.cn/music/photo_new/T001R150x150M000002J4UUk29y8BY.webp”
},
{
“country”: “内地”,
“singer_id”: 1199300,
“singer_mid”: “0013RsPD3Xs0FG”,
“singer_name”: “半阳”,
“singer_pic”: “http://y.gtimg.cn/music/photo_new/T001R150x150M0000013RsPD3Xs0FG.webp”
},

这里…代表绝对哪里有就匹配到哪相当于xpath的//singlist下的singer_name很好找到可以先打印一下看错误再改依次将歌手名歌手国家等打印出来因为他们的个数都是相同的采用枚举的方式将每一个按行写入csv格式

with open('spider2.csv','a+',encoding='utf-8',newline='') as filecsv:第一个参数是文件名，第二个参数是追加读方式，第三个是编码方式避免乱码，第四个是控制空行删除空行,会自动生成这个csv文件成功后看有没有多出来的csv文件打开后

这样一个简单的异步加载的数据的爬取就成功了