问题描述： (2021年7月12日17:9:46)爬取豆瓣展示数据

1： 查看触发条件：（点击加载更多即可）

request豆瓣 the post 豆瓣_Windows

二： 查看 Fiddler 数据， 可以看到 get 请求

request豆瓣 the post 豆瓣_Chrome_02

三： 如果不用 header， 会报茶壶错误（418）

my_agent = "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.114 Safari/537.36"

问题解决： 代码直接展示

# -*- coding=utf-8 -*-

"""
豆瓣电影 get json 数据
第一页数据
https://movie.douban.com/j/new_search_subjects?sort=U&range=0,10&tags=&start=20
    webform  数据
sort	U
range	0,10
tags
start	20
第二页数据
https://movie.douban.com/j/new_search_subjects?sort=U&range=0,10&tags=&start=40
    webform 数据
sort	U
range	0,10
tags
start	40

    变化，每次多 20 条数据
"""

# 1: 引包
import urllib.request

# 2: 请求得到响应
target_url = "https://movie.douban.com/j/new_search_subjects?sort=U&range=0,10&tags=&start=20"

my_agent = "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.114 Safari/537.36"
user_headers = {
    "User-Agent": my_agent
}

req = urllib.request.Request(url=target_url, headers=user_headers)
resp = urllib.request.urlopen(req)

# 3： 获取状态码
print(resp.getcode())
# 4: 获取内容
print(resp.read().decode("utf-8"))

运行结果：

200
{"data":[{"directors":["陈思诚"],"rate":"5.4","cover_x":2143,"star":"25","title":"唐人街探案3","url":"https:\/\/movie.douban.com\/subject\/27619748\/","casts":["王宝强","刘昊然","妻夫木聪","托尼·贾","长泽雅美"],"cover":"https://img2.doubanio.com\/view\/photo\/s_ratio_poster\/public\/p2622388983.jpg","id":"27619748","cover_y":3000},{"directors":["詹姆斯·卡梅隆"],"rate":"9.4","cover_x":2015,"star":"45","title":"泰坦尼克号","url":"https:\/\/movie.douban.com\/subject\/1292722\/","casts":["莱昂纳多·迪卡普里奥","凯特·温丝莱特","比利·赞恩","凯西·贝茨","弗兰西丝·费舍"],"cover":"https://img9.doubanio.com\/view\/photo\/s_ratio_poster\/public\/p457760035.jpg","id":"1292722","cover_y":3000},{"directors":["凯特·绍特兰"],"rate":"6.6","cover_x":864,"star":"35","title":"黑寡妇","url":"https:\/\/movie.douban.com\/subject\/25828589\/","casts":["斯嘉丽·约翰逊","弗洛伦丝·皮尤","蕾切尔·薇兹","大卫·哈伯","艾尔·安德森"],"cover":"https://img1.doubanio.com\/view\/photo\/s_ratio_poster\/public\/p2665872718.jpg","id":"25828589","cover_y":1280},{"directors":["涅提·蒂瓦里"],"rate":"9.0","cover_x":1414,"star":"45","title":"摔跤吧！爸爸","url":"https:\/\/movie.douban.com\/subject\/26387939\/","casts":["阿米尔·汗","法缇玛·萨那·纱卡","桑亚·玛荷塔","阿帕尔夏克提·库拉那","沙克希·坦沃"],"cover":"https://img1.doubanio.com\/view\/photo\/s_ratio_poster\/public\/p2401676338.jpg","id":"26387939","cover_y":2048},{"directors":["李·昂克里奇","阿德里安·莫利纳"],"rate":"9.1","cover_x":1080,"star":"45","title":"寻梦环游记","url":"https:\/\/movie.douban.com\/subject\/20495023\/","casts":["安东尼·冈萨雷斯","盖尔·加西亚·贝纳尔","本杰明·布拉特","阿兰娜·乌巴赫","芮妮·维克托"],"cover":"https://img9.doubanio.com\/view\/photo\/s_ratio_poster\/public\/p2614500706.jpg","id":"20495023","cover_y":1512},{"directors":["殷飞","王磊"],"rate":"","cover_x":1078,"star":"0","title":"不说再见","url":"https:\/\/movie.douban.com\/subject\/30284901\/","casts":["任嘉伦","张钧甯","寇世勋","张晞临","刘恩佑"],"cover":"https://img1.doubanio.com\/view\/photo\/s_ratio_poster\/public\/p2542861359.jpg","id":"30284901","cover_y":1932},{"directors":["罗伯特·泽米吉斯"],"rate":"9.5","cover_x":1872,"star":"50","title":"阿甘正传","url":"https:\/\/movie.douban.com\/subject\/1292720\/","casts":["汤姆·汉克斯","罗宾·怀特","加里·西尼斯","麦凯尔泰·威廉逊","莎莉·菲尔德"],"cover":"https://img2.doubanio.com\/view\/photo\/s_ratio_poster\/public\/p2372307693.jpg","id":"1292720","cover_y":2546},{"directors":["约翰·卡拉辛斯基"],"rate":"6.5","cover_x":3957,"star":"35","title":"寂静之地2","url":"https:\/\/movie.douban.com\/subject\/30206311\/","casts":["艾米莉·布朗特","基里安·墨菲","米利森特·西蒙兹","诺亚·尤佩","约翰·卡拉辛斯基"],"cover":"https://img1.doubanio.com\/view\/photo\/s_ratio_poster\/public\/p2651298629.jpg","id":"30206311","cover_y":5793},{"directors":["吕克·贝松"],"rate":"9.4","cover_x":658,"star":"45","title":"这个杀手不太冷","url":"https:\/\/movie.douban.com\/subject\/1295644\/","casts":["让·雷诺","娜塔莉·波特曼","加里·奥德曼","丹尼·爱罗","彼得·阿佩尔"],"cover":"https://img2.doubanio.com\/view\/photo\/s_ratio_poster\/public\/p511118051.jpg","id":"1295644","cover_y":980},{"directors":["曾国祥"],"rate":"8.2","cover_x":5906,"star":"40","title":"少年的你","url":"https:\/\/movie.douban.com\/subject\/30166972\/","casts":["周冬雨","易烊千玺","尹昉","周也","吴越"],"cover":"https://img2.doubanio.com\/view\/photo\/s_ratio_poster\/public\/p2572166063.jpg","id":"30166972","cover_y":8268},{"directors":["新海诚"],"rate":"8.4","cover_x":1428,"star":"40","title":"你的名字。","url":"https:\/\/movie.douban.com\/subject\/26683290\/","casts":["神木隆之介","上白石萌音","长泽雅美","市原悦子","成田凌"],"cover":"https://img1.doubanio.com\/view\/photo\/s_ratio_poster\/public\/p2395733377.jpg","id":"26683290","cover_y":2000},{"directors":["罗伯·莱纳"],"rate":"9.1","cover_x":986,"star":"45","title":"怦然心动","url":"https:\/\/movie.douban.com\/subject\/3319755\/","casts":["玛德琳·卡罗尔","卡兰·麦克奥利菲","瑞贝卡·德·莫妮","安东尼·爱德华兹","约翰·马奥尼"],"cover":"https://img1.doubanio.com\/view\/photo\/s_ratio_poster\/public\/p501177648.jpg","id":"3319755","cover_y":1466},{"directors":["郭帆"],"rate":"7.9","cover_x":1786,"star":"40","title":"流浪地球","url":"https:\/\/movie.douban.com\/subject\/26266893\/","casts":["屈楚萧","吴京","李光洁","吴孟达","赵今麦"],"cover":"https://img2.doubanio.com\/view\/photo\/s_ratio_poster\/public\/p2545472803.jpg","id":"26266893","cover_y":2500},{"directors":["陈凯歌"],"rate":"9.6","cover_x":600,"star":"50","title":"霸王别姬","url":"https:\/\/movie.douban.com\/subject\/1291546\/","casts":["张国荣","张丰毅","巩俐","葛优","英达"],"cover":"https://img3.doubanio.com\/view\/photo\/s_ratio_poster\/public\/p2561716440.jpg","id":"1291546","cover_y":889},{"directors":["姜文"],"rate":"8.9","cover_x":1500,"star":"45","title":"让子弹飞","url":"https:\/\/movie.douban.com\/subject\/3742360\/","casts":["姜文","葛优","周润发","刘嘉玲","陈坤"],"cover":"https://img1.doubanio.com\/view\/photo\/s_ratio_poster\/public\/p1512562287.jpg","id":"3742360","cover_y":2200},{"directors":["彼得·威尔"],"rate":"9.3","cover_x":997,"star":"45","title":"楚门的世界","url":"https:\/\/movie.douban.com\/subject\/1292064\/","casts":["金·凯瑞","劳拉·琳妮","艾德·哈里斯","诺亚·艾默里奇","娜塔莎·麦克艾霍恩"],"cover":"https://img2.doubanio.com\/view\/photo\/s_ratio_poster\/public\/p479682972.jpg","id":"1292064","cover_y":1500},{"directors":["克里斯托弗·诺兰"],"rate":"9.3","cover_x":4500,"star":"45","title":"盗梦空间","url":"https:\/\/movie.douban.com\/subject\/3541415\/","casts":["莱昂纳多·迪卡普里奥","约瑟夫·高登-莱维特","艾利奥特·佩吉","汤姆·哈迪","渡边谦"],"cover":"https://img2.doubanio.com\/view\/photo\/s_ratio_poster\/public\/p2616355133.jpg","id":"3541415","cover_y":6563},{"directors":["克雷格·吉勒斯佩"],"rate":"7.0","cover_x":6759,"star":"35","title":"黑白魔女库伊拉","url":"https:\/\/movie.douban.com\/subject\/26703121\/","casts":["艾玛·斯通","艾玛·汤普森","乔尔·弗莱","保罗·沃尔特·豪泽","约翰·麦克雷"],"cover":"https://img9.doubanio.com\/view\/photo\/s_ratio_poster\/public\/p2653946775.jpg","id":"26703121","cover_y":10160},{"directors":["陈凯歌","张一白","管虎","薛晓路","徐峥","宁浩","文牧野"],"rate":"7.7","cover_x":5906,"star":"40","title":"我和我的祖国","url":"https:\/\/movie.douban.com\/subject\/32659890\/","casts":["黄渤","张译","韩昊霖","杜江","葛优"],"cover":"https://img3.doubanio.com\/view\/photo\/s_ratio_poster\/public\/p2567998580.jpg","id":"32659890","cover_y":8268},{"directors":["楼健"],"rate":"8.1","cover_x":1080,"star":"40","title":"御赐小仵作","url":"https:\/\/movie.douban.com\/subject\/32581207\/","casts":["苏晓彤","王子奇","杨廷东","赵尧珂","王彦鑫"],"cover":"https://img2.doubanio.com\/view\/photo\/s_ratio_poster\/public\/p2652208493.jpg","id":"32581207","cover_y":1656}]}