python 高性能编程 异步并发编程 asyncio
异步并发编程-相关概念
- 同步:按照顺序执行:js同步的实现
- 异步:非同步。一起执行,但是异步不一定就是多线程,比如JavaScript 的运行时是跑在单线程上的,所有同步任务都在主线程上执行,形成一个执行栈(execution context stack)。主线程之外,还存在一个”任务队列”(task queue)。只要异步任务有了运行结果,就在”任务队列”之中放置一个事件。一旦”执行栈”中的所有同步任务执行完毕,系统就会读取”任务队列”,看看里面有哪些事件。那些对应的异步任务,于是结束等待状态,进入执行栈,开始执行。主线程不断重复上面的第三步。
# 输出3个“i=3”
for (var i = 0; i < 3; i++) {
setTimeout(function () {
console.log("i =", i)
}, 100)
}
- 多线程:实现异步的一种方法:python多线程threading
- 协程:协程也是实现异步的一种方法。函数执行到一半,切换到另一个函数,在一个线程中,asyncio可以实现在io等待的过程中执行其他函数。
pip install greenlet
from greenlet import greenlet
def test1():
print 12
gr2.switch()
print 34
def test2():
print 56
gr1.switch()
print 78
gr1 = greenlet(test1)
gr2 = greenlet(test2)
gr1.switch()
异步并发编程-相关概念
# 协程函数的执行方法的“格式”
asyncio.get_event_loop().run_until_complete(main())
asyncio 遇到io自动切换
import asyncio
@asyncio.coroutine
def func1():
print(1)
yield from asyncio.sleep(2) # 遇到IO耗时操作,自动化切换到tasks中的其他任务
print(2)
@asyncio.coroutine
def func2():
print(3)
yield from asyncio.sleep(2) # 遇到IO耗时操作,自动化切换到tasks中的其他任务
print(4)
tasks = [
asyncio.ensure_future( func1() ),
asyncio.ensure_future( func2() )
]
loop = asyncio.get_event_loop()
loop.run_until_complete(asyncio.wait(tasks))
结果: 1 3 2 4
async await 关键字(遇到async的函数是,执行到await的耗时部分停止,去执行其他函数,当await的耗时部分执行完成只后再继续执行async标记的函数)
import asyncio
@asyncio.coroutine
def func1():
print(1)
yield from asyncio.sleep(2) # 遇到IO耗时操作,自动化切换到tasks中的其他任务
print(2)
@asyncio.coroutine
def func2():
print(3)
yield from asyncio.sleep(2) # 遇到IO耗时操作,自动化切换到tasks中的其他任务
print(4)
tasks = [
asyncio.ensure_future( func1() ),
asyncio.ensure_future( func2() )
]
loop = asyncio.get_event_loop()
loop.run_until_complete(asyncio.wait(tasks))
网络部分
aiohttp
https://docs.aiohttp.org/en/stable/每天学Python-通过aiohttp模块实现HTTP高并发处理
self._session = aiohttp.ClientSession(loop=self._loop, timeout=aiohttp.ClientTimeout(total=10))
self._websocket: Optional[aiohttp.ClientWebSocketResponse] = None
message: aiohttp.WSMessage
async for message in websocket:
await self._on_ws_message(message)
# 至少成功处理1条消息
retry_count = 0
async def run_multi_client():
clients = [blivedm.BLiveClient(room_id) for room_id in TEST_ROOM_IDS]
handler = MyHandler()
for client in clients:
client.add_handler(handler)
client.start()
try:
await asyncio.gather(*(
client.join() for client in clients
))
async def main():
await run_single_client()
await run_multi_client()
handler - urllib的open方法
- 定制请求头
- 代理访问
- cookie(也可以抓包cookie然后放到请求里)
基础用法
import urllib.request
url = 'http://www.baidu.com'
headers = {'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/101.0.4951.67 Safari/537.36'}
#(1)创建handler对象
handler = urllib.request.HTTPHandler()
#(2) opener对象,不适用urlopen方法了
opener = urllib.request.build_opener(handler)
# (3)构建请求对象
request = urllib.request.Request(url=url,headers=headers)
#(4)调用open方法,发送请求
response = opener.open(request)
content = response.read().decode('utf-8')
print(content)
代理(中介)
使用多个代理来访问服务器。(代理需要代理云服务提供商提供,再有了代理ip和端口只后,可以再电脑中设置代理:浏览器设置=》高级设置=》代理设置=》局域网=》为LAN使用代理,试用一下)
import urllib.request
url = 'http://*'
headers = {'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/101.0.4951.67 Safari/537.36'}
proxies = {'http':'ip:port'}
handler = urllib.request.ProxyHandler(proxies = proxies)
opener = urllib.request.build_opener(handler)
request = urllib.request.Request(url=url,headers=headers)
response = opener.open(request)
content = response.read().decode('utf-8')
cookie 模拟登录
import urllib.request
import urllib.parse
import http.cookiejar
cookie =http.cookiejar.CookieJar()
handler = urllib.request.HttpCookieProcessor(cookie)
opener = urllib.request.build_opener(handler)
posturl = 'http://*'
formdata = {'*': '*','*': '*','*': '*','*': '*','*': '*'}
headers = {'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/101.0.4951.67 Safari/537.36'}
request = urllib.request.Request(url=posturl,headers=headers)
formdata = urllib.parse.urlencode(formdata).encode()
response = opener.open(request,data = formdata )
# content = response.read().decode('utf-8')
get_url = 'http://*'
request = urllib.request.Request(url=get_url ,headers=headers)
response = opener.open(request)
content = response.read().decode('utf-8')
print(content)
参考与更多
asyncio异步并发部分