python request不阻塞 python queue阻塞

转载

mob64ca140761a4 2023-10-24 09:59:33

文章标签 python request不阻塞 python mac mac Queue.get方法 Queue的常见问题 文章分类 Python 后端开发

Mac下Python 多谢进程假死或卡死或阻塞的情况

python中，队列是线程间最常用的交换数据的形式。queue模块是提供队列操作的模块，虽然简单易用，但是不小心的话，还是会出现一些意外。

1. 阻塞模式

q = queue.Queue(10)  #创建一个队列

......
       for i in range(10):
               q.put('A')
               time.sleep(0.5)

这是一段极其简单的代码（另有两个线程也在操作队列q），我期望每隔0.5秒写一个'A'到队列中，但总是不能如愿：间隔时间有时会远远超过0.5秒。原来，Queue.put（）默认有 block = True 和 timeou 两个参数。当 block = True 时，写入是阻塞式的，阻塞时间由 timeou 确定。当队列q被（其他线程）写满后，这段代码就会阻塞，直至其他线程取走数据。Queue.put（）方法加上 block=False 的参数，即可解决这个隐蔽的问题。但要注意，非阻塞方式写队列，当队列满时会抛出 exception Queue.Full 的异常。

注意坑：定义队列数固定后，后续写入超出队列设置数Windows会卡死不会继续执行。

2. 无法捕获 exception Queue.Empty 的异常

while True:
    ......
    try:
        data = q.get()
    except Queue.Empty:
        break

原始需求为队列为空时退出循环，但实际运行起来，却陷入了死循环。

这个问题和上面有点类似：Queue.get（）默认的也是阻塞方式读取数据，队列为空时，不会抛出 except Queue.Empty ，而是进入阻塞直至超时。

其他人的博客：加上block=False 的参数，问题无法解决会造成队列数据丢失。Mac运行Queue.Empty()会出现判空异常，相同的判空在Windows运行正常。

3. Queue常用方法汇总

Queue.Queue(maxsize=0)   FIFO， 如果maxsize小于1就表示队列长度无限
Queue.LifoQueue(maxsize=0)   LIFO， 如果maxsize小于1就表示队列长度无限
Queue.qsize()   返回队列的大小 
Queue.empty()   如果队列为空，返回True,反之False 
Queue.full()   如果队列满了，返回True,反之False
Queue.get([block[, timeout]])   读队列，timeout等待时间 
Queue.put(item, [block[, timeout]])   写队列，timeout等待时间 
Queue.queue.clear()   清空队列

4、Mac下的坑

1、Queue.qsize() 返回队列的大小，不过在 Mac OS 上没法运行。

2、Queue.get()搭配Queue.empty()做队列判空处理无法捕获异常会出现bug

3、出现问题尝试进程守护无效

python进程之守护进程

主进程创建子进程，然后将该进程设置成守护自己的进程，守护进程就好比崇祯皇帝身边的老太监，崇祯皇帝已死老太监就跟着殉葬了；

关于守护进程需要强调两点：
其一：守护进程会在主进程代码执行结束后就终止
其二：守护进程内无法再开启子进程,否则抛出异常：AssertionError: daemonic processes are not allowed to have children
如果我们有两个任务需要并发执行，那么开一个主进程和一个子进程分别去执行就ok了，如果子进程的任务在主进程任务结束后就没有存在的必要了，那么该子进程应该在开启前就被设置成守护进程。主进程代码运行结束，守护进程随即终止;

from multiprocessing import Process
import time
import random
def task(name):
    print('%s is piaoing' %name)
    time.sleep(random.randrange(1,3))
    print('%s is piao end' %name)
if __name__ == '__main__':
    p=Process(target=task,args=('egon',))
    p.daemon=True #一定要在p.start()前设置,设置p为守护进程,禁止p创建子进程,并且父进程代码执行结束,p即终止运行
    p.start()
    print('主') #只要终端打印出这一行内容，那么守护进程p也就跟着结束掉了

5、调试遇到的异常信息

异常一

KeyboardInterrupt
Process MyProcess-2:
Process MyProcess-3:
2020-12-22T11:18:45Z
Process MyProcess-4:
KeyboardInterrupt
Process MyProcess-1:
2020-12-22T11:18:45Z
Traceback (most recent call last):
  File "/Users/benjamin/Desktop/Python/Test/testHttpGevent/testGevent.py", line 73, in <module>
Traceback (most recent call last):
    main()
  File "/Users/benjamin/Desktop/Python/Test/testHttpGevent/testGevent.py", line 33, in warpper
Traceback (most recent call last):
  File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap
    self.run()
  File "/Users/benjamin/Desktop/Python/Test/testHttpGevent/testGevent.py", line 23, in run
    gevent_list = [gevent.spawn(request_url,self.que.get()) for i in range(10)]
  File "/Users/benjamin/Desktop/Python/Test/testHttpGevent/testGevent.py", line 23, in <listcomp>
    gevent_list = [gevent.spawn(request_url,self.que.get()) for i in range(10)]
  File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/multiprocessing/queues.py", line 96, in get
    with self._rlock:
  File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/multiprocessing/synchronize.py", line 95, in __enter__
    return self._semlock.__enter__()
KeyboardInterrupt
  File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap
    self.run()
  File "/Users/benjamin/Desktop/Python/Test/testHttpGevent/testGevent.py", line 24, in run
    gevent.joinall(gevent_list)
  File "src/gevent/greenlet.py", line 1002, in gevent._gevent_cgreenlet.joinall
  File "src/gevent/greenlet.py", line 1012, in gevent._gevent_cgreenlet.joinall
  File "src/gevent/_hub_primitives.py", line 250, in gevent._gevent_c_hub_primitives.wait_on_objects
Traceback (most recent call last):
  File "src/gevent/_hub_primitives.py", line 287, in gevent._gevent_c_hub_primitives.wait_on_objects
  File "src/gevent/_hub_primitives.py", line 185, in gevent._gevent_c_hub_primitives._WaitIterator.__next__
  File "src/gevent/_hub_primitives.py", line 176, in gevent._gevent_c_hub_primitives._WaitIterator.__next__
  File "src/gevent/_waiter.py", line 192, in gevent._gevent_c_waiter.MultipleWaiter.get
  File "src/gevent/_waiter.py", line 151, in gevent._gevent_c_waiter.Waiter.get
  File "src/gevent/_greenlet_primitives.py", line 61, in gevent._gevent_c_greenlet_primitives.SwitchOutGreenletWithLoop.switch
  File "src/gevent/_greenlet_primitives.py", line 61, in gevent._gevent_c_greenlet_primitives.SwitchOutGreenletWithLoop.switch
  File "src/gevent/_greenlet_primitives.py", line 65, in gevent._gevent_c_greenlet_primitives.SwitchOutGreenletWithLoop.switch
  File "src/gevent/_gevent_c_greenlet_primitives.pxd", line 35, in gevent._gevent_c_greenlet_primitives._greenlet_switch
    func(*args,**kwargs)
KeyboardInterrupt
  File "/Users/benjamin/Desktop/Python/Test/testHttpGevent/testGevent.py", line 68, in main
  File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap
    self.run()
  File "/Users/benjamin/Desktop/Python/Test/testHttpGevent/testGevent.py", line 23, in run
    gevent_list = [gevent.spawn(request_url,self.que.get()) for i in range(10)]
  File "/Users/benjamin/Desktop/Python/Test/testHttpGevent/testGevent.py", line 23, in <listcomp>
    gevent_list = [gevent.spawn(request_url,self.que.get()) for i in range(10)]
  File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/multiprocessing/queues.py", line 96, in get
    with self._rlock:
  File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/multiprocessing/synchronize.py", line 95, in __enter__
    return self._semlock.__enter__()
KeyboardInterrupt
    [i.join() for i in task_list]
  File "/Users/benjamin/Desktop/Python/Test/testHttpGevent/testGevent.py", line 68, in <listcomp>
Traceback (most recent call last):
    [i.join() for i in task_list]
  File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/multiprocessing/process.py", line 149, in join
  File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap
    self.run()
  File "/Users/benjamin/Desktop/Python/Test/testHttpGevent/testGevent.py", line 23, in run
    gevent_list = [gevent.spawn(request_url,self.que.get()) for i in range(10)]
  File "/Users/benjamin/Desktop/Python/Test/testHttpGevent/testGevent.py", line 23, in <listcomp>
    gevent_list = [gevent.spawn(request_url,self.que.get()) for i in range(10)]
  File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/multiprocessing/queues.py", line 97, in get
    res = self._recv_bytes()
  File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/multiprocessing/connection.py", line 216, in recv_bytes
    buf = self._recv_bytes(maxlength)
  File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/multiprocessing/connection.py", line 414, in _recv_bytes
    buf = self._recv(4)
  File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/multiprocessing/connection.py", line 379, in _recv
    chunk = read(handle, remaining)
KeyboardInterrupt
    res = self._popen.wait(timeout)
  File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/multiprocessing/popen_fork.py", line 47, in wait
    return self.poll(os.WNOHANG if timeout == 0.0 else 0)
  File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/multiprocessing/popen_fork.py", line 27, in poll
    pid, sts = os.waitpid(self.pid, flag)
  File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/gevent/os.py", line 380, in waitpid
    get_hub().wait(new_watcher)
  File "src/gevent/_hub_primitives.py", line 46, in gevent._gevent_c_hub_primitives.WaitOperationsGreenlet.wait
  File "src/gevent/_hub_primitives.py", line 55, in gevent._gevent_c_hub_primitives.WaitOperationsGreenlet.wait
  File "src/gevent/_waiter.py", line 151, in gevent._gevent_c_waiter.Waiter.get
  File "src/gevent/_greenlet_primitives.py", line 61, in gevent._gevent_c_greenlet_primitives.SwitchOutGreenletWithLoop.switch
  File "src/gevent/_greenlet_primitives.py", line 61, in gevent._gevent_c_greenlet_primitives.SwitchOutGreenletWithLoop.switch
  File "src/gevent/_greenlet_primitives.py", line 65, in gevent._gevent_c_greenlet_primitives.SwitchOutGreenletWithLoop.switch
  File "src/gevent/_gevent_c_greenlet_primitives.pxd", line 35, in gevent._gevent_c_greenlet_primitives._greenlet_switch
KeyboardInterrupt

异常二

Process MyProcess-2:
Traceback (most recent call last):
  File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap
    self.run()
  File "/Users/benjamin/Desktop/Python/Test/testHttpGevent/testGevent.py", line 23, in run
    gevent_list = [gevent.spawn(request_url,self.que.get(False)) for i in range(10)]
  File "/Users/benjamin/Desktop/Python/Test/testHttpGevent/testGevent.py", line 23, in <listcomp>
    gevent_list = [gevent.spawn(request_url,self.que.get(False)) for i in range(10)]
  File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/multiprocessing/queues.py", line 103, in get
    raise Empty
_queue.Empty

本文章为转载内容，我们尊重原作者对文章享有的著作权。如有内容错误或侵权问题，欢迎原作者联系我们进行内容更正或删除文章。