python多线程起消费者和生产者 python生产者消费者问题

转载

ctaxnews 2023-09-24 15:46:18

文章标签 python多线程起消费者和生产者 python 数据插入数据写数据 文章分类 Python 后端开发

假设有一个公共队列，生产者向队列中写数据，消费者从队列中读数据。当队列中没有任何数据的时候，消费者应该停止运行并等待(wait），而不是继续尝试读取数据而引发读取空队列的异常。而当生产者在队列中加入数据之后，应该有一个渠道去告诉（notify）消费者。然后消费者可以再次从队列中进行读取，而IndexError不再出现。

消费者：

from threading import Condition
 
condition = Condition()
 
class ConsumerThread(Thread):
    def run(self):
        global queue
        while True:
            condition.acquire()
            if not queue:
                print "Nothing in queue, consumer is waiting"
                condition.wait()
                print "Producer added something to queue and notified the consumer"
            num = queue.pop(0)
            print "Consumed", num 
            condition.release()
            time.sleep(random.random())
from threading import Condition
 
condition = Condition()
 
class ConsumerThread(Thread):
    def run(self):
        global queue
        while True:
            condition.acquire()
            if not queue:
                print "Nothing in queue, consumer is waiting"
                condition.wait()
                print "Producer added something to queue and notified the consumer"
            num = queue.pop(0)
            print "Consumed", num 
            condition.release()
            time.sleep(random.random())

生产者：

class ProducerThread(Thread):
    def run(self):
        nums = range(5)
        global queue
        while True:
            condition.acquire()
            num = random.choice(nums)
            queue.append(num)
            print "Produced", num 
            condition.notify()
            condition.release()
            time.sleep(random.random())
class ProducerThread(Thread):
    def run(self):
        nums = range(5)
        global queue
        while True:
            condition.acquire()
            num = random.choice(nums)
            queue.append(num)
            print "Produced", num 
            condition.notify()
            condition.release()
            time.sleep(random.random())

输出：

Produced 3
Consumed 3
Produced 1
Consumed 1
Produced 4
Consumed 4
Produced 3
Consumed 3
Nothing in queue, consumer is waiting
Produced 2
Producer added something to queue and notified the consumer
Consumed 2
Nothing in queue, consumer is waiting
Produced 2
Producer added something to queue and notified the consumer
Consumed 2
Nothing in queue, consumer is waiting
Produced 3
Producer added something to queue and notified the consumer
Consumed 3
Produced 4
Consumed 4
Produced 1
Consumed 1
Produced 3
Consumed 3
Produced 1
Consumed 1
Produced 4
Consumed 4
Produced 3
Consumed 3
Nothing in queue, consumer is waiting
Produced 2
Producer added something to queue and notified the consumer
Consumed 2
Nothing in queue, consumer is waiting
Produced 2
Producer added something to queue and notified the consumer
Consumed 2
Nothing in queue, consumer is waiting
Produced 3
Producer added something to queue and notified the consumer
Consumed 3
Produced 4
Consumed 4
Produced 1
Consumed 1

解释：

对于消费者，在消费前检查队列是否为空。
如果为空，调用condition实例的wait()方法。
消费者进入wait()，同时释放所持有的lock。
除非被notify，否则它不会运行。
生产者可以acquire这个lock，因为它已经被消费者release。
当调用了condition的notify()方法后，消费者被唤醒，但唤醒不意味着它可以开始运行。
notify()并不释放lock，调用notify()后，lock依然被生产者所持有。
生产者通过condition.release()显式释放lock。
消费者再次开始运行，现在它可以得到队列中的数据而不会出现IndexError异常。

为队列增加大小限制

生产者不能向一个满队列继续加入数据。

它可以用以下方式来实现：

在加入数据前，生产者检查队列是否为满。
如果不为满，生产者可以继续正常流程。
如果为满，生产者必须等待，调用condition实例的wait()。
消费者可以运行。消费者消耗队列，并产生一个空余位置。
然后消费者notify生产者。
当消费者释放lock，消费者可以acquire这个lock然后往队列中加入数据。

最终版本如下：

from threading import Thread, Condition
import time
import random
 
queue = []
MAX_NUM = 10
condition = Condition()
 
class ProducerThread(Thread):
    def run(self):
        nums = range(5)
        global queue
        while True:
            condition.acquire()
            if len(queue) == MAX_NUM:
                print "Queue full, producer is waiting"
                condition.wait()
                print "Space in queue, Consumer notified the producer"
            num = random.choice(nums)
            queue.append(num)
            print "Produced", num
            condition.notify()
            condition.release()
            time.sleep(random.random())
 
class ConsumerThread(Thread):
    def run(self):
        global queue
        while True:
            condition.acquire()
            if not queue:
                print "Nothing in queue, consumer is waiting"
                condition.wait()
                print "Producer added something to queue and notified the consumer"
            num = queue.pop(0)
            print "Consumed", num
            condition.notify()
            condition.release()
            time.sleep(random.random())
 
ProducerThread().start()
ConsumerThread().start()
from threading import Thread, Condition
import time
import random
 
queue = []
MAX_NUM = 10
condition = Condition()
 
class ProducerThread(Thread):
    def run(self):
        nums = range(5)
        global queue
        while True:
            condition.acquire()
            if len(queue) == MAX_NUM:
                print "Queue full, producer is waiting"
                condition.wait()
                print "Space in queue, Consumer notified the producer"
            num = random.choice(nums)
            queue.append(num)
            print "Produced", num
            condition.notify()
            condition.release()
            time.sleep(random.random())
 
class ConsumerThread(Thread):
    def run(self):
        global queue
        while True:
            condition.acquire()
            if not queue:
                print "Nothing in queue, consumer is waiting"
                condition.wait()
                print "Producer added something to queue and notified the consumer"
            num = queue.pop(0)
            print "Consumed", num
            condition.notify()
            condition.release()
            time.sleep(random.random())
 
ProducerThread().start()
ConsumerThread().start()

输出结果：

Produced 0
Consumed 0
Produced 0
Produced 4
Consumed 0
Consumed 4
Nothing in queue, consumer is waiting
Produced 4
Producer added something to queue and notified the consumer
Consumed 4
Produced 3
Produced 2
Consumed 3
Produced 0
Consumed 0
Produced 0
Produced 4
Consumed 0
Consumed 4
Nothing in queue, consumer is waiting
Produced 4
Producer added something to queue and notified the consumer
Consumed 4
Produced 3
Produced 2
Consumed 3

Queue封装了Condition的行为，如wait()，notify()，acquire()。

利用Queue可以很方便的实现以上功能：

from threading import Thread
import time
import random
from Queue import Queue
 
queue = Queue(10)
 
class ProducerThread(Thread):
    def run(self):
        nums = range(5)
        global queue
        while True:
            num = random.choice(nums)
            queue.put(num)
            print "Produced", num
            time.sleep(random.random())
 
class ConsumerThread(Thread):
    def run(self):
        global queue
        while True:
            num = queue.get()
            queue.task_done()
            print "Consumed", num
            time.sleep(random.random())
 
ProducerThread().start()
ConsumerThread().start()
from threading import Thread
import time
import random
from Queue import Queue
 
queue = Queue(10)
 
class ProducerThread(Thread):
    def run(self):
        nums = range(5)
        global queue
        while True:
            num = random.choice(nums)
            queue.put(num)
            print "Produced", num
            time.sleep(random.random())
 
class ConsumerThread(Thread):
    def run(self):
        global queue
        while True:
            num = queue.get()
            queue.task_done()
            print "Consumed", num
            time.sleep(random.random())
 
ProducerThread().start()
ConsumerThread().start()

解释：