本篇沿用上篇的代码,请先阅读上篇文章
轮询不能保证执行时间的效率最大化
用轮询分配消息的方式只能在消息条数上保证公平,并没有兼顾每个消息的执行时间。这样就可能导致这样的情况:
消费者A正在执行一个非常耗时的任务,预估耗时1小时;消费者B执行的任务只要0.1s就完成,由于是按照任务数分配,就会造成消费者A有一堆任务在后面排队,而消费都B却闲得心里有点慌...
生产者投递一个长时间的任务以及五个短时间的任务
☁ rabbitMq [master] ⚡ python new_task.py long time.......................
[x] Sent 'long time.......................'
☁ rabbitMq [master] ⚡ python new_task.py one .
[x] Sent 'one .'
☁ rabbitMq [master] ⚡ python new_task.py two .
[x] Sent 'two .'
☁ rabbitMq [master] ⚡ python new_task.py three .
[x] Sent 'three .'
☁ rabbitMq [master] ⚡ python new_task.py four .
[x] Sent 'four .'
☁ rabbitMq [master] ⚡ python new_task.py five .
[x] Sent 'five .'
消费者1对任务的接受情况:
☁ rabbitMq [master] ⚡ python worker.py
[*] Waiting for messages. To exit press CTRL+C
[x] Received b'long time.......................'
[x] Done
[x] Received b'two .'
[x] Done
[x] Received b'four .'
[x] Done
在第一个长时间任务未完成之前,后面的两个短时间任务只能一直在等待...
排队等待消费者2对任务的接受情况:
☁ rabbitMq [master] ⚡ python worker.py
[*] Waiting for messages. To exit press CTRL+C
[x] Received b'one .'
[x] Done
[x] Received b'three .'
[x] Done
[x] Received b'five .'
[x] Done
消费者2很快就完成了分配的任务,然后就无所事事地看着消费者1忙着狗...
设置消费者的预读取数
这样的分配明显不公平,数目上的绝对公平忽视了处理效率的区别。更好的方案应该是在分配任务时,根据哪个消费者空闲,就优先分配给此消费者
具体的配置:
worker.py
channel.basic_qos(prefetch_count=1) # 同一时刻,不要发送超过一条消息给一个消费者
channel.basic_consume(callback,
queue='task_queue')
执行情况:
消费者1: 专注于处理耗时的任务
☁ rabbitMq [master] ⚡ python worker.py
[*] Waiting for messages. To exit press CTRL+C
[x] Received b'long time.......................'
[x] Done
消费者2: 将零散的小任务都处理了
☁ rabbitMq [master] ⚡ python worker.py
[*] Waiting for messages. To exit press CTRL+C
[x] Received b'one .'
[x] Done
[x] Received b'two .'
[x] Done
[x] Received b'three .'
[x] Done
[x] Received b'four .'
[x] Done
[x] Received b'five .'
[x] Done
如此调整,才能实现在总的执行时间最优化
网友评论