python错误BrokenProcessPool

报错信息如下

> A process in the process pool was terminated abruptly while the future was running or pending.

> Rasie Queue.Full

python # 线程池入口 def ThreadPool(self, val): with ThreadPoolExecutor(max_workers=Thread_number) as executor2: tasks = [executor2.submit(self.extract_dict, data) for data in val] for future in as_completed(tasks): if future.result() is None or future.result() == None: continue for result in future.result(): print('result: %s' % (result)) executor2.shutdown() # 进程池入口 def ProcessPool(self, redis_list_key): ''' :param Process_number: 即开启线程的个数 :return: None ''' result_list = self.get_list(redis_list_key) q = queue.Queue() q.put(result_list) with ProcessPoolExecutor(max_workers=Process_number) as executor: while not q.empty(): val = q.get() for result in executor.map(self.ThreadPool, val): if result is None or result == None: continue print('ProcessPoolresult: %s' % (result)) executor.shutdown(wait=True)
问题就在于线程池模块中并没有返回对象 ,也就是我所多线程开启的程序extract_dict没有返回return,所以导致下面的future.result()
会导致出现异常 爆出BrokenProcessPool的错误

python if future.result() is None or future.result() == None: continue for result in future.result(): print('result: %s' % (result))


所以将代码调整为如下所示:


python # 线程池入口 def ThreadPool(self, val): with ThreadPoolExecutor(max_workers=Thread_number) as executor2: tasks = [executor2.submit(self.extract_dict, data) for data in val] as_completed(tasks) executor2.shutdown() # 进程池入口 def ProcessPool(self, redis_list_key): ''' :param Process_number: 即开启线程的个数 :return: None ''' result_list = self.get_list(redis_list_key) q = queue.Queue() q.put(result_list) with ProcessPoolExecutor(max_workers=Process_number) as executor: while not q.empty(): val = q.get() for result in executor.map(self.ThreadPool, val): if result is None or result == None: continue print('ProcessPoolresult: %s' % (result)) executor.shutdown(wait=True)

直接使用 as_completed(tasks)对象,不需要对返回值进行处理

as_completed(tasks) 是 concurrent.futures的一个模块

python from concurrent.futures import as_completed


   转载规则


《python错误BrokenProcessPool》 旋律JOJO哒 采用 知识共享署名 4.0 国际许可协议 进行许可。
  目录