运行dask程序报错:Task exception was never retrieved

标签:#dask##python##编程# 时间:2021/06/07 22:42:03 作者:小木

运行本地dask集群的时候出错,报错信息如下:

  1. Task exception was never retrieved
  2. future: <Task finished coro=<_wrap_awaitable() done, defined at D:\Program Files\Python3.7\lib\asyncio\tasks.py:592> exception=RuntimeError('\n An attempt has been made to start a new process before the\n current process has finished its bootstrapping phase.\n\n This probably means that you are not using fork to start your\n child processes and you have forgotten to use the proper idiom\n in the main module:\n\n if __name__ == \'__main__\':\n freeze_support()\n ...\n\n The "freeze_support()" line can be omitted if the program\n is not going to be frozen to produce an executable.')>
  3. Traceback (most recent call last):
  4. File "D:\Program Files\Python3.7\lib\asyncio\tasks.py", line 599, in _wrap_awaitable
  5. return (yield from awaitable.__await__())
  6. File "D:\Program Files\Python3.7\lib\site-packages\distributed\nanny.py", line 291, in start
  7. response = await self.instantiate()
  8. File "D:\Program Files\Python3.7\lib\site-packages\distributed\nanny.py", line 374, in instantiate
  9. result = await self.process.start()
  10. File "D:\Program Files\Python3.7\lib\site-packages\distributed\nanny.py", line 567, in start
  11. await self.process.start()
  12. File "D:\Program Files\Python3.7\lib\site-packages\distributed\process.py", line 34, in _call_and_set_future
  13. res = func(*args, **kwargs)
  14. File "D:\Program Files\Python3.7\lib\site-packages\distributed\process.py", line 202, in _start
  15. process.start()
  16. File "D:\Program Files\Python3.7\lib\multiprocessing\process.py", line 112, in start
  17. self._popen = self._Popen(self)
  18. File "D:\Program Files\Python3.7\lib\multiprocessing\context.py", line 223, in _Popen
  19. return _default_context.get_context().Process._Popen(process_obj)
  20. File "D:\Program Files\Python3.7\lib\multiprocessing\context.py", line 322, in _Popen
  21. return Popen(process_obj)
  22. File "D:\Program Files\Python3.7\lib\multiprocessing\popen_spawn_win32.py", line 33, in __init__
  23. prep_data = spawn.get_preparation_data(process_obj._name)
  24. File "D:\Program Files\Python3.7\lib\multiprocessing\spawn.py", line 143, in get_preparation_data
  25. _check_not_importing_main()
  26. File "D:\Program Files\Python3.7\lib\multiprocessing\spawn.py", line 136, in _check_not_importing_main
  27. is not going to be frozen to produce an executable.''')
  28. RuntimeError:
  29. An attempt has been made to start a new process before the
  30. current process has finished its bootstrapping phase.
  31. This probably means that you are not using fork to start your
  32. child processes and you have forgotten to use the proper idiom
  33. in the main module:
  34. if __name__ == '__main__':
  35. freeze_support()
  36. ...
  37. The "freeze_support()" line can be omitted if the program
  38. is not going to be frozen to produce an executable.

提示Task exception was never retrieved。

代码如下:

  1. import dask.dataframe as dd
  2. from dask.distributed import Client, LocalCluster
  3. import pandas as pd
  4. cluster = LocalCluster(dashboard_address=None)
  5. client = Client(cluster)
  6. df = pd.DataFrame([[1, 1, 1], [2, 2, 2]])
  7. df = dd.from_pandas(df, npartitions=1)
  8. print(df.head())

这个问题在dask社区中有讨论( https://github.com/modin-project/modin/issues/843 ),具体原因还不是特别清楚,讨论过程看可能与modin或者ray有关系,解决方法很简单,加入main方法即可:

  1. import dask.dataframe as dd
  2. from dask.distributed import Client, LocalCluster
  3. import pandas as pd
  4. if __name__ == '__main__':
  5. cluster = LocalCluster(dashboard_address=None)
  6. client = Client(cluster)
  7. df = pd.DataFrame([[1, 1, 1], [2, 2, 2]])
  8. df = dd.from_pandas(df, npartitions=1)
  9. print(df.head())
欢迎大家关注DataLearner官方微信,接受最新的AI技术推送
Back to Top