https://github.com/lucacappelletti94/downloaders
Python package to handle download of multiple types of files.
https://github.com/lucacappelletti94/downloaders
Last synced: 3 months ago
JSON representation
Python package to handle download of multiple types of files.
- Host: GitHub
- URL: https://github.com/lucacappelletti94/downloaders
- Owner: LucaCappelletti94
- License: mit
- Created: 2020-12-18T15:43:10.000Z (over 4 years ago)
- Default Branch: main
- Last Pushed: 2023-12-21T18:55:03.000Z (over 1 year ago)
- Last Synced: 2024-05-01T15:42:40.326Z (about 1 year ago)
- Language: Python
- Size: 248 KB
- Stars: 4
- Watchers: 2
- Forks: 2
- Open Issues: 0
-
Metadata Files:
- Readme: README.rst
- License: LICENSE
Awesome Lists containing this project
README
downloaders
=========================================================================================
|pip| |downloads| |github|
Python package to handle the download of multiple types of files.
How do I install this package?
----------------------------------------------
As usual, just download it using pip:
.. code:: shell
pip install downloaders
Usage examples
----------------------------------------------
.. code:: python
from downloaders import BaseDownloader
downloader = BaseDownloader()
urls = [...]
downloader.download(urls)
Troubleshooting
-----------------------------------------------
MacOS multiprocessing nightmare fuel
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Cupertino has a gift for us: somehow multiprocessing on MacOS in some specific
astral configurations that I have yet to fully understand, it will crash with the
following error:
.. code:: bash
The process has forked and you cannot use this CoreFoundation functionality safely. You MUST exec().
Break on __THE_PROCESS_HAS_FORKED_AND_YOU_CANNOT_USE_THIS_COREFOUNDATION_FUNCTIONALITY___YOU_MUST_EXEC__() to debug.
Apparently, this can be easily fixed by changing the way multiprocessing spawns
processes, that is:
.. code:: python
import platform, multiprocessing
if platform.system() == "Darwin":
multiprocessing.set_start_method('spawn')
The aforementioned solution was proposed on `this StackOverflow question `__.
.. |pip| image:: https://badge.fury.io/py/downloaders.svg
:target: https://badge.fury.io/py/downloaders
:alt: Pypi project
.. |downloads| image:: https://pepy.tech/badge/downloaders
:target: https://pepy.tech/project/downloaders
:alt: Pypi total project downloads
.. |github| image:: https://github.com/lucacappelletti94/downloaders/actions/workflows/python.yml/badge.svg
:target: https://github.com/lucacappelletti94/downloaders/actions
:alt: Github Actions