https://github.com/mosquito/forklib
Fork the single process easily
https://github.com/mosquito/forklib
fork linux posix python
Last synced: 12 months ago
JSON representation
Fork the single process easily
- Host: GitHub
- URL: https://github.com/mosquito/forklib
- Owner: mosquito
- License: apache-2.0
- Created: 2018-07-25T11:52:16.000Z (almost 8 years ago)
- Default Branch: master
- Last Pushed: 2020-05-12T09:58:51.000Z (about 6 years ago)
- Last Synced: 2025-04-09T17:34:47.906Z (over 1 year ago)
- Topics: fork, linux, posix, python
- Language: Python
- Size: 27.3 KB
- Stars: 4
- Watchers: 2
- Forks: 1
- Open Issues: 0
-
Metadata Files:
- Readme: README.rst
- License: LICENSE
Awesome Lists containing this project
README
forklib
=======
Fork the single process easily
Basic example
+++++++++++++
.. code-block:: python
import forklib
import logging
import os
from time import sleep
logging.basicConfig(level=logging.DEBUG)
def run():
print(
"Proceess #{id} has PID: {pid}".format(
id=forklib.get_id(),
pid=os.getpid()
)
)
sleep(1)
def main():
print("Master proccess has PID: {0}".format(os.getpid()))
forklib.fork(4, run)
if __name__ == '__main__':
main()
This code makes 4 forks. When you try to run it you will see something like this ::
Master proccess has PID: 39485
DEBUG:forklib.forking:Starting 4 processes
Proceess #1 has PID: 39487
Proceess #0 has PID: 39486
Proceess #2 has PID: 39488
Proceess #3 has PID: 39489
DEBUG:forklib.forking:Child with PID: 39487 Number: 1 exited normally
DEBUG:forklib.forking:Child with PID: 39489 Number: 3 exited normally
DEBUG:forklib.forking:Child with PID: 39488 Number: 2 exited normally
DEBUG:forklib.forking:Child with PID: 39486 Number: 0 exited normally
Forkme will be control forks. When subprocess will be killed or will exit with
non-zero code it will be restarted immediately. e.g.::
Master proccess has PID: 7579
INFO:forklib:Starting 4 processes
Proceess #0 has PID: 7580
Proceess #1 has PID: 7581
Proceess #2 has PID: 7582
Proceess #3 has PID: 7583
WARNING:forklib:Child with PID: 7580 Number: 0 killed by signal 9, restarting
Proceess #0 has PID: 7584
``async_callback`` example
++++++++++++++++++++++++++
.. code-block:: python
import asyncio
import forklib
import logging
import os
from time import sleep
logging.basicConfig(level=logging.DEBUG)
def run():
print(
"Proceess #{id} has PID: {pid}".format(
id=forklib.get_id(),
pid=os.getpid()
)
)
sleep(1)
async def amain():
await asyncio.sleep(0.5)
print("Async callback finished")
def main():
print("Master proccess has PID: {0}".format(os.getpid()))
forklib.fork(
4, run,
async_callback=amain,
# Wait all incomplete async tasks, otherwise cancel (default)
wait_async_callback = True,
)
if __name__ == '__main__':
main()
``thread_callback`` example
+++++++++++++++++++++++++++
.. code-block:: python
import asyncio
import logging
import os
from time import sleep
from threading import Event
import forklib
logging.basicConfig(level=logging.DEBUG)
def run():
print(
"Proceess #{id} has PID: {pid}".format(
id=forklib.get_id(),
pid=os.getpid(),
),
)
sleep(3)
exit_event = Event()
def thread_callback():
while not exit_event.is_set():
sleep(0.5)
print("Thread callback making great stuff")
print("Thread callback finished")
async def async_callback():
await asyncio.sleep(5)
print("Async callback finished")
def main():
print("Master proccess has PID: {0}".format(os.getpid()))
forklib.fork(
4, run,
thread_callback=thread_callback,
# Wait theread_callback, otherwise exit (default)
# Note: You have to be careful when using this option.
# Thread cancellation is impossible in the general case and you must
# implement your own way of thread exit notification for example
# like following one using exit_callback and threading.Event
wait_thread_callback=True,
# Notifying thread_callback about exit.
exit_callback=exit_event.set,
async_callback=async_callback,
# Wait async_callback, otherwise cancel incomplete tasks (default)
wait_async_callback=True
)
if __name__ == "__main__":
main()
Parallel iteration
++++++++++++++++++
You can load the large array of elements on the memory and process it in
multiple processes. After forking the memory will not be copied, instead
of the copy-on-write mechanism will be used.
.. code-block:: python
from forklib import fork_map
import logging
logging.basicConfig(level=logging.INFO)
def map_func(item):
return item + 1
def main():
for item in fork_map(map_func, range(20000), workers=10):
print(item)
if __name__ == '__main__':
main()
Versioning
++++++++++
This software follows `Semantic Versioning`_
.. _Semantic Versioning: http://semver.org/