https://github.com/heavenvolkoff/cpu_count
Calculate the number of CPUs available taking into account system constraints
https://github.com/heavenvolkoff/cpu_count
constraints cpu-count cpus docker monkey-patch python
Last synced: 10 months ago
JSON representation
Calculate the number of CPUs available taking into account system constraints
- Host: GitHub
- URL: https://github.com/heavenvolkoff/cpu_count
- Owner: HeavenVolkoff
- License: bsd-3-clause
- Created: 2019-10-18T18:17:11.000Z (over 6 years ago)
- Default Branch: master
- Last Pushed: 2019-10-23T17:15:35.000Z (over 6 years ago)
- Last Synced: 2025-04-10T04:40:42.354Z (10 months ago)
- Topics: constraints, cpu-count, cpus, docker, monkey-patch, python
- Language: Python
- Size: 31.3 KB
- Stars: 9
- Watchers: 1
- Forks: 0
- Open Issues: 1
-
Metadata Files:
- Readme: README.md
- License: LICENSE
Awesome Lists containing this project
README
# cpu_count
[](https://github.com/HeavenVolkoff/cpu_count)
[](https://pypi.org/project/cpu-count/)
[](https://pypi.org/project/cpu-count/)
[](https://opensource.org/licenses/BSD-3-Clause)
Modified version of python's `cpu_count` that takes into account system
constraints to calculate the number of available CPUs
## Motivation
The Python standard library offers an implementation of cpu_count that returns
the real number of CPUs even when they are not actually available to be used by
the python process (due to constraints such as CPU affinity or CPU scheduler
configurations). This is the preferred behaviour for most applications. However,
when the interest is the amount of CPUs available for data processing, that
approach could be misleading. Especially when it is used behind the scenes such
as by the `concurrent.futures.Executor` when defining its defaults.
The purpose of this module is to provide this functionality in an API equal to
the standard implementation. By taking into account the described constraints,
this implementation attempts to return the amount of usable CPUs that are
available. If no constraint is identified the result will be the same as the
standard implementation.
## How to install
```
pip install cpu_count
```
## How to use
### As an external module
This is the standard way. Just import and call `cpu_count`
```python
from cpu_count import cpu_count
print(cpu_count())
# $> 8
```
### Monkey-patch standard lib
This an alternative way that replaces python's standard `cpu_count` with the
one from this module (Affected internal modules are `posix`, `os` and
`multiprocessing`). The advantage of this approach is not needing to port any
code, just import and call `setup_monkey_patch` ate the begin of your
application and everything will just work™.
_Note: This will also impact the behaviour of standard libraries that use this
function_
```python
import os
from cpu_count.monkey_patch import setup_monkey_patch
print(os.cpu_count())
# $> 12
setup_monkey_patch()
print(os.cpu_count())
# $> 8
```
#### Limitation
This approach has one limitation: it can't replace previous code that
imported the standard implementation using `from os import cpu_count`.
```python
from os import cpu_count
from cpu_count.monkey_patch import setup_monkey_patch
print(cpu_count())
# $> 12
setup_monkey_patch()
print(cpu_count())
# $> 12
```
### System wide monkey-patch
This approach also replaces python's standard `cpu_count`. However instead of
calling this module's `setup_monkey_patch` in your application code, it will be
called at python startup. For this to work you need to create a file called
`cpu_count_monkey_patch.pth` at your python's global or local site-package's
folder with the following content:
```python
import cpu_count; cpu_count.monkey_patch.setup_monkey_patch()
```
_NOTE: This approach is specially useful when creating container images of
python applications. An example of using this on a Dockerfile can be found
[here](tests/docker/Dockerfile)_
## TODO
- [ ] Add logic for Realtime Scheduler constraint
- [ ] Create unit tests
## Contributions
All contributions are very welcome.
Code styles is defined in the [Editorconfig](.editorconfig) file. Besides that
I use [black](https://github.com/psf/black) and
[isort](https://github.com/timothycrosley/isort) for auto-format, their
configurations are defined in the [Editorconfig](.editorconfig) and the
[black.toml](black.toml) files respectively. Also I use
[mypy](https://github.com/timothycrosley/isort) for statical type check, it's
configurations are in the [.mypy](.mypy) file.
## Acknowledgement
Thanks to [@tomMoral](https://github.com/tomMoral) for his
[loky](https://github.com/tomMoral/loky) project, from which I took the base
code for this implementation.
Thanks to [@Lothiraldan](https://github.com/Lothiraldan) for his in-depth guide
on how to monkey patch python: [python-production-monkey-patching](https://gitpitch.com/Lothiraldan/python-production-monkey-patching/master),
which helped me a lot when constructing this module.
## License
BSD 3-Clause “New” or “Revised” License
See [LICENSE](./LICENSE)