{"id":14982422,"url":"https://github.com/spratiher9/exelog","last_synced_at":"2026-03-17T18:36:33.432Z","repository":{"id":57427543,"uuid":"432455930","full_name":"Spratiher9/Exelog","owner":"Spratiher9","description":"Enabling meticulous logging for your Spark Applications","archived":false,"fork":false,"pushed_at":"2021-11-28T14:31:37.000Z","size":28,"stargazers_count":5,"open_issues_count":1,"forks_count":1,"subscribers_count":1,"default_branch":"master","last_synced_at":"2025-07-23T15:07:37.121Z","etag":null,"topics":["analytics","apache","apache-spark","aws","azure","bigdata","databricks","gcp","logging","pyspark","python","spark","spark-utils"],"latest_commit_sha":null,"homepage":"https://pypi.org/project/exelog/0.0.1/","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/Spratiher9.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2021-11-27T12:32:16.000Z","updated_at":"2023-09-28T11:18:34.000Z","dependencies_parsed_at":"2022-09-19T19:50:43.029Z","dependency_job_id":null,"html_url":"https://github.com/Spratiher9/Exelog","commit_stats":null,"previous_names":[],"tags_count":1,"template":false,"template_full_name":null,"purl":"pkg:github/Spratiher9/Exelog","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Spratiher9%2FExelog","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Spratiher9%2FExelog/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Spratiher9%2FExelog/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Spratiher9%2FExelog/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/Spratiher9","download_url":"https://codeload.github.com/Spratiher9/Exelog/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Spratiher9%2FExelog/sbom","scorecard":{"id":133253,"data":{"date":"2025-08-04","repo":{"name":"github.com/Spratiher9/Exelog","commit":"5afa29ad8062eae1dd34743259d2a54d5237ea7d"},"scorecard":{"version":"v5.2.1-28-gc1d103a9","commit":"c1d103a9bb9f635ec7260bf9aa0699466fa4be0e"},"score":1.7,"checks":[{"name":"Packaging","score":-1,"reason":"packaging workflow not detected","details":["Warn: no GitHub/GitLab publishing workflow detected."],"documentation":{"short":"Determines if the project is published as a package that others can easily download, install, easily update, and uninstall.","url":"https://github.com/ossf/scorecard/blob/c1d103a9bb9f635ec7260bf9aa0699466fa4be0e/docs/checks.md#packaging"}},{"name":"Dangerous-Workflow","score":-1,"reason":"no workflows found","details":null,"documentation":{"short":"Determines if the project's GitHub Action workflows avoid dangerous patterns.","url":"https://github.com/ossf/scorecard/blob/c1d103a9bb9f635ec7260bf9aa0699466fa4be0e/docs/checks.md#dangerous-workflow"}},{"name":"SAST","score":0,"reason":"no SAST tool detected","details":["Warn: no pull requests merged into dev branch"],"documentation":{"short":"Determines if the project uses static code analysis.","url":"https://github.com/ossf/scorecard/blob/c1d103a9bb9f635ec7260bf9aa0699466fa4be0e/docs/checks.md#sast"}},{"name":"Token-Permissions","score":-1,"reason":"No tokens found","details":null,"documentation":{"short":"Determines if the project's workflows follow the principle of least privilege.","url":"https://github.com/ossf/scorecard/blob/c1d103a9bb9f635ec7260bf9aa0699466fa4be0e/docs/checks.md#token-permissions"}},{"name":"Maintained","score":0,"reason":"0 commit(s) and 0 issue activity found in the last 90 days -- score normalized to 0","details":null,"documentation":{"short":"Determines if the project is \"actively maintained\".","url":"https://github.com/ossf/scorecard/blob/c1d103a9bb9f635ec7260bf9aa0699466fa4be0e/docs/checks.md#maintained"}},{"name":"Code-Review","score":0,"reason":"Found 0/17 approved changesets -- score normalized to 0","details":null,"documentation":{"short":"Determines if the project requires human code review before pull requests (aka merge requests) are merged.","url":"https://github.com/ossf/scorecard/blob/c1d103a9bb9f635ec7260bf9aa0699466fa4be0e/docs/checks.md#code-review"}},{"name":"Pinned-Dependencies","score":-1,"reason":"no dependencies found","details":null,"documentation":{"short":"Determines if the project has declared and pinned the dependencies of its build process.","url":"https://github.com/ossf/scorecard/blob/c1d103a9bb9f635ec7260bf9aa0699466fa4be0e/docs/checks.md#pinned-dependencies"}},{"name":"Binary-Artifacts","score":10,"reason":"no binaries found in the repo","details":null,"documentation":{"short":"Determines if the project has generated executable (binary) artifacts in the source repository.","url":"https://github.com/ossf/scorecard/blob/c1d103a9bb9f635ec7260bf9aa0699466fa4be0e/docs/checks.md#binary-artifacts"}},{"name":"CII-Best-Practices","score":0,"reason":"no effort to earn an OpenSSF best practices badge detected","details":null,"documentation":{"short":"Determines if the project has an OpenSSF (formerly CII) Best Practices Badge.","url":"https://github.com/ossf/scorecard/blob/c1d103a9bb9f635ec7260bf9aa0699466fa4be0e/docs/checks.md#cii-best-practices"}},{"name":"Security-Policy","score":0,"reason":"security policy file not detected","details":["Warn: no security policy file detected","Warn: no security file to analyze","Warn: no security file to analyze","Warn: no security file to analyze"],"documentation":{"short":"Determines if the project has published a security policy.","url":"https://github.com/ossf/scorecard/blob/c1d103a9bb9f635ec7260bf9aa0699466fa4be0e/docs/checks.md#security-policy"}},{"name":"Fuzzing","score":0,"reason":"project is not fuzzed","details":["Warn: no fuzzer integrations found"],"documentation":{"short":"Determines if the project uses fuzzing.","url":"https://github.com/ossf/scorecard/blob/c1d103a9bb9f635ec7260bf9aa0699466fa4be0e/docs/checks.md#fuzzing"}},{"name":"License","score":10,"reason":"license file detected","details":["Info: project has a license file: LICENSE:0","Info: FSF or OSI recognized license: Apache License 2.0: LICENSE:0"],"documentation":{"short":"Determines if the project has defined a license.","url":"https://github.com/ossf/scorecard/blob/c1d103a9bb9f635ec7260bf9aa0699466fa4be0e/docs/checks.md#license"}},{"name":"Signed-Releases","score":-1,"reason":"no releases found","details":null,"documentation":{"short":"Determines if the project cryptographically signs release artifacts.","url":"https://github.com/ossf/scorecard/blob/c1d103a9bb9f635ec7260bf9aa0699466fa4be0e/docs/checks.md#signed-releases"}},{"name":"Branch-Protection","score":0,"reason":"branch protection not enabled on development/release branches","details":["Warn: branch protection not enabled for branch 'master'"],"documentation":{"short":"Determines if the default and release branches are protected with GitHub's branch protection settings.","url":"https://github.com/ossf/scorecard/blob/c1d103a9bb9f635ec7260bf9aa0699466fa4be0e/docs/checks.md#branch-protection"}},{"name":"Vulnerabilities","score":0,"reason":"11 existing vulnerabilities detected","details":["Warn: Project is vulnerable to: PYSEC-2020-92 / GHSA-hj5v-574p-mj7c","Warn: Project is vulnerable to: PYSEC-2022-42969","Warn: Project is vulnerable to: PYSEC-2023-44 / GHSA-329j-jfvr-rhr6","Warn: Project is vulnerable to: PYSEC-2022-42976 / GHSA-43xg-8wmj-cw8h","Warn: Project is vulnerable to: PYSEC-2022-236 / GHSA-4x9r-j582-cgr8","Warn: Project is vulnerable to: PYSEC-2018-25 / GHSA-6mqq-8r44-vmjc","Warn: Project is vulnerable to: PYSEC-2017-147 / GHSA-8rhc-48pp-52gr","Warn: Project is vulnerable to: PYSEC-2022-186 / GHSA-9rr6-jpg7-9jg6","Warn: Project is vulnerable to: PYSEC-2019-114 / GHSA-fp5j-3fpf-mhj5","Warn: Project is vulnerable to: PYSEC-2020-95 / GHSA-wgx7-jwwm-cgjv","Warn: Project is vulnerable to: PYSEC-2023-72"],"documentation":{"short":"Determines if the project has open, known unfixed vulnerabilities.","url":"https://github.com/ossf/scorecard/blob/c1d103a9bb9f635ec7260bf9aa0699466fa4be0e/docs/checks.md#vulnerabilities"}}]},"last_synced_at":"2025-08-16T05:43:40.830Z","repository_id":57427543,"created_at":"2025-08-16T05:43:40.830Z","updated_at":"2025-08-16T05:43:40.830Z"},"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":273183228,"owners_count":25059812,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-09-01T02:00:09.058Z","response_time":120,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["analytics","apache","apache-spark","aws","azure","bigdata","databricks","gcp","logging","pyspark","python","spark","spark-utils"],"created_at":"2024-09-24T14:05:23.247Z","updated_at":"2026-03-17T18:36:33.372Z","avatar_url":"https://github.com/Spratiher9.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"[comment]: \u003c\u003e (\u003ca href=\"https://ibb.co/jhTdDPZ\"\u003e\u003cimg src=\"https://i.ibb.co/6Ym2F7J/xLogs.gif\" alt=\"Exelog: Meticulous logging for Apache Spark\" border=\"0\"\u003e\u003c/a\u003e)\n\n![Exelog: Meticulous logging for Apache Spark](https://raw.githubusercontent.com/Spratiher9/Files/master/xLogs.gif)\n\n![PyPI - Python Version](https://img.shields.io/pypi/pyversions/exelog)\n[![PyPI](https://img.shields.io/pypi/v/exelog)](https://pypi.org/project/exelog/)\n\n# Exelog: Meticulous logging for Apache Spark\n\n### _Enabling meticulous logging for Spark Applications_\n\n`Exelog` is a refactored logging module that provides a decorator based approach to ensure standard Python logging from\nPySpark Executor nodes also.\n\nDrop a 🌟 if this project helps and you would like to see more development on it.\n\n### Installation\n\n```shell\npip install exelog\n```\n\n## Why?\n\n### _The problem: logging from Spark executors doesn't work with normal logging_\n\nIn Apache Spark, the actual data processing is done in what's called \"executors\", which are separate processes that are\nseparate from the \"driver\" program. For end users, full control is only over the driver process, but not so much over\nthe executor processes.\n\nFor example, in the PySpark driver program we can set up standard python logging as desired, but this setup is not\nreplicated in the executor processes. There are no out-of-the-box logging handlers for executors, so all logging\nmessages from executors are lost. Since Python 3.2 however, when there are no handlers, there is still a \"improvised\"\nhandler that will show `warning()` and `error()` messages in their bare format on standard error, but for proper logging\nwe probably need something more flexible and powerful than that.\n\nIllustration in interactive PySpark shell:\n```python\n    \u003e\u003e\u003e import os, logging\n    \u003e\u003e\u003e logging.basicConfig(level=logging.INFO)\n    \n    \u003e\u003e\u003e def describe():\n    ...     return \"pid: {p}, root handlers: {h}\".format(p=os.getpid(), h=logging.root.handlers)\n    ... \n    \u003e\u003e\u003e describe()\n    'pid: 8915, root handlers: [\u003cStreamHandler \u003cstderr\u003e (NOTSET)\u003e]'\n\n    \u003e\u003e\u003e sc.parallelize([4, 1, 7]).map(lambda x: describe()).collect()\n    ['pid: 9111, root handlers: []', 'pid: 9128, root handlers: []', 'pid: 9142, root handlers: []']\n```\nThe initial `describe()` happens in the driver and has root handlers because of the `basicConfig()` beforehand. However,\nthe `describe()` calls in the `map()` happen in separate executor processes (note the different PIDs) and got no root\nhandlers.\n\n## Exelog: Logging Spark executor execution one decorator at a time\n\nVarious ways for setting up logging for executors may be found on the internet. It usually entails sending and loading a\nseparate file containing logging configuration code. Depending on the use case, managing this file may be difficult. One\nof the approaches\nis [here](https://community.cloudera.com/t5/Support-Questions/Logging-from-Pyspark-executor/td-p/212210)\n\nIn contrast, `Exelog` takes a decorator based approach. We just have to decorate the data processing functions which we\nare passing to `map()`, `filter()`, `sortBy()`, etc.\n\nA very minimal example:\n```python\n    @exelog.enable_info_logging\n    def process(x):\n        logger.info(\"Got {x}\".format(x=x))\n        return x * x\n    \n    result = rdd.map(process).collect()\n```\nWhat will happen here is that the first time `process()` is called in the executor, basic logging is set up with `INFO`\nlevel, so that logging messages are not lost.\n\n### Options and finetuning\n\nThe `enable_exelog` decorator will do a basic logging setup using\n[`logging.basicConfig()`](https://docs.python.org/3/library/logging.html#logging.basicConfig), and desired options can\nbe directly provided to the decorator as illustrated in the following example using the interactive PySpark shell:\n```python\n    \u003e\u003e\u003e import logging\n    \u003e\u003e\u003e from exelog import enable_exelog\n    \u003e\u003e\u003e logger = logging.getLogger(\"example\")\n    \n    \u003e\u003e\u003e @enable_exelog(level=logging.INFO)\n    ... def process(x):\n    ...     logger.info(\"Got {x}\".format(x=x))\n    ...     return x * x\n    ... \n    \u003e\u003e\u003e sc.parallelize(range(5)).map(process).collect()\n    INFO:example:Got 0\n    INFO:example:Got 1\n    INFO:example:Got 3\n    INFO:example:Got 2\n    INFO:example:Got 4\n    [0, 1, 4, 9, 16]\n```\nTo improve readability or code reuse, you can of course predefine decorators:\n```python\n    with_logging = enable_exelog(\n        level=logging.INFO,\n        format=\"[%(process)s/%(name)s] %(levelname)s %(message)s\"\n    )\n    \n    @with_logging\n    def process(x):\n        ...\n```\n`exelog` also defines some simple predefined decorators:\n```python\n    # Predefined decorator for stderr/NOTSET logging\n    enable_notset_logging = enable_exelog(level=logging.NOTSET)\n    \n    # Predefined decorator for stderr/DEBUG logging\n    enable_debug_logging = enable_exelog(level=logging.DEBUG)\n    \n    # Predefined decorator for stderr/INFO logging\n    enable_info_logging = enable_exelog(level=logging.INFO)\n    \n    # Predefined decorator for stderr/WARN logging\n    enable_warn_logging = enable_exelog(level=logging.WARN)\n    \n    # Predefined decorator for stderr/ERROR logging\n    enable_error_logging = enable_exelog(level=logging.ERROR)\n    \n    # Predefined decorator for stderr/CRITICAL logging\n    enable_critical_logging = enable_exelog(level=logging.CRITICAL)\n```\n### Fine-grained logging set up\n\nIf the `logging.basicConfig()` API is not flexible enough for your desired setup, you can also inject more advanced\nsetup code with the `initialized_call` decorator. This decorator is not limited to logging setup, it just expects a\ncallable (that can be called without arguments). A very simple example:\n\n    @exelog.initialized_call(lambda: print(\"Executor logging enabled\"))\n    def process(x):\n        ....\n\nThis will print \"Executor logging enabled\" the first time the `process` function is called in each executor.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fspratiher9%2Fexelog","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fspratiher9%2Fexelog","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fspratiher9%2Fexelog/lists"}