{"id":13576771,"url":"https://github.com/smacke/pyccolo","last_synced_at":"2025-04-04T15:06:54.424Z","repository":{"id":41497276,"uuid":"438517026","full_name":"smacke/pyccolo","owner":"smacke","description":"Declarative instrumentation for Python.","archived":false,"fork":false,"pushed_at":"2025-03-17T06:09:09.000Z","size":386,"stargazers_count":88,"open_issues_count":0,"forks_count":3,"subscribers_count":3,"default_branch":"master","last_synced_at":"2025-03-28T14:07:39.008Z","etag":null,"topics":["ast","declarative","instrumentation","interpreter","metaprogramming","python","tracing"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/smacke.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":"docs/CODE_OF_CONDUCT.md","threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2021-12-15T06:16:16.000Z","updated_at":"2025-03-17T06:09:12.000Z","dependencies_parsed_at":"2024-06-19T20:02:49.324Z","dependency_job_id":"8f4d05f0-e964-4ef9-a105-431937136ccb","html_url":"https://github.com/smacke/pyccolo","commit_stats":{"total_commits":334,"total_committers":1,"mean_commits":334.0,"dds":0.0,"last_synced_commit":"e5080a83c6fccbca74602e0ffad33b0fc8edeedf"},"previous_names":[],"tags_count":69,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/smacke%2Fpyccolo","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/smacke%2Fpyccolo/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/smacke%2Fpyccolo/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/smacke%2Fpyccolo/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/smacke","download_url":"https://codeload.github.com/smacke/pyccolo/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":247198449,"owners_count":20900079,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["ast","declarative","instrumentation","interpreter","metaprogramming","python","tracing"],"created_at":"2024-08-01T15:01:13.902Z","updated_at":"2025-04-04T15:06:54.399Z","avatar_url":"https://github.com/smacke.png","language":"Python","funding_links":[],"categories":["Python"],"sub_categories":[],"readme":"Pyccolo\n=======\n\n[![CI Status](https://github.com/smacke/pyccolo/workflows/pyccolo/badge.svg)](https://github.com/smacke/pyccolo/actions)\n[![Checked with mypy](http://www.mypy-lang.org/static/mypy_badge.svg)](http://mypy-lang.org/)\n[![codecov](https://codecov.io/gh/smacke/pyccolo/branch/master/graph/badge.svg?token=MGORH1IXLO)](https://codecov.io/gh/smacke/pyccolo)\n[![License: BSD3](https://img.shields.io/badge/License-BSD3-maroon.svg)](https://opensource.org/licenses/BSD-3-Clause)\n[![Python Versions](https://img.shields.io/pypi/pyversions/pyccolo.svg)](https://pypi.org/project/pyccolo)\n[![PyPI Version](https://img.shields.io/pypi/v/pyccolo.svg)](https://pypi.org/project/pyccolo)\n\nPyccolo (pronounced like the instrument \"piccolo\") is a library for declarative\ninstrumentation in Python; i.e., it lets you specify the *what* of the\ninstrumentation you wish to perform, and takes care of the *how* for you.  It\naims to be *ergonomic*, *composable*, and *portable*, by providing an intuitive\ninterface, making it easy to layer multiple levels of instrumentation, and\nallowing the same code to work across multiple versions of Python (3.6 to\n3.12), with few exceptions. Portability across versions is accomplished by\nembedding instrumentation at the level of source code (as opposed to\nbytecode-level instrumentation).\n\nPyccolo can be used (and has been used) to implement various kinds of dynamic analysis\ntools and other instrumentation:\n- Code coverage (see [pyccolo/examples/coverage.py](https://github.com/smacke/pyccolo/blob/master/pyccolo/examples/coverage.py))\n- Syntactic macros such as quasiquotes (like [MacroPy's](https://macropy3.readthedocs.io/en/latest/reference.html#quasiquote)) or quick lambdas; see [pyccolo/examples/quasiquote.py](https://github.com/smacke/pyccolo/blob/master/pyccolo/examples/quasiquote.py) and [pyccolo/examples/quick_lambda.py](https://github.com/smacke/pyccolo/blob/master/pyccolo/examples/quick_lambda.py)\n- Syntax-augmented Python (3.8 and up, see [pyccolo/examples/optional_chaining.py](https://github.com/smacke/pyccolo/blob/master/pyccolo/examples/optional_chaining.py))\n- Dynamic dataflow analysis performed by [ipyflow](https://github.com/ipyflow/ipyflow)\n- Tools to perform (most) imports lazily (see [pyccolo/examples/lazy_imports.py](https://github.com/smacke/pyccolo/blob/master/pyccolo/examples/lazy_imports.py))\n- Tools to uncover [semantic memory leaks](http://ithare.com/java-vs-c-trading-ub-for-semantic-memory-leaks-same-problem-different-punishment-for-failure/)\n- \\\u003cYour tool here!\u003e\n\n## Install\n\n```bash\npip install pyccolo\n```\n\n## Hello World\n\nBelow is a simple script that uses Pyccolo to print \"Hello, world!\" before\nevery statement that executes:\n\n```python\nimport pyccolo as pyc\n\n\nclass HelloTracer(pyc.BaseTracer):\n    @pyc.before_stmt\n    def handle_stmt(self, *_, **__):\n        print(\"Hello, world!\")\n\n\nif __name__ == \"__main__\":\n    with HelloTracer:\n        # prints \"Hello, world!\" 11 times\n        pyc.exec(\"for _ in range(10): pass\")\n```\n\nInstrumentation is provided by a *tracer class* that inherit from\n`pyccolo.BaseTracer`. This class rewrites Python source code with\ninstrumentation that triggers whenever events of interest occur, such as when a\nstatement is about to execute. By registering a handler with the associated\nevent (with the `@pyc.before_stmt` decorator, in this case), we can enrich our\nprograms with additional observability, or even alter their behavior\naltogether.\n\n### What is up with `pyc.exec(...)`?\n\nA program's abstract syntax tree is fixed at import / compile time, and when\nour script initially started running, the tracer was not active, so unquoted\nPython in the same file will lack instrumentation. It is possible to instrument\nmodules at import time, but only when the imports are performed inside a\ntracing context. Thus, we must quote any code appearing in the same module\nwhere the tracer class was defined in order to instrument it.\n\n## Composing tracers\n\nA core feature of Pyccolo is that its instrumentation is *composable*.  It's\nusually tricky to use two or more `ast.NodeTransformer` classes simultaneously\n--- sometimes you can just have one inherit from the other, but if they both\ndefine `visit` methods for the same AST node type, then typically you would\nneed to define a bespoke node transformer that uses logic from each base\ntransformer, handling corner cases to resolve incompatibilities.  With Pyccolo,\nyou simply compose the context managers of each tracer class whose\ninstrumentation you wish to use, and everything usually Just\nWorks\u003csup\u003eTM\u003c/sup\u003e:\n\n```python\nwith tracer1:\n    with tracer2:\n        pyc.exec(...)\n```\n\n## Compatibility with sys.settrace(...)\n\nPyccolo is designed to support not only AST-level instrumentation, but also\ninstrumentation involving Python's [built in tracing\nutilities](https://docs.python.org/3/library/sys.html#sys.settrace).\nTo use it, you simply register handlers for one of the corresponding\nPyccolo events (`call`, `line`, `return_`, `exception`, or `opcode`).\nHere's a minimal example:\n\n```python\nimport pyccolo as pyc\n\n\nclass SysTracer(pyc.BaseTracer):\n    @pyc.call\n    def handle_call(self, *_, **__):\n        print(\"Pushing a stack frame!\")\n\n    @pyc.return_\n    def handle_return(self, *_, **__):\n        print(\"Popping a stack frame!\")\n\n\nif __name__ == \"__main__\":\n    with SysTracer:\n        def f():\n            def g():\n                return 42\n            return g()\n        # push, push, pop, pop\n        answer_to_life_universe_everything = f()\n```\n\nNote that we didn't need to use `pyc.exec(...)` in the above example, because Python's built-in\ntracing does not involve any AST-level transformations. If, however, we had registered handlers\nfor other events, such as `pyc.before_stmt`, we would need to use `pyc.exec(...)` to ensure those\nhandlers get called, when running code in the same file where our tracer class is defined.\n\n### What if I'm already using sys.settrace(...) with my own tracing function?\n\nPyccolo is designed to be *composable*, and should execute both your tracing function as well\nas any handlers defined in any active Pyccolo tracers. For example Pyccolo's unit tests for\n`call` and `return` events work even when [coverage.py](https://coverage.readthedocs.io/)\nis active (and without breaking it), which also uses Python's built-in tracing utilities.\n\n## Instrumenting Imported Modules\n\nInstrumentation is opt-in for modules imported within tracing contexts. To determine whether\na module gets instrumented, the method `should_instrument_file(...)` is called with the module's\ncorresponding filename as input. For example:\n\n```python\nclass MyTracer(pyc.BaseTracer):\n    def should_instrument_file(self, filename: str) -\u003e bool:\n        return filename.endswith(\"foo.py\")\n    \n    # handlers, etc. defined below\n    ...\n\nwith MyTracer:\n    import foo  # contents of `foo` module get instrumented\n    import bar  # contents of `bar` module do not get instrumented\n```\n\nImports are instrumented by registering a custom finder / loader with `sys.meta_path`.\nThis loader ignores cached bytecode (which may possibly be uninstrumented), and avoids\ngenerating *new* cached bytecode (which would be instrumented, possibly causing confusion\nlater when instrumentation is not desired).\n\n## Command Line Interface\n\nYou can execute arbitrary scripts with instrumentation enabled with the `pyc` command line tool.\nFor example, to use the `OptionalChainer` tracer defined in [pyccolo/examples/optional_chaining.py](https://github.com/smacke/pyccolo/blob/master/pyccolo/examples/optional_chaining.py),\nyou can call `pyc` as follows, given some example script `bar.py`:\n\n```python\n# bar.py\nbar = None\n# prints `None` since bar?.foo coalesces to `None`\nprint(bar?.foo)\n```\n\n```bash\n\u003e pyc bar.py -t pyccolo.examples.OptionalChainer\n```\n\nYou can also run `bar` as a module (indeed, `pyc` performs this internally when provided a file):\n\n```bash\n\u003e pyc -m bar -t pyccolo.examples.OptionalChainer\n```\n\nNote that you can specify multiple tracer classes after the `-t` argument;\nin case you were not already aware, Pyccolo is composable! :)\n\nThe above example demonstrates a tracer class that performs syntax augmentation on its\ninstrumented Python source to modify the default Python syntax. This feature is available\nonly on Python \u003e= 3.8 for now and is lacking documentation for the moment, but you can\nsee some examples in the [test_syntax_augmentation.py](https://github.com/smacke/pyccolo/blob/master/test/test_syntax_augmentation.py) unit tests.\n\n## More Events\n\nPyccolo handlers can be registered for many kinds of events. Some of the more common ones are:\n- `pyc.before_stmt`, emitted before a statement executes;\n- `pyc.after_stmt`, emitted after a statement executes;\n- `pyc.before_attribute_load`, emitted in [load contexts](https://docs.python.org/3/library/ast.html#ast.Load) before an attribute is accessed;\n- `pyc.after_attribute_load`, emitted in load contexts after an attribute is accessed;\n- `pyc.load_name`, emitted when a variable is used in a load context (e.g. `foo` in `bar = foo.baz`);\n- `pyc.call` and `pyc.return_`, two non-AST trace events built-in to Python.\n\nThere are many different Pyccolo events, and more are always being added. See\n[pyccolo/trace_events.py](https://github.com/smacke/pyccolo/blob/master/pyccolo/trace_events.py)\nfor a full list.\n\nNote that, for AST events, Python source is only transformed to emit some event\nwhen there is at least one tracer active that has at least one handler\nregistered for that event. This prevents the transformed source from becoming\nextremely bloated when only a few events are needed.\n\n## Handler Interface\n\nEvery Pyccolo handler is passed four positional arguments:\n1. The return value, for instrumented expressions;\n2. The AST node (or node id, if using `register_raw_handler(...)`, or `None`, for `sys` events);\n3. The stack frame, at the point where instrumentation kicks in;\n4. The event (useful when the same handler is registered for multiple events).\n\nSome events pass additional keyword arguments, which I'm still in the process\nof documenting, but the above four tend to suffice for most use cases.\n\nNot every handler receives a return value; for example, this argument is always\n`None` for `pyc.after_stmt` handlers. For certain handlers, the return value\ncan be overridden. For example, by returning a value in a\n`pyc.before_attribute_load`, we override the object whose attribute is\naccessed. If we return nothing or `None`, then we do not override this object.\n(If we actually want to override it as `None` for some reason, then we can\nreturn `pyc.Null`.) For a particular event, handler return values compose with\nother handlers defined on the same tracer class as well as with handlers\ndefined on other tracer classes.\n\n## Performance\n\nPyccolo instrumentation adds significant overhead to Python. In some\ncases, this overhead can be partially mitigated if, for example, you only need\ninstrumentation the first time a statement runs. In such cases, you can\ndeactivate instrumentation after, e.g., the first time a function executes, or\nafter the first iteration in a loop for that respective function or loop, so\nthat further calls (iterations, respectively) use uninstrumented code with all\nthe mighty performance of native Python. This is implemented by activating\n\"guards\" associated with the function or loop, as in the below example:\n\n```python\nclass TracesOnce(pyc.BaseTracer):\n    @pyc.register_raw_handler((pyc.after_for_loop_iter, pyc.after_while_loop_iter))\n    def after_loop_iter(self, *_, guard, **__):\n        self.activate_guard(guard)\n\n    @pyc.register_raw_handler(pyc.after_function_execution)\n    def after_function_exec(self, *_, guard, **__):\n        self.activate_guard(guard)\n```\n\nSubsequent calls / iterations will be instrumented only after calling\n`self.deactivate_guard(...)` on the associated function / loop guard.\n\n## License\nCode in this project licensed under the [BSD-3-Clause License](https://opensource.org/licenses/BSD-3-Clause).\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsmacke%2Fpyccolo","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fsmacke%2Fpyccolo","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsmacke%2Fpyccolo/lists"}