{"id":23283585,"url":"https://github.com/micrologai/microlog","last_synced_at":"2025-08-25T00:32:10.944Z","repository":{"id":179892335,"uuid":"625957162","full_name":"micrologai/microlog","owner":"micrologai","description":"Microlog - A continuous profiler/logger for Python with microscopically small recordings, no overhead, and super-fast rendering using PyScript","archived":false,"fork":false,"pushed_at":"2023-11-17T19:27:57.000Z","size":8044,"stargazers_count":16,"open_issues_count":1,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2024-12-20T00:37:18.931Z","etag":null,"topics":["microlog","profile","profiler","pyscript","python","python-3"],"latest_commit_sha":null,"homepage":"https://github.com/micrologai/microlog","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"other","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/micrologai.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null}},"created_at":"2023-04-10T13:42:18.000Z","updated_at":"2024-08-04T21:29:26.000Z","dependencies_parsed_at":"2023-11-17T14:43:44.694Z","dependency_job_id":null,"html_url":"https://github.com/micrologai/microlog","commit_stats":null,"previous_names":["micrologai/microlog"],"tags_count":1,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/micrologai%2Fmicrolog","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/micrologai%2Fmicrolog/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/micrologai%2Fmicrolog/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/micrologai%2Fmicrolog/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/micrologai","download_url":"https://codeload.github.com/micrologai/microlog/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":230852614,"owners_count":18290082,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["microlog","profile","profiler","pyscript","python","python-3"],"created_at":"2024-12-20T01:22:42.290Z","updated_at":"2024-12-22T15:56:54.053Z","avatar_url":"https://github.com/micrologai.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# microlog.ai\n\nA live demo of the _Microlog_ Python UI, running on PyScript with MicroPython, can be found at [micrologai.github.io/microlog](https://micrologai.github.io/microlog/). \n\n_Microlog_ is a continuous profiler and logger for the Python language that\nexplains application behavior using interactive graphs and AI. \nIt makes understanding complex applications easy, reducing support costs\nand shortening production problems, increasing application quality, and minimizing outages.\n\n_Microlog_ has extremely low runtime overhead (~1%) and exceptionally fast rendering (~20ms).\nIt saves logs and performance profiles on the local file system. The logs are\ncompressed exceptionally well, resulting in a remarkably low 0.5MB per hour of recording.\n\nThis project is written in 100% Python. The recorder is a Python module that uses a separate thread to sample \nperformance and record logs. The UI is written in Python as well, rendered in the browser by Python code using\nPyScript. As a result, the identical Python classes encode _and_ decode the recordings, avoiding the need\nfor cumbersome cross-language data modeling. \n\n_Microlog_ is open source, with an available commercial license. We welcome extensions to _Microlog_ from\nthe Python performance community, such as \nrecording of special events, new optimizations related to PyScript,\nor centralization of recordings into central storage systems, using `rsync`, `scp`, or `Google Drive`.\n\n# Installing microlog.ai\n\nTo install _Microlog_ from pypi run:\n```console\npip install micrologai\n```\n\nTo install _Microlog_ globally using a `sitecustomize.py`, run:\n\n```console\ngit3 clone https://github.com/micrologai/microlog\ncd micrologai/microlog\npython3 setup.py install\n```\n\n# How to use microlog.ai\n\nIf you used the setup command shown above, _Microlog_ is enabled as a continuous profiler for all Python processes running on that Python VM. \nTo use microlog manually, use:\n```python\nimport microlog\n\nwith microlog.enabled():\n    # run any Python code\n```\n\nThe part of _Microlog_ that records the execution can be found in [microlog/tracer.py](microlog/tracer.py#L89). It starts a new thread and samples the other threads at regular intervals, see [sample](microlog/tracer.py#L240). The tracer also sets up wrappers for logging and print statements. The logs are compressed and saved when _Microlog_ is stopped, by [microlog/log.py](microlog/log.py#L94).\n\nTo give you an idea of the features of _Microlog_, you could run all the examples. This does assume you set up microlog globally. In that case, run:\n\n```console\nsh examples/runall.sh\n```\n\nThis runs for a minute and eventually produces 13 logs. You will see lines appear looking like this:\n\n```console\n📈 Microlog ··· 26.3s ··· 4.6KB ··· examples-memory ··· http://127.0.0.1:4000/log/examples-memory/2023_07_12_10_24_53 🚀\n```\n\nThis shows how long the app ran, the size of the (compressed) log, its name, and a URL to view the result.\nThe report URL is rendered by the _Microlog_ server implemented in [microlog/server.py](microlog/server.py).  If it is not yet running,\nyou can start it as follows:\n\n```console\npython3 microlog/server.py\n```\n\n# The Microlog.ai UI \n\nA live demo of the _Microlog_ UI can be found at [micrologai.github.io/microlog](https://micrologai.github.io/microlog/). The UI is\nwritten almost entirely in Python, see [dashboard](dashboard). The _Microlog_ UI runs in the browser using PyScript. \n\nTo describe the UI features of _Microlog_, we will look at the output of the [examples\\memory.py](examples\\memory.py) example (the live preview is at\n[GitHub Pages](https://micrologai.github.io/microlog/#examples-memory/2023_07_31_11_16_19/)):\n\n![Example run of microlog](https://github.com/micrologai/microlog/raw/main/microlog/images/overview.png)\n\nThe main elements of the UI are:\n\n - `Log list`, at the left, showing currently available logs on the local machine.\n - `Timeline`, the starting point for analysis of your application:\n    - `Status`, providing insights into CPU, memory, and I/O.\n    - `Timescale`, with ticks for second elapsed since start.\n    - `Flamegraph`, showing the result of the continuous profiler. \n - `Design`, showing a graphical rendering of the structure of the application.\n - `Log`, listing all print and logging output and statistics in a chronological order.\n - `Explanation`, giving a human language description of your code, using OpenAIs' ChatGPT.\n - `Tips`, linking to best practices and tips for the modules used in your code.\n\n\n## Timeline Navigation and Zoom\n\nUsing the mouse, the dashboard can be panned and zoomed. More details will be shown when zoomed in deeper:\n\n![Example run of microlog](https://github.com/micrologai/microlog/raw/main/microlog/images/zoomedin.png)\n\nIn the above example, we panned the flame graph by grabbing it with the mouse and zoomed in using the scroll wheel on the mouse.\n\nIn addition, we clicked on a method call in the flame graph, which is now highlighted in red. A moveable popup dialog shows details about the method, such as the average CPU during the call. A CPU percentage below 100% means the process is involved\nin reading or writing files on the local disk, loading or sending data over sockets, loading new modules (requiring disk I/O), async or thread synchronization, or other system-level event handling using `select` or event handlers. \n\nA low CPU typically indicates a bottleneck and warrants in-depth investigation.\n\n## Timeline Anomaly Detection\n\nWhen a method is selected in the flame graph, the popup shows information about similar calls detected in the same run, showing when they ran and how long they ran. _Microlog_ also uses anomaly detection to highlight methods you may want to investigate in more detail. In the screenshot below, the average call duration is 1 second, and four calls were more than 50% over the average. \n\nAutomatic anomaly detection, call stack analysis, and process health indicators offered by _Microlog_ allow you to debug performance/quality issues quickly.\n\n![Anomaly detection in the Microlog UI](https://github.com/micrologai/microlog/raw/main/microlog/images/anomaly.png)\n\n## Timeline Detecting expensive I/O or Starved Processes\n\nThe top bar shows general statistics for the process, such as CPU and number of modules loaded over time. \nNote that a low CPU in the top bar tends to indicate I/O took place at that moment.\n\n![Microlog's status bar](https://github.com/micrologai/microlog/raw/main/microlog/images/status.png)\n\n## Timeline Integrating Profiling with Logging\n\nLog entries are shown as visual markers in the top bar. Because _Microlog_ shows log entries on the timeline, analyzing problems becomes much easier than with normal logs. No more scrolling page after page to find a stack trace. With _Microlog_, they appear as easy-to-see stop signs:\n\n![Log entries in the status bar](https://github.com/micrologai/microlog/raw/main/microlog/images/error-log.png)\n\n## Formatting Logs with Markdown\n\nLog entries can be formatted using Markdown to make it easier to show important information to the reader.\n\n![Using markdown for log entries](https://github.com/micrologai/microlog/raw/main/microlog/images/markdown.png)\n\n# Logging \n\n_Microlog_ detects calls to `print` and `logging`. Those calls are automatically intercepted\nadded to the _Microlog_ event log.  \n\nManual log entries can be inserted into Microlog using `info`, `warn`, `debug`, and `error`:\n\n```python\nprint(\"Add a log entry to microlog with an info marker...\")\nprint(\"... or as an error marker.\", stream=sys.stderr)\n\nimport logging\nlogger = logging.Logger(\"Microlog Demo\")\n\nlogger.info(\"Add a log entry to microlog with an info marker...\")\nlogger.debug(\"... or a bug marker...\")\nlogger.warning(\"... or a warning marker...\")\nlogger.error(\"... or an error marker.\")\n   \nmicrolog.info(\"Add something to the log explicitly...\")\nmicrolog.warning(\"... as a warning...\")\nmicrolog.debug(\"... as a debug message...\")\nmicrolog.error(\"... as an error.\")\n```\n\n# Design\n\nThe Design tab analyzes the runtime call graph and draws a structural diagram of the underlying design of your application.\nHere is an example for `examples\\go.py`:\n\n![Microlog's innovative design graph](https://github.com/micrologai/microlog/raw/main/microlog/images/design-go.png)\n\n# Log\n\nThe log tab contains a chronological listing of all print and logging output, statistics, and analysis performed by _Microlog_ in a more traditional linear log style:\n\n![Microlog's traditional log, with a twist](https://github.com/micrologai/microlog/raw/main/microlog/images/log.png)\n\nThe links shown in lightblue directly take you to the source file where the print took place. This makes it extremely easy to figure out what code prints what exactly. For the `files.py` example, the log shows that this program leaked one file descriptor. That same report is also shown in the timeline when you click at the warning icon at the end of the run. \nThe popup also shows a link to the source location where the file was opened:\n\n![Detecting leaked file descriptors](https://github.com/micrologai/microlog/raw/main/microlog/images/fd-leak.png)\n\n## Log - Memory Leaks\n\nIn addition to checking for leaked file descriptors, _Microlog_ aims to detect memory leaks. Those objects that were\nallocated but cannot be garbage-collected as they are either reachable from a module or are involed in a reference cycle that\ncannot be broken. The following report shows the top-10 offenders for the `dataframes.py` example, that uses Pandas dataframes.\n\n![Detecting memory leaks](https://github.com/micrologai/microlog/raw/main/microlog/images/memory-leak.png)\n\n# Explanation\n\nThe explanation tab shows a human-language explanation provided by OpenAI's `ChatGPT` to explain the design and implementation behind the application being monitored by _Microlog_. This analysis is not performed on the source code, but on a condensed call graph generated from the performance log that was recorded by _Microlog_. Here is an example of the what `ChatGPT` thinks of\nour `examples\\go.py` execution:\n\n![ChatGPT's simple explanation of complex Python code](https://github.com/micrologai/microlog/raw/main/microlog/images/chatgpt.png)\n\n# Tips\n\nTo discover best practices, performance tips, or tutorials for modules being used by the application, _Microlog_ offers quick\nlinks to general information sources, such as search engines and Q\u0026A sites:\n\n![Tips make you a better Python developer](https://github.com/micrologai/microlog/raw/main/microlog/images/tips.png)\n\n\n# Developer Notes\n\n## Run all unit tests\n\n```\npython3 -m unittest discover tests\n```\n\n\n## Upload new version to PyPi\n\nFirst build the package into a source distribution and a Python wheel:\n```console\npython3 -m pip install --user --upgrade setuptools wheel twine build\npython3 -m build\n```\n\nThen verify whether the build works for pypi:\n```console\ntwine check dist/*\n```\n\nThen upload to the pypi test environment:\n```console\ntwine upload --repository pypitest dist/*\n```\n\nFinally, if the pypi test upload appears to work fine, run:\n```console\ntwine upload dist/*\n```\n\n# License\n\n_Microlog_ is released under version 1 of the [Server Side Public License (SSPL)](LICENSE).\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmicrologai%2Fmicrolog","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fmicrologai%2Fmicrolog","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmicrologai%2Fmicrolog/lists"}