{"id":31252477,"url":"https://github.com/admirito/pymongowatch","last_synced_at":"2026-05-14T23:03:41.613Z","repository":{"id":57456160,"uuid":"461201131","full_name":"admirito/pymongowatch","owner":"admirito","description":"A pymongo extension for watching and auditing the database operations","archived":false,"fork":false,"pushed_at":"2022-03-29T07:49:08.000Z","size":193,"stargazers_count":1,"open_issues_count":3,"forks_count":0,"subscribers_count":1,"default_branch":"master","last_synced_at":"2025-09-23T06:33:39.733Z","etag":null,"topics":["analytics","auditing","logging","mongodb","mongodb-driver","pymongo"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"other","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/admirito.png","metadata":{"files":{"readme":"README.org","changelog":null,"contributing":null,"funding":null,"license":"LICENSE.org","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2022-02-19T13:32:48.000Z","updated_at":"2022-11-07T10:51:03.000Z","dependencies_parsed_at":"2022-09-06T10:12:53.541Z","dependency_job_id":null,"html_url":"https://github.com/admirito/pymongowatch","commit_stats":null,"previous_names":[],"tags_count":4,"template":false,"template_full_name":null,"purl":"pkg:github/admirito/pymongowatch","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/admirito%2Fpymongowatch","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/admirito%2Fpymongowatch/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/admirito%2Fpymongowatch/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/admirito%2Fpymongowatch/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/admirito","download_url":"https://codeload.github.com/admirito/pymongowatch/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/admirito%2Fpymongowatch/sbom","scorecard":{"id":166892,"data":{"date":"2025-08-11","repo":{"name":"github.com/admirito/pymongowatch","commit":"14b10cd1e88a93ec94dcf6abda8aad3255c08f83"},"scorecard":{"version":"v5.2.1-40-gf6ed084d","commit":"f6ed084d17c9236477efd66e5b258b9d4cc7b389"},"score":2.7,"checks":[{"name":"Packaging","score":-1,"reason":"packaging workflow not detected","details":["Warn: no GitHub/GitLab publishing workflow detected."],"documentation":{"short":"Determines if the project is published as a package that others can easily download, install, easily update, and uninstall.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#packaging"}},{"name":"Code-Review","score":0,"reason":"Found 0/30 approved changesets -- score normalized to 0","details":null,"documentation":{"short":"Determines if the project requires human code review before pull requests (aka merge requests) are merged.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#code-review"}},{"name":"Maintained","score":0,"reason":"0 commit(s) and 0 issue activity found in the last 90 days -- score normalized to 0","details":null,"documentation":{"short":"Determines if the project is \"actively maintained\".","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#maintained"}},{"name":"Token-Permissions","score":-1,"reason":"No tokens found","details":null,"documentation":{"short":"Determines if the project's workflows follow the principle of least privilege.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#token-permissions"}},{"name":"Dangerous-Workflow","score":-1,"reason":"no workflows found","details":null,"documentation":{"short":"Determines if the project's GitHub Action workflows avoid dangerous patterns.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#dangerous-workflow"}},{"name":"Pinned-Dependencies","score":-1,"reason":"no dependencies found","details":null,"documentation":{"short":"Determines if the project has declared and pinned the dependencies of its build process.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#pinned-dependencies"}},{"name":"SAST","score":0,"reason":"no SAST tool detected","details":["Warn: no pull requests merged into dev branch"],"documentation":{"short":"Determines if the project uses static code analysis.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#sast"}},{"name":"Binary-Artifacts","score":10,"reason":"no binaries found in the repo","details":null,"documentation":{"short":"Determines if the project has generated executable (binary) artifacts in the source repository.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#binary-artifacts"}},{"name":"CII-Best-Practices","score":0,"reason":"no effort to earn an OpenSSF best practices badge detected","details":null,"documentation":{"short":"Determines if the project has an OpenSSF (formerly CII) Best Practices Badge.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#cii-best-practices"}},{"name":"Security-Policy","score":0,"reason":"security policy file not detected","details":["Warn: no security policy file detected","Warn: no security file to analyze","Warn: no security file to analyze","Warn: no security file to analyze"],"documentation":{"short":"Determines if the project has published a security policy.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#security-policy"}},{"name":"Fuzzing","score":0,"reason":"project is not fuzzed","details":["Warn: no fuzzer integrations found"],"documentation":{"short":"Determines if the project uses fuzzing.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#fuzzing"}},{"name":"Vulnerabilities","score":10,"reason":"0 existing vulnerabilities detected","details":null,"documentation":{"short":"Determines if the project has open, known unfixed vulnerabilities.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#vulnerabilities"}},{"name":"License","score":9,"reason":"license file detected","details":["Info: project has a license file: LICENSE.org:0","Warn: project license file does not contain an FSF or OSI license."],"documentation":{"short":"Determines if the project has defined a license.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#license"}},{"name":"Branch-Protection","score":0,"reason":"branch protection not enabled on development/release branches","details":["Warn: branch protection not enabled for branch 'master'"],"documentation":{"short":"Determines if the default and release branches are protected with GitHub's branch protection settings.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#branch-protection"}},{"name":"Signed-Releases","score":0,"reason":"Project has not signed or included provenance with any releases.","details":["Warn: release artifact v1.0.0 not signed: https://api.github.com/repos/admirito/pymongowatch/releases/63002596","Warn: release artifact v0.3.0 not signed: https://api.github.com/repos/admirito/pymongowatch/releases/61092469","Warn: release artifact v0.2.0 not signed: https://api.github.com/repos/admirito/pymongowatch/releases/60687054","Warn: release artifact v0.1.0 not signed: https://api.github.com/repos/admirito/pymongowatch/releases/60131619","Warn: release artifact v1.0.0 does not have provenance: https://api.github.com/repos/admirito/pymongowatch/releases/63002596","Warn: release artifact v0.3.0 does not have provenance: https://api.github.com/repos/admirito/pymongowatch/releases/61092469","Warn: release artifact v0.2.0 does not have provenance: https://api.github.com/repos/admirito/pymongowatch/releases/60687054","Warn: release artifact v0.1.0 does not have provenance: https://api.github.com/repos/admirito/pymongowatch/releases/60131619"],"documentation":{"short":"Determines if the project cryptographically signs release artifacts.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#signed-releases"}}]},"last_synced_at":"2025-08-16T15:10:27.691Z","repository_id":57456160,"created_at":"2025-08-16T15:10:27.691Z","updated_at":"2025-08-16T15:10:27.691Z"},"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":33046775,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-05-13T13:14:54.681Z","status":"online","status_checked_at":"2026-05-14T02:00:06.663Z","response_time":57,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["analytics","auditing","logging","mongodb","mongodb-driver","pymongo"],"created_at":"2025-09-23T06:29:26.667Z","updated_at":"2026-05-14T23:03:41.607Z","avatar_url":"https://github.com/admirito.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"* PyMongoWatch\n\n=pymongowatch= is an extension for the MongoDB's Python driver [[https://pymongo.readthedocs.io/en/stable/][pymongo]]\nwhich enable watching and auditing the underlying operations.\n\nMongoDB Community Server doesn't support [[https://docs.mongodb.com/manual/core/auditing/][auditing]] and the [[https://docs.mongodb.com/manual/tutorial/manage-the-database-profiler/][database\nprofiler]] is not very flexible. Basically your only option for\nfiltering is to collect operations that take longer than a certain\nvalue. You can't easily separate the insert log from the query log or\nyou cannot extract the operations only for a certain collection or a\ncertain set of queries.\n\nFurthermore, operations at database level are not always the same as\noperations in an application. A very simple operation like fetching\ndocuments with [[https://pymongo.readthedocs.io/en/stable/api/pymongo/collection.html#pymongo.collection.Collection.find][find]] may be translated to several [[https://docs.mongodb.com/manual/reference/command/getMore/][getMore]] operations at\ndatabase level.\n\nTherefore, if you want to analyze the user behavior by studying the\ndatabase queries, a very simple auditing log for each user search\ncontaining the query itself (possibly with deleting value fields to\nprotect the users' privacy), the number of the documents read from the\ndatabase and its time is all you need.\n\nDoing this at database level is not always feasible and doing it\nmanually at application level may not be very desirable for the\ndevelopers. This is when an intermediate tool at library level comes\nhandy.\n\n=pymongowatch= can detect and audit operations at application level\nwithout any modifications to existing applications.\n\n** Quick Setup\n\nFirst you have to install =pymongowatch=. It is available at [[https://pypi.org/project/pymongowatch/][Python\nPackage Index (PyPI)]] and can be installed with [[https://pip.pypa.io/en/stable/][pip]]. At a [[https://docs.python.org/3/tutorial/venv.html][virtual\nenvironment]] or system-wide:\n\n#+begin_src shell\npip install pymongowatch\n#+end_src\n\n[[https://github.com/admirito/pymongowatch#other-installation-options][Other installation options]] such as Debian packages are available, too.\n\nAfter basic installation, =pymongo.watcher= sub-package will be\navailable and you have two options to go on:\n\n*** Option 1: Call the watcher at application startup\n\nThis approach is recommended if you have no restrictions on modifying\nthe pymongo-based application and you don't want any hacks on the\n=pymongo= itself.\n\n#+begin_src python\nfrom pymongo import watcher\n\nwatcher.patch_pymongo()\nwatcher.add_logging_handlers()  # Adds a logging.StreamHandler by default\n#+end_src\n\nOf course, you can pass your own customized logging handlers to\n=add_logging_handlers= and add filters and formatters as you wish. See\nthe following sections for more information.\n\n*** Option 2: No application modification\n\nIf you don't own the application or you don't want to even add a\nsingle line of code to your existing application, =pymongo_mask= which\nis included in the =pymongowatch= package comes handy.\n\n=pymongo_mask= is a directory with a module =pymongo.py= which\nemulates the =pymongo= package by using the real installed =pymongo=;\nSo if you add =pymongo_mask= directory in your Python application\n[[https://docs.python.org/3/library/sys.html#sys.path][search path for modules]]--in somewhere with more priority than where\nthe real =pymongo= is installed--the mask will be imported in your\napplication instead of the real =pymongo=. Nevertheless the mask will\nwork as a proxy to the real installed =pymongo= and your application\nshould work as usual.\n\nThe importation of the pymongo mask will also trigger\n=watcher.patch_pymongo()= automatically, so the watchers will be\nenabled without any need to change your application source code or\neven =pymongo= library itself.\n\nFurthermore, the pymongo mask will apply the configuration at\n=etc/pymongowatch.yaml= with [[https://docs.python.org/3/library/logging.config.html#logging.config.dictConfig][logging.config.dictConfig]] to the loggers\nof the watchers. A sample [[https://github.com/admirito/pymongowatch/blob/master/etc/pymongowatch.yaml][etc/pymongowatch.yaml]] is included in the\n=pymongowatch= package but you can modify it as needed. The [[https://en.wikipedia.org/wiki/YAML][yaml]]\nconfiguration follows the [[https://docs.python.org/3/library/logging.config.html#dictionary-schema-details][logging dictionary schema]].\n\nFinally, =pymongowatch= package includes a =pymongowatch-install-mask=\nscript which can help you enable the pymongo mask with ease:\n\n#+begin_src shell\npymongowatch-install-mask  # follow the interactive instructions\n#+end_src\n\n** Usage\n\n=pymongowatch= implements classes such as =WatchCursor= (a subclass of\npymongo's =cursor=) which if used instead of pymongo's =cursor= can\nemit logs as specified. To do so, you need some workaround to make\npymongo's methods to use =pymongowatch= classes instead of its\ndefaults. The =patch_pymongo= function is the method that will call\nall the patch methods in =pymongowatch= classes to replace the default\npymongo underlying classes:\n\n#+begin_src python\nfrom pymongo import watcher\n\nwatcher.patch_pymongo()\n#+end_src\n\nAll the =pymongowatch= watcher classes will use the logger retrieved\nwith =pymongo.watcher.logger.get_log_emitter()= to emit their\nlogs. Each module will emit the logs with their [[https://docs.python.org/3/tutorial/modules.html][__name__]]. So for\nexample, all the cursor logs will be emitted under the\n=pymongo.watcher.cursor=. So you can apply specific logging\nconfiguration for each module or as all the logs has the prefix\n=pymongo.watcher= as their logger name you can apply global\nconfiguration with that name.\n\n#+begin_src python\nimport logging\n\n# The following code is not the recommended method for adding handlers\n# to the pymongowatch loggers and is for illustrative purposes only\n\nglobal_logger = logging.getLogger(\"pymongo.watcher\")\n# add a stream log handler globally to watch all the logs on the\n# console\nglobal_logger.addHandler(logging.StreamHandler())\n\ncursor_logger = logging.getLogger(\"pymongo.watcher.cursor\")\n# add a file handler specifically for cursor logs to store them in\n# file, too.\ncursor_logger.addHandler(logging.FileHandler(\"/tmp/watch.log\"))\n#+end_src\n\nOf course you can apply [[https://docs.python.org/3/library/logging.html#filter-objects][filters]] and [[https://docs.python.org/3/library/logging.html#formatter-objects][formatters]] or add more [[https://docs.python.org/3/library/logging.html#handler-objects][handlers]] as\nyou wish but there are some details that you should take care of. The\neasiest way to achieve this is by leveraging the\n=add_logging_handlers= method:\n\n#+begin_src python\nfrom pymongo import watcher\n\n# Add two handlers to get all the watcher logs both in file and\n# console\nconsole_handler = logging.StreamHandler()\nglobal_handler = logging.FileHanlder(\"/tmp/watcher-all.log\")\nwatcher.add_logging_handlers(console_handler, global_handler)\n\n# Add a more customized handler for cursor logs\ncursor_simple_handler = logging.FileHanlder(\"/tmp/watcher-cursor-simple.log\")\nwatcher.add_logging_handlers(\n    cursor_simple_handler,\n    logger_name=\"pymongo.watcher.cursor\",\n    formatter=\"{name} - {watch}\")\n\n# Add a more customized handler for cursor logs\ncursor_customized_handler = logging.FileHanlder(\n    \"/tmp/watcher-cursor-customized.log\")\nwatcher.add_logging_handlers(\n    cursor_customized_handler,\n    logger_name=\"pymongo.watcher.cursor\",\n    formatter=\"{asctime} {name}.{watch.Collection} - {watch.Query} fetched \"\n              \"{watch.RetrievedCount} in {watch.RetrieveTime} seconds\")\n\n# Add another handler to log the full information for cursors in csv\ncursor_csv_handler = logging.FileHanlder(\n    \"/tmp/watcher-cursor-csv.log\")\nwatcher.add_logging_handlers(\n    cursor_csv_handler,\n    logger_name=\"pymongo.watcher.cursor\",\n    formatter=\"{asctime},{name},{watch.csv}\")\n#+end_src\n\nNote that using =add_logging_handlers= has not only the advantage of\nsimplicity for adding formatters, but also take care of automatically\nadding an extra [[https://docs.python.org/3/library/logging.handlers.html#queuehandler][logging.handlers.QueueHandler]] and\n[[https://docs.python.org/3/library/logging.handlers.html#queuelistener][logging.handlers.QueueListener]] for each handler to overcome some log\nmutation issues we discuss later. It will also add a couple of\ncustomized [[https://docs.python.org/3/library/logging.html#filter-objects][filters]] to the =QueueHandler= that are necessary to fully\ntake advantage of =pymongowatch=.\n\nYou can set the log format by using [[https://docs.python.org/3/library/logging.html#formatter-objects][formatters]] either by passing a\nstring as the =formatter= option to the =add_logging_handlers= or by\ncreating a =formatter= object and using the handler's [[https://docs.python.org/3/library/logging.html#logging.Handler.setFormatter][setFormatter]]\nmethod directly. In either case the recommended way is to use ={=\n[[https://docs.python.org/3/library/logging.html#logging.Formatter][style]]. Specially if you want to access inner values with dot notation\ne.g. =watch.Query= or =watch.Collection= other styles such as =%= and\n=$= (e.g. =%(watch.Query)s= or =${watch.Query}=) will *NOT* work.\n\nAnother useful feature of Python =logging= module is its [[https://docs.python.org/3/library/logging.html#filter-objects][filters]]. You\ncan use =filter= objects not only to filter unwanted logs but also to\nmodify the ones that you want. =add_logging_handlers= will\nautomatically add some basic filters (such as =AddPymongoResults=\nwhich will extract information about =pymongo= operations results and\nadd them to the related log), but you can also add your own filters.\n\n=pymongo.watcher.filters= module implements a couple of useful\nconfigurable filters such as =ExpressionFilter=, =ExecuteFilter= and\n=RateFilter=. The [[./etc/pymongowatch.yaml][etc/pymongowatch.yaml]] has some examples on how to\nuse them.\n\nYou can also develop your own logging filters by sub-classing\n[[https://docs.python.org/3/library/logging.html#filter-objects][logging.Filter]] as usual and access the =record.watch= in [[https://docs.python.org/3/library/logging.html#logging.Filter.filter][filter]] method\nfor investigating or modifying the watcher attributes such as =DB=,\n=Collection=, =Query=, etc.\n\n#+begin_src python\nimport logging\n\nclass SlowQueriesOnNewsCollectionFilter(logging.Filter):\n    def filter(self, record):\n        watch = getattr(record, \"watch\", {})\n        return (watch.get(\"Collection\") == \"news\" and\n                watch.get(\"RetrieveTime\", 0) \u003e 10)\n#+end_src\n\nOr you can add filters to modify the logs:\n\n#+begin_src python\nimport logging\n\ndef remove_private_data(data):\n    if isinstance(data, dict):\n        return {k: remove_private_data(v) for k, v in data.items()}\n    elif isinstance(data, list):\n        return [remove_private_data(i) for i in data]\n    return None\n\nclass UserPrivacyFilter(logging.Filter):\n    def filter(self, record):\n        watch = getattr(record, \"watch\", {})\n        watch.update(remove_private_data(watch))\n        return True\n#+end_src\n\nDon't forget to add the defined filters to you handlers:\n\n#+begin_src python\nmy_handler.addFilter(SlowQueriesOnNewsCollectionFilter)\nmy_handler.addFilter(UserPrivacyFilter)\n#+end_src\n\nLastly, you can use Python's great [[https://docs.python.org/3/library/logging.config.html#module-logging.config][logging.conig]] module and specially\nthe new flexible [[https://docs.python.org/3/library/logging.config.html#logging.config.dictConfig][logging.config.dictConfig]] method to apply all the\n=handlers=, =formatters= and =filters= in a single configuration\nfile. Custom =pymongowatch= configuration can also be applied in the\nsame dictionary.\n\n=pymongowatch= has even the required filters implemented in\n=pymongo.watcher.filters= module. To see the examples for the\n=dictConfig= configuration with watcher filters refer to the\n=etc/pymongowatch.yaml= file which will be installed via\n=pymongowatch= (if you are using a virtual environment, it would be\ninside the venv directory).\n\n** dictConfig configuration\n\nThere is a =pymongo.watcher.dictConfig= method that accepts a\nconfiguration dictionary with almost [[https://docs.python.org/3/library/logging.config.html#configuration-dictionary-schema][the same specification]] as\n=logging.config.dictConfig=. But it will apply custom configuration\nrequired by =pymongowatch= itself.\n\nThe =pymongo.watcher.dictConfig= will not apply the intended logging\nconfigurations such as =handlers= and =formatters=. Instead, it reads\nthe special key =watchers= from the dictionary and apply the related\nconfiguration for =pymongowatch=. Although you can use the a single\ndictionary with both logging configuration and =watchers= key and\napply both =pymongo.watcher.dictConfig= and\n=logging.config.dictConfig=.\n\nFor more information about the =pymongowatch= configurations please\nrefer to the =watchers= key inside the example =etc/pymongowatch.yaml=\nfile that will be installed alongside the package.\n\n** Mutable vs Immutable Logs\n\nMutable Logs? Is that a thing?\n\nUsually the good thing about logs is that they are immutable. So if\nyou see a log you can trust it. This is always true when some atomic\noperation happens and you have no concerns about the start and end\ntime of the operation (and you don't have access to a time machine to\ntravel to the past and change what happened).\n\nBut what if you start an operation which we have no idea when will it\nend? Suppose we have queried a very large database for a very slow\nquery that may take some time to get back the full results. Also, we\nmay use a cursor in our application to fetch data and the application\nhas some delays itself and we don't want the slowness of the\napplication to affect the database auditing.\n\nThese are the sort of challenges that =pymongo.watcher.WatchQueue=\ntries to fix.\n\n=pymongowatch= uses =pymongo.watcher.logger.WatchMessage= instead of\nstrings as log messages as described in [[https://docs.python.org/3/howto/logging.html#using-arbitrary-objects-as-messages][using arbitrary objects as\nmessages]] in Python's logging HOWTO. =WatchMessage= is a sub-class of\nPython's dictionaries which are mutable objects.\n\n=WatchMessage= instances are the ={watch}= templates in the format\nstrings that we saw earlier. They are a =dict= so you can access log\nattributes with =[]= access e.g. =watch[\"Query\"]=. For more\nconvenience while using log formatters =WatchMessage= provides\nattribute access with dot notation e.g. =watch.Query= and one of the\nreasons why ={= style formatting (which let you use dot notation\naccess) is recommended for logging formatters.\n\nA =WatchMessage= is a mutable object and the watcher classes need to\nmodify the logs from time to time. For example if you fetch more items\nfrom a cursor, they can update the attributes such as =RetrieveTime=\nor =RetrievedCount=. So we have flexibility and it is =pymongowatch=\nusers decision when to emit the final immutable log with the logger\nhandlers.\n\nIn the old versions of =pymongowatch=, the watcher classes modified\nthe =WatchMessage= even after the emitting the log, leaving the entire\nburden for consistency on the =WatchQueue=. This approach soon became\nproblematic in multiprocessing environments where the =WatchMessage=\nin the emitter process differs from the one in the queue process.\n\nIn the newer versions, =WatchMessage= has a unique identifier\n=WatchID= for the ongoing operation. The watcher classes will not only\nmutate the previous =WatchMessage= but also re-emit the log again and\nagain on each update (with an incremental meta-data named\n=Iteration=). In this way, even if the log has already stored\nsomewhere as an immutable object (for example in a file as a log\nrecord or in a separate process as an object that cannot see the\nmodifications to the original =WatchMessage=) can still receive the\nupdates.\n\nThe pitfall is now we may have thousands of updates and consequently\nthousands of logs for each operation. But this problem can easily be\nhandled by =WatchQueue= or an external tool such as\n=pymongowatch-csv=.\n\n=WatchQueue= alongside a [[https://docs.python.org/3/library/logging.handlers.html#queuehandler][QueueHandler]] is the right tool to make sure\nwe handle logs at right time. A =WatchQueue= works like a priority\nqueue in which the earliest logs has the higher priority but it can\nalso aggregate the updates for each operation based on the =WatchID=.\n\n=WatchQueue= will update the old iterations with the newer ones and\nreleases the logs when they have the =final= iteration mark or when\nthey have expired. The =final= mark is a special case of =Iteration=\nmeta-data in each log that indicates the database operation has\nfinished and the watcher class will not modify this =WatchMessage=\n(with the specified =WatchID=) anymore.\n\nAlso it is important to note that the highest priority item (for\nretrieval) in the =WatchQueue= is not always the earliest generated\nlog. First, the different operations may take different times, and\nsecond, the watcher classes may assign different expiration times to\neach operation.\n\nFinally you can set a =forced_delay_sec= to add delay to all the logs\nin the queue (for example if you are using logging for analytics and\nyou do care more about accuracy than delay for the logs). This is an\noptional keyword arguments that both =WatchQueue= constructor and\n=add_logging_handlers= method that we saw earlier accepts.\n\nThe =add_logging_handlers= has a =with_queue= optional argument which\nif is =True= (the default), will use\n=pymongo.watcher.setup_queue_handler= to setup a [[https://docs.python.org/3/library/logging.handlers.html#queuehandler][QueueHandler]]\nalongside a started [[https://docs.python.org/3/library/logging.handlers.html#queuelistener][QueueListener]] for each handler you specify with a\n=WatchQueue= so you usually don't have to worry about log mutation if\nyou use =add_logging_handlers= to add your handlers to watcher\nloggers.\n\nA =pymongowatch-csv= script is also available after the =pymongowatch=\ninstallation that can be used to aggregate all the iterations of a\nsingle operation into a single row when you use the =csv= format:\n\n#+begin_src shell\npymongowatch-csv aggr /tmp/watch.csv \u003e /tmp/aggregated-watch.csv\n#+end_src\n\n** Multiprocessing\n\nAnother useful application of [[https://docs.python.org/3/library/logging.handlers.html#queuehandler][QueueHandler]] is its use in [[https://docs.python.org/3/howto/logging-cookbook.html#logging-to-a-single-file-from-multiple-processes][logging to a\nsingle file from multiple processes]]. For this application you have to\nuse a multiprocessing =Queue= alongside the\n=QueueHandler=. Fortunately =WatchQueue= also supports\nmultiprocessing. All you have to do is to pass =True= as the\n=enable_multiprocessing= argument.\n\nThis is useful for example when you have a =pymongo= based web\napplication with several web server processes and you need a process\nsafe method to store the logs in a single file. Its worth noting that\nin such environments using the default value for\n=enable_multiprocessing= i.e. =False= will result in total failure in\nlogging because the default =Queue= is not multiprocessing and the\n=QueueHandler= and the =QueueListener= will use different queues in\ndifferent processes.\n\nPassing =True= as =enable_multiprocessing= in =WatchQueue= constructor\nwill makes the constructor to build and return a [[https://docs.python.org/3/library/multiprocessing.html#proxy-objects][proxy object]] from a\n[[https://docs.python.org/3/library/multiprocessing.html#customized-managers][customized manager]] which has its own dedicated process. You can also\npass =enable_multiprocessing= to =setup_queue_handler= and\n=add_logging_handlers=:\n\n#+begin_src python\n# Extra keyword arguments of `add_logging_handlers` will be passed to\n# the newly created WatchQueue for each handler.\nwatcher.add_logging_handlers(enable_multiprocessing=True)\n#+end_src\n\n** Pymongo Versions\n\n=pymongowatch= is not a standalone MongoDB library and it relies on\nthe the MongoDB's Python driver [[https://pymongo.readthedocs.io/en/stable/][pymongo]]. But does the pymongo's\nversion matter?\n\n=pymongowatch= has been tested with the recent versions of =pymongo=\ni.e. =3.10= and the newer =4= series but you can use it for other\nversions at your own risk. If you have any problems you can open an\nissue at the [[https://github.com/admirito/pymongowatch/issues][project's issue tracker]].\n\nOne known difference between =pymongo= versions is that they handle\noperation closing differently. For example, =4= series close the\ncursors more intelligently and you can usually see the =cursor= logs\nvery fast without any need to a explicit timeout whereas any =3=\nseries usually an explicit timeout is required.\n\n** Other Installation Options\n\n*** Debian Packages\n\nIf you are a [[https://www.debian.org/][Debian]]-based GNU/Linux distribution user you are in luck!\nThere is a Debian package maintained in the [[https://github.com/admirito/pymongowatch/tree/debian][project's debian branch]]\nthat can make your installation even easier.\n\nYou can find the binary packages at [[https://launchpad.net/~mrazavi/+archive/ubuntu/pymongowatch][mrazavi's pymongowatch PPA]] and to\ninstall it on Ubuntu:\n\n#+begin_src shell\nsudo add-apt-repository ppa:mrazavi/pymongowatch\nsudo apt update\n\nsudo apt install python3-pymongowatch\n#+end_src\n\n** Roadmap\n\nThe project [[https://github.com/admirito/pymongowatch/issues][issue tracker]] is the main location to maintain the details\nfor the future development. But here the cardinal points will be\nreviewed briefly.\n\n- NOTE :: If you see an ugly TODO list below with oversize items, it's\n  not even clear which items are DONE and which ones are still TODO,\n  maybe that is because this document is written in [[https://orgmode.org/][org-mode]] but you\n  are seeing a bad render e.g. in [[https://github.com/github/markup/blob/master/README.md#markups][GitHub]] or a converted\n  reStructuredText format e.g. because [[https://packaging.python.org/en/latest/guides/making-a-pypi-friendly-readme/][the lack of org-mode support in\n  PyPI]].\n  \n  That doesn't make org-mode less lovable or inferior. Anyway [[https://karl-voit.at/2017/09/23/orgmode-as-markup-only/][org-mode\n  is one of the most reasonable markup languages to use for text]]. Why\n  not to use it and brag about it?\n\n#+TODO: TODO(t) DOING(i) | DONE(d)\n\n*** DONE Support queries with find\n*** DONE Support for basic collection operations [[https://github.com/admirito/pymongowatch/issues/3][#3]]\n*** TODO Support for CommandCursor [[https://github.com/admirito/pymongowatch/issues/4][#4]]\n*** DONE Packaging\n**** DONE Implement pip package\n**** DONE Implement debian package\n**** TODO Automatic execution of tests\n*** DOING Tests\n**** DOING Unit Tests\n***** DONE Old Implementation\n***** TODO Update with the latest code\n*** DOING Deployment Tests\n***** DONE Basic Implementation\n***** TODO Add more complex tests\n*** DONE Support for multiprocessing\n*** DONE Improve mutable logs [[https://github.com/admirito/pymongowatch/issues/2][#2]]: add a unique ID for each operation\n*** TODO Documentation\n**** DONE Add a README\n**** TODO Add Sphinx generated documents\n**** TODO Create an online API reference\n\n** About\n\nThe =pymongowatch= has developed mainly by [[https://github.com/admirito/][Mohammad Razavi]].\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fadmirito%2Fpymongowatch","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fadmirito%2Fpymongowatch","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fadmirito%2Fpymongowatch/lists"}