{"id":15222436,"url":"https://github.com/manesioz/rilly","last_synced_at":"2026-03-27T04:46:43.954Z","repository":{"id":53548055,"uuid":"227624212","full_name":"manesioz/rilly","owner":"manesioz","description":"Distributed change data capture (CDC) framework for Google BigQuery","archived":false,"fork":false,"pushed_at":"2020-01-15T03:32:03.000Z","size":29,"stargazers_count":1,"open_issues_count":0,"forks_count":0,"subscribers_count":0,"default_branch":"master","last_synced_at":"2025-11-27T18:44:36.981Z","etag":null,"topics":["change-data-capture","distributed-systems","google-bigquery","kafka","pubsub","python3"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/manesioz.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2019-12-12T14:21:32.000Z","updated_at":"2024-03-12T12:45:27.000Z","dependencies_parsed_at":"2022-08-28T19:11:04.042Z","dependency_job_id":null,"html_url":"https://github.com/manesioz/rilly","commit_stats":null,"previous_names":[],"tags_count":2,"template":false,"template_full_name":null,"purl":"pkg:github/manesioz/rilly","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/manesioz%2Frilly","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/manesioz%2Frilly/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/manesioz%2Frilly/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/manesioz%2Frilly/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/manesioz","download_url":"https://codeload.github.com/manesioz/rilly/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/manesioz%2Frilly/sbom","scorecard":{"id":614898,"data":{"date":"2025-08-11","repo":{"name":"github.com/manesioz/rilly","commit":"db50f4a1ee12441a99fb0db4e645495b66d945b2"},"scorecard":{"version":"v5.2.1-40-gf6ed084d","commit":"f6ed084d17c9236477efd66e5b258b9d4cc7b389"},"score":3,"checks":[{"name":"Pinned-Dependencies","score":-1,"reason":"no dependencies found","details":null,"documentation":{"short":"Determines if the project has declared and pinned the dependencies of its build process.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#pinned-dependencies"}},{"name":"Packaging","score":-1,"reason":"packaging workflow not detected","details":["Warn: no GitHub/GitLab publishing workflow detected."],"documentation":{"short":"Determines if the project is published as a package that others can easily download, install, easily update, and uninstall.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#packaging"}},{"name":"Dangerous-Workflow","score":-1,"reason":"no workflows found","details":null,"documentation":{"short":"Determines if the project's GitHub Action workflows avoid dangerous patterns.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#dangerous-workflow"}},{"name":"SAST","score":0,"reason":"no SAST tool detected","details":["Warn: no pull requests merged into dev branch"],"documentation":{"short":"Determines if the project uses static code analysis.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#sast"}},{"name":"Binary-Artifacts","score":10,"reason":"no binaries found in the repo","details":null,"documentation":{"short":"Determines if the project has generated executable (binary) artifacts in the source repository.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#binary-artifacts"}},{"name":"Code-Review","score":0,"reason":"Found 0/18 approved changesets -- score normalized to 0","details":null,"documentation":{"short":"Determines if the project requires human code review before pull requests (aka merge requests) are merged.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#code-review"}},{"name":"Maintained","score":0,"reason":"0 commit(s) and 0 issue activity found in the last 90 days -- score normalized to 0","details":null,"documentation":{"short":"Determines if the project is \"actively maintained\".","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#maintained"}},{"name":"Token-Permissions","score":-1,"reason":"No tokens found","details":null,"documentation":{"short":"Determines if the project's workflows follow the principle of least privilege.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#token-permissions"}},{"name":"CII-Best-Practices","score":0,"reason":"no effort to earn an OpenSSF best practices badge detected","details":null,"documentation":{"short":"Determines if the project has an OpenSSF (formerly CII) Best Practices Badge.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#cii-best-practices"}},{"name":"Security-Policy","score":0,"reason":"security policy file not detected","details":["Warn: no security policy file detected","Warn: no security file to analyze","Warn: no security file to analyze","Warn: no security file to analyze"],"documentation":{"short":"Determines if the project has published a security policy.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#security-policy"}},{"name":"Vulnerabilities","score":10,"reason":"0 existing vulnerabilities detected","details":null,"documentation":{"short":"Determines if the project has open, known unfixed vulnerabilities.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#vulnerabilities"}},{"name":"Fuzzing","score":0,"reason":"project is not fuzzed","details":["Warn: no fuzzer integrations found"],"documentation":{"short":"Determines if the project uses fuzzing.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#fuzzing"}},{"name":"License","score":10,"reason":"license file detected","details":["Info: project has a license file: LICENSE:0","Info: FSF or OSI recognized license: MIT License: LICENSE:0"],"documentation":{"short":"Determines if the project has defined a license.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#license"}},{"name":"Signed-Releases","score":-1,"reason":"no releases found","details":null,"documentation":{"short":"Determines if the project cryptographically signs release artifacts.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#signed-releases"}},{"name":"Branch-Protection","score":0,"reason":"branch protection not enabled on development/release branches","details":["Warn: branch protection not enabled for branch 'master'"],"documentation":{"short":"Determines if the default and release branches are protected with GitHub's branch protection settings.","url":"https://github.com/ossf/scorecard/blob/f6ed084d17c9236477efd66e5b258b9d4cc7b389/docs/checks.md#branch-protection"}}]},"last_synced_at":"2025-08-21T03:38:38.282Z","repository_id":53548055,"created_at":"2025-08-21T03:38:38.282Z","updated_at":"2025-08-21T03:38:38.282Z"},"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":28069389,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-12-27T02:00:05.897Z","response_time":58,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["change-data-capture","distributed-systems","google-bigquery","kafka","pubsub","python3"],"created_at":"2024-09-28T15:12:00.211Z","updated_at":"2025-12-27T02:20:00.754Z","avatar_url":"https://github.com/manesioz.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"\u003cp align=\"center\"\u003e\n  Distributed change data capture (CDC) platform for Google BigQuery\n  \u003cbr\u003e\u003cbr\u003e\n  \u003cimg src=\"https://github.com/manesioz/rilly/blob/master/assets/rilly.png\"\u003e\n\u003c/p\u003e\n\n### What is Change Data Capture? \nChange data capture (CDC) is a set of software design patterns used to determine (and track) the data that has changed so that action can be taken using the changed data. Instead of continuously polling a database for changes (which is costly if you do it often and inaccurate if you don't), `rilly` uses the log-based approach (as does [`debezium`](https://debezium.io) and all other major CDC frameworks). \n\n\n### Why `rilly`?\nThere is currently no CDC plug-in for BigQuery that I am aware of, and certainly none for Python. The goal of this package is to be as simple and non-opinionated as possible to allow developers to have full control over how they want to stream and parse their change events. \n\n\n### Installation \n\n```python\npip install rilly\n```\n\n### Authentication \nThis library uses Google's PubSub and Stackdriver APIs, so follow the authentication process [here](https://cloud.google.com/pubsub/docs/reference/libraries#setting_up_authentication). \n\n\n### Usage \nSay you want to track all update/delete/insert events in your BigQuery dataset. After authenticating the Google Python Client APIs: \n\n```python\nfrom rilly import logging, stream\n\n#create a PubSub topic to send your change events to \nstream.create_pubsub_topic('my-project-id', 'pubsub-topic') \n\n#create sink to send logs to PubSub topic\nlogging.create_sink('sink-id', 'my-project-id', 'my-dataset-id', pubsub_topic='pubsub-topic') \n\n#custom callback function to perform some action on each event\ndef custom_callback(message: str) -\u003e str: \n    print('Received message data: {}'.format(message))\n    return message \n    \n#create subscription to PubSub topic, apply custom_callback() to each streamed log\nstream.subscribe('my-project-id', 'pubsub-topic', 'cdc-subscription', 30, custom_callback) \n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmanesioz%2Frilly","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fmanesioz%2Frilly","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmanesioz%2Frilly/lists"}