{"id":24088003,"url":"https://github.com/tinybirdco/demo-audit-log","last_synced_at":"2025-05-05T21:41:11.041Z","repository":{"id":38236344,"uuid":"475412920","full_name":"tinybirdco/demo-audit-log","owner":"tinybirdco","description":null,"archived":false,"fork":false,"pushed_at":"2022-06-08T16:50:49.000Z","size":291,"stargazers_count":5,"open_issues_count":0,"forks_count":1,"subscribers_count":2,"default_branch":"main","last_synced_at":"2025-03-30T23:51:13.507Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/tinybirdco.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2022-03-29T11:33:30.000Z","updated_at":"2025-01-15T11:03:23.000Z","dependencies_parsed_at":"2022-09-04T22:11:27.501Z","dependency_job_id":null,"html_url":"https://github.com/tinybirdco/demo-audit-log","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/tinybirdco%2Fdemo-audit-log","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/tinybirdco%2Fdemo-audit-log/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/tinybirdco%2Fdemo-audit-log/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/tinybirdco%2Fdemo-audit-log/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/tinybirdco","download_url":"https://codeload.github.com/tinybirdco/demo-audit-log/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":252581821,"owners_count":21771573,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2025-01-10T03:56:52.942Z","updated_at":"2025-05-05T21:41:10.999Z","avatar_url":"https://github.com/tinybirdco.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Using Tinybird with an Audit Log\n\nThis repository contains the data project —[datasources](./datasources), and [endpoints](./endpoints)— and [data-generator](./data-generator) scripts for an audit log example of using Tinybird.\n\nCheck out this 5 minute [video](https://youtu.be/NlGtrTGH9dQ) to see how to work with audit logs and Tinybird or this 60 minute [video](https://www.youtube.com/watch?v=C9859uVd6uc) of a live coding session walking you through the whole process.\n\nTo clone the repository:\n\n`git clone git@github.com:tinybirdco/demo-audit-log.git`\n\n`cd demo-audit-log`\n\n## Working with the Tinybird CLI\n\nTo start working with data projects as if they were software projects, let's install the Tinybird CLI in a virtual environment.\nCheck the [CLI documentation](https://docs.tinybird.co/cli.html) for other installation options and troubleshooting.\n\n```bash\nvirtualenv -p python3 .e\n. .e/bin/activate\npip install tinybird-cli\ntb auth --interactive\n```\n\nChoose your region: __1__ for _us-east_, __2__ for _eu_\n\nGo to your workspace, copy a token with admin rights and paste it. A new `.tinyb` file will be created.  \n\n## Project description\n\n```bash\n├── datasources\n│   ├── audit_log_hfi.datasource\n│   ├── companies.datasource\n│   └── fixtures\n│       └── companies.csv\n├── endpoints\n│   ├── api_audit_log_enriched.pipe\n│   ├── api_audit_log_params.pipe\n│   └── api_count_per_type.pipe\n```\n\nIn the `/datasources` folder we have two Data Sources:\n- audit_log_hfi: where we'll be sending audit log events with a python script\n- companies: where we will store the information about the companies. The content (a CSV file) is in the fixtures subfolder.\n\nAnd three .pipe files in the `/endpoints` folder:\n- api_audit_log_enriched: a pipe that counts the number of events per company from _audit_log_hfi_ and joins the result with the info in _companies_\n- api_audit_log_params: en example with several [dynamic parameters](https://guides.tinybird.co/guide/using-dynamic-parameters-for-changing-aggregation-types-on-the-fly)\n- api_count_per_type: an example of pivoting rows to columns using arrays.\n\nNote:\nTypically, in big projects, we split the .pipe files across two folders: /pipes and /endpoints\n- `/pipes` where we store the pipes ending in a datasource, that is, [materialized views](https://guides.tinybird.co/guide/materialized-views)\n- `/endpoints` for the pipes that end in API endpoints. \n\n## Pushing the data project to your Tinybird workspace\n\nPush the data project —datasources, pipes and fixtures— to your workspace.\n\n```bash\ntb push --fixtures\n```\n  \nYour data project is ready for realtime analysis. You can check the UI's Data flow to see how it looks.\n\n![Data flow](data_flow.jpg?raw=true \"Data flow in UI\")\n\n## Ingesting data using high-frequency ingestion (HFI)\n\nLet's add some data through the [HFI endpoint](https://www.tinybird.co/guide/high-frequency-ingestion).\n\nTo do that we have created a python script to generate and send dummy events.\n\n```bash\npip install click faker\npython3 data-generator/audit_log_events.py --repeat 100\n```\n\nFeel free to play with the parameters. You can check them with `python3 data-generator/audit_log_events.py --help`\n\n## Token security\n\nYou now have your Data Sources and pipes that end in API endpoints. \n\nThe endpoints need a [token](https://www.tinybird.co/guide/serverless-analytics-api) to be consumed. You should not expose your admin token, so let's create one with more limited scope.\n\n```bash\npip install jq\n\nTOKEN=$(cat .tinyb | jq '.token'| tr -d '\"')\nHOST=$(cat .tinyb | jq '.host'| tr -d '\"')\n\ncurl -H \"Authorization: Bearer $TOKEN\" \\\n-d \"name=endpoints_token\" \\\n-d \"scope=PIPES:READ:api_audit_log_enriched\" \\\n-d \"scope=PIPES:READ:api_audit_log_params\" \\\n-d \"scope=PIPES:READ:api_count_per_type\" \\\n$HOST/v0/tokens/\n```\n\nYou will receive a response similar to this:\n\n```json\n{\n    \"token\": \"\u003cthe_newly_ceated_token\u003e\",\n    \"scopes\": [\n        {\n            \"type\": \"PIPES:READ\",\n            \"resource\": \"api_audit_log_enriched\",\n            \"filter\": \"\"\n        },\n        {\n            \"type\": \"PIPES:READ\",\n            \"resource\": \"api_audit_log_params\",\n            \"filter\": \"\"\n        },\n        {\n            \"type\": \"PIPES:READ\",\n            \"resource\": \"api_count_per_type\",\n            \"filter\": \"\"\n        }\n    ],\n    \"name\": \"endpoints_token\"\n}\n```\n\nIf you want to create a token to share just `api_audit_log_params` with, let's say, the company with company_id 1, you can do so with the **row level security**:\n\n```bash\ncurl -H \"Authorization: Bearer $TOKEN\" \\\n-d \"name=comp_1_token\" \\\n-d \"scope=PIPES:READ:api_audit_log_params\" \\\n-d \"scope=DATASOURCES:READ:audit_log_hfi:company_id=1\" \\\n$HOST/v0/tokens/\n```\n\nThis project shows just some of the features of Tinybird. If you have any questions, come along and join our community [Slack](https://join.slack.com/t/tinybird-community/shared_invite/zt-yi4hb0ht-IXn9iVuewXIs3QXVqKS~NQ)!\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ftinybirdco%2Fdemo-audit-log","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Ftinybirdco%2Fdemo-audit-log","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ftinybirdco%2Fdemo-audit-log/lists"}