{"id":24088000,"url":"https://github.com/tinybirdco/demo_logs","last_synced_at":"2026-06-11T08:31:44.186Z","repository":{"id":38237628,"uuid":"475799823","full_name":"tinybirdco/demo_logs","owner":"tinybirdco","description":null,"archived":false,"fork":false,"pushed_at":"2022-06-08T15:40:47.000Z","size":32,"stargazers_count":0,"open_issues_count":0,"forks_count":1,"subscribers_count":2,"default_branch":"main","last_synced_at":"2025-02-27T05:25:37.068Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/tinybirdco.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2022-03-30T09:02:41.000Z","updated_at":"2022-03-30T15:23:36.000Z","dependencies_parsed_at":"2022-09-04T21:54:14.233Z","dependency_job_id":null,"html_url":"https://github.com/tinybirdco/demo_logs","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/tinybirdco/demo_logs","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/tinybirdco%2Fdemo_logs","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/tinybirdco%2Fdemo_logs/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/tinybirdco%2Fdemo_logs/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/tinybirdco%2Fdemo_logs/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/tinybirdco","download_url":"https://codeload.github.com/tinybirdco/demo_logs/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/tinybirdco%2Fdemo_logs/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":34190583,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-05-26T15:22:16.424Z","status":"online","status_checked_at":"2026-06-11T02:00:06.485Z","response_time":57,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2025-01-10T03:56:52.934Z","updated_at":"2026-06-11T08:31:44.164Z","avatar_url":"https://github.com/tinybirdco.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Using Tinybird with a Demo Log\n\nThis repository contains the data project —[datasources](./datasources), and [endpoints](./endpoints)— and [data-generator](./data-generator) scripts for a log example of using Tinybird.\n\nTo clone the repository:\n\n`git clone git@github.com:tinybirdco/demo_logs.git`\n\n`cd demo_logs`\n\n## Working with the Tinybird CLI\n\nTo start working with data projects as if they were software projects, let's install the Tinybird CLI in a virtual environment.\nCheck the [CLI documentation](https://docs.tinybird.co/cli.html) for other installation options and troubleshooting.\n\n```bash\nvirtualenv -p python3 .e\n. .e/bin/activate\npip install tinybird-cli\ntb auth --interactive\n```\n\nChoose your region: __1__ for _us-east_, __2__ for _eu_\n\nGo to your workspace, copy a token with admin rights and paste it. A new `.tinyb` file will be created.  \n\n## Project description\n\n```bash\n├── datasources\n│   ├── build_log.datasource\n│   ├── lambda_log.datasource\n│   └── rewrite_log.datasource\n├── endpoints\n│   ├── count_log.pipe\n│   ├── get_filter_values.pipe\n│   └── query_logs.pipe  \n```\n\nIn the `/datasources` folder we have three Data Sources:\n\n- build_log: where we'll be sending build log events.\n- lambda_log: where we'll be sending lambda log events.\n- rewrite_log: where we'll be sending rewrite log events.\n\nAnd three .pipe files in the `/endpoints` folder:\n\n- query_logs: Retrieve timestamp and message data every events filtered by channel, project_id, deployment_id and log_level.\n- count_log: Retrieve the number of rows in all the logs.\n- get_filter_values_channel: Retrieve distinct values for channel, project, deploymert and log level.\n\nNote:\nTypically, in big projects, we split the .pipe files across two folders: /pipes and /endpoints\n\n- `/pipes` where we store the pipes ending in a datasource, that is, [materialized views](https://guides.tinybird.co/guide/materialized-views)\n- `/endpoints` for the pipes that end in API endpoints.\n\n## Authentication\n\nYou'll need cli-authentication before executing the next steps [cli-authenticate](https://docs.tinybird.co/cli/workspaces.html#authenticate), given that the token will be readed from the file .tinyb.\n\n```bash\ntb auth\nCopy the admin token from https://ui.tinybird.co/tokens and paste it here: \u003cpasted token\u003e\n** Auth successful!\n** Configuration written to .tinyb file, consider adding it to .gitignore\n```\n\nHere you must paste your Tinybird token.\n\n## Pushing the data project to your Tinybird workspace\n\nPush the data project —datasources, pipes and fixtures— to your workspace.\n\n```bash\ntb push\n```\n\nYour data project is ready for realtime analysis. You can check the UI's Data flow to see how it looks.\n\n![Data_flow](https://user-images.githubusercontent.com/51535157/161029357-bed71b97-0900-469c-8653-17169b7b57e3.png)\n\n## Ingesting data using high-frequency ingestion (HFI)\n\nLet's add some data through the [HFI endpoint](https://www.tinybird.co/guide/high-frequency-ingestion).\n\nTo do that we have created a python script to generate and send dummy events.\n\n```bash\npython3 data_generator/demo_log_events.py --datasource build_log --sample 100000 --events 100 --silent\n```\n\nFeel free to play with the parameters. You can check them with `python3 data_generator/demo_log_events.py --help`\n\n## Advanced Token security\n\nYou now have your Data Sources and pipes that end in API endpoints.\n\nThe endpoints need a [token](https://www.tinybird.co/guide/serverless-analytics-api) to be consumed. You should not expose your admin token, so let's create one with more limited scope.\n\n```bash\npip install jq\n\nTOKEN=$(cat .tinyb | jq '.token'| tr -d '\"')\nHOST=$(cat .tinyb | jq '.host'| tr -d '\"')\n\ncurl -H \"Authorization: Bearer $TOKEN\" \\\n-d \"name=endpoints_token\" \\\n-d \"scope=PIPES:READ:query_logs\" \\\n-d \"scope=PIPES:READ:count_log\" \\\n-d \"scope=PIPES:READ:get_filter_values\" \\\n$HOST/v0/tokens/\n```\n\nYou will receive a response similar to this:\n\n```json\n{\n    \"token\": \"\u003cthe_newly_ceated_token\u003e\",\n    \"scopes\": [\n        {\n            \"type\": \"PIPES:READ\",\n            \"resource\": \"query_logs\",\n            \"filter\": \"\"\n        },\n        {\n            \"type\": \"PIPES:READ\",\n            \"resource\": \"count_log\",\n            \"filter\": \"\"\n        },\n        {\n            \"type\": \"PIPES:READ\",\n            \"resource\": \"get_filter_values\",\n            \"filter\": \"\"\n        }\n    ],\n    \"name\": \"endpoints_token\"\n}\n```\n\nIf you want to create a token to share just `query_logs` with, let's say, the channel with value build, you can do so with the __row level security__:\n\n```bash\ncurl -H \"Authorization: Bearer $TOKEN\" \\\n-d \"name=build_logs_token\" \\\n-d \"scope=PIPES:READ:query_logs\" \\\n-d \"scope=DATASOURCES:READ:query_logs:channel=build\" \\\n$HOST/v0/tokens/\n```\n\nThis project shows just some of the features of Tinybird. If you have any questions, come along and join our community [Slack](https://join.slack.com/t/tinybird-community/shared_invite/zt-yi4hb0ht-IXn9iVuewXIs3QXVqKS~NQ)!\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ftinybirdco%2Fdemo_logs","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Ftinybirdco%2Fdemo_logs","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ftinybirdco%2Fdemo_logs/lists"}