{"id":27914996,"url":"https://github.com/onflow/flow-data-sources","last_synced_at":"2025-05-06T15:33:21.195Z","repository":{"id":273007606,"uuid":"918418191","full_name":"onflow/Flow-Data-Sources","owner":"onflow","description":null,"archived":false,"fork":false,"pushed_at":"2025-05-03T00:24:34.000Z","size":54283,"stargazers_count":3,"open_issues_count":0,"forks_count":1,"subscribers_count":4,"default_branch":"main","last_synced_at":"2025-05-03T01:38:35.995Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/onflow.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2025-01-17T22:24:02.000Z","updated_at":"2025-05-03T00:24:41.000Z","dependencies_parsed_at":"2025-04-25T01:37:17.299Z","dependency_job_id":null,"html_url":"https://github.com/onflow/Flow-Data-Sources","commit_stats":null,"previous_names":["onflow/flow-data-sources"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/onflow%2FFlow-Data-Sources","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/onflow%2FFlow-Data-Sources/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/onflow%2FFlow-Data-Sources/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/onflow%2FFlow-Data-Sources/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/onflow","download_url":"https://codeload.github.com/onflow/Flow-Data-Sources/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":252713045,"owners_count":21792421,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2025-05-06T15:33:20.586Z","updated_at":"2025-05-06T15:33:21.184Z","avatar_url":"https://github.com/onflow.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Flow Data Sources\n\nThis repository contains a Python script that updates daily a list of Flow-related sites, GitHub repositories, and GitHub discussions and converts them into **Markdown** files. The resulting `.md` files are intended for **AI ingestion**, **Retrieval-Augmented Generation (RAG)** pipelines, chatbots, or any other knowledge base platform that benefits from structured text.\n\n---\n\n## Table of Contents\n\n- [Purpose](#purpose)\n- [How It Works](#how-it-works)\n  - [1. Normal Docs Sites (HTML → MD)](#1-normal-docs-sites-html--md)\n  - [2. GitHub Repos (Raw Code)](#2-github-repos-raw-code)\n  - [3. GitHub Discussions (Q\u0026A Text Only)](#3-github-discussions-qa-text-only)\n- [Usage](#usage)\n  - [Requirements](#requirements)\n  - [Running the Scraper](#running-the-scraper)\n- [Modifying the List of Sites](#modifying-the-list-of-sites)\n- [Output Structure](#output-structure)\n- [Scheduling \u0026 Automation](#scheduling-automation)\n\n---\n\n## Purpose\n\nWe want a single repository that **periodically crawls** all relevant Flow ecosystem content—**documentation**, **code examples**, and **community discussions**—and stores them in a consolidated **Markdown** format. You can then feed these files into:\n\n- **ChatGPT plugins** (for enhanced Q\u0026A)\n- **Retrieval-Augmented Generation** (indexing and searching them in a vector database)\n- **Discord/Telegram bots** that cite official doc sections\n- **Any** other knowledge base for advanced Q\u0026A or search.\n\n---\n\n## How It Works\n\nThe Python script performs **domain-limited BFS** (Breadth-First Search) and specialized scraping logic based on each URL:\n\n### 1. Normal Docs Sites (HTML → MD)\n\n- **Non-GitHub** URLs are treated as “normal” websites.  \n- The script fetches each page and removes `\u003cscript\u003e`, `\u003cstyle\u003e`, `\u003cnoscript\u003e` tags.\n- Then it uses [`markdownify`](https://pypi.org/project/markdownify/) to convert the **remaining HTML** into **Markdown**.  \n- It recurses only within the **same domain** to avoid crawling unrelated pages.\n\n### 2. GitHub Repos (Raw Code)\n\n- For **GitHub repo** links like `https://github.com/onflow/flow-ft/`, the script visits:\n  - The **repo root**\n  - `tree/(main|master)/...` subdirectories\n  - `blob/(main|master)/...` file pages\n- **Files** with certain extensions (like `.cdc`, `.md`, `.json`, etc.) or **any** `README` are downloaded in their **raw** form from `raw.githubusercontent.com`.  \n- The file contents are saved in a `.md` file, wrapped in triple backticks for easy code parsing.\n\n### 3. GitHub Discussions (Q\u0026A Text Only)\n\n- For `https://github.com/orgs/onflow/discussions`, the script:\n  - Crawls the **listing** pages, discovers discussion links like `/orgs/onflow/discussions/1330`\n  - For each thread, it extracts **only** the text from user posts (skipping the GitHub UI) and converts it to Markdown.\n- This yields `.md` files containing the **original question** and **comments/replies**.\n\n---\n\n## Usage\n\n### Requirements\n\n1. **Python 3.7+**  \n2. [`requests`](https://pypi.org/project/requests/), [`beautifulsoup4`](https://pypi.org/project/beautifulsoup4/), [`markdownify`](https://pypi.org/project/markdownify/)\n\nInstall all dependencies:\n\npip install requests beautifulsoup4 markdownify\n\n### Running the Scraper\n\nClone or download this repo locally.\n\nIn the repo directory, run:\n\n```bash\npython scraper.py\n```\n\nThe script will crawl each site listed in `SITES` (inside `scraper.py`) and output the results under `scraped_docs/`.\n\n### Modifying the List of Sites\n\nInside `scraper.py`, near the top, you’ll see:\n\n```python\nSITES = [\n    \"https://developers.flow.com/\",\n    \"https://academy.ecdao.org/en/cadence-by-example\",\n    ...\n    \"https://github.com/onflow/flow-ft/\",\n    ...\n    \"https://github.com/orgs/onflow/discussions\"\n]\n```\n\n- Add a docs site by appending its URL if it’s not on GitHub.\n- Add a GitHub repo by appending the base URL (e.g. \"\u003chttps://github.com/onflow/another-repo\u003e\").\n- Add another GitHub Discussions page if needed.\n- Remove any site by deleting or commenting out its line.\n\nFor private sites or repos, you may need authentication tokens/cookies to see content that’s not public.\n\n### Merging\n\nYou can merge all the `.md` files into a single file or a file containing only the essentials (removing code blocks, etc.).  \nThat will be useful for indexing or searching or being used in a chatbot.\n\n```bash\npython merge.py\n```\n\n### Output Structure\n\nAfter a successful run, you’ll see:\n\n```bash\nscraped_docs/\n  ├─ developers_flow_com/\n  │   ├─ index.md\n  │   ├─ docs_tutorial_somepage.md\n  │   └─ ...\n  ├─ github_com_onflow_flow_ft/\n  │   ├─ blob_main_contracts_exampletoken_cdc.md\n  │   ├─ ...\n  ├─ github_com_orgs_onflow_discussions/\n  │   ├─ discussion_1330.md\n  │   ├─ discussion_1514.md\n  │   └─ ...\n  └─ ...\nmerged_docs/\n  ├─ all_merged.md\n  └─ essentials_merged.md\n```\n\n- Docs directories for each site\n- Repos with code files in `.md` (wrapped code blocks)\n- Discussions as `discussion_\u003cid\u003e.md`, each containing Q\u0026A text.\n\n### Scheduling Automation\n\nThe script can be scheduled to run daily using **GitHub Actions**.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fonflow%2Fflow-data-sources","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fonflow%2Fflow-data-sources","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fonflow%2Fflow-data-sources/lists"}