{"id":28330830,"url":"https://github.com/thomsendrake/ainews-source-extractor","last_synced_at":"2026-04-30T13:31:55.297Z","repository":{"id":294969316,"uuid":"987754028","full_name":"ThomsenDrake/ainews-source-extractor","owner":"ThomsenDrake","description":"Extracts all URLs from the most recent AI News issue (from news.smol.ai) and prepares them for seamless import into Google's NotebookLM","archived":false,"fork":false,"pushed_at":"2025-05-22T22:27:50.000Z","size":40,"stargazers_count":6,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"master","last_synced_at":"2025-06-03T04:03:06.346Z","etag":null,"topics":["automation","markdown","notebooklm","python","web-scraping"],"latest_commit_sha":null,"homepage":"","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/ThomsenDrake.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":".github/FUNDING.yml","license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null},"funding":{"github":["ThomsenDrake"],"patreon":null,"open_collective":null,"ko_fi":null,"tidelift":null,"community_bridge":null,"liberapay":null,"issuehunt":null,"lfx_crowdfunding":null,"polar":null,"buy_me_a_coffee":null,"thanks_dev":null,"custom":null}},"created_at":"2025-05-21T14:33:49.000Z","updated_at":"2025-05-27T00:24:18.000Z","dependencies_parsed_at":"2025-05-22T23:53:31.753Z","dependency_job_id":"ea43beec-c28e-492d-b002-71de4a3506b0","html_url":"https://github.com/ThomsenDrake/ainews-source-extractor","commit_stats":null,"previous_names":["thomsendrake/ainews-source-extractor"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/ThomsenDrake/ainews-source-extractor","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ThomsenDrake%2Fainews-source-extractor","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ThomsenDrake%2Fainews-source-extractor/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ThomsenDrake%2Fainews-source-extractor/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ThomsenDrake%2Fainews-source-extractor/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/ThomsenDrake","download_url":"https://codeload.github.com/ThomsenDrake/ainews-source-extractor/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ThomsenDrake%2Fainews-source-extractor/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":260658756,"owners_count":23043435,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["automation","markdown","notebooklm","python","web-scraping"],"created_at":"2025-05-26T17:42:19.786Z","updated_at":"2026-04-30T13:31:55.292Z","avatar_url":"https://github.com/ThomsenDrake.png","language":"Python","funding_links":["https://github.com/sponsors/ThomsenDrake"],"categories":[],"sub_categories":[],"readme":"# AI News Source Extractor\n\n## Description\nAI News Link Scraper extracts all URLs from the most recent AI News issue (from news.smol.ai) and prepares them for seamless import into Google's NotebookLM. It organizes sources into a dedicated folder, separates non-social URLs into a `sources.txt`, and generates individual markdown files for quoted tweet content.\n\n## Features\n* **Folder Generation:** Creates a timestamped folder for each issue’s sources.\n* **sources.txt:** Lists all URLs from the issue, excluding `twitter.com`, `x.com`, and `discord.com`.\n* **Tweet Markdown:** Saves the full text of each quoted tweet as a separate markdown file.\n* **WebSync Ready:** `sources.txt` can be pasted directly into the [WebSync for NotebookLM](https://chromewebstore.google.com/detail/websync-full-site-importe/hjoonjdnhagnpfgifhjolheimamcafok) Chrome extension to auto-import into NotebookLM.\n\n## Installation\n```bash\ngit clone https://github.com/ThomsenDrake/ainews-source-extractor.git\ncd ainews-source-extractor\npython3 -m venv venv\nsource venv/bin/activate    # On Windows: venv\\Scripts\\activate\npip install -r requirements.txt\n```\n\n## Usage\nSimply run the main scraper:\n```bash\npython build_issue.py\n```\nThis will:\n1. Generate a folder named with the current date for the latest AI News issue.\n2. Create `sources.txt` inside that folder, containing all non-social URLs.\n3. Produce individual `.md` files for each tweet quoted in the issue.\n\n## Roadmap\n* Improve URL-filtering logic to separate `twitter.com`, `x.com`, and `discord.com` links.\n* Build `discord_scraper.py` to fetch and save referenced Discord messages as markdown.\n* Parameterize the output folder path and issue source URL for greater flexibility.\n\n## Contributing\nContributions welcome! Fork, branch, and submit a pull request.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fthomsendrake%2Fainews-source-extractor","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fthomsendrake%2Fainews-source-extractor","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fthomsendrake%2Fainews-source-extractor/lists"}