{"id":26123618,"url":"https://github.com/guillempuche/news_crawler","last_synced_at":"2026-07-11T22:31:03.872Z","repository":{"id":280515538,"uuid":"942255120","full_name":"guillempuche/news_crawler","owner":"guillempuche","description":"Scrape news from Olot town hall (https://www.olot.cat) with TypeScript and Crawlee. Collects summaries and full articles, stored in separate datasets.","archived":false,"fork":false,"pushed_at":"2025-04-25T21:21:56.000Z","size":181,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"main","last_synced_at":"2026-02-10T19:14:13.354Z","etag":null,"topics":["biomejs","crawlee","crawler","news-crawler","olot","townhall","yarn-berry"],"latest_commit_sha":null,"homepage":"","language":"TypeScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/guillempuche.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2025-03-03T20:26:49.000Z","updated_at":"2025-04-25T21:22:00.000Z","dependencies_parsed_at":null,"dependency_job_id":"38214aac-486b-4083-9a9e-077b32266803","html_url":"https://github.com/guillempuche/news_crawler","commit_stats":null,"previous_names":["guillempuche/news_crawler"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/guillempuche/news_crawler","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/guillempuche%2Fnews_crawler","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/guillempuche%2Fnews_crawler/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/guillempuche%2Fnews_crawler/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/guillempuche%2Fnews_crawler/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/guillempuche","download_url":"https://codeload.github.com/guillempuche/news_crawler/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/guillempuche%2Fnews_crawler/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":35377012,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-05-26T15:22:16.424Z","status":"online","status_checked_at":"2026-07-11T02:00:05.354Z","response_time":104,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["biomejs","crawlee","crawler","news-crawler","olot","townhall","yarn-berry"],"created_at":"2025-03-10T15:53:47.579Z","updated_at":"2026-07-11T22:31:03.859Z","avatar_url":"https://github.com/guillempuche.png","language":"TypeScript","funding_links":[],"categories":[],"sub_categories":[],"readme":"# News Crawler\n\nWelcome to the Olot News Crawler! This project scrapes news from the Olot town hall website (`https://www.olot.cat/ajuntament/comunicacio/actualitat.htm`) using TypeScript and [Crawlee](https://crawlee.dev/). It collects summaries from the news list pages and full details from individual articles, storing them in separate datasets for easy access.\n\nThis guide will help you set up and run the project step-by-step, even if you're new to coding!\n\n---\n\n## Prerequisites\n\nBefore you start, you’ll need to install a few tools on your computer:\n\n### 1. Install Node.js\n\n- **What it is:** Node.js is a runtime that lets us run JavaScript (and TypeScript) outside a browser. This project requires Node.js version 22 or higher.\n- **How to install:**\n  1. Go to the [Node.js website](https://nodejs.org/).\n  2. Download the \"LTS\" version for your operating system (Windows, macOS, or Linux). As of March 2025, this should be v22.x.x or later.\n  3. Run the installer and follow the prompts (accept defaults if unsure).\n  4. Open a terminal (Command Prompt on Windows, Terminal on macOS/Linux) and check the version:\n\n     ```\n     node --version\n     ```\n\n     You should see something like `v22.x.x`. If not, reinstall Node.js.\n\n### 2. Enable Yarn Berry (via Corepack)\n\n- **What it is:** Yarn is a package manager that installs project dependencies. We’re using Yarn Berry (v4.6.0), which comes built into Node.js via Corepack.\n- **How to enable:**\n  1. In your terminal, run:\n\n     ```\n     corepack enable\n     ```\n\n     This activates Corepack, which manages Yarn versions.\n  2. The project specifies Yarn 4.6.0 in `package.json`, so you don’t need to install it manually—Corepack will handle it when you set up the project.\n\n### 3. Install Git\n\n- **What it is:** Git is a version control system to clone this project from a repository.\n- **How to install:**\n  1. Go to the [Git website](https://git-scm.com/downloads).\n  2. Download and install Git for your OS (accept defaults).\n  3. Verify installation:\n\n     ```\n     git --version\n     ```\n\n     You should see `git version x.x.x`.\n\n---\n\n## Getting Started\n\nFollow these steps to set up and run the project:\n\n### 1. Clone the Repository\n\n- Open your terminal and navigate to a folder where you want the project:\n\n  ```\n  cd path/to/your/folder\n  ```\n\n- Clone the repo (replace `\u003crepo-url\u003e` with the actual URL, e.g., from GitHub):\n\n  ```\n  git clone \u003crepo-url\u003e olot-news-crawler\n  cd olot-news-crawler\n  ```\n\n### 2. Install Dependencies\n\n- Run this command to install all project dependencies using Yarn 4.6.0:\n\n  ```\n  yarn install\n  ```\n\n- What happens:\n  - Corepack detects `\"packageManager\": \"yarn@4.6.0...\"` in `package.json` and downloads Yarn 4.6.0.\n  - Yarn installs libraries like `crawlee` and `typescript`.\n- Verify Yarn version:\n\n  ```\n  yarn --version\n  ```\n\n  It should say `4.6.0`. If not, ensure `corepack enable` was run.\n\n### 3. Run the Crawler\n\n- Start the crawler in development mode (it auto-reloads if you edit the code):\n\n  ```\n  yarn dev\n  ```\n\n- What it does:\n  - Crawls `https://www.olot.cat/ajuntament/comunicacio/actualitat.htm`.\n  - Collects news summaries and article details.\n  - Saves data to `storage/datasets/`.\n- To stop it, press `Ctrl + C` in the terminal.\n\n### 4. Build for Production (Optional)\n\n- If you want a compiled version:\n\n  ```\n  yarn build\n  ```\n\n- This creates a `dist/` folder with JavaScript files you can run with `node dist/index.js`.\n\n### 5. Lint and Format Code (Optional)\n\n- To check and fix code style:\n\n  ```\n  yarn lint\n  ```\n\n- Uses Biome to keep the code consistent.\n\n---\n\n## Project Structure\n\nHere’s what’s in the project:\n\n- **`src/`**: Source code.\n  - `index.ts`: Entry point that runs the crawler.\n  - `crawler.ts`: Crawling logic using [Crawlee](https://crawlee.dev/).\n  - `types.ts`: Defines the `NewsItem` data structure.\n- **`storage/`**: Where data is saved.\n  - `datasets/news_summaries/`: Summaries from list pages (title, date, summary).\n  - `datasets/news_articles/`: Full article details (title, date, summary, full text, image URL).\n  - `request_queues/default/`: Tracks URLs being crawled.\n- **`package.json`**: Lists dependencies and scripts.\n- **`.yarnrc.yml`**: Configures Yarn settings.\n- **`tsconfig.json`**: TypeScript configuration.\n- **`.gitignore`**: Ignores files like `node_modules/` from Git.\n\n---\n\n## Understanding the Output\n\nWhen you run `yarn dev`, the crawler:\n\n1. Visits the news list page and its paginated versions (e.g., `?pg=2`).\n2. Extracts summaries from each news item on the list and saves them to `storage/datasets/news_summaries/`.\n3. Enqueues and visits each article URL, extracting full details and saving them to `storage/datasets/news_articles/`.\n\n### Sample Data\n\n- **Summary (`news_summaries/000000001.json`):**\n\n  ```json\n  {\n    \"url\": \"https://www.olot.cat/pl217/ajuntament/comunicacio/actualitat/id8690/nova-edicio-del-casal-esportiu-de-setmana-santa.htm\",\n    \"title\": \"Nova edició del casal esportiu de Setmana Santa\",\n    \"date\": \"2/3/2025\",\n    \"summary\": \"L’Àrea d’Esports de l’Ajuntament d’Olot ha organitzat un any més el casal esportiu de Setmana Santa.\"\n  }\n  ```\n\n- **Article (`news_articles/000000001.json`):**\n\n  ```json\n  {\n    \"url\": \"https://www.olot.cat/pl217/ajuntament/comunicacio/actualitat/id8690/nova-edicio-del-casal-esportiu-de-setmana-santa.htm\",\n    \"title\": \"Nova edició del casal esportiu de Setmana Santa\",\n    \"date\": \"2 de març de 2025\",\n    \"summary\": \"L’Àrea d’Esports de l’Ajuntament d’Olot ha organitzat un any més el casal esportiu de Setmana Santa.\",\n    \"fullText\": \"L’Àrea d’Esports de l’Ajuntament d’Olot ha organitzat un any més el casal esportiu de Setmana Santa. Enguany, aquest casal multiesportiu dirigit a infants nascuts entre el 2013 i el 2020 porta per nom Casal Setmana Santa Esportiva 2025 i es realitzarà del 14 al 17 d’abril de 9 a 13 h.\\n[...]\",\n    \"imageUrl\": \"https://www.olot.cat/media/site1/cache/images/post-casal-setmana-santa-esportiva-2025.jpg\"\n  }\n  ```\n\n---\n\n## Troubleshooting\n\n- **Node.js Version Error:**\n  - If you see `node: command not found` or a version \u003c 22, reinstall Node.js.\n- **Yarn Install Fails:**\n  - Ensure `corepack enable` was run. Check `node --version` is 22+.\n  - Run `yarn install` again.\n- **Crawler Stops Early:**\n  - The `maxRequestsPerCrawl` is set to 50 for testing. Edit `src/crawler.ts` to increase it (e.g., 100) or remove it for a full crawl.\n- **No Data in `storage/`:**\n  - Check terminal logs. If it says \"No summaries found\" or \"No data extracted,\" the website structure might have changed. Tell me, and we’ll fix it!\n\n---\n\n## Customizing the Project\n\n- **Change the Crawl Limit:** In `src/crawler.ts`, adjust `maxRequestsPerCrawl`:\n\n  ```typescript\n  maxRequestsPerCrawl: 100, // Crawl more pages\n  ```\n\n- **Add More Data:** Edit the article handler in `crawler.ts` to extract extra fields (e.g., related links `\u003cul class=\"llista-links\"\u003e`).\n- **Run Without Watching:** Use `tsx src/index.ts` instead of `yarn dev`.\n\n---\n\n## Next Steps\n\n- Explore the data in `storage/datasets/` using a text editor or JSON viewer.\n- Learn TypeScript basics to tweak the code: [TypeScript Docs](https://www.typescriptlang.org/docs/).\n- Check out Crawlee for more features: [Crawlee Docs](https://crawlee.dev/docs/introduction).\n\n---\n\n### Notes for You\n\n- **File Creation:** Save this as `README.md` in the project root and commit it:\n\n  ```\n  echo \"# Olot News Crawler\" \u003e README.md\n  # Paste the content above into README.md using your editor\n  git add README.md\n  git commit -m \"Add README guide for setup and usage\"\n  git push\n  ```\n\n- **Assumptions:** Assumes your friend will get the repo URL from you (e.g., GitHub). Replace `\u003crepo-url\u003e` with the actual link when sharing.\n- **Tone:** Kept it friendly and simple, avoiding jargon where possible.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fguillempuche%2Fnews_crawler","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fguillempuche%2Fnews_crawler","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fguillempuche%2Fnews_crawler/lists"}