{"id":17239331,"url":"https://github.com/rehacktive/news_aggregator","last_synced_at":"2025-03-26T02:36:56.181Z","repository":{"id":149775031,"uuid":"387000708","full_name":"rehacktive/news_aggregator","owner":"rehacktive","description":null,"archived":false,"fork":false,"pushed_at":"2021-07-18T17:46:47.000Z","size":17,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":3,"default_branch":"master","last_synced_at":"2025-01-30T23:44:19.929Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/rehacktive.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2021-07-17T17:31:16.000Z","updated_at":"2021-07-18T17:46:49.000Z","dependencies_parsed_at":"2023-07-22T22:15:59.374Z","dependency_job_id":null,"html_url":"https://github.com/rehacktive/news_aggregator","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/rehacktive%2Fnews_aggregator","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/rehacktive%2Fnews_aggregator/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/rehacktive%2Fnews_aggregator/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/rehacktive%2Fnews_aggregator/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/rehacktive","download_url":"https://codeload.github.com/rehacktive/news_aggregator/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":245577700,"owners_count":20638361,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-10-15T05:48:26.914Z","updated_at":"2025-03-26T02:36:56.159Z","avatar_url":"https://github.com/rehacktive.png","language":"Go","funding_links":[],"categories":[],"sub_categories":[],"readme":"# news-aggregator\n\nThe news-aggregator, based on 2 microservices, allows the user to define a list of sources (RSS atom feeds) and fetch them periodically, storing the content in the database, and a way to fetch the content, previously stored, for the original requirement: an app showing news from a list of public feeds.\n\nIt uses Postgres (containerized) as database.\n\nThe 2 microservices:\n\n* **feedIngestionService**: periodically gets a list of feed URLs from the database, gets the articles from all the sources and stores them in the database;\n\n  The list of feeds is stored in the database, and a category is associated to each different source, allowing later the possibility to filter news by category.\n\n  The fetch interval can be specified as an env variable, default is 1 minute.\n\n  *This  feed list could be managed in a CRUD-way by the endpoint (not implemented)*\n\n* **newsfeedService**: exposes 3 different endpoints, related to the original requirements, to get a list of news stored in the database, filter them by category and get a list of categories (used by the previous filter).\n\n---\n\n## Install\n\nWith a [correctly configured](https://golang.org/doc/install#testing) Go toolchain:\n\n```sh\ngit clone https://github.com/rehacktive/news_aggregator.git\n```\n\n## How to run it\n\nIf you have docker and docker-compose, you can just:\n\n```sh\ndocker-compose up\n```\n\nAnother option is to install  Postgres locally (port 5432 on localhost) and then run both services with the *run.sh* script present in both folders.\n\n\n\n---\n\n## How to use the endpoints\n\nOnce both services are up and running, after the defined interval (default is 1 minute) the database should be populated with the articles from the predefined feeds.\n\nWhen the content is available on the database, the first endpoint returns a list of news, sorted by date:\n\n```sh\n$ curl http://localhost:8880/news | jq .\n  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current\n                                 Dload  Upload   Total   Spent    Left  Speed\n100 19934    0 19934    0     0  9733k      0 --:--:-- --:--:-- --:--:-- 9733k\n{\n  \"news\": [\n    {\n      \"Id\": 21,\n      \"Title\": \"Covid-19: PM and chancellor self-isolate after rapid U-turn\",\n      \"URL\": \"https://www.bbc.co.uk/news/uk-57879730\",\n      \"GUID\": \"https://www.bbc.co.uk/news/uk-57879730\",\n      \"Description\": \"It comes after anger over \\\"VIP testing\\\" allowing them to work following Sajid Javid's positive test.\",\n      \"Category\": 1,\n      \"PubDate\": \"2021-07-18T17:27:23Z\"\n    },\n    {\n      \"Id\": 112,\n      \"Title\": \"Cavendish wins Tour de France green jersey - but narrowly misses out on record 35 stage wins\",\n      \"URL\": \"http://news.sky.com/story/tour-de-france-mark-cavendish-wins-green-jersey-but-narrowly-misses-out-on-record-35-stage-wins-12358666\",\n      \"GUID\": \"http://news.sky.com/story/tour-de-france-mark-cavendish-wins-green-jersey-but-narrowly-misses-out-on-record-35-stage-wins-12358666\",\n      \"Description\": \"British cyclist Mark Cavendish has won the green jersey in this year's Tour de France.\",\n      \"Category\": 2,\n      \"PubDate\": \"2021-07-18T16:57:00Z\"\n    },\n    ...\n```\n\nTo \"resolve\" the category name another endpoint is available:\n\n```sh\n$ curl http://localhost:8880/category | jq .\n  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current\n                                 Dload  Upload   Total   Spent    Left  Speed\n100    70  100    70    0     0  70000      0 --:--:-- --:--:-- --:--:-- 70000\n{\n  \"categories\": [\n    {\n      \"Id\": 1,\n      \"Name\": \"BBC News\"\n    },\n    {\n      \"Id\": 2,\n      \"Name\": \"Sky News\"\n    }\n  ]\n}\n\n```\n\nit's also now possible use the category ID to filter the news:\n\n```sh\n$ curl http://localhost:8880/news/2 | jq .\n  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current\n                                 Dload  Upload   Total   Spent    Left  Speed\n100  6453    0  6453    0     0  3150k      0 --:--:-- --:--:-- --:--:-- 3150k\n{\n  \"news\": [\n    {\n      \"Id\": 112,\n      \"Title\": \"Cavendish wins Tour de France green jersey - but narrowly misses out on record 35 stage wins\",\n      \"URL\": \"http://news.sky.com/story/tour-de-france-mark-cavendish-wins-green-jersey-but-narrowly-misses-out-on-record-35-stage-wins-12358666\",\n      \"GUID\": \"http://news.sky.com/story/tour-de-france-mark-cavendish-wins-green-jersey-but-narrowly-misses-out-on-record-35-stage-wins-12358666\",\n      \"Description\": \"British cyclist Mark Cavendish has won the green jersey in this year's Tour de France.\",\n      \"Category\": 2,\n      \"PubDate\": \"2021-07-18T16:57:00Z\"\n    },\n    {\n      \"Id\": 3,\n      \"Title\": \"Hamilton wins British Grand Prix after collision with Verstappen who was taken to hospital\",\n      \"URL\": \"http://news.sky.com/story/lewis-hamilton-wins-british-grand-prix-after-collision-with-max-verstappen-who-was-taken-to-hospital-12358585\",\n      \"GUID\": \"http://news.sky.com/story/lewis-hamilton-wins-british-grand-prix-after-collision-with-max-verstappen-who-was-taken-to-hospital-12358585\",\n      \"Description\": \"Lewis Hamilton has won the British Grand Prix following a race which saw him and F1 title rival Max Verstappen collide on the opening lap.\",\n      \"Category\": 2,\n      \"PubDate\": \"2021-07-18T15:58:00Z\"\n    },\n    {\n      \"Id\": 5,\n      \"Title\": \"Man arrested after teenager riding e-scooter dies following hit-and-run\",\n      \"URL\": \"http://news.sky.com/story/teenager-riding-e-scooter-dies-after-hit-and-run-12358583\",\n      \"GUID\": \"http://news.sky.com/story/teenager-riding-e-scooter-dies-after-hit-and-run-12358583\",\n      \"Description\": \"A man has been arrested after a 16-year-old boy was killed in a hit-and-run incident while riding an e-scooter in south-east London, police say.\",\n      \"Category\": 2,\n      \"PubDate\": \"2021-07-18T15:58:00Z\"\n    },\n    ...\n```\n\n## \n\n\n\n\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Frehacktive%2Fnews_aggregator","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Frehacktive%2Fnews_aggregator","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Frehacktive%2Fnews_aggregator/lists"}