{"id":19038448,"url":"https://github.com/tonytw1/whakaoko","last_synced_at":"2025-04-23T19:46:09.320Z","repository":{"id":38380543,"uuid":"11073883","full_name":"tonytw1/whakaoko","owner":"tonytw1","description":"RSS aggregator with JSON API","archived":false,"fork":false,"pushed_at":"2024-10-12T17:43:35.000Z","size":2314,"stargazers_count":6,"open_issues_count":7,"forks_count":1,"subscribers_count":3,"default_branch":"main","last_synced_at":"2025-04-18T04:54:27.594Z","etag":null,"topics":["mongodb","rss","rss-aggregator"],"latest_commit_sha":null,"homepage":"","language":"JavaScript","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/tonytw1.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2013-06-30T16:51:27.000Z","updated_at":"2024-10-12T17:43:39.000Z","dependencies_parsed_at":"2024-05-03T21:43:48.771Z","dependency_job_id":"a4a686a4-6256-4cb0-9bb0-d388494a91a4","html_url":"https://github.com/tonytw1/whakaoko","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/tonytw1%2Fwhakaoko","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/tonytw1%2Fwhakaoko/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/tonytw1%2Fwhakaoko/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/tonytw1%2Fwhakaoko/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/tonytw1","download_url":"https://codeload.github.com/tonytw1/whakaoko/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":250502646,"owners_count":21441269,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["mongodb","rss","rss-aggregator"],"created_at":"2024-11-08T22:03:25.820Z","updated_at":"2025-04-23T19:46:09.266Z","avatar_url":"https://github.com/tonytw1.png","language":"JavaScript","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Whakaoko (verb. to listen)\n\nA service for aggregating 3rd party content from RSS feeds into an easy to consume local JSON or RSS feed.\n\nProvides a [local API](src/main/resources/static/openapi.json) for other applications to consume \nthe aggregated content via a single HTTP/JSON interface.\n\n\n![Screenshot of an aggregated channel of RSS feeds](channel.png)\n\n\n## Good at\n\nVery good at the use case of aggregating a list of related RSS feeds into a combined channel.\n\n\n## Features\n\n### Resolving short URLs\n\nExpands feedburner and tinyurl URLs into canonical full URLs.\n\n\n### Minimises HTTP traffic\n\nUses etag and last modified header conditional GETs to minimise traffic.\n\nThe feedcache documentation on [implementing conditional HTTP GET](https://feedcache.readthedocs.io/en/latest/howto/index.html#implementing-conditional-http-get) describes the technique well.\n\n### GeoRSS aware\n\nPreserves GeoRSS location data.\n\n## Implementation\n\nKotlin application. \nMongoDB 3.6 is used for storage. \nMemcached for caching of resolved short URLs.\n\n\n## Concepts\n\n### Subscriptions\n\nA subscription represents a source of 3rd party content.\nThe following sources are currently supported.\n\n- RSS / ATOM feeds\n\nContent, media RSS and geoRSS tags.\n\n\n### Channels\n\nA channel is a collection of related subscriptions.\nThe content from each subscription in the channel is aggregated together.\n\n\n\n### Output formats\n\nContent is output is either JSON or RSS.\n\nThe RSS format encodes images as media RSS tags and geotags as geoRSS tags.\n\n\n\n### API end points\n\n##### GET /{userid}/channels\n\nLists the channels defined for this user.\n\n\n#### GET /{userid}/channels/{channelid}\n\nDetails for a specific channel.\n\n\n#### GET /{userid}/channels/{channelid}/subscriptions\n\nLists the subscriptions into this channel.\n\n\n#### GET /{userid}/channels/{channelid}/items\n\n| Parameter\t | \tDescription      |\n|------------|-------------------|\n| format     | json / rss        |\n| page       | pagination number |\n\t\nShow content for this channel.\n\nContains all content items received from all of the subscriptions in this channel.\n\n\n#### GET //subscriptions/{subscriptionid}\n\nDetails for a specific subscription.\n\n```\n{\n  \"id\": \"wellynews-feed-78885da5bcdcde79d46c2cb5c6d936d0\",\n  \"name\": \"Coming Events – Friends of Te Papa\",\n  \"url\": \"https://www.friendsoftepapa.org.nz/events/feed/\",\n  \"channelId\": \"wellynews\",\n  \"username\": \"tonytw1\",\n  \"classifications\": [\n    \"ok\"\n  ],\n  \"lastRead\": \"2023-06-04T15:12:10Z\",\n  \"latestItemDate\": \"2023-10-09T22:00:00Z\",\n  \"etag\": \"W/\\\"da7aa1ec5b7aaf30fe2dce55c58a44fb\\\"\",\n  \"httpStatus\": 200,\n  \"itemCount\": 64,\n  \"lastModified\": 1685671239000\n}\n```\n\n| Field          | Description                                                                                                                                                                                           |\n|----------------|-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|\n| lastRead       | When this subscription was last read                                                                                                                                                                  |\n| latestItemDate | The date of the latest item seen in the feeds response. \u003cbr/\u003eThis may be different from the date of the most recently persisted feed item if content on a given GUID has been updated or republished. |\n\n#### GET /{userid}/subscriptions/{subscriptionid}/items\n\n| Parameter\t | \tDescription      |\n|------------|-------------------|\n| format     | json / rss        |\n| page       | pagination number |\n\nShow content from this subscription.\n\n\n#### POST /{username}/subscriptions/feeds\n\n| Parameter\t | \tDescription                               |\n|------------|--------------------------------------------|\n| channel    | The channel id to add this subscription to |\n| url        | The url of an RSS or ATOM feed             |\n\t\nRequest a new subscription to a RSS or ATOM feed url.\n\n\n# Setup\n\n## Run locally\n\nStart Mongo and Memcached dependencies as Docker images:\n\n```\ndocker-compose -f docker/docker-compose.yml up\n```\n\nRun using the Spring Boot plugin:\n\n```\nmvn spring-boot:run\n```\n\nThe service will be visible on localhost port 8080.\n\n\n## Create a user\n\nUsing the user interface:\n```\nhttp://localhost:8080/ui/newuser\n```\n\nor curl:\n```\ncurl -XPOST http://localhost:8080/users?username=new-user\n```\n\n## Generate an access token\n\nUsing the user interface sign in as your new user.\n\n```\nhttp://localhost:8080/ui\n```\n\nThen click Generate to generate an access token.\n\n![Generate a token](generate-token.png)\n\n\n## Create a channel\n\nEither using the UI or with an API call.\n\n```\ncurl -X POST http://localhost:8080/channels -H \"Authorization: Bearer YOUR_TOKEN\" -H \"Content-Type: application/json\" -d \"{\\\"name\\\":\\\"A channel\\\"}\"\n```\n\nNote the returned channel id. You will use it in the next request.\n\n\n## Create a feed subscription\n\nEither using the UI or with an API call.\n\n```\ncurl -X POST http://localhost:8080/subscriptions -H \"Authorization: Bearer YOUR_TOKEN\" -H  \"Content-Type: application/json\" -d \"{\\\"channel\\\":\\\"YOUR_CHANNEL_ID\\\",\\\"url\\\":\\\"http://www.victoria.ac.nz/home/about/newspubs/news/newslatest/news-rss-feed\\\"}\"\n\\\",\\\"url\\\":\\\"http://www.victoria.ac.nz/home/about/newspubs/news/newslatest/news-rss-feed\\\"}\"\n```\n\nNote the returned subscription id. You will use it in the next request\n\n## Read feed items\n\n```\ncurl http://localhost:8080/subscriptions/YOUR_SUBSCRIPTION_ID/items | json_pp\n```\n\n![Feed items JSON](feeditems.png)\n\n\n# Local development\n\nMaven build.\n\n```\ndocker-compose -f docker/docker-compose.yml up\nmvn clean install\n```\n\n\n## Cloud build\n\n```\ngcloud components install cloud-build-local\ncloud-build-local --config=cloudbuild.yaml --dryrun=false --push=false .\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ftonytw1%2Fwhakaoko","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Ftonytw1%2Fwhakaoko","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ftonytw1%2Fwhakaoko/lists"}