{"id":42942256,"url":"https://github.com/wechuli/rss_feeds_fetcher","last_synced_at":"2026-01-30T20:10:19.617Z","repository":{"id":114650764,"uuid":"242664794","full_name":"wechuli/rss_feeds_fetcher","owner":"wechuli","description":"rss feeds project that fetches rss feeds, stores in a database and enables full text search to be performed","archived":false,"fork":false,"pushed_at":"2020-03-03T07:03:48.000Z","size":5223,"stargazers_count":0,"open_issues_count":0,"forks_count":1,"subscribers_count":0,"default_branch":"master","last_synced_at":"2026-01-13T02:52:17.897Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/wechuli.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2020-02-24T06:39:35.000Z","updated_at":"2020-03-03T07:03:51.000Z","dependencies_parsed_at":"2024-06-20T15:29:12.643Z","dependency_job_id":"44d9704e-61aa-426e-9fe4-29365ac2c042","html_url":"https://github.com/wechuli/rss_feeds_fetcher","commit_stats":null,"previous_names":[],"tags_count":1,"template":false,"template_full_name":null,"purl":"pkg:github/wechuli/rss_feeds_fetcher","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/wechuli%2Frss_feeds_fetcher","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/wechuli%2Frss_feeds_fetcher/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/wechuli%2Frss_feeds_fetcher/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/wechuli%2Frss_feeds_fetcher/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/wechuli","download_url":"https://codeload.github.com/wechuli/rss_feeds_fetcher/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/wechuli%2Frss_feeds_fetcher/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":28918235,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-01-30T19:10:10.838Z","status":"ssl_error","status_checked_at":"2026-01-30T19:06:40.573Z","response_time":66,"last_error":"SSL_connect returned=1 errno=0 peeraddr=140.82.121.5:443 state=error: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2026-01-30T20:10:19.104Z","updated_at":"2026-01-30T20:10:19.609Z","avatar_url":"https://github.com/wechuli.png","language":"Go","funding_links":[],"categories":[],"sub_categories":[],"readme":"## Introduction\n\nThe project fetches RSS Feeds from BBC and CNN and enables the feeds to be accessed and searched through a RESTful web service. Feeds are stored in a Postgres database and full text search can be performed on them. The search functionality is exposed as an API endpoint that accepts a search keyword and returns a JSON array of RSS feeds that matched the search criteria.\n\n## Running the Project\n\n1. The project makes use of a Postgres database to store the rss feeds and uses Postgres's full text search capability to search through the feeds . A running Postgres database will need to be set up before the application is run. A sample script to create a Postgres database is provided below:\n\n```SQL\n\n\nCREATE DATABASE rssfeeds;\nCREATE USER rssuser WITH PASSWORD 'password';\n\nALTER ROLE rssuser SET client_encoding TO 'utf8';\nALTER ROLE rssuser SET default_transaction_isolation TO 'read committed';\nALTER ROLE rssuser SET timezone TO 'UTC';\n\n\nGRANT ALL PRIVILEGES ON DATABASE rssfeeds TO rssuser;\n\nCREATE TABLE feeds(\n    title text NOT NULL ,\n    description text,\n    pubdate varchar,\n    link varchar PRIMARY KEY\n);\n\n\n```\n\nThe table does not need to be created before-hand, once the application is started, the necessary table will be created if it does not exist.\n\n2.  The project can be run using:\n\n        go run main.go app.go\n\nThis initializes a http server on port 8080 while concurrently fetching new RSS Feeds from CNN and BBC and storing them on the database.\n\nThe database connection details are queried from the environment variables, so they will need to be setup before hand. Refer to code snippet below to name the environment variables correctly:\n\n```GO\nhost, user, password, dbname := os.Getenv(\"RSS_DB_HOST\"), os.Getenv(\"RSS_DB_USERNAME\"), os.Getenv(\"RSS_DB_PASSWORD\"), os.Getenv(\"RSS_DB_NAME\")\n```\n\n3. There is only one route - for searching for the feeds `/search` which allows a `POST` method with a JSON body describing the search term\n\n**sample request**\n\n`POST /search`\n\n```JSON\n{\n\t\"term\":\"africa\"\n}\n```\n\n**sample response**\n\n```JSON\n\n[\n    {\n        \"title\": \"CNN Travel's 20 best places to visit in 2020\",\n        \"description\": \"Whether you want to relax on a remote island off the coast of Africa, ride Germany's coolest trains or spot howling monkeys in South America, there is much to explore heading into a new decade in 2020.\u003cimg src=\\\"http://feeds.feedburner.com/~r/rss/edition_world/~4/AmgOSHUmQzE\\\" height=\\\"1\\\" width=\\\"1\\\" alt=\\\"\\\"/\u003e\",\n        \"pubdate\": \"Mon, 06 Jan 2020 23:10:01 GMT\",\n        \"link\": \"http://rss.cnn.com/~r/rss/edition_world/~3/AmgOSHUmQzE/index.html\"\n    },\n    {\n        \"title\": \"Britain seeks closer economic ties with Africa following Brexit \",\n        \"description\": \"Britain is leaving the European Union on Friday, starting the clock on an 11-month transition period during which the country will try to sign as many new trade deals as possible. African countries are a prime target.\u003cimg src=\\\"http://feeds.feedburner.com/~r/rss/edition_africa/~4/VrZIYMYDIxU\\\" height=\\\"1\\\" width=\\\"1\\\" alt=\\\"\\\"/\u003e\",\n        \"pubdate\": \"Fri, 31 Jan 2020 16:23:35 GMT\",\n        \"link\": \"http://rss.cnn.com/~r/rss/edition_africa/~3/VrZIYMYDIxU/index.html\"\n    }\n]\n\n```\n\n## Application Structure\n\nTo separate concerns, the project was implemented using two packages(additional to `main`) :- `models` and `rss`\n\n### Package rss\n\nPackage rss implements functions concerned with fetching and parsing rss feeds from CNN and BBC (it can fetch rss feeds from any rss enabled website) and returning it in a format ready to be stored in the database (where the `models` package takes over)\n\n- **`func FetchRssFeedRaw`**\n\n  ```GO\n\n    func FetchRssFeedRaw(url string) (string, error)\n  ```\n\n  `FetchRssFeedRaw` fetches the raw rss feeds (usually in xml) and returns a string representation of the same.\n\n- **`func ParseRawRssString`**\n\n  ```GO\n\n    func ParseRawRssString(rawRssString string) ([]models.Feed, error)\n  ```\n\n`ParseRawRssString` receives a raw string rss website and parses it to extract the rss feeds represented in a convenient `Feed` struct, defined in the package `models`. It returns a slice of feeds extracted from the website.\n\n### Package models\n\nPackage models implements types and functions related to reading and writing the rss feeds to the database. It expects feeds to have already been formatted correctly (from the rss package).\n\n- **`type Feed`**\n\n  ```GO\n  type Feed struct {\n\tTitle       string `json:\"title\"`\n\tDescription string `json:\"description\"`\n\tPubDate     string `json:\"pubdate\"`\n\tLink        string `json:\"link\"`\n  }\n\n  ```\n\n  `Feed` struct is a minimal representation of the rss feeds, with just enough information to enable searching. JSON tags are included to allow marshalling and unmarshalling when working with the web servive\n\n- **`func StoreRssFeeds`**\n\n  ```GO\n  func StoreRssFeeds(db *sql.DB, feeds []Feed) error{...}\n\n  ```\n\n  `StoreRssFeeds` takes a slice of `Feeds` and runs a query on the database(pointed at by the `db` parameter) to store the Feeds.\n\n- **`func SearchRssFeeds`**\n\n  ```GO\n\n  func SearchRssFeeds(db *sql.DB, searchString string) ([]Feed, error){...}\n\n  ```\n\n  `SearchRssFeeds` performs a full text search on the database (pointed at by the `db` parameter) for the phrase specified in the `searchString` string parameter and returns a slice of Feeds that match the search criteria.\n\n## Testing\n\nTo test the project run `go test -v`.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fwechuli%2Frss_feeds_fetcher","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fwechuli%2Frss_feeds_fetcher","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fwechuli%2Frss_feeds_fetcher/lists"}