{"id":22152132,"url":"https://github.com/findinpath/search-alert","last_synced_at":"2026-04-11T22:03:19.255Z","repository":{"id":112791026,"uuid":"276574769","full_name":"findinpath/search-alert","owner":"findinpath","description":"Proof of concept project on implementing both near-real-time \u0026 batched search agent functionality","archived":false,"fork":false,"pushed_at":"2020-07-20T08:24:14.000Z","size":416,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"master","last_synced_at":"2025-01-29T18:32:11.352Z","etag":null,"topics":["cassandra","elasticsearch","kafka","kafka-consumer","percolator"],"latest_commit_sha":null,"homepage":"","language":"Kotlin","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/findinpath.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2020-07-02T07:11:38.000Z","updated_at":"2020-07-20T08:24:17.000Z","dependencies_parsed_at":null,"dependency_job_id":"6614bfb4-6ba2-4290-9ec9-adf3da0e5d92","html_url":"https://github.com/findinpath/search-alert","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/findinpath%2Fsearch-alert","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/findinpath%2Fsearch-alert/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/findinpath%2Fsearch-alert/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/findinpath%2Fsearch-alert/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/findinpath","download_url":"https://codeload.github.com/findinpath/search-alert/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":245277467,"owners_count":20589154,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["cassandra","elasticsearch","kafka","kafka-consumer","percolator"],"created_at":"2024-12-02T00:47:20.872Z","updated_at":"2026-04-11T22:03:19.176Z","avatar_url":"https://github.com/findinpath.png","language":"Kotlin","funding_links":[],"categories":[],"sub_categories":[],"readme":"Search alert \n============\n\n\nThis is a proof of concept project on implementing both:\n\n- near real time (NRT)\n- batched (hourly/daily)\n\nsearch alert functionality.\n\n\n## Introduction\n\nA search alert is as [Google Alerts](https://www.google.com/alerts) puts it a way to :\n\n\u003e Monitor the web for interesting new content \n\n\nWhen having a search index at disposal, at the time new content is being indexed, the system\ncan checked whether it fits to any of the search alerts configured and if so\nnotify the user about it.\n\n\nThis system is no match for the scale of the \n[Google Alerts](https://www.google.com/alerts) system, but it is rather a\nproof of concept on how this functionality can be achieved in a decent fashion\nfor websites of medium size.\n\nIn the same fashion as Google Alerts, this system should be able to handle both:\n\n- immediate (as it happens)\n- batched (hourly, daily, weekly)\n\nsearch alert notifications.\n\n\n## Technological stack\n\nThe tech stack on which the system behind this project is built is composed of the following components:\n\n- [Apache Kafka](https://kafka.apache.org/) for streaming data between the components of the system\n- [Elasticsearch](https://www.elastic.co/elasticsearch/) distributed, RESTful search engine \n- [Apache Cassandra](https://cassandra.apache.org/) for storing information about the search alert messages sent\nin order to avoid sending duplicated messages (e.g.: for a batched search alert on a specific hour/day).\n\n\n## Architecture\n\nThe architecture of this proof of concept system is relatively simple:\n\n![architecture](images/search-alert.png)\n\nFor the sake of better understanding the purpose of this system we'll consider that this\nsystem handles the search alert functionality for a news website.\nThe users of the website that have interest for a certain topic \ncan register a search alert to be notified when new articles matching their criteria \nare published on the platform. \nIn case of having matches, the search alerts can be configured by the users to be notified\nimmediately/ hourly/ daily about new results. \n\nWhen a news article is being published on the news platform, it will also be checked against the registered\nsearch alerts to look for matches.\n\nThere is a different type of handling between the immediate, and the batched search alerts.\n\n### Immediate search alerts\n\nThe immediate search alerts that match a newly published article are going to be pushed by the _percolator_ \ncomponent to the `immediate` topic which is being handled almost on the spot by the \n_immediate-messenger_ component to notify the user about a new possible match.\n\nThe _immediate-messenger_ component runs indefinitely.\n\n### Batched search alerts\n\nThe batched  search alert notifications need to be _\"parked\"_ until their corresponding messaging period(hour/day) elapses.\n\nConsidering that a search alert is configured to hourly notify a user about new articles matching a specific\nsearch criteria, when an article is being published on  `2020-06-19 09:21:05` on the marketplace platform,\nthe search alert should then notify the user at `2020-06-19 10:00:00`. In this case, the search alert \ninformation is going to be pushed to the topic `hourly_1592553600000` (`1592553600000` corresponds \nto `2020-06-19 10:00:00`).\nIn case that the search alert would have been configured to delivery notifications daily, then\nthe search alert information would be then published to the topic `daily_1592604000000` (`1592604000000` \ncorresponds to `2020-06-20 00:00:00`)\n\nThe consumption of the entries from the `hourly_1592553600000` topic will start at the earliest \nat `2020-06-19 10:00:00`.\nThe consumption of the entries from the `daily_1592604000000` topic will start at the earliest\nat `2020-06-20 00:00:00`.\n\n\nThe _batched-messenger_ component is responsible of handling the batched\nsearch alert hits from a specified topic that contain entries for the hourly/daily search alerts.\nThe consumption from the topic corresponding to a specific hour/day time window will begin only \nafter the time window has elapsed.  \nOnce the consumption of these topics reaches the end of the topic, the _batched-messenger_ component\ncan end its runtime.\n\n\nOnce all the partitions of the topic `hourly_1592553600000` are read by the _batched-messenger_ component,\nthe batched topic is considered obsolete (and can be eventually be deleted) and not being taken \nanymore into consideration for consumption.\n\n **OPTIONAL** In case of finding a matching article for a batched search alert, the _percolator_ component is responsible to _\"pause\"_ \n the search alert until its current batching period (hour/day) elapses in order to avoid doing unnecessary matches\n against new incoming articles.\n\n\n#### Orchestrating the batched search alerts\n\nAs mentioned previously, the batched  search alert notifications need to be _\"parked\"_ until their corresponding \nmessaging period(hour/day) elapses.\n\nAt the beginning of each batched time window (hour/day) should be therefor started one or multiple instances\nof the _batched-messenger_ component for the batched topics on which the consumer offset of the  _batched-messenger_\nlags behind.\n\nDepending on the amount of the batched search alert hits from the topic that need to be processed, the\norchestrator could then choose how many instances of the _batched-messenger_ component to spawn.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ffindinpath%2Fsearch-alert","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Ffindinpath%2Fsearch-alert","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ffindinpath%2Fsearch-alert/lists"}