{"id":18649854,"url":"https://github.com/ashwanthkumar/finder","last_synced_at":"2025-10-05T01:18:17.295Z","repository":{"id":25918221,"uuid":"29359218","full_name":"ashwanthkumar/finder","owner":"ashwanthkumar","description":"Wayback like system for any datasets.","archived":false,"fork":false,"pushed_at":"2015-01-17T09:00:26.000Z","size":228,"stargazers_count":2,"open_issues_count":0,"forks_count":2,"subscribers_count":4,"default_branch":"master","last_synced_at":"2025-07-21T06:10:32.741Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"","language":"Scala","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":"PredixDev/MobileExample-Microservice-CommandProcessor","license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/ashwanthkumar.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2015-01-16T17:23:24.000Z","updated_at":"2015-01-18T14:48:55.000Z","dependencies_parsed_at":"2022-08-24T07:40:21.000Z","dependency_job_id":null,"html_url":"https://github.com/ashwanthkumar/finder","commit_stats":null,"previous_names":[],"tags_count":1,"template":false,"template_full_name":null,"purl":"pkg:github/ashwanthkumar/finder","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ashwanthkumar%2Ffinder","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ashwanthkumar%2Ffinder/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ashwanthkumar%2Ffinder/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ashwanthkumar%2Ffinder/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/ashwanthkumar","download_url":"https://codeload.github.com/ashwanthkumar/finder/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ashwanthkumar%2Ffinder/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":271574431,"owners_count":24783319,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-08-22T02:00:08.480Z","response_time":65,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-07T06:40:53.178Z","updated_at":"2025-10-05T01:18:12.254Z","avatar_url":"https://github.com/ashwanthkumar.png","language":"Scala","funding_links":[],"categories":[],"sub_categories":[],"readme":"[![Build Status](https://snap-ci.com/ashwanthkumar/finder/branch/master/build_image)](https://snap-ci.com/ashwanthkumar/finder/branch/master)\n\n# finder\nFinder is a hobby project to build a [Wayback](https://github.com/iipc/openwayback/) like system without converting our existing crawled data which are in SequenceFiles to WARC / ARC formats. Architecture is very much inspired from Wayback but with a few changes. Finder not only supports HTML but can provide key+timestamp based access to any dataset. These are very typical in the [λ Architecture](http://lambda-architecture.net/) world, where raw immutable dataset is the fundamental assumption.\n\n## Getting Started\n* Run the `SampleDatagenerator` program from your fav. IDE\n* By default it would create a folder in the project structure called \"example-user-data\"\n* Run the `FinderService` from `services` module\n* Example Queries\n  * [http://localhost:7070/user/search/Tanek](http://localhost:7070/user/search/Tanek)\n  * [http://localhost:7070/user/get/Tanek/timestamp_from_previous_search](http://localhost:7070/user/get/Tanek/timestamp_from_previous_search)\n\n## Dependencies\nTo build datasets for your models, use the following dependency on finder-core.\n\nFor Maven,\n```xml\n\u003cdependency\u003e\n  \u003cgroupId\u003ein.ashwanthkumar\u003c/groupId\u003e\n  \u003cartifactId\u003efinder-core_2.10\u003c/artifactId\u003e\n  \u003cversion\u003e0.0.1\u003c/version\u003e\n\u003c/dependency\u003e\n```\n\nFor SBT,\n```sbt\nlibraryDependencies += \"in.ashwanthkumar\" %% \"finder-core\" % \"0.0.1\"\n```\n\n\n## Finder HTTP Service\nFinder comes with a simple microservice that gives you access to the dataset.\n```\nGET  /:dataset/search/:key  Searches for a given key in the given dataset. Key is prefix matched with the key that's used in the dataset while indexing.\nGET  /:dataset/get/:key/:timestamp Returns the underlying record from the Dataset.\n```\n\n## More Info\nYou can define a custom Dataset like\n```scala\nimport finder.spec.Dataset\nimport finder.util.JSONUtil\n\ncase class User(name: String, age: Int, timestamp: Long)\n\nclass UserDataset extends Dataset[User] {\n  override def key(input: User): String = input.name\n  /**\n   * Timestamp of the record in UTC timezone\n   */\n  override def timestamp(input: User): Long = input.timestamp\n  override def serialize(input: User): Array[Byte] = JSONUtil.toJson(input).getBytes\n  override def deserialize(input: Array[Byte]): User = JSONUtil.readAs(classOf[User])(new String(input))\n}\n```\n\nAll the dataset goes with a configuration in `application.conf` like\n```hocon\nfinder {\n  datasets {\n    descriptors: [{\n      name: \"user\"\n      impl: \"finder.models.UserDataset\"\n      index {\n        params {\n          db-location: \"user-db\"\n        }\n      }\n    }]\n  }\n}\n```\n\nFinder uses an Index to store and retrieve metadata information regarding every record. By default we use LevelDB backed index, but again it is pluggable. Most likely we could use something like [ElephantDB](https://github.com/nathanmarz/elephantdb) to query the index.\n\n\n## TODOs\n- \u003cs\u003eBuild a HTTP service to provide access to these datasets\u003c/s\u003e\n- ElephantDB Adaptors\n- Partitioned Indexes - Comes for free with ElephantDB but for local LevelDB instances we might want to build something on our own\n- Versioned datasets - Each version could be a different run of the view in Batch Layer\n\n## License\nLicensed under the Apache License, Version 2.0: http://www.apache.org/licenses/LICENSE-2.0\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fashwanthkumar%2Ffinder","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fashwanthkumar%2Ffinder","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fashwanthkumar%2Ffinder/lists"}