{"id":15508271,"url":"https://github.com/jponge/dice-proxy","last_synced_at":"2026-07-14T12:15:06.592Z","repository":{"id":2702628,"uuid":"3696298","full_name":"jponge/dice-proxy","owner":"jponge","description":"A simple transparent HTTP proxy that detects common search engine queries, and puts query data into a MongoDB database.","archived":false,"fork":false,"pushed_at":"2012-03-12T17:07:19.000Z","size":125,"stargazers_count":1,"open_issues_count":0,"forks_count":2,"subscribers_count":2,"default_branch":"master","last_synced_at":"2025-06-05T13:03:33.657Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"","language":"Scala","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/jponge.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2012-03-12T15:11:48.000Z","updated_at":"2013-11-05T11:20:24.000Z","dependencies_parsed_at":"2022-08-29T10:51:17.515Z","dependency_job_id":null,"html_url":"https://github.com/jponge/dice-proxy","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/jponge/dice-proxy","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jponge%2Fdice-proxy","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jponge%2Fdice-proxy/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jponge%2Fdice-proxy/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jponge%2Fdice-proxy/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/jponge","download_url":"https://codeload.github.com/jponge/dice-proxy/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jponge%2Fdice-proxy/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":280747650,"owners_count":26383947,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-10-24T02:00:06.418Z","response_time":73,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-10-02T09:36:50.921Z","updated_at":"2025-10-24T05:42:50.972Z","avatar_url":"https://github.com/jponge.png","language":"Scala","funding_links":[],"categories":[],"sub_categories":[],"readme":"# DICE HTTP Proxy\n\nThis is a simple transparent HTTP proxy that detects common search engine queries, and puts query data into a MongoDB database.\n\nWritten by [Julien Ponge](http://julien.ponge.info/).\n\n## License\n\n    Copyright 2012 Julien Ponge, Institut National des Sciences Appliquées de Lyon.\n\n    Licensed under the Apache License, Version 2.0 (the \"License\");\n    you may not use this file except in compliance with the License.\n    You may obtain a copy of the License at\n\n        http://www.apache.org/licenses/LICENSE-2.0\n\n    Unless required by applicable law or agreed to in writing, software\n    distributed under the License is distributed on an \"AS IS\" BASIS,\n    WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n    See the License for the specific language governing permissions and\n    limitations under the License.\n\n## Supported sites\n\n* Google\n* Bing\n* Yahoo!\n* Wikipedia\n\n## What works and not\n\n* The proxy works for any HTTP connection. It also detects requests on other ports than 80.\n* It does not support HTTPS connections.\n* Some streaming / websocket connections as used by sites such as Youtube won't work through this proxy.\n\n## Dependencies\n\nRuntime:\n\n* [Scala 2.9.1](http://www.scala-lang.org/)\n* [MongoDB](http://www.mongodb.org/)\n\nDevelopment:\n\n* [Apache BuildR](http://buildr.apache.org/)\n* [Ruby](http://www.ruby-lang.org/) (required to run BuildR)\n* [Twitter Finagle](http://twitter.github.com/finagle/) (automatically fetched by BuildR)\n* [Netty](http://netty.io/) (automatically fetched by BuildR as Finagle dependency)\n* [Casbash](http://api.mongodb.org/scala/casbah/current/) (automatically fetch by BuildR)\n* (other misc. dependencies fetched as dependencies of the projects above)\n\nThose are fairly easy to install dependencies.\n\nApache BuildR can be installed using Ruby gems: `gem install buildr`. Scala is automatically downloaded by BuildR to compile, but it remains best to install it system-wide for the runtime.\n\nIf you are on MacOSX using [Homebrew](http://mxcl.github.com/homebrew/), then installing the dependencies is as simple as:\n\n    brew install scala\n    brew install mongodb\n    gem install buildr\n\n## Building\n\nBuilding is based on the common BuildR tasks, such as `buildr compile` or `buildr package`.\n\nOther tasks are available too (see `buildr -T` to list them all):\n\n* `buildr execute` launches the application\n* `buildr dist` will create a distributable ZIP image of the application along with a `run.sh` script.\n\n## Running\n\nUse the distributable image and launch `run.sh`.\n\nAlternatively you may assemble a classpath from the dependencies being put in the exploded image folder, and run the `dice.searchengine.httpproxy.SearchEngineHttpProxy` main class.\n\nYou will also need a working MongoDB instance. In development mode, I suggest that you use it with the provided configuration file called `mongod.conf`:\n\n    mkdir db-mongodb\n    mongod run --config=mongod.conf\n\nYou can tweak the MongoDB configuration as you want to add replication and sharding support. This is all transparent to the HTTP proxy application, and you can even start a cluster of those!\n\n## Use it with a web browser\n\nIf you don't know how to configure a proxy for your web browser then Google is your friend.\n\n## Recognizing new search engines\n\nA search engine is defined as a partial function through the following trait:\n\n```scala\ntrait SearchEngineProcessor extends PartialFunction[String, SearchEngineQuery] {\n\n  /**\n   * Regular expression to check is a URI corresponds to those a given search engine.\n   */\n  def searchEngineTest: Regex\n\n  /**\n   * Regular expression to extract a query string from a URI.\n   */\n  def queryExtractor: Regex\n\n  /**\n   * Regular expression to split a query string into keywords.\n   */\n  def keywordSplitter: Regex\n  \n  /**\n   * Symbolic name for the search engine.\n   */\n  def name: String\n\n  def isDefinedAt(uri: String) = searchEngineTest.findFirstIn(uri).isDefined\n\n  def apply(uri: String): SearchEngineQuery = {\n    val query = queryExtractor.findFirstMatchIn(uri).get.group(1)\n    val keywords = keywordSplitter.split(query)\n\n    SearchEngineQuery(query, keywords)\n  }\n}\n```\n\nThe variance is captured by regular expressions. For example here is how Google queries can be captured:\n\n```scala\nclass GoogleSearch extends SearchEngineProcessor {\n  val searchEngineTest = \"www.google.*q=.*\".r\n  val queryExtractor = \"q=([^\u0026]*)\".r\n  val keywordSplitter = \"(%20)|(\\\\+)\".r\n  val name = \"google\"\n}\n```\n\nProcessors can then be elegantly chained as partial functions, then lifted to form a single function returning an optional type, such as in:\n\n```scala\nval searchEngineProcessor = (google orElse bing orElse yahoo orElse wikipedia).lift\n```\n\nThus:\n\n```scala\nsearchEngineProcessor(\"http://www.autosport.com/\")\n=\u003e None\n\nsearchEngineProcessor(\"http://www.google.com/?q=les+muscles+merguez+party\")\n=\u003e Some(SearchEngineQuery(\n     \"les+muscles+merguez+party\",\n     Seq(\"les\", \"muscles\", \"merguez\", \"party\"),\n     \"google\"\n   ))\n```\n\n## Why *xyz*?\n\n* Twitter Finagle: because it is a fairly well-tested asynchronous server stack, and it works on the JVM.\n* Scala: because of Twitter Finagle and that their Java API was much more verbose. At last, there are people who write maintainable Scala code, see [Effective Scala](http://twitter.github.com/effectivescala/).\n* BuildR: because I can't get my head around Scala's SBT, and because Maven was just too rigid for the need.\n* MongoDB: because it is a solid NoSQL database, and that their Scala driver API (Casbah) is so easy.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fjponge%2Fdice-proxy","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fjponge%2Fdice-proxy","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fjponge%2Fdice-proxy/lists"}