{"id":20124919,"url":"https://github.com/ezhulenev/marketdb","last_synced_at":"2025-05-06T17:34:00.102Z","repository":{"id":2509268,"uuid":"3484570","full_name":"ezhulenev/marketdb","owner":"ezhulenev","description":"Market time series database","archived":false,"fork":false,"pushed_at":"2015-01-14T15:18:17.000Z","size":466,"stargazers_count":31,"open_issues_count":0,"forks_count":12,"subscribers_count":5,"default_branch":"master","last_synced_at":"2023-03-22T17:12:34.017Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"","language":"Scala","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/ezhulenev.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2012-02-19T09:56:16.000Z","updated_at":"2022-03-05T06:42:50.000Z","dependencies_parsed_at":"2022-09-07T03:11:21.938Z","dependency_job_id":null,"html_url":"https://github.com/ezhulenev/marketdb","commit_stats":null,"previous_names":[],"tags_count":0,"template":null,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ezhulenev%2Fmarketdb","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ezhulenev%2Fmarketdb/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ezhulenev%2Fmarketdb/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ezhulenev%2Fmarketdb/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/ezhulenev","download_url":"https://codeload.github.com/ezhulenev/marketdb/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":224517388,"owners_count":17324407,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-13T19:53:54.803Z","updated_at":"2024-11-13T19:53:56.248Z","avatar_url":"https://github.com/ezhulenev.png","language":"Scala","funding_links":[],"categories":["数据库"],"sub_categories":["Spring Cloud框架"],"readme":"# MarketDb\n\n## Where to get it\n\nTo get the latest version of the library, add the following to your SBT build:\n\n``` scala\nresolvers += \"Scalafi Bintray Repo\" at \"http://dl.bintray.com/ezhulenev/releases\"\n```\n\nAnd use following library dependency:\n\n```\nlibraryDependencies +=  \"com.scalafi.marketdb\" %% \"marketdb-api\" % \"0.0.1\"\n```\n\n## What is MarketDb\n\nMarketDb is a distributed, scalable Time Series Database written on top of HBase, inspired by [OpenTSDB](https://github.com/OpenTSDB/opentsdb), focused on Financial Market time series. MarketDb was written to address a common need: store, index and serve market time series (trades and orders) collected from market data providers and make this data conveniently accessible for backtesting and strategies simulation.\n\nMarketDb written in scala and provides functional [Iteratee](http://jsuereth.com/scala/2012/02/29/iteratees.html) style timeseries processing.\n\n## Quick Start\n1. Install \u0026 Run [HBase](http://hbase.apache.org/). (How to install HBase on Windows could be found [here](http://hbase.apache.org/))\n2. Install \u0026 Run [Kestrel](https://github.com/robey/kestrel) - distributed message queue which is used to deliver market data to MarketDb\n3. Setup environment variables:\n\n        HBASE_HOME=[path to HBase installation directory]        \n        TRADES_TABLE=[table name for trades time series, default='market-trades']\n        ORDERS_TABLE=[table name for orders time series, default='market-orders']\n        UID_TABLE=[table name for generated UID, default='market-uid']\n        COMPRESSION=[HBase compression, default='NONE']        \n   More about compression levels could be found [here](http://wiki.apache.org/hadoop/UsingLzoCompression)\n   \n\n4. Checkout \u0026 build MarketDb\n\n        $ git clone git://github.com/Ergodicity/marketdb.git\n        $ cd ./marketdb\n        $ sbt package\n\n5. Execute commands from MarketDb root directory:\n\n        $ ./install/create_tables.sh - create tables\n        $ ./install/create_test_tables.sh - create tables for integration tests (with prefix 'test-')\n    \n    \n6. Create MarketDb configuration.  \n   MarketDb uses [Ostrich](https://github.com/twitter/ostrich) for configuration, hence you need to create Scala configuration before running. \n\n   ##### Configuration for local HBase \u0026 Kestrel\n\n        import com.ergodicity.marketdb.core.MarketDb\n        import com.ergodicity.marketdb.{MarketDbConfig, KestrelLoader, KestrelConfig}\n        import com.twitter.ostrich.admin.config._\n        import java.net.InetSocketAddress\n        \n        new MarketDbConfig {\n          // HBase connection: ZookeeperQuorumUrl\n          connection = Connection(\"localhost\")\n        \n          // HBase tables\n          tradesTable = \"market-trades\"\n          ordersTable = \"market-orders\"\n          uidTable = \"market-uid\"\n        \n          // Socked address used for exposing MarketDbService\n          socketAddress = Some(new InetSocketAddress(10333))\n          \n          \n          // Connection to running Kestrel (possibly multiple applications)\n          // And queue names to consume Trades \u0026 Orders\n          val kestrelLoaderService = (marketDB: MarketDb) =\u003e {\n            new KestrelLoader(marketDB, KestrelConfig(Seq(\"localhost:22133\"), \n                              \"trades\", \"orders\", hostConnectionLimit = 30))\n          }\n    \n          // Services to start right after MarketDb itself\n          services = Seq(kestrelLoaderService)\n        \n          // HTTP port for Ostrich admin console\n          admin.httpPort = 9000\n          \n          // Ostrich statistics\n          admin.statsNodes = new StatsConfig {\n            reporters = new JsonStatsLoggerConfig {\n              loggerName = \"stats\"\n              serviceName = \"marketDB\"\n            } :: new TimeSeriesCollectorConfig\n          }\n        }\n\n\n\n7. Run MarketDb\n\n            $ java -jar marketdb-0.1-SNAPSHOT.jar -f config.scala\n            \n   At this point you can access the MarketDb's admin page through 127.0.0.1:9000 (if it's running on your local machine).\n\n\n## Filling MarketDb\n\nMarketDb consumes market data from Kestrel queues used in configuration. How to write data into Kestrel queues could be found in [finagle-kestrel api documentation](http://twitter.github.com/finagle/api/finagle-kestrel/index.html) and [Finagle examples](https://github.com/twitter/finagle/blob/master/README.md#api-reference-documentation)\n\nFor particular code examples you may also take a look into marketdb-loader project which provides convinient way to load historical data into MarketDb from \"Russian Trading System\" Stock Exchange [trades result archive](http://www.rts.ru/s638).\n\n\n## Querying MarketDb\n\nMarketDb exposes binary API using [Twitter's Finagle](https://github.com/twitter/finagle) library to get Timeseries for further processing.\n\nTimeseries processing relies on Enumeration based I/O with Iteratees, based on [Scalaz 6.0.4 Iteratees](https://github.com/scalaz/scalaz/blob/release/6.0.4/example/src/main/scala/scalaz/example/ExampleIteratee.scala). Rúnar Óli wrote a good [article](http://apocalisp.wordpress.com/2010/10/17/scalaz-tutorial-enumeration-based-io-with-iteratees/) about this approach for composable \u0026 functional input processing.\n\n##### Scala MarketDb Client Implementation\n    import com.ergodicity.marketdb.api._\n    import com.ergodicity.marketdb.model._\n    import com.ergodicity.marketdb.iteratee.TimeSeriesEnumerator._\n    import java.net.InetSocketAddress\n    import com.twitter.finagle.builder.ClientBuilder\n    import com.twitter.util.Future\n    import org.joda.time.DateTime\n    import org.scala_tools.time.Implicits._\n    \n    \n    object MarketDbClient {\n      val market = Market(\"FORTS\")\n      val security = Security(\"RTS-3.13\")\n      val interval = new DateTime(2013, 1, 8, 10, 0) to new DateTime(2013, 1, 8, 19, 0)\n    \n      def main(args: Array[String]) {\n        // Wrap the raw MarketDb service in a Client decorator. The client provides\n        // a convenient procedural interface for accessing the MarketDb server.\n        val marketdb = ClientBuilder()\n          .codec(MarketDbCodec)\n          .hosts(new InetSocketAddress(10033))\n          .hostConnectionLimit(1)\n          .build()\n    \n        val config: Future[MarketDbConfig] = marketdb(GetMarketDbConfig).map(_.asInstanceOf[MarketDbConfig])\n        val trades: Future[Trades] = marketdb(ScanTrades(market, security, interval)).map(_.asInstanceOf[Trades])\n        \n        (config join trades) onSuccess {\n          case (MarketDbConfig(connection), Trades(timeSeries)) =\u003e\n            implicit val marketDbReader = new MarketDbReader(connection)\n    \n            val enumerator = TimeSeriesEnumerator(timeSeries)\n            val iterv = enumerator.enumerate(MarketIteratees.counter[TradePayload])\n            val count = iterv.map(_.run)()\n            \n            System.out.println(\"Trades count = \" + count + \", for given interval = \" + interval)\n    \n        } onFailure {\n          case cause =\u003e System.out.println(\"Failed with cause: \" + cause)\n        } ensure {\n          marketdb.release()\n        }\n      }\n    }\n    \n    \n## Monitoring MarketDb\nYou can monitor MarketDb status and statistics using Ostrich admin console, which is configured using MarketDbConfig (admin.httpPort). See \u003ca href=\"#configuration-for-local-hbase--kestrel\"\u003eexample cofniguration\u003c/a\u003e\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fezhulenev%2Fmarketdb","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fezhulenev%2Fmarketdb","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fezhulenev%2Fmarketdb/lists"}