{"id":24587149,"url":"https://github.com/urbanairship/datacube","last_synced_at":"2025-04-05T20:07:29.300Z","repository":{"id":2664499,"uuid":"3655776","full_name":"urbanairship/datacube","owner":"urbanairship","description":"Multidimensional data storage with rollups for numerical data","archived":false,"fork":false,"pushed_at":"2024-01-13T00:30:22.000Z","size":4702,"stargazers_count":266,"open_issues_count":11,"forks_count":61,"subscribers_count":133,"default_branch":"master","last_synced_at":"2025-03-29T19:03:47.634Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"http://urbanairship.com","language":"Java","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/urbanairship.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"COPYING","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2012-03-08T01:55:05.000Z","updated_at":"2024-11-28T18:16:41.000Z","dependencies_parsed_at":"2025-02-14T12:10:30.189Z","dependency_job_id":"f1eb9914-03dd-4d7e-952c-30dcc5e868a1","html_url":"https://github.com/urbanairship/datacube","commit_stats":null,"previous_names":[],"tags_count":36,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/urbanairship%2Fdatacube","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/urbanairship%2Fdatacube/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/urbanairship%2Fdatacube/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/urbanairship%2Fdatacube/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/urbanairship","download_url":"https://codeload.github.com/urbanairship/datacube/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":247393569,"owners_count":20931812,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2025-01-24T06:16:52.829Z","updated_at":"2025-04-05T20:07:29.267Z","avatar_url":"https://github.com/urbanairship.png","language":"Java","funding_links":[],"categories":["数据库"],"sub_categories":[],"readme":"## Introduction\n\nA data cube is an abstraction for counting things in complicated ways ([Wikipedia](http://en.wikipedia.org/wiki/OLAP_cube)). This project is a Java implementation of a data cube backed by a pluggable database backend.\n\nThe purpose of a data cube is to store aggregate information about large numbers of data points. The data cube stores aggregate information about interesting subsets of the input data points. For example, if you're writing a web server log analyzer, your input points could be log lines, and you might be interested in keeping a count for each browser type, each browser version, OS type, OS version, and other attributes. You might also be interested in counts for a particular *combination* of (browserType,browserVersion,osType), (browserType,browserVersion,osType,osVersion), etc. It's a challenge to quickly add and change counters without wasting time writing database code and reprocessing old data into new counters. A data cube helps you keep these counts. You declare what you want to count, and the data cube maintains all the counters as you supply new data points.\n\nA bit more mathily, if your input data points have N attributes, then the number of counters you may have to store is the product of the cardinalities of all N attributes in the worst case. The goal of the datacube project is to help you maintain these counters in a simple declarative way without any nested switch statements or other unpleasantness.\n\nUrban Airship uses the datacube project to support its analytics stack for mobile apps. We handle about ~10K events per second per node.\n\nRequires JDK 1.6.\n\n## Features\n - Performance: high-speed asynchronous batching IO backend\n - Bulk loading with Hadoop MapReduce\n - Pluggable database interface\n\n\n## IO\n\nEach input data point may affect multiple counts in the data cube. For example, if you're counting events with a timestamp, a single event may increment the count for its hour, day, month, and year, ending up with four increments that must be applied to the database. Updating the database for each of these increments wouldn't scale to thousands of events per second, so we use the standard trick of batching counter updates in the client. When an input data point is given to the data cube, it updates a batch in memory for each of the affected counters. Periodically the batches are flushed to the backing database. If a single counter is incremented multiple times in the same batch, the increments are combined into a single database update.\n\nTODO parameters to tune, implementation details to explain parameters\n\n## Bulk loading / backfilling\n\n\n## Database backend\n\nA data cube can be backed by any database that supports a key-value interface and allows iterating over keys. To add support for a new database backend, implement the DbHarness interface and optionally the IdService interface. See HBaseDbHarness.java and HBaseIdService.java for examples. If you add support for a new database, we'd love to have you contribute your work back into the datacube project.\n\nCurrently HBase is the only supported backing database.\n\n## Example\n\n```java\nIdService idService = new CachingIdService(5, new MapIdService());\nConcurrentMap\u003cBoxedByteArray,byte[]\u003e backingMap = \n        new ConcurrentHashMap\u003cBoxedByteArray, byte[]\u003e();\n        \nDbHarness\u003cLongOp\u003e dbHarness = new MapDbHarness\u003cLongOp\u003e(backingMap, LongOp.DESERIALIZER, \n        CommitType.READ_COMBINE_CAS, idService);\n\nHourDayMonthBucketer hourDayMonthBucketer = new HourDayMonthBucketer();\n\nDimension\u003cDateTime\u003e time = new Dimension\u003cDateTime\u003e(\"time\", hourDayMonthBucketer, false, 8);\nDimension\u003cString\u003e zipcode = new Dimension\u003cString\u003e(\"zipcode\", new StringToBytesBucketer(), \n        true, 5);\n        \nDataCubeIo\u003cLongOp\u003e cubeIo = null;\nDataCube\u003cLongOp\u003e cube;\n        \nRollup hourAndZipRollup = new Rollup(zipcode, time, HourDayMonthBucketer.hours);\nRollup dayAndZipRollup = new Rollup(zipcode, time, HourDayMonthBucketer.days);\nRollup hourRollup = new Rollup(time, HourDayMonthBucketer.hours);\nRollup dayRollup = new Rollup(time, HourDayMonthBucketer.days);\n        \nList\u003cDimension\u003c?\u003e\u003e dimensions =  ImmutableList.\u003cDimension\u003c?\u003e\u003eof(time, zipcode);\nList\u003cRollup\u003e rollups = ImmutableList.of(hourAndZipRollup, dayAndZipRollup, hourRollup,\n        dayRollup);\n        \ncube = new DataCube\u003cLongOp\u003e(dimensions, rollups);\n\ncubeIo = new DataCubeIo\u003cLongOp\u003e(cube, dbHarness, 1, Long.MAX_VALUE, SyncLevel.FULL_SYNC);\n        \nDateTime now = new DateTime(DateTimeZone.UTC);\n        \n// Do an increment of 5 for a certain time and zipcode\ncubeIo.writeSync(new LongOp(5), new WriteBuilder(cube)\n        .at(time, now)\n        .at(zipcode, \"97201\"));\n        \n// Do an increment of 10 for the same zipcode in a different hour of the same day\nDateTime differentHour = now.withHourOfDay((now.getHourOfDay()+1)%24);\ncubeIo.writeSync(new LongOp(10), new WriteBuilder(cube)\n        .at(time, differentHour)\n        .at(zipcode, \"97201\"));\n\n// Read back the value that we wrote for the current hour, should be 5 \nOptional\u003cLongOp\u003e thisHourCount = cubeIo.get(new ReadBuilder(cube)\n         .at(time, HourDayMonthBucketer.hours, now)\n        .at(zipcode, \"97201\"));\nAssert.assertTrue(thisHourCount.isPresent());\nAssert.assertEquals(5L, thisHourCount.get().getLong());\n        \n// Read back the value we wrote for the other hour, should be 10\nOptional\u003cLongOp\u003e differentHourCount = cubeIo.get(new ReadBuilder(cube)\n        .at(time, HourDayMonthBucketer.hours, differentHour)\n        .at(zipcode, \"97201\"));\nAssert.assertTrue(differentHourCount.isPresent());\nAssert.assertEquals(10L, differentHourCount.get().getLong());\n\n// The total for today should be the sum of the two increments\nOptional\u003cLongOp\u003e todayCount = cubeIo.get(new ReadBuilder(cube)\n        .at(time, HourDayMonthBucketer.days, now)\n        .at(zipcode, \"97201\"));\nAssert.assertTrue(todayCount.isPresent());\nAssert.assertEquals(15L, todayCount.get().getLong());\n```\n\n## Quickstart\n\nAdd datacube to your maven build. (TODO upload to a public repo)\n\nFigure out your dimensions. These are the attributes of your incoming data points. Some examples of dimensions are time, latitude, and browser version. Create one Dimension object for each dimension. Use these dimensions to instantiate a data cube\n\nYou can skip using an IdService for now. This is an optional optimization for dimensions have that have long coordinates with low cardinality. For example, if you have a \"country\" dimension, the country name might be dozens of characters long, but there are only a few bytes of entropy. You could assign integers to countries and only use a few bytes to represent a country coordinate.\n\nCreate one Rollup object for each kind of counter you want to keep. For example, if you want to keep a counter of web hits by (time,browser), this would be one Rollup object.\n\nCreate a DbHarness object that will handle writing to the database. Currently, only HBaseDbHarness exists.\n\nCreate a DataCubeIo object, passing your DataCube object and your DbHarness.\n\nInsert data points into your cube by passing them to DataCubeIo.writeSync().\n\nRead back your rollup values by calling DataCubeIo.get().\n\n## Building\n\nThe POM is configured to build with specific versions of HBase and Hadoop. If your versions differ from those in the POM, you can override the versions by passing hbaseVersion and hadooopVersion. For example:\n\n```\n$ mvn -DhbaseVersion=0.90.6 -DhadoopVersion=0.20.2 package\n```\n\nThe build artifact jars each have a classifier of the form hbase${hbaseVersion}-hadoop${hadoopVersion}, so you can depend on them in another project by doing something like:\n\n```\n\u003cdependency\u003e\n  \u003cgroupId\u003ecom.urbanairship\u003c/groupId\u003e\n  \u003cartifactId\u003edatacube\u003c/artifactId\u003e\n  \u003cversion\u003e${datacube.version}\u003c/version\u003e\n  \u003cclassifier\u003ehbase0.94.0-hadoop1.0.3\u003c/classifier\u003e\n\u003c/dependency\u003e\n```\n\nThe main build artifact jar (without a classifer) uses the default HBase and Hadoop versions, which may change between datacube releases.\n\nYou can pass -DhadoopVersion and -DhbaseVersion to maven to choose which version of Haodop and HBase to depend on. Hadoop 2 is not yet supported since the artifact names are different. For example: \n\nTo build against your own version of HBase or Hadoop, just add your repository to the POM and pass -DhbaseVersion or -DhadoopVersion to the datacube build.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Furbanairship%2Fdatacube","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Furbanairship%2Fdatacube","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Furbanairship%2Fdatacube/lists"}