https://github.com/indix/dfs-datastores

Dead-simple vertical partitioning, compression, appends, and consolidation of data on a distributed filesystem.
https://github.com/indix/dfs-datastores

Last synced: 9 months ago
JSON representation

Dead-simple vertical partitioning, compression, appends, and consolidation of data on a distributed filesystem.

• Host: GitHub
• URL: https://github.com/indix/dfs-datastores
• Owner: indix
• License: bsd-3-clause
• Created: 2013-04-18T17:05:16.000Z (over 12 years ago)
• Default Branch: master
• Last Pushed: 2021-03-23T05:42:51.000Z (almost 5 years ago)
• Last Synced: 2025-03-25T13:39:31.621Z (10 months ago)
• Language: Java
• Homepage:
• Size: 379 KB
• Stars: 0
• Watchers: 64
• Forks: 6
• Open Issues: 1
• Metadata Files:
  • Readme: README.md
  • License: LICENSE

Awesome Lists containing this project

README

          
[![Build Status](https://snap-ci.com/indix/dfs-datastores/branch/master/build_image)](https://snap-ci.com/indix/dfs-datastores/branch/master)

# dfs-datastores

A dramatically simpler and more powerful way to store records on a distributed filesystem.

To run the tests:

```scala

sbt test

```

To publish the changes

* Update the minor / majar version in version.sbt

* `sbt publish` to publish the artifact to artifactory

### Using Incremental PailConsolidate

Please make sure to add this argument to the PailConsolidate when you want to run the incremental version. The reason for doing so is because by default the S3N implementation does full recursive directory and file fetch without an ability to control the listing limit.

```

-Dfs.s3n.impl=com.indix.hadoop.fs.s3native.LimitedListingS3NFileSystem

```