https://github.com/tuplejump/snackfs-release
The GA Release of SnackFS
https://github.com/tuplejump/snackfs-release
Last synced: 5 months ago
JSON representation
The GA Release of SnackFS
- Host: GitHub
- URL: https://github.com/tuplejump/snackfs-release
- Owner: tuplejump
- Created: 2014-03-19T06:17:11.000Z (about 11 years ago)
- Default Branch: master
- Last Pushed: 2015-07-09T12:32:04.000Z (almost 10 years ago)
- Last Synced: 2024-08-04T01:07:49.309Z (8 months ago)
- Language: Scala
- Size: 761 KB
- Stars: 13
- Watchers: 7
- Forks: 5
- Open Issues: 2
-
Metadata Files:
- Readme: README.md
Awesome Lists containing this project
- awesome-data-engineering - SnackFS - SnackFS is our bite-sized, lightweight HDFS compatible file system built over Cassandra. (File System)
README
[SnackFS @ Calliope](http://tuplejump.github.io/calliope/snackfs.html)
# SnackFS
SnackFS is our bite-sized, lightweight HDFS compatible FileSystem built over Cassandra.
With it's unique fat driver design it requires no additional SysOps or setup on the Cassanndra Cluster. All you have to do is point to your Cassandra cluster and you are ready to go.
As SnackFS was written as a dropin replacement for HDFS, your existing HDFS backed applications not only run as-is on SnackFS, but they also run faster!
SnackFS cluster is also more resilient than a HDFS cluster as there is no SPOF like the NameNode.
## Prerequisites
1. SBT : It can be set up from the instructions [here](http://www.scala-sbt.org/release/docs/Getting-Started/Setup.html#installing-sbt).
2. Cassandra(v1.2.12) : Instructions can be found [here](http://wiki.apache.org/cassandra/GettingStarted). An easier alternative would be using [CCM](https://github.com/pcmanus/ccm)
## Using SnackFS
### Use the binary
* You can download the SnackFS distribution built with [Scala 2.9.x here](http://bit.ly/1eKV1ae) and [Scala 2.10.x here](http://bit.ly/1jI7vVw)
* To add SnackFS to your SBT project use,
For SBT
```scala
"com.tuplejump" %% "snackfs" % "0.6.1-EA"
```
* To add SnackFS to your Maven project use,
with Scala 2.9.3 use,
```xml
com.tuplejump
snackfs_2.9.3
0.6.1-EA
```
And with Scala 2.10.3,
```xml
com.tuplejump
snackfs_2.10
0.6.1-EA
```
### Build from Source
1. Checkout the source from http://github.com/tuplejump/snackfs
2. To build SnackFS distribution run sbt's dist command in the project directory
```
[snackfs]$ sbt dist
```
This will result in a "snackfs-{version}.tgz" file in the "target" directory of "snackfs".
Extract "snackfs-{version}.tgz" to the desired location.
3. Start Cassandra (default setup for snackfs assumes its a cluster with 3 nodes)
4. It is possible to configure the file system by updating core-site.xml.
The following properties can be added.
* snackfs.cassandra.host (default 127.0.0.1)
* snackfs.cassandra.port (default 9160)
* snackfs.consistencyLevel.write (default QUORUM)
* snackfs.consistencyLevel.read (default QUORUM)
* snackfs.keyspace (default snackfs)
* snackfs.subblock.size (default 8 MB (8 * 1024 * 1024))
* snackfs.block.size (default 128 MB (128 * 1024 * 1024))
* snackfs.replicationFactor (default 3)
* snackfs.replicationStrategy (default org.apache.cassandra.locator.SimpleStrategy)
5. SnackFS Shell provides the fs commands similar to Hadoop Shell. For example to create a directory,
```
[Snackfs(extracted)]$bin/snackfs -mkdir snackfs:///random
```
###To build and use with Hadoop
1. Setup Apache Hadoop v1.0.4.(http://hadoop.apache.org/#Getting+Started). The base directory will be referred as 'hadoop-1.0.4' in the following steps.
2. Execute the following commands in the snackfs project directory.
```2.
[snackfs]$ sbt package
```
This will result in a "snackfs_<scala_version>-<version>.jar" file in the "target/scala-<scala_version>" directory of "snackfs".
Copy the jar to 'hadoop-1.0.4/lib'.
3. Copy all the jars in snackfs/lib_managed and scala-library-<scala_version>.jar
(located at '~/.ivy2/cache/org.scala-lang/scala-library/jars') to 'hadoop-1.0.4/lib'.
4. Copy snackfs/src/main/resources/core-site.xml to 'hadoop-1.0.4/conf'
5. Start Cassandra (default setup for snackfs assumes its a cluster with 3 nodes)
6. Hadoop fs commands can now be run using snackfs. For example,
```
[hadoop-1.0.4]$ bin/hadoop fs -mkdir snackfs:///random
```
###To configure logging,
#### In System Environment
Set SNACKFS_LOG_LEVEL in the Shell to one of the following Values
* DEBUG
* INFO
* ERROR
* ALL
* OFF
Default value if not set if ERROR
####In code (for further control/tuning)
If you want your logs in a File, update LogConfiguration.scala like below
```scala
val config = new LoggerFactory("", Option(Level.ALL), List(FileHandler("logs")), true)
```
The arguments for LoggerFactory are
1. node - Name of the logging node. ("") is the top-level logger.
2. level - Log level for this node. Leaving it None implies the parent logger's level.
3. handlers - Where to send log messages.
4. useParents - indicates if log messages are passed up to parent nodes.To stop at this node level, set it to false
Additional logging configuration details can be found [here](https://github.com/twitter/util/tree/master/util-logging#configuring)