https://github.com/bartekdobija/hive-omniture-storage-handler

Hive Storage Handler for Omniture clickstream log transformation
https://github.com/bartekdobija/hive-omniture-storage-handler

Last synced: about 1 month ago
JSON representation

Hive Storage Handler for Omniture clickstream log transformation

• Host: GitHub
• URL: https://github.com/bartekdobija/hive-omniture-storage-handler
• Owner: bartekdobija
• Created: 2016-09-05T19:56:58.000Z (almost 8 years ago)
• Default Branch: master
• Last Pushed: 2016-10-05T00:27:39.000Z (over 7 years ago)
• Last Synced: 2024-03-21T18:15:35.617Z (3 months ago)
• Language: Java
• Homepage:
• Size: 1.88 MB
• Stars: 0
• Watchers: 1
• Forks: 0
• Open Issues: 0
• Metadata Files:
  • Readme: README.md

Lists

• awesome-hive - Omniture

README

        
# Hive Omniture Storage Handler

Hive storage handler and SerDe implementation intended for Omniture clickstream log transformation.

The handler creates all header and metadata information, including data file locations, based on the manifest file provided by Adobe

with logs and used in the HiveQL's TBLPROPERTIES.

This is an early alpha version based on the [omniture-clickstream](https://github.com/bartekdobija/omniture-clickstream) library.

###Configuration:

hdfs://, s3:// and file:// schemes are supported

```sql

-- optionally set Amazon S3 credentials

SET system:aws.accessKeyId=;

SET system:aws.secretKey=;

-- optionally set S3 HTTP proxy configuration

SET system:http.proxyHost=;

SET system:http.proxyPort=;

CREATE EXTERNAL TABLE omniture_table

ROW FORMAT SERDE 'com.github.bartekdobija.omniture.serde.OmnitureSerDe'

STORED BY 'com.github.bartekdobija.omniture.handler.OmnitureStorageHandler'

TBLPROPERTIES (

  "manifest.file" = 'file://omniture-logs/manifest.txt',

  -- "manifest.file" = 'hdfs://namenode/omniture-logs/manifest.txt',

  -- "manifest.file" = 's3://bucket/omniture-logs/manifest.txt',

  "metadata.lookuptable.enabled" = 'true'

);

```

This first version does not implement partition handling.

[![Build Status](https://travis-ci.org/bartekdobija/hive-omniture-storage-handler.svg?branch=master)](https://travis-ci.org/bartekdobija/hive-omniture-storage-handler)