https://github.com/riskiq/solr-map-reduce

Utilities for creation of Solr indexes using mapreduce
https://github.com/riskiq/solr-map-reduce

hadoop solr

Last synced: 3 months ago
JSON representation

Utilities for creation of Solr indexes using mapreduce

• Host: GitHub
• URL: https://github.com/riskiq/solr-map-reduce
• Owner: RiskIQ
• License: apache-2.0
• Created: 2018-03-13T21:16:16.000Z (almost 7 years ago)
• Default Branch: master
• Last Pushed: 2021-06-14T20:39:59.000Z (over 3 years ago)
• Last Synced: 2024-10-31T21:02:22.936Z (3 months ago)
• Topics: hadoop, solr
• Language: Java
• Size: 1.15 MB
• Stars: 3
• Watchers: 8
• Forks: 1
• Open Issues: 1
• Metadata Files:
  • Readme: README.md
  • License: LICENSE

Awesome Lists containing this project

README

        
# Solr MapReduce

[![Build Status](https://secure.travis-ci.org/RiskIQ/solr-map-reduce.png?branch=master)](http://travis-ci.org/RiskIQ/solr-map-reduce)

## What?

This library provides utilities for creating Solr indexes using mapreduce.

## Why?

The code contained in this repository was initially taken from [version 6.5.1](https://github.com/apache/lucene-solr/tree/releases/lucene-solr/6.5.1) 

of the [apache lucene-solr](https://github.com/apache/lucene-solr) codebase.  Starting with version 6.6.0, these modules 

were dropped from the codebase, as its maintainers were no longer interested in supporting them. 

As heavy users of this library, we feel that these tools are still useful and valuable, and that they should continue to 

be maintained and made available to the open-source community.  

## How?

Maven:

```xml

    com.riskiq

    solr-map-reduce

    8.4.1.2

```

Gradle:

```groovy

compile group: 'com.riskiq', name: 'solr-map-reduce', version: '8.4.1.2'

```

Example lifted from [the original readme](https://github.com/apache/lucene-solr/tree/releases/lucene-solr/6.5.1/solr/contrib/map-reduce):

```bash

# Build an index with map-reduce and deploy it to SolrCloud

source $solr_distrib/example/scripts/map-reduce/set-map-reduce-classpath.sh

$hadoop_distrib/bin/hadoop --config $hadoop_conf_dir jar \

$solr_distrib/dist/solr-map-reduce-*.jar -D 'mapred.child.java.opts=-Xmx500m' \

-libjars "$HADOOP_LIBJAR" --morphline-file readAvroContainer.conf \

--zk-host 127.0.0.1:9983 --output-dir hdfs://127.0.0.1:8020/outdir \

--collection $collection --log4j log4j.properties --go-live \

--verbose "hdfs://127.0.0.1:8020/indir"

```