Ecosyste.ms: Awesome
An open API service indexing awesome lists of open source software.
https://github.com/dwyl/meteor-search
🔍 SPIKE of full-text search in MongoDB using Meteor
https://github.com/dwyl/meteor-search
meteor meteorjs mongodb search
Last synced: 3 months ago
JSON representation
🔍 SPIKE of full-text search in MongoDB using Meteor
- Host: GitHub
- URL: https://github.com/dwyl/meteor-search
- Owner: dwyl
- Created: 2014-03-26T14:05:50.000Z (almost 11 years ago)
- Default Branch: master
- Last Pushed: 2015-03-30T15:20:08.000Z (almost 10 years ago)
- Last Synced: 2024-10-01T05:07:31.169Z (4 months ago)
- Topics: meteor, meteorjs, mongodb, search
- Language: JavaScript
- Homepage:
- Size: 215 KB
- Stars: 40
- Watchers: 6
- Forks: 3
- Open Issues: 0
-
Metadata Files:
- Readme: README.md
Awesome Lists containing this project
README
# Meteor Search
## Notes
1. This is **"Spike" Code**. It ***works*** but has **Zero Unit Tests**
so is for ***Learning Purposes Only***!
2. Was written for the *previous version* of Meteor and requires
some tweaking for **Meteor 1.0** (which we are not using)...
## WHY
People want to find content. Good ***search is essential***.
## What
Full-text search. Using ***Native*** MongoDB Commands.
Example app:
## How
### Step 1 - *Get* "*Real*" MongoDB
By default, Meteor starts up its own instance of MongoDB, this does not have
full-text indexing/search, so you need to ***go native***.
If you don't already have a "Real" installation of MongoDB on your system,
install it with **HomeBrew**:
```
brew update
brew install mongodb
```
If you're not using Mac ...
- Linux: http://docs.mongodb.org/manual/tutorial/install-mongodb-on-linux/
- Windows? *ouch*... http://www.geekyprojects.com/tutorials/how-to-use-virtualbox-tutorial/
#### Startup MongoDB with textSearchEnabled=true
In a terminal/console window startup up your mongod databse with the following command:
```
mongod --dbpath ~/code/meteor-search/.meteor/local/db --setParameter textSearchEnabled=true
```
**Notes**:
- **--dbpath** speficies where your data is. Your data directory will be different,
my project is at **~/code/meteor-search** (replace for what ever yours is)
- **--setParameter** lets you enable full-text search at run time. (you can do this in a
config file if you prefer see: http://docs.mongodb.org/manual/reference/configuration-options/)
> Confirm its working by visiting: http://localhost:28017/ (in your browser)
**More info** on enabling text search: http://docs.mongodb.org/manual/tutorial/enable-text-search/
#### Start Meteor with the "Real" MongoDB
Because we are using "real" Mongo we need to specifiy it when starting meteor.
(This only applies while you are developing in production you would set an ENV
)
```
MONGO_URL="mongodb://localhost:27017/meteor" meteor
```
If the app starts up ok, its ***game on***!
(otherwise *submit a bug* to this repo and I will wil try to assist you!)
### Step 2 - Get (Test) Content
#### Seed Content (From Twitter Streaming API)
When you boot this app it will access the Twitter Streaming API and fetch
thousands of tweets for you to search through (*locally*).
(leave it running for a few minutes and you will get 10k posts.
Or a few hours if you want hundreds of thousands to stress test search)
If you want ***LOTS*** of content very quickly, change the KEYWORD to **news**.
If you want ***INSANE*** amounts of (*noisy*) data
(to symulate *high volume*), use:
```
var KEYWORDS = "katie, justin, kim, beyonce, miley, Obama, 1DWorld, OMG, FML, breaking, news";
```
### Step 3 - Index the Text (in MongoDB)
Once you have some content you need to ensure that MongoDB is indexing it.
Thankfully this is quite *easy* with MongoDB's **ensureIndex** method.
In our case we are simply going to index the post's **body** field:
```
db.posts.ensureIndex( { body: "text" },{ background:true } );
```
> There's a **startup** script that does this automatically for you at: **server/indexes.js**
Depending on how much data you have already inserted, this may take some time...
I had 92k posts (tweets) in my DB when I ran it and it took less than 10 seconds!
More detail on **ensureIndex**:
- http://docs.mongodb.org/manual/core/index-text/
- http://docs.mongodb.org/manual/reference/method/db.collection.ensureIndex/
#### Query to Search for a Keyword
```
db.posts.runCommand( "text", { search: "paypal" } )
```
While this works fine in RoboMongo:
Meteor does not support the **runCommand** method:
So ...
>> http://stackoverflow.com/questions/17159626/implementing-mongodb-2-4s-full-text-search-in-a-meteor-app/18258688#18258688
### Step 4 - Highlighting Hashtags (Clickable)
I wrote a simple regular expression to turn hashtagged keywords into links.
Instead of polluting the raw data with links (and bloating our records)
We finde/replace the #keywords at render time (client-side) using
a Handlebars template helper method:
```
// place this code in your main.js or inside an Meteor.isClient block
Handlebars.registerHelper('highlight', function(text) {
var hashtagPattern = /\s*(#\w*)/gi,
link = "/search/",
m, match, matches = [], t, url ='';
// initial check for hashtag in text
if(text.indexOf("#") !== -1) {
// find all #keywords (that have hashtags)
while ( (match = hashtagPattern.exec(text)) ) {
matches.push(match[0]);
}
// replace any #keywords with #keywords
for(var j=0; j < matches.length; j++) {
m = matches[j].replace(/\s/g, "");
// console.log('match',m);
url = link+m;
url = url.replace('#',"").toLowerCase(); // remove hashtag for lookup
t = " "+m+" "; // replace with
replace = new RegExp("\\s*("+m+")", 'gi');
text = text.replace(replace, t);
}
}
return text;
});
```
**Note**: the *link pattern* is *hard-coded* `/search/:keywords` and method is not
chainable so *far from perfect*! Send a pull-request if you improve on it. :-)
## Notes
### Setting Up Full Text Search on MongoHQ
MongoHQ does **not** have (full) text indexing **enabled by default**.
But they were quick to help me when I asked for it:
https://twitter.com/nelsonic/statuses/451758108285489152
You will need to set up your indexes *manually* with a command
(either in your Mongo Client - We use RoboMongo - or the Web Interface)
Once that is set up you are good to go.
### Searching Through Your Posts
In RoboMongo (or what ever MongoDB Client of your choice) use the following
command to search through your collection (on the field(s) you specified as searchable)
```
db.COLLECTION.runCommand( "text", { search: "KEYWORDS" } );
// e.g:
db.posts.runCommand( "text", { search: "learn" } );
```
Search results returned in **0.15 seconds** when I had **327k posts** in
the collection:
### Iron Router
This project uses Iron Router for url routing.
If you are not familiar with it
(you *should* be if you're serious about using Meteor), read:
- Tutorial: http://www.manuel-schoebel.com/blog/iron-router-tutorial
- Docs: https://github.com/EventedMind/iron-router
- [Discover Meteor](https://www.discovermeteor.com/) Uses Iron Router extensively.
Read it for a good step-by-step intro.
- Advanced: https://properapp.com/meteor/advanced-routing-in-meteor-navigation-state-w/#.Uz8BIS9dVX4
```
mrt add iron-router
```
See the **routes.js** file for more detail on how I've wired this up to
accept request in the form: `http://yoursite.com/search/:keywords`
### Research
- MeteorPedia Full-text Search: http://www.meteorpedia.com/read/Fulltext_search
- Using Elastic Search with MongoDB: https://github.com/matteodem/meteor-easy-search
- Meteor (MongoDB) Full-text search: http://stackoverflow.com/questions/14567856/full-text-search-with-meteor-js-and-mongodb (non-answer!)
- Lunr JS "Simple" full-text search in your browser: http://lunrjs.com/
(but our data is in MongoDB so not much use...)
#### Deploying to Heroku?
- Heroku ENVironment variables (in Node.js Apps): https://devcenter.heroku.com/articles/nodejs-support