https://github.com/neo4jrb/kvitter
Example application using neo4j.rb
https://github.com/neo4jrb/kvitter
Last synced: 5 months ago
JSON representation
Example application using neo4j.rb
- Host: GitHub
- URL: https://github.com/neo4jrb/kvitter
- Owner: neo4jrb
- Created: 2011-11-01T21:32:23.000Z (over 13 years ago)
- Default Branch: master
- Last Pushed: 2013-01-29T07:57:29.000Z (about 12 years ago)
- Last Synced: 2024-03-25T21:13:11.926Z (about 1 year ago)
- Language: JavaScript
- Size: 279 KB
- Stars: 45
- Watchers: 5
- Forks: 3
- Open Issues: 2
-
Metadata Files:
- Readme: README.rdoc
Awesome Lists containing this project
README
== Neo4j.rb example application
This application shows how to use neo4j together with rails 3.1 by parsing Twitter feeds and creating a social graph.
It first shows some basic operations which you are probably used to from the ActiveRecord/ActiveModel API.
In the last section we take real advantage of the neo4j engine by implementing a recommendation algorithm for finding new twitter users.
Have fun and feel free to clone it.
=== Installation
Make sure you have Java JDK 1.6+ installed
Install latest JRuby, example
rvm use jruby-1.6.7
Install Rails (>= 3.1.1)
gem install rails
=== 1. Create a rails project
Use my rails template which will disable active record and enable neo4j instead
rails new kvitter -m http://andreasronge.github.com/neo4j/rails.rb
Edit the Gemfile and add the twitter gem
cd kvitter
emacs Gemfile # and add
gem 'twitter', '1.7.2'
Download all the dependencies
bundle
=== 2. Scaffold
Run the following commands:
rails generate scaffold Tweet text:string link:string date:datetime tweet_id:string --indices tweet_id date text --has_n tags mentions links --has_one tweeted_by:tweeted
rails generate scaffold User twid:string link:string --indices twid --has_n tweeted follows knows used_tags mentioned_from:mentions
rails generate scaffold Link url:string --indices url --has_n tweets:links short_urls:redirected_link --has_one redirected_link
rails generate scaffold Tag name:string --indices name --has_n tweets:tags used_by_users:used_tags
There is nothing magical happening here like neo4j configuration or migrations.
It only creates plain new Ruby classes (models, controllers and views) and routing.
The relationships and properties are only specified in the model classes.
=== 3. Start Rails
Test the basic crud operations
rails s
Open browser: http://localhost:3000/tags
=== 4 Search using Twitter API
Add the following:
==== app/controllers/tag_controller.rb:
The following code does a twitter search and parses the result.
It creates and connects the Tweet, Link, User and Tag model classes.
def search
@tag = Tag.find(params[:id])
search = Twitter::Search.new
result = search.hashtag(@tag.name)
curr_page = 0
while curr_page < 2 do
result.each do |item|
parsed_tweet_hash = Tweet.parse(item)
next if Tweet.find_by_tweet_id(parsed_tweet_hash[:tweet_id])
tweet = Tweet.create!(parsed_tweet_hash)
twid = item['from_user'].downcase
user = User.find_or_create_by(:twid => twid)
user.tweeted << tweet
user.save
parse_tweet(tweet, user)
end
result.fetch_next_page
curr_page += 1
end
redirect_to @tag
end
def parse_tweet(tweet, user)
tweet.text.gsub(/(@\w+|https?:\/\/[a-zA-Z0-9\-\.~\:\?#\[\]\!\@\$&,\*+=;,\/]+|#\w+)/).each do |t|
case t
when /^@.+/
t = t[1..-1].downcase
next if t.nil?
other = User.find_or_create_by(:twid => t)
user.knows << other unless t == user.twid || user.knows.include?(other)
user.save
tweet.mentions << other
when /#.+/
t = t[1..-1].downcase
tag = Tag.find_or_create_by(:name => t)
tweet.tags << tag unless tweet.tags.include?(tag)
user.used_tags << tag unless user.used_tags.include?(tag)
user.save
when /https?:.+/
link = Link.find_or_create_by(:url => t)
tweet.links << (link.redirected_link || link)
end
end
tweet.save!
end
==== app/models/tweet.rb:
Change index on text to fulltext lucene:
property :text, :type => String, :index => :fulltext
Add a to_s and parse method and change the index type of text to fulltext (we need that later on, see below).
def to_s
text.gsub(/(@\w+|https?\S+|#\w+)/,"").strip
end
def self.parse(item)
{:tweet_id => item['id_str'],
:text => item['text'],
:date => Time.parse(item['created_at']),
:link => "http://twitter.com/#{item['from_user']}/statuses/#{item['id_str']}"
}
end
Notice : in Neo4j it is not necessary to specify the types of properties. By setting :type => String we force that each Tweet object will store the property 'type' as strings.
==== config/routes.rb:
change
resource :tags
to
resources :tags do
get :search, :on => :member
end
==== app/views/tags/show.html.erb
Add a button to the view:
<%= button_to "Search", [:search, @tag], :method => :get %>
Test the application now by opening a browser http://localhost:3000/tags
Create a new tag
and press the button 'search'
You will now found tweets, users and links
=== Follow URL shortenings
When you looked at all the links, most of them are short urls like http://t.co
Since we are more interested in the real link and who has tweeted about them
we must follow the URL by doing a HTTP head request.
We use a before save callback to create a the real link which correspond to where
the short url is directed to. The short link and the real link are connected in a redirected_link relationship.
==== app/models/link.rb
class Link < Neo4j::Rails::Model
property :url, :type => String, :index => :exact
has_n(:tweets).from(:links)
has_n(:short_urls).from(:redirected_link)
has_one :redirected_link
# Add the following:
before_save :create_redirect_link
SHORT_URLS = %w[t.co bit.ly ow.ly goo.gl tiny.cc tinyurl.com doiop.com readthisurl.com memurl.com tr.im cli.gs short.ie kl.am idek.net short.ie is.gd hex.io asterl.in j.mp].to_set
def to_s
url
end
private
def self.short_url?(url)
domain = url.split('/')[2]
domain && SHORT_URLS.include?(domain)
end
def create_redirect_link
return if !self.class.short_url?(url)
uri = URI.parse(url)
http = Net::HTTP.new(uri.host, uri.port)
http.read_timeout = 200
req = Net::HTTP::Head.new(uri.request_uri)
res = http.request(req)
redirect = res['location']
if redirect && url != redirect
self.redirected_link = Link.find_or_create_by(:url => redirect.strip)
end
rescue Timeout::Error
puts "Can't acccess #{url}"
rescue Error
puts "Can't call #{url}"
rescue Net::HTTPBadResponse
puts "Bad response for #{url}"
end
==== app/views/links/show.html.erb
In order to show both outgoing redirected_link and incoming redirected_link (by using short_urls method) add the following:
Short Urls:
<% @link.short_urls.each do |link| %>
<%= link_to link, link %>
<% end %>
<% if @link.redirected_link %>
Redirects to
<%= link_to @link.redirected_link, @link.redirected_link %>
<% end %>
=== Don't display URL shortening
The index page shows all the links, including links like http://bit.ly and the the real one.
To only return the real URLs we can use rules, which is a bit similar to scope in ActiveRecord.
==== app/models/links.rb
rule(:real) { redirected_link.nil?}
This means that it will group all links under the rule :real which does not have a redirected_link
To return all those nodes, use the class method #real. Notice you can do some really interesting queries using rules and cypher together, see "Rules-Cypher":https://github.com/andreasronge/neo4j/wiki/Neo4j%3A%3AWrapper-Rules-and-Functions
Btw, the Neo4j::Rails::Model#all method is also implemented as a rule.
You can also chain rules, just like scopes in Active Record.
==== app/controllers/links_controller.rb
def index
@links = Link.real
respond_to do |format|
format.html # index.html.erb
format.json { render :json => @links }
end
end
Since the rules do work by creating relationships when new nodes are created/updated/deleted we must do another
search or stop the rails server and delete the database 'db/neo4j-developement'
=== Pagination
The list of all tweets (http://localhost:3000/tweets ) does not look good. It needs some pagination.
Add the two gem in Gemfile:
gem 'neo4j-will_paginate', :git => 'git://github.com/andreasronge/neo4j-will_paginate.git'
gem 'will_paginate'
==== app/controllers/tweet_controller.rb
Neo4j.rb comes included with the will_paginate gem.
Change the index method
@tweets = Tweet.all
to
@tweets = Tweet.all.paginate(:page => params[:page], :per_page => 10)
Pagination is support for all traversals and lucene queries.
==== app/views/tweets/index.html.erb
Add the following line before the table:
<%= will_paginate(@tweets) %>
=== Searching and Sorting
Lets say we want to sort the tweets by the text. Lucene has two types of indexes: exact and fulltext.
The exact index is perfect for keywords while the fulltext is for longer texts.
We have already changed the index of text to fulltext for the app/models/tweets.rb
See http://lucene.apache.org/java/3_0_0/queryparsersyntax.html for the query syntax.
==== app/views/tweets/index.html.erb
Add the following form before the table
<%= form_for(:tweets, :method => :get) do |f| %>
<%= text_field_tag :query %>
<%= f.submit "Search" %>
<% end %>
==== app/controllers/tweets_controller.rb
In the index method we now should handle the query parameter.
Add the following before the respond_to.
def index
query = params[:query]
if query.present?
@tweets = Tweet.all("text:#{query}", :type => :fulltext).paginate(:page => params[:page], :per_page => 10)
else
@tweets = Tweet.all.paginate(:page => params[:page], :per_page => 10)
end
# respond_to ...
end
Test it !
=== Add SVG Visualization !
Would it not be cool to visualize who knows who ?
==== Download the D3 Javascript library
Download the D3 version 2.4.6 javascript library from https://github.com/mbostock/d3/archives/master.
Unzip it and move the following files to the app/assets/javascript folder
* d3.geom.js
* d3.js
* d3.layout.js
==== app/controllers/users_controller.rb
Create a JSON API needed by the javascript
def index
@users = User.all
respond_to do |format|
format.html # index.html.erb
format.json do
nodes = @users.map{|u| {:name => u.twid, :value => u.tweeted.size}}
links = []
@users.each do |user|
links += user.knows.map {|other| { :source => nodes.find_index{|n| n[:name] == user.twid}, :target => nodes.find_index{|n| n[:name] == other.twid}}}
end
render :json => {:nodes => nodes, :links => links}
end
end
end
Test the API, open a browser http://localhost:3000/users.json
it should return something like this:
{"nodes":[{"name":"geoaxis","value":1},{"name":"peterneubauer","value":1},{"name":"dolugen","value":1},{"name":"neo4j","value":8},{"name":"emileifrem","value":1},{"name":"linnbar_consult","value":1},{"name":"skillsmatter","value":0},{"name":"swgoof","value":1},{"name":"bytor99999","value":0},{"name":"teropaananen","value":1},{"name":"tobyorourke","value":1},{"name":"sannegrinovero","value":1},{"name":"emmanuelbernard","value":0},{"name":"jimwebber","value":0},{"name":"ianrobinson","value":0},{"name":"mesirii","value":1},{"name":"kings13y","value":0},{"name":"chitrasen","value":1},{"name":"einarhreindal","value":1},{"name":"smunchang","value":1},{"name":"technige","value":1},{"name":"josh_adell","value":1},{"name":"mmuekk","value":1},{"name":"asbkar","value":1},{"name":"noppanit","value":1},{"name":"roddare","value":1},{"name":"tanyaespe","value":1},{"name":"erikusaj","value":1},{"name":"druidjaidan","value":1}],"links":[{"source":0,"target":1},{"source":2,"target":3},{"source":2,"target":4},{"source":3,"target":7},{"source":3,"target":4},{"source":3,"target":8},{"source":3,"target":9},{"source":3,"target":10},{"source":3,"target":11},{"source":3,"target":12},{"source":3,"target":13},{"source":3,"target":14},{"source":3,"target":1},{"source":3,"target":16},{"source":4,"target":9},{"source":5,"target":6},{"source":7,"target":4},{"source":9,"target":4},{"source":10,"target":4},{"source":11,"target":12},{"source":11,"target":13},{"source":11,"target":14},{"source":15,"target":9},{"source":15,"target":4},{"source":17,"target":9},{"source":17,"target":4},{"source":18,"target":6},{"source":19,"target":4},{"source":19,"target":3},{"source":20,"target":21},{"source":22,"target":6},{"source":23,"target":6},{"source":24,"target":1},{"source":25,"target":1},{"source":26,"target":4},{"source":26,"target":3},{"source":27,"target":6},{"source":28,"target":6}]}
==== app/assets/javascripts/users.js
Add the following coffeescript (be careful with the indentation):
# Place all the behaviors and hooks related to the matching controller here.
# All this logic will automatically be available in application.js.
# You can use CoffeeScript in this file: http://jashkenas.github.com/coffee-script/
$(document).ready ->
w = 1260
h = 1300
fill = d3.scale.category20()
vis = d3.select("#graph").append("svg:svg").attr("width", w).attr("height", h)
d3.json("/users.json", (json) ->
force = d3.layout.force()
.charge(-320)
.linkDistance(160)
.nodes(json.nodes)
.links(json.links)
.size([w, h])
.start()
link = vis.selectAll("line.link")
.data(json.links)
.enter().append("svg:line")
.attr("class", "link")
.style("stroke-width", (d) -> Math.sqrt(d.value))
.attr("x1", (d) -> d.source.x)
.attr("y1", (d) -> d.source.y)
.attr("x2", (d) -> d.target.x)
.attr("y2", (d) -> d.target.y)
node = vis.selectAll("g.node")
.data(json.nodes)
.enter().append("svg:g")
.attr("transform", (d) -> "translate(" + d.x + "," + d.y + ")")
.attr("class", "node")
.call(force.drag)
node.append("svg:circle")
.attr("r", (d) -> if d.value > 25 then 50 else d.value*2 + 5)
.style("fill", (d) -> '#fea')
node.append("svg:title").text((d) -> d.name)
node.append("svg:text")
.attr("text-anchor", "middle")
.attr("dy", ".3em")
.text((d) -> d.name)
vis.style("opacity", 1e-6)
.transition()
.duration(0)
.style("opacity", 1)
force.on("tick", ->
link.attr("x1", (d) -> d.source.x).attr("y1", (d) -> d.source.y).attr("x2", (d) -> d.target.x).attr("y2", (d) -> d.target.y)
node.attr("transform", (d) -> "translate(" + d.x + "," + d.y + ")")
)
)
==== app/views/users/index.html.erb
Add the following at the bottom of the file
==== app/assets/javascripts/application.js
Make sure things are loading in the correct order.
//= require jquery
//= require jquery_ujs
//= require d3
//= require d3.geom
//= require d3.layout
//= require users
==== app/assets/stylesheets/application.css
circle.node {
stroke: #fff;
stroke-width: 1.5px;
}
line.link {
stroke: #999;
stroke-opacity: .6;
}
==== Test it
Open a browser http://localhost:3000/users and scroll down
=== Better Navigation for Users
==== app/views/users/show.erb
Add the following before the link_to lines
Used tags
<% @user.used_tags.each do |tag| %>
<%= link_to tag.name, tag %>
<% end %>
Knows:
<% @knows.each do |user| %>
<%= link_to user, user %>
<% end %>
<%= will_paginate(@knows) %>
Mentioned from:
<% @mentioned_from.each do |tweet| %>
<%= link_to tweet, tweet %>
<% end %>
Tweets
<% @user.tweeted.each do |tweet| %>
<%= link_to tweet, tweet %>
<% end %>
==== app/controllers/users_controller.rb
Change the show function to:
def show
@user = User.find(params[:id])
@knows = @user.knows.paginate(:page => params[:page], :per_page => 10)
@mentioned_from = @user.mentioned_from
respond_to do |format|
format.html # show.html.erb
format.json { render :json => @user }
end
end
Notice that we do pagination of a traversal (knows).
=== Better Navigation for Tweets
==== app/views/tweets/show.html.rb
Show the relationships from and to a tweet:
Text:
<%= @tweet.text %>
Link:
<%= link_to @tweet.link, @tweet.link %>
Date:
<%= @tweet.date %>
Tweet:
<%= @tweet.tweet_id %>
Tweeted by
<%= link_to @tweet.tweeted_by, @tweet.tweeted_by %>
Tags:
<% @tweet.tags.each do |tag| %>
<%= link_to tag.name, tag %>
<% end %>
Links:
<% @tweet.links.each do |link| %>
<%= link_to link, link %>
<% end %>
=== Better Navigation for Tags
==== app/views/tags/show.html.erb
Tweets:
<% @tweets.each do |tweet| %>
<%= link_to tweet, tweet %>
<% end %>
<%= will_paginate(@tweets) %>
==== app/controller/
Add the following line in the show method after the Tag.find line
@tweets = @tag.tweets.paginate(:page => params[:page], :per_page => 10)
=== Recommendations
==== app/controllers/user_controllers.rb
The following algorithm works like this:
1. Get all the users who are also using my tags.
2. For each of those users get their used tags and compare with mine.
3. Select the user who has the most similar tags to mine.
Add the following at the bottom of the file user_controllers.rb
private
def recommend(user)
my_tags = user.used_tags.to_a
my_friends = user.knows.to_a
# we are here using the raw java API - that's why using _java_node, raw and wrapper
other_users = user._java_node.outgoing(:used_tags).incoming(:used_tags).raw.depth(2).filter{|path| path.length == 2 && !my_friends.include?(path.end_node)}
# for all those users, find the person who has the max number of same tags as I have
found = other_users.max_by{|friend| (friend.outgoing(:used_tags).raw.map{|tag| tag[:name]} & my_tags).size }
found && found.wrapper # load the ruby wrapper around the neo4j java node
end
==== app/controllers/user_controllers.rb
Add one line to the show method:
def show
@user = User.find(params[:id])
@recommend = recommend(@user)
respond_to do |format|
format.html # show.html.erb
format.json { render :json => @user }
end
end
==== app/views/users/show.html.erb
Display a recommendation if available
<% if @recommend %>
Recommend
<%= link_to @recommend.twid, @recommend %>
<% end %>