REST, Ruby On Rails, CouchDB and Me – Part 6 Getting The Data Into And Out Of CouchDB   1 comment

Part 0 – REST, Ruby On Rails, CouchDB and Me

Part 1 – Ruby, The Command Line Version

Part 2 – Aptana IDE For Ruby

Part 3 CouchDB Up and Running on Windows

Part 4 – CouchDB, Curl and RUBY

Part 5 – Getting The Data Ready for CouchDB

Part 7 – JQUERY,JPlayer and HTML5

 [This just in: couchDB and Microsoft MVC 3: here]

Our mission in today’s post: 

  • Serialize the Concert object to a JSON string (i.e. a JSON document);
  • Do POST requests to insert  a JSON document for each concert into the couchdb database;
  • Create couchDB views to allow optimized data retrieval; and
  • Create a couchDB view to optimize retrieval recordings for all years for an arbitrary Month and Day (this duplicates the data provided by the “Grateful Dead Shows on This Day In History” selection in the Internet Archive.

Our last post outlined the formal structure of our document and the JSON format of the document we intend to post to our couchdb database.  Also in that post we outlined the RUBY code used to collect and clean the data we accessed from the Internet Archive.  In order to place our concert information into our couchdb database we need to transform our RUBY objects into proper JSON format and then call an HTTP POST to place our resource (document) into the database.  Why a POST rather than a PUT command you ask?  Better to ask a true RESTafarian.  In REST PUT is used to create a resource (in couchdb terms this is the database) and POST is used to modify a resource (in our case create a document within the database – which modifies the database I guess).  Hmmmm.

Why JSON?  Ask These Guys:

dsc_0029

To transform a RUBY object into JSON format requires the use of a RUBY GEM: json.  This GEM comes in two flavors, a pure RUBY form (slower) and a C form (faster, but it requires that you have a working RUBY development stack, GNU C compiler etc. deployed on your local machine).  Since we are happy with the speed of the pure RUBY form.  The downside of this GEM is that it will not serialize pure RUBY objects as is.  To do that you need to code custom serialization handlers. By default, the JSON serializer will work directly on primitive (native) types (strings, ints, what have you), and the simple structures: arrays and hash tables.  So our first task will be to transform our data into hash table format.  Assume a RUBY array of concert objects (@selectList) and a parallel array of data for each track (tracks). In pseudo code we have:

@selectList.each do |recording|
tracks=GDConcertTracks.new(recording)
tracks.getTrackDetails

   jsonString=makeJSON(recording,tracks)
PostRecording(jsonString)
end

 

Our method makeJSON takes the RUBY objects and returns a JSON string as:

def makeJSONFull(recording,tracks)
tList=Hash.new()
tList[“_id”]=recording.date
tList[“IAKey”]=recording.uri
tList[“description”]=recording.description
tList[“venue”]=recording.title
tList[“pubdate”]=recording.pubdate
tList[“cm”]=recording.cm
tList[“sb”]=recording.sb
tList[“mx”]=recording.mx
tTracks=Array.new()
tracks.each do |t|
tItem=Hash.new()
tItem[“title”]=t.title
tItem[“time”]=t.time
tItem[“uri”]=t.trackURI
tItem[“track”]=t.track #this could be derived as offset – 1
tTracks[tTracks.count]=tItem
end
tList[“tracks”]=tTracks
return JSON.generate(tList)
end

where JSON.generate(tList) is the JSON GEM method generate (to serialize) and tList is a Hash containing only primitive types (strings) and a Hash list of track data.  The track data contains only primitive types (strings).  We pass the returned string (which contains our data in JSON format) to our routine PostRecording(jsonString).  This Routine looks like:

def PostRecording(jsonString)
uri=”
http://127.0.0.1:5984/deadbase/”
begin
responseBody=open(uri,:method=> :post, :body => jsonString,”Content-Type” => “application/json”).read
puts ‘POST Response Success: ‘ + responseBody
rescue OpenURI::HTTPError => the_error
puts ‘##########Post Response Error: ‘ + the_error.io.status[0]
end
end

HiREST-LowREST1

OK this routine depends on the RUBY GEM: rest-open-uri .  This GEM is a single file of RUBY code which depends on the RUBY components: uri and net/http (these are part of the base RUBY system).  “rest-open-uri” extends the base RUBY HTTP components for the HTTP GET verb  and extends them to include all of the HTTP verbs (PUT, POST, DELETE and HEAD)  necessary to implement any REST system.  The open method, as used here:

open(uri,:method=> :post, :body => jsonString,”Content-Type” => “application/json”).read

uri: the address of our couchdb database

Content-Type: our MIME type for this call (“application/json”)

body: our document

method: the HTTP verb POST
If this POST command fails we want to capture the error and (for this example) display the error:

rescue OpenURI::HTTPError => the_error
puts ‘##########Post Response Error: ‘ + the_error.io.status[0]

Actual error recovery is left as an exercise for the reader.

 

OK, we now have data in our database.  In our case this is is 2,014 documents (i.e. 2,014 separate Grateful Dead concerts).  Using Futon we see the database as:

clip_image001

In Futon, we can look at our data:

clip_image001[7]

And we can drill in on an individual document (for example the primordial recording on 1965-11-01):

clip_image001[9]

 

Of course what we need to able to return our document with a simple HTTP GET request.  Further we will want to return only those fields we will actually need from a document (i.e. we need to shape our data to minimize bandwidth).  We do that in couchdb with a view which is defined in a design document.  We can define multiple views in a single design document and can have multiple design documents in a given database. 

For our first view lets set the task to return all documents which occurs on a given month and day (regardless of year).  This is the filter. We want to shape our data to return only a subset of fields: the venue, the IAKey and the document tracks.  The key for this view will be the month+day (in MMDD format).  Our design document might look like:

 {
“_id”: “_design/basic”,
“_rev”: “19-fd2c9b34d2536ce1f187ab2d4e5413de”,
“views”: {
“MonthDay”: {
“map”: “function(doc){emit(doc._id.substr(5,2)+doc._id.substr(8,2),[doc.venue , doc.IAKey, doc.tracks ])}”
}
}
}

 

What this view (called MonthDay) does is maps each document (doc) into a key (doc._id.substr(5,2)+doc._id.substr(8,2)) and an array of return fields:

[doc.venue , doc.IAKey, doc.tracks ]

Note that doc.tracks itself returns an array of tracks.

Using Futon to test our work and find the results for all New Years Day Grateful Dead Concerts.  The first document looks like:

 

clip_image001[11]

(additional lines are omitted from this screen snap).

 

As an HTTP Get Verb we would write:

http://127.0.0.1:5984/deadbase/_design/basic/_view/MonthDay?startkey=%2201-01%22

This command users the

URI: http://127.0.0.1:5984/deadbase

The View: _design/basic/_view/MonthDay

startkey: this is a key word parameter field which tells couchdb to go to the b-tree which represents the MonthDay view and finds the first Key match and continues through the tree returning all Key matches.

If we wanted a range of Keys (say all holiday season Concerts) we can use the startkey and the endkey parameters:

http://127.0.0.1:5984/deadbase/_design/basic/_view/MonthDay?startkey=%221224%22&endkey=%221969-12-31%22

startkey” gives us a starting key in the b-tree and returns all documents until (and including) the b-tree key defined by endkey.

 

JSConf-eu-Alexander-Lang--Writing-apps-on-the-edge-with-CouchDB-e8451113

For more details see the Definitive Guide to CouchDB chapter: Finding Your Data With Views.  The key to simple view definition is defining an appropriate lookup key and defining what data to return.  We can define a simple listing view as:

"Jump": { "map": "function(doc){emit(doc._id,'['+doc.venue+'] ' +'http://www.archive.org/details/'+doc.IAKey)}" },
This returns a lookup key equal to the original key used in the database (doc._id) and the data as venue (doc.venue)
and a URL for the concert recording ('http://www.archive.org/details/'+doc.IAKey). Now if we want to return all
concerts in August, 1969 we can issue the GET:
http://127.0.0.1:5984/deadbase/_design/basic/_view/Jump?startkey=%221969-08-00%22&endkey=%221969-08-31%22

This will return the rows as:{“total_rows”:2013,”offset”:156,”rows”:[ {“id”:”1969-08-02″,”key”:”1969-08-02″,”value”:”[Family Dog at the Great Highway] http://www.archive.org/details/gd69-08-02.sbd.miller.30651.sbeok.flacf”}, {“id”:”1969-08-03″,”key”:”1969-08-03″,”value”:”[Family Dog at the Great Highway] http://www.archive.org/details/gd1969-08-03.sbd.miller.30652.sbeok.flac16″}, {“id”:”1969-08-16″,”key”:”1969-08-16″,”value”:”[Woodstock Music] http://www.archive.org/details/gd1969-08-16.sbd.gmb.fixed.95918.flac16″}, {“id”:”1969-08-21″,”key”:”1969-08-21″,”value”:”[Aqua Theater] http://www.archive.org/details/gd1969-08-21.sbd-part.tremblay.1170.sbeok.shnf”}, {“id”:”1969-08-23″,”key”:”1969-08-23″,”value”:”[Pelletier Farm] http://www.archive.org/details/gd1969-08-23.sbd.lai.6639.shnf”}, {“id”:”1969-08-28″,”key”:”1969-08-28″,”value”:”[Family Dog at the Great Highway] http://www.archive.org/details/gd69-08-28.sbd.lepley.4234.sbeok.shnf”}, {“id”:”1969-08-29″,”key”:”1969-08-29″,”value”:”[Family Dog at the Great Highway] http://www.archive.org/details/gd1969-08-29.sbd.lai.9179.sbefail.shnf”}, {“id”:”1969-08-30″,”key”:”1969-08-30″,”value”:”[Family Dog at the Great Highway]

]}

Get it? Good.

 
6011428946_f722b3cb71
What Would Dweelze Do?

Now let’s return to RUBY and write a web page which will display concert and track data for any given day in Grateful Dead History. Using the view:MonthView as defined above:

“MonthDay”: {
“map”: “function(doc){emit(doc._id.substr(5,2)+doc._id.substr(8,2),[doc.venue , doc.IAKey, doc.tracks ])}”
}

Lets return data using RUBY for concerts for all years for the Month and day of todays Date.  In Ruby we need to define a controller and a erb HTML file to accomplish this.  Lets say our page will be called player.  Our controller might look like:

require ‘rest-open-uri’

require ‘json’

class PlayerController < ApplicationController
def player
mmdd=Date.today.to_s[5..-1]
mmdd=mmdd.gsub(“-“,””)
base=’
http://127.0.0.1:5984/deadbase/
    view=’_design/basic/_view/MonthDay’
url=base+view+’?key=’
url=url+’%22’+mmdd+’%22′
jsonString=returnJSONString(url)
@parm=jsonString
end
def returnJSONString(url)
jsonString=”
open (url) do |x|
x.each_line do |y|
jsonString=jsonString+y
end
end
return jsonString;
end 

end

This returns the JSON returned from couchdb unaltered as a parameter to the erb page for player (player.html.erb).  If we wanted to work with the data within RUBY we would need to change the JSON back into a RUBY format by calling:

JSON.parse(jsonString)

For our purposes, however, we want to pass the data directly to the browser and will process the data using JavaScript this will in almost all cases be faster than processing the JSON back to RUBY and then formatting the data on the erb page.  In our first pass the route map for our page will be simple:

map.connect ‘player’, :controller => ‘player’, :action => ‘player’

and our first pass at an HTML page will look like this:

<!–DOCTYPE HTML PUBLIC “-//W3C//DTD HTML 4.01//EN” “http://www.w3.org/TR/html4/strict.dtd>
<html>
<head>
<meta http-equiv=”Content-Type” content=”text/html; charset=iso-8859-1″ />
<title>Player: On This Day In Grateful Dead History</title>
<script type=”text/javascript”>
gdData=<%=  @parm  %> ;
</script>
</head>
<body>
<h3>On This Day In Grateful Dead History</h3>
</body>
</html>

This page does absolutely NOTHING…  Except gets our JSON data to the Javascript on the browser page.  And that, dear reader is as far as we need to go with today’s post.

5719769521_0380972a03




			

One response to “REST, Ruby On Rails, CouchDB and Me – Part 6 Getting The Data Into And Out Of CouchDB

Subscribe to comments with RSS.

  1. Pingback: Hey Flickr, Where Did My Statistics Go? The CouchDB Connection. Part III | Cloud2013 Or Bust

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: