Archive for the ‘Ruby’ Category

On Public Access to the Internet Archive’s Grateful Dead Collection   1 comment

Although I have covered this topic in a technical blog some time ago, the programmatic approach is not for everyone.  As one correspondent wrote:

Cloud,

I just want to listen to Dark Star on my IPOD, not get a computer science degree.

paul6

Well, if you just want to listen to Dark Star (or Fire On the Mountain for that matter) just go here or here.  Right Click on any track listed, select SAVE AS and Bear’s your Uncle.  But if your needs go deeper and you don’t want to write code; here dear reader is a simple explanation involving nothing more technical than a browser (Chrome will do fine but use Internet Explorer will work also) and a text editor (notepad for example).

Know Your Rights

The first thing we do, let’s kill all the lawyers…

                                              Shakespeare  ‘Henry VI,” Part II

Please read the statement from the Internet Archive on use of the Grateful Dead recordings stored on the Internet Archive here.  Understanding and interpretation of same is up to you, not me. I am not a lawyer, I don’t even play one on television, thank you.

PBS680505-01-FP

Doing a Single Track

Let’s say we like an early Dark Star, say:  Grateful Dead Live at Carousel Ballroom on 1968-01-17 for example.  Cursing to the Internet Archive we find the concert page for the Soundboard version we like the best:

image

Opps, No downloads on this concert. Let’s take a closer look at the browser screen, at the URL text box at the top:

Picture2

the “Https://archive.org/details” is the same on all concerts but the other part is the unique key for this particular version of this concert:

gd1968-01-17.sbd.jeff.3927.shnf

I will call this the IAKEY.  We will use this key to delve into the Internet Archive a little more deeply.  Copy this text string from the URL text box and copy it into a Notepad text document.  Now we need a list of all the mp3 files associated with this recording of this concert. Internet Archive stores these in an xml file uniquely defined for each recording. Copy the following template string into a second Notepad text document:

http://www.archive.org/download/{IAKEY}/{IAKEY}_files.xml

Now using text replace the string “{IAKEY} with the IAKey we got perviously.  When you are done the string will look like:

http://www.archive.org/download/gd1968-01-17.sbd.jeff.3927.shnf/gd1968-01-17.sbd.jeff.3927.shnf_files.xml

Open a new browser window and copy this new string into the URL text box and press enter.  Here is what you get back:

Picture3

Search for “Dark Star” on the screen and locate the mp3 file name I will call this the TrackMP3 (ignore any mp3 entries with the digits 64 in the mp3 file name).  In this case the mp3 for Dark Star is:

gd68-01-17d1t03_vbr.mp3

Open up your Notepad document as paste this string into it:

http://www.archive.org/download/{IAKEY}/{TrackMP3}

Now in this string replace {IAKEY} with gd1968-01-17.sbd.jeff.3927.shnf as we did before and replace {TrackMP3} with gd68-01-17d1t03_vbr.mp3

The final text string will now look like:

http://www.archive.org/download/gd1968-01-17.sbd.jeff.3927.shnf/gd68-01-17d1t03_vbr.mp3

Open up a new browser tab and paste this string into it and press enter.  There you are Dark Star will start playing and just RIGHT CLICK and Select “Save As” and Bear’s your uncle.

r733929_5943031

Doing a Group of Tracks

This gets tedious right away.  You can improve this process to skip the part where the file starts playing first by using a little HTML code.  I am not going to explain the whole process but it works like this:

follow the steps above and get a couple (or all) of the TrackMP3 strings.  “Turn on Your Love Light” has an TrackMP3 of gd68-01-17d1t01_vbr.mp3.  Cryptical Envelopment has a TrackMP3 of gd68-01-17d2t01_vbr.mp3 and  Dark Star we already know.

Open a text editor and enter the lines:

<html><body>

<a href=http://www.archive.org/download/{IAKEY}/{TrackMP3}> {title}</a>

<a href=http://www.archive.org/download/{IAKEY}/{TrackMP3}> {title}</a>

<a href=http://www.archive.org/download/{IAKEY}/{TrackMP3}> {title}</a> 

</body><html>

Now replace {IAKey} and {TrackMP3} and {title} in each line with the strings you got above and save the text file with the extension HTML (NOT as a txt file).  Load the HTML file into the browser (drag and drop will work) and you will see in the browser:

Turn on Your Love Light

Cryptical Envelopment

Dark Star

Right click on each of these in turn and click Save As… I hope you are getting the picture now.

tumblr_mg3mwmD25A1qabpu5o1_500

Microsoft MVC 3 and CouchDB – Low Level Get Calls   1 comment

I have written elsewhere on couchdb on Windows and using Ruby on Rails to interface to this system.  These posts can be found here:couchdb

Part 0 – REST, Ruby On Rails, CouchDB and Me

Part 1 – Ruby, The Command Line Version

Part 2 – Aptana IDE For Ruby

Part 3 CouchDB Up and Running on Windows

Part 4 – CouchDB, Curl and RUBY

Part 5 – Getting The Data Ready for CouchDB

Part 6 – Getting The Data Into And Out Of CouchDB

Part 7 – JQUERY,JPlayer and HTML5

In my work life I work in a Microsoft shop which for us means Microsoft servers for the back end and (mostly) pure HTML/AJAX frontends.  We are transitioning towards using Microsoft MVC 3 to provide HTTP end points for our AJAX calls.  Here are some notes from my POC work in this area.  My couch data consists of documents describing Grateful Dead concerts stored on the great site Internet Archive, if you have never visited the Internet Archive, please do so.  I back engineered the meta data of IA’s extensive collection of Dead concerts (over 2,000 concert recordings).  Visit the Grateful Dead Archive Home at the Internet Archive here.

CouchDB Documents and Views

I stored the meta data into a local couchdb (running on Windows XP).  The basic document I am storing is a master detail set for the ‘best’ recording for each Dead concert.  The Master part of the document contains the date, venue and other data of the concert and the detail set is an array of meta data on each song preformed during the concert.  As is traditional with couchdb, the documents are represented as JSON strings.  Here is what the document for the UR recording (1965-11-01) found on the IA:

{

“_id”: “1965-11-01”,tumblr_ld4jfoNw7F1qai6ym

“_rev”: “1-6ea272d20d7fc80e51c1ba53a5101ac1”,

“mx”: false,

“pubdate”: “2009-03-14”,

“sb”: true,

“venue”: “various”,

“tracks”: [

{

“uri”: “http://www.archive.org/download/gd1965-11- 01.sbd.bershaw.5417.sbeok.shnf/Acid4_01_vbr.mp3”,

“track”: “01”,

“title”: “Speed Limit”,

“time”: “09:48”

},

{

“uri”: “http://www.archive.org/download/gd1965-11-01.sbd.bershaw.5417.sbeok.shnf/Acid4_02_vbr.mp3&#8221;,

“track”: “02”,

“title”: “Neil Cassidy Raps”,

“time”: “02:19”

}

]

}

Couchdb allow the creation of views which are binary trees with user defined Keys and user defined sub sets of the document data.  If one wanted to return the venue and the tracks for each concert for a given Month and Day (across all years) the view created in couchdb would look like:

“MonthDay”: {

“map”: “function(doc){emit(doc._id.substr(5,2)+doc._id.substr(8,2),[doc.venue , doc.IAKey, doc.tracks ])}”

}

This view allows us to use and HTTP GET to pass in a monthday key (e.g. “1101”) and get back (as a JSON array)

the date (MMDDYY: doc._id.substr(5,2)+doc._id.substr(8,2))

the venue (doc.venue);

the AI URI of the concert (doc.IAKey); and

an array of track data (doc.tracks)

MVC URL Routing Maps

Although we could call couchdb directly from the browser, we normally work through a gateway system for security, so we will build a shim to sit between the browser and couchdb.  This allows us to flow the authentication / authorization stack separately from couchdb’s security system.  In MS MVC we can create a new HTTP endpoint for AJAX calls (our shim) is a very simple manner. Let’s create an endpoint which will look like:

http:\\{our server path}\DeadBase\MonthDay\{month}\{day}

where vacuum_routing

http:\\{our server path}\DeadBase\MonthDay\111

would request month:11 and day:01 concerts.  In MVC we can declare this routing as:

routes.MapRoute(

“MyMonthDay”,

“{controller}/{action}/{month}/{day}”, 

new { controller = “DeadBase”, action = “RestMonthDay”,null} );

Done.  Interestingly in MVC 3 this route definition will accept either the form:

http:\\{our server path}\DeadBase\MonthDay\{month}\{day} ; or

http:\\{our server path}\DeadBase\MonthDay?month=”??”&day=”??”

In the second form,  parameter order does not matter, but case does; quotation marks are optional and need to be dealt with internally by the action method.

either of these call will resolve to the same controller and method.

MVC Controller and Method HandlerMVC

We now need to create the shim which will be the target for the Http Endpoint.  In C# this looks like:

public class DeadBaseController : Controller

public string RestMonthDay( string month, string day )
{
//our shim code goes here

      }

    }

We able to use string as our return type because we will be calling couchdb which returns a string from of JSON by default.  As a side note if we wanted to use MVC 3 to return JSON from a native C# object our controller method takes a different form:

public JsonResult GetStateList()

{

List<ListItem> list = new List<ListItem>() {

new ListItem() { Value = “1”, Text = “VA” },

new ListItem() { Value = “2”, Text = “MD” },

new ListItem() { Value = “3”, Text = “DC” } };

return this.Json(list);

}

Our AJAX call from the browser does not need to know any of these details.  Here is one way to code the call in JavaScript using JQuery:

var url = urlBase + “?” + args;ajax

$.ajax({

url: url,

dataType: ‘json’,

success: okCallBack,

error: nookCallBack

});

function okCallBack(data) {

gdData = data;

//do something useful here

}

function nookCallBack(xhr, ajaxOptions, errorThrown) {

alert(“ErrorText:” + errorThrown + ” ” + “Error Code:” + xhr.status);

}

}

From Handler to CouchDB in C#

Here is the rest of the generic C# code to go from the Handler to CouchDB and back.

Clean the parameters and pass the call to a generic couchDB GET caller:mvc

image

Format the view name and parameter into couchdb format  and pass to the low level couchDB caller:

image

Classic Framework HTTP code to make the HTTP GET and return the results as a string back up the call stack:

image

We could (and did) take our Browser code from the Ruby on Rails project above and with minimum changes call our MVC shim.

Simple clean and fun.

Occupy your mind2

QCon San Francisco 2011   1 comment

QCON the software conference hosted by InfoQ will be meeting in San Francisco next month form November 14 through November 16.  I attended last years conference and am looking forward to attending again this year.

I am also a regular attendee of the Microsoft MIX Conferences.  Even though both of these conferences focus on Web development the contrast between these two conferences could not be greater.  First off MIX is much larger and is of course devoted to all things Microsoft.  MVC 3 was the big push this year at MIX.  This is a very strong development approach for Microsoft doing its ‘embrace and extend’ dance which it does so well.  In this case source is Ruby on Rails and its approach to standard MVC development.  MVC 3 (and Microsoft) approaches the Web from the perspective of the corporate developer of (basically) client server architecture.  But it is not a bad or evil effort.  Indeed the improved and streamlined http pipeline used by IIS for MVC is fast, the tools development environment are well thought out and, once you drop down a level the low level support for Rest(ful) approaches, JSON and HTML templates is impressive.  In addition to JSON and JQUERY, Microsoft is also a strong supporter of the emergent ODATA standard.  I recommend MIX (and the Channel 9 videos of the conference) to anyone working with or considering Microsoft development tools. I always learn new things are gain important information on how to advance the web at MIX.  You can read more details on the sessions here.

In terms of big metal companies MVC 3 and Framework 4.0 are much stronger than anything Java EE has to offer. The biggest problem Microsoft has is that it can not seem to ship its HTML5 compatible browser and so it’s development systems do not optimize for (or even in some cases take advantage of) the strongest and newest features of HTML5.  In addition, try as they will two things Microsoft will never be is cutting edge or free.  Over in the LAMP and Rails  and NOSQL world QCon is offers a look at how the world of the web will be (or at least could be) if any of the independent developers who make up most of QCON’s speakers and attendee’s are able to hit the mark with the next big thing.  It’s always a mixed bag of nuts at QCon, a nice mixture of visionaries and hucksters, Rastafarians and Agile advocates.  I like this conference because in addition to providing me with some alternative voices to the Google and Microsoft and Oracle, it also forces me to both re-evaluate the way I am doing things and to think independently about HOW we can do web development.  And San Francisco is a much better venue than La$ Wage$.  This is a hacker fest without the emphasis on cool technique not how to create the next big thing (product or Brand).  This is NOT Web 2.0 Summit which is about venture capitalism defining the web.  Alexia Tsotsis will not be covering this.

By the way, if you are not reading InfoQ on the web regularly you ARE missing out.

REST, Ruby On Rails, CouchDB and Me – Part 6 Getting The Data Into And Out Of CouchDB   1 comment

Part 0 – REST, Ruby On Rails, CouchDB and Me

Part 1 – Ruby, The Command Line Version

Part 2 – Aptana IDE For Ruby

Part 3 CouchDB Up and Running on Windows

Part 4 – CouchDB, Curl and RUBY

Part 5 – Getting The Data Ready for CouchDB

Part 7 – JQUERY,JPlayer and HTML5

 [This just in: couchDB and Microsoft MVC 3: here]

Our mission in today’s post: 

  • Serialize the Concert object to a JSON string (i.e. a JSON document);
  • Do POST requests to insert  a JSON document for each concert into the couchdb database;
  • Create couchDB views to allow optimized data retrieval; and
  • Create a couchDB view to optimize retrieval recordings for all years for an arbitrary Month and Day (this duplicates the data provided by the “Grateful Dead Shows on This Day In History” selection in the Internet Archive.

Our last post outlined the formal structure of our document and the JSON format of the document we intend to post to our couchdb database.  Also in that post we outlined the RUBY code used to collect and clean the data we accessed from the Internet Archive.  In order to place our concert information into our couchdb database we need to transform our RUBY objects into proper JSON format and then call an HTTP POST to place our resource (document) into the database.  Why a POST rather than a PUT command you ask?  Better to ask a true RESTafarian.  In REST PUT is used to create a resource (in couchdb terms this is the database) and POST is used to modify a resource (in our case create a document within the database – which modifies the database I guess).  Hmmmm.

Why JSON?  Ask These Guys:

dsc_0029

To transform a RUBY object into JSON format requires the use of a RUBY GEM: json.  This GEM comes in two flavors, a pure RUBY form (slower) and a C form (faster, but it requires that you have a working RUBY development stack, GNU C compiler etc. deployed on your local machine).  Since we are happy with the speed of the pure RUBY form.  The downside of this GEM is that it will not serialize pure RUBY objects as is.  To do that you need to code custom serialization handlers. By default, the JSON serializer will work directly on primitive (native) types (strings, ints, what have you), and the simple structures: arrays and hash tables.  So our first task will be to transform our data into hash table format.  Assume a RUBY array of concert objects (@selectList) and a parallel array of data for each track (tracks). In pseudo code we have:

@selectList.each do |recording|
tracks=GDConcertTracks.new(recording)
tracks.getTrackDetails

   jsonString=makeJSON(recording,tracks)
PostRecording(jsonString)
end

 

Our method makeJSON takes the RUBY objects and returns a JSON string as:

def makeJSONFull(recording,tracks)
tList=Hash.new()
tList[“_id”]=recording.date
tList[“IAKey”]=recording.uri
tList[“description”]=recording.description
tList[“venue”]=recording.title
tList[“pubdate”]=recording.pubdate
tList[“cm”]=recording.cm
tList[“sb”]=recording.sb
tList[“mx”]=recording.mx
tTracks=Array.new()
tracks.each do |t|
tItem=Hash.new()
tItem[“title”]=t.title
tItem[“time”]=t.time
tItem[“uri”]=t.trackURI
tItem[“track”]=t.track #this could be derived as offset – 1
tTracks[tTracks.count]=tItem
end
tList[“tracks”]=tTracks
return JSON.generate(tList)
end

where JSON.generate(tList) is the JSON GEM method generate (to serialize) and tList is a Hash containing only primitive types (strings) and a Hash list of track data.  The track data contains only primitive types (strings).  We pass the returned string (which contains our data in JSON format) to our routine PostRecording(jsonString).  This Routine looks like:

def PostRecording(jsonString)
uri=”
http://127.0.0.1:5984/deadbase/&#8221;
begin
responseBody=open(uri,:method=> :post, :body => jsonString,”Content-Type” => “application/json”).read
puts ‘POST Response Success: ‘ + responseBody
rescue OpenURI::HTTPError => the_error
puts ‘##########Post Response Error: ‘ + the_error.io.status[0]
end
end

HiREST-LowREST1

OK this routine depends on the RUBY GEM: rest-open-uri .  This GEM is a single file of RUBY code which depends on the RUBY components: uri and net/http (these are part of the base RUBY system).  “rest-open-uri” extends the base RUBY HTTP components for the HTTP GET verb  and extends them to include all of the HTTP verbs (PUT, POST, DELETE and HEAD)  necessary to implement any REST system.  The open method, as used here:

open(uri,:method=> :post, :body => jsonString,”Content-Type” => “application/json”).read

uri: the address of our couchdb database

Content-Type: our MIME type for this call (“application/json”)

body: our document

method: the HTTP verb POST
If this POST command fails we want to capture the error and (for this example) display the error:

rescue OpenURI::HTTPError => the_error
puts ‘##########Post Response Error: ‘ + the_error.io.status[0]

Actual error recovery is left as an exercise for the reader.

 

OK, we now have data in our database.  In our case this is is 2,014 documents (i.e. 2,014 separate Grateful Dead concerts).  Using Futon we see the database as:

clip_image001

In Futon, we can look at our data:

clip_image001[7]

And we can drill in on an individual document (for example the primordial recording on 1965-11-01):

clip_image001[9]

 

Of course what we need to able to return our document with a simple HTTP GET request.  Further we will want to return only those fields we will actually need from a document (i.e. we need to shape our data to minimize bandwidth).  We do that in couchdb with a view which is defined in a design document.  We can define multiple views in a single design document and can have multiple design documents in a given database. 

For our first view lets set the task to return all documents which occurs on a given month and day (regardless of year).  This is the filter. We want to shape our data to return only a subset of fields: the venue, the IAKey and the document tracks.  The key for this view will be the month+day (in MMDD format).  Our design document might look like:

 {
“_id”: “_design/basic”,
“_rev”: “19-fd2c9b34d2536ce1f187ab2d4e5413de”,
“views”: {
“MonthDay”: {
“map”: “function(doc){emit(doc._id.substr(5,2)+doc._id.substr(8,2),[doc.venue , doc.IAKey, doc.tracks ])}”
}
}
}

 

What this view (called MonthDay) does is maps each document (doc) into a key (doc._id.substr(5,2)+doc._id.substr(8,2)) and an array of return fields:

[doc.venue , doc.IAKey, doc.tracks ]

Note that doc.tracks itself returns an array of tracks.

Using Futon to test our work and find the results for all New Years Day Grateful Dead Concerts.  The first document looks like:

 

clip_image001[11]

(additional lines are omitted from this screen snap).

 

As an HTTP Get Verb we would write:

http://127.0.0.1:5984/deadbase/_design/basic/_view/MonthDay?startkey=%2201-01%22

This command users the

URI: http://127.0.0.1:5984/deadbase

The View: _design/basic/_view/MonthDay

startkey: this is a key word parameter field which tells couchdb to go to the b-tree which represents the MonthDay view and finds the first Key match and continues through the tree returning all Key matches.

If we wanted a range of Keys (say all holiday season Concerts) we can use the startkey and the endkey parameters:

http://127.0.0.1:5984/deadbase/_design/basic/_view/MonthDay?startkey=%221224%22&endkey=%221969-12-31%22

startkey” gives us a starting key in the b-tree and returns all documents until (and including) the b-tree key defined by endkey.

 

JSConf-eu-Alexander-Lang--Writing-apps-on-the-edge-with-CouchDB-e8451113

For more details see the Definitive Guide to CouchDB chapter: Finding Your Data With Views.  The key to simple view definition is defining an appropriate lookup key and defining what data to return.  We can define a simple listing view as:

"Jump": { "map": "function(doc){emit(doc._id,'['+doc.venue+'] ' +'http://www.archive.org/details/'+doc.IAKey)}" },
This returns a lookup key equal to the original key used in the database (doc._id) and the data as venue (doc.venue)
and a URL for the concert recording ('http://www.archive.org/details/'+doc.IAKey). Now if we want to return all
concerts in August, 1969 we can issue the GET:
http://127.0.0.1:5984/deadbase/_design/basic/_view/Jump?startkey=%221969-08-00%22&endkey=%221969-08-31%22

This will return the rows as:{“total_rows”:2013,”offset”:156,”rows”:[ {“id”:”1969-08-02″,”key”:”1969-08-02″,”value”:”[Family Dog at the Great Highway] http://www.archive.org/details/gd69-08-02.sbd.miller.30651.sbeok.flacf&#8221;}, {“id”:”1969-08-03″,”key”:”1969-08-03″,”value”:”[Family Dog at the Great Highway] http://www.archive.org/details/gd1969-08-03.sbd.miller.30652.sbeok.flac16&#8243;}, {“id”:”1969-08-16″,”key”:”1969-08-16″,”value”:”[Woodstock Music] http://www.archive.org/details/gd1969-08-16.sbd.gmb.fixed.95918.flac16&#8243;}, {“id”:”1969-08-21″,”key”:”1969-08-21″,”value”:”[Aqua Theater] http://www.archive.org/details/gd1969-08-21.sbd-part.tremblay.1170.sbeok.shnf&#8221;}, {“id”:”1969-08-23″,”key”:”1969-08-23″,”value”:”[Pelletier Farm] http://www.archive.org/details/gd1969-08-23.sbd.lai.6639.shnf&#8221;}, {“id”:”1969-08-28″,”key”:”1969-08-28″,”value”:”[Family Dog at the Great Highway] http://www.archive.org/details/gd69-08-28.sbd.lepley.4234.sbeok.shnf&#8221;}, {“id”:”1969-08-29″,”key”:”1969-08-29″,”value”:”[Family Dog at the Great Highway] http://www.archive.org/details/gd1969-08-29.sbd.lai.9179.sbefail.shnf&#8221;}, {“id”:”1969-08-30″,”key”:”1969-08-30″,”value”:”[Family Dog at the Great Highway]

]}

Get it? Good.

 
6011428946_f722b3cb71
What Would Dweelze Do?

Now let’s return to RUBY and write a web page which will display concert and track data for any given day in Grateful Dead History. Using the view:MonthView as defined above:

“MonthDay”: {
“map”: “function(doc){emit(doc._id.substr(5,2)+doc._id.substr(8,2),[doc.venue , doc.IAKey, doc.tracks ])}”
}

Lets return data using RUBY for concerts for all years for the Month and day of todays Date.  In Ruby we need to define a controller and a erb HTML file to accomplish this.  Lets say our page will be called player.  Our controller might look like:

require ‘rest-open-uri’

require ‘json’

class PlayerController < ApplicationController
def player
mmdd=Date.today.to_s[5..-1]
mmdd=mmdd.gsub(“-“,””)
base=’
http://127.0.0.1:5984/deadbase/
    view=’_design/basic/_view/MonthDay’
url=base+view+’?key=’
url=url+’%22’+mmdd+’%22′
jsonString=returnJSONString(url)
@parm=jsonString
end
def returnJSONString(url)
jsonString=”
open (url) do |x|
x.each_line do |y|
jsonString=jsonString+y
end
end
return jsonString;
end 

end

This returns the JSON returned from couchdb unaltered as a parameter to the erb page for player (player.html.erb).  If we wanted to work with the data within RUBY we would need to change the JSON back into a RUBY format by calling:

JSON.parse(jsonString)

For our purposes, however, we want to pass the data directly to the browser and will process the data using JavaScript this will in almost all cases be faster than processing the JSON back to RUBY and then formatting the data on the erb page.  In our first pass the route map for our page will be simple:

map.connect ‘player’, :controller => ‘player’, :action => ‘player’

and our first pass at an HTML page will look like this:

<!–DOCTYPE HTML PUBLIC “-//W3C//DTD HTML 4.01//EN” “http://www.w3.org/TR/html4/strict.dtd>
<html>
<head>
<meta http-equiv=”Content-Type” content=”text/html; charset=iso-8859-1″ />
<title>Player: On This Day In Grateful Dead History</title>
<script type=”text/javascript”>
gdData=<%=  @parm  %> ;
</script>
</head>
<body>
<h3>On This Day In Grateful Dead History</h3>
</body>
</html>

This page does absolutely NOTHING…  Except gets our JSON data to the Javascript on the browser page.  And that, dear reader is as far as we need to go with today’s post.

5719769521_0380972a03




	

REST, Ruby On Rails, CouchDB and Me – Part 4 – CURL on Windows And Ruby POST   Leave a comment

Part 0 – REST, Ruby On Rails, CouchDB and Me

Part 1 – Ruby, The Command Line Version

Part 2 – Aptana IDE For Ruby

Part 3 CouchDB Up and Running on Windows

Part 4 – CouchDB, Curl and RUBY

Part 5 – Getting The Data Ready for CouchDB

Part 6 – Getting The Data Into And Out Of CouchDB

Part 7 – JQUERY,JPlayer and HTML5

In The Post:

  • CURL and Couchdb
  • Documents Design and Otherwise
  • Posting Documents to couchDB Using Ruby

If you are like me you have spent some time with the free ebook: CouchDB The Definitive Guide.  If you are a windows user you may have run into some problems with the examples given in the chapter  on “Design Documents”.  Specifically they don’t work ‘out of the box’.  The examples in that chapter show us how to: create a database, to create and post a design document and to post a document to the database.  These examples use  CURL in a command shell.

OmniVortex

Since we are running Windows first we need to install CURL on our system.  Either set your system path to include the CURL executable. We can get a windows version here.  Use the version labeled DOS, Win32- MSVC or Win64 depending on your system. We assume here that couchDB has been installed successfully on your system. Now open a ‘command prompt’ on your system.  If you must have a UNIX type shell you need to install CYWIN or some other UNIX emulator for Windows.  If you are using the Aptana IDE like me you need to create an “external command” to open a command shell within Aptana.  This figure illustrates the setup within the Aptana IDE to do this:

image

In the command shell you can create a couchdb Database using a POST command and CURL.  Couchdb is RESTful so we use a PUT command for all actions which CREATE a resource, of which a database is one example.  The format of the command is:

curl -X PUT http://{couchdb}/{yourdatabasename}I want to create a database named deadbase so on my system this command and response looks like:

C:\Documents and Settings\dredfield\My Documents\Aptana Studio Workspace\couchDB01

>curl -X PUT http://127.0.0.1:5984/deadbase

{“ok”:true}

The where “{“ok”:true}” is the response body of the http response to my put command.  Confirm your work by starting a browser and navigating to Futon user interface to your couchdb installation.  On my system this url is:

http://127.0.0.1:5984/_utils/index.html

you should see something like this:

image

CURL and Documents

OK, now lets make a design document for this database and PUT that document to the new database.  With slight modifications to the example given in CouchDB The Definitive Guide my first cut at a design document looks like this:

{

     “_id” : “_design/example”,

     “views” : {

        “View00” : {

       “map” : “function(doc){emit(doc._id, doc.date)}”

        }

  }

}

This is a JSON formatted document.  Initial syntax checking is up to you.  Basically couchDB will accept anything within the outer brackets whether or not it is formatted as usable JSON or not.  We have several options for checking syntax.  There are free online syntax checkers like JSONLint.  The interface to JSONLint looks like:

clip_image001

An installable open source JSON checker and visualizing tool, JSON View is available here.  JSON View’s output looks like:

clip_image001[10]

Now that we know our syntax is correct (if not the logic of the design document – more on this in the next installment) we can PUT this document to our database.  We can have more than one design document in a given database.  The name (id) of this document is “_design/example”.  where “_design” tells couchdb this is indeed a design document and its name is “example”.   My document is named mydesign.json on my file system.  The CURL command to PUT this into the database looks like:

curl -X PUT http://127.0.0.1:5984/deadbase/_design/example -d @mydesign.json

couchdb will respond:

{“ok”:true,”id”:”_design/example”,”rev”:”1-45f081a3f681b28ce7a0bdc5db216e74″}

Note here that this is NOT the syntax shown in CouchDB The Definitive Guide.  The syntax there will not work in a windows shell (i.e. command prompt).  Even when you have syntax correct JSON document  and the correct format of the PUT statement on Windows you may recieve an error message from CURL complaining about UTF8 errors within the document and have a PUT failure.  The problem here is that the Windows file system supports several encoding schemes and various windows programs save documents in to different default encoding.  If you are using Notepad.exe to create your files be sure to save the files in ANSIformat.

 
Check your work using the FUTON interface locate the “_design/example document” in deadbase

clip_image001[12]

Double click on the document:

clip_image001[16]

Note that “views” is a “Field” within the document.  Select the “Source” tab  and take a look inside the document:

clip_image002[4]

Now lets POST a document into the database.  Since we have not defined any validation fields we can push anything into the database.  Even documents which consist of just “{}”.  CouchDB defines only one innate restriction:

If a document defines the id field (“_id”) then the value of _id must not conflict with an existing value of the ID field of ANY other document in the database.

If the document does not define an ID field, couchDB will generate an ID (as a UUID) and apply it to the document.  You can supply your own ID values.  If you can either generate your own value  (Ruby can generate a GUID for you) or you can request a GUID from couchdb with a GET command.  See this page for more information.  In the sample program I will be developing for this series I will be using a ‘natural key’ – that is a key whose value has an actual meaning (a Social Security is such a natural key for example, but please never use this).  If you try to POST a document and use a duplicate key you will get back a 409 status code for the error.

The document I will be using in the next post looks like this:

{

“_id” : “1972-07-22”,

“IAKey” : “gd1972-07-22.sbd.miller.94112.sbeok.flac16”,

“description” : “Set 1 Bertha Me And My Uncle You Win Again Jack Straw Bird Song Beat It On Down The Line Sugaree Black Throated …

“pubdate”: “2008-08-15”,

“sb”: true,

“cm”: true,

“mx”: false,

“venue”: “Paramount Northwest Theatre”,

}

If I save this document as ConcertRecord.json I can use CURL to POST this document as:

curl -H “Content-Type: application/json” -X POST http://127.0.0.1:5984/deadbase/ -d @ConcertRecord.json

and couchdb will reply with an HTTP status 200 and a response body of:

{“ok”:true,”id”:”1972-07-22″,”rev”:”1-01a182f329c40ba3bab4b13695d0a098″}

In couchDB Futon this document looks like:

clip_image001[20]

Note that the order of the fields is set by couchDB not the order in the first loaded document.

Ruby At Last

OK, enough of the command shell let’s do some couchDB work using RUBY.  I am going to access couchDB from a fairly low level within Ruby in these posts.  There are several ActiveRecord type GEMS which will interface with couchDB but my focus here will be on: (1)  speed of access and (2) transferability of knowledge between Ruby access and direct Javascript/Browser access to couchDB.

Here’s a minimum of what we need to POST a document to a couchdb using RUBY.

The GEMS for

JSON : This will always load the Ruby based version of the JSON module.  If you want to have ‘pure’ JSON (i.e. a C based module you will need to have the Ruby/Windows DEVKit installed on your system.  For our purposes the ‘pure’ version is not necessary.

REST-OPEN-URI:  This extends open-uri by using the net/http  and the uri GEMs to cover all of the REST verbs (GET, POST, PUT and DELETE).  This is a very light install and is only lightly documented.

Here is the basic plan:

Assume we have a RUBY object (call it “rec”) which includes, among other things the fields we want to POST into the deadbase as a deadbase document like the one developed above.  We first need to convert the fields into a JSON string and then to POST the JSON string into the deadbase.  The JSON GEM is used to achive the first goal and REST-Open-URI is used to accomplish the second.

JSON Strings:

The JSON GEM will only serialize Ruby base types (strings, numbers and bools and HASH objects).  The JSON GEM is quite limited in that it will not serialize a Ruby object derived from the base RUBY object  into a JSON string, even if that object consists only of base types and Hash objects.  Although you may extend JSON we did not choose to do so. Rather we will create a simple Hash object and populate it manually via Ruby code with the fields we want to use for a document. Simply this could look like:

def makeJSON(rec)

thing=Hash.new()  #we know that JSON can serialize this type of object

thing[“_id”]=rec.date

thing[“IAKey”]=rec.uri

thing[“description”]=rec.description

thing[“venue”]=rec.title

thing[“pubdate”]=rec.pubdate

thing[“cm”]=rec.cm

thing[“sb”]=rec.sb

thing[“mx”]=rec.mx

return JSON.generate(thing)  #this returns a JSON String

end

REST-OPEN_URI:

Our POST routine will use the output form makeJSON and POST the JSON string to the deadbase.  In simple for this routine looks like:

def PostRecording(jsonString)

uri=”http://127.0.0.1:5984/deadbase/”   #this is our database

begin

responseBody=open(uri,:method=> :post, :body => jsonString,”Content-Type” => “application/json”).read

puts ‘POST Response Success: ‘ + responseBody

end

rescue

OpenURI::HTTPError => the_error

puts ‘Post Response Error: ‘ + the_error.io.status[0]

end

end

The key line is, of course:

responseBody=open(uri,:method=> :post, :body => jsonString,”Content-Type” => “application/json”).read

If we ran this line as:

responseBody=open(uri,:method=> :post, :body => jsonString).read

we would get an http Status code for an “Invalid Media Type”.  That’s because the default “Content-Type” for POST commands is “application/xxx-form” which is the typical format of a HTML “form” involved in a POST from a web browser.  We are far from a browser here and our “Content-Type” needs to be “application/json”.  The way to add Headers to the POST is to provide one or more key/value pairs with the desired header information.  Hence:

“Content-Type” => “application/json”

and the correct Ruby line is:

responseBody=open(uri,:method=> :post, :body => jsonString,”Content-Type” => “application/json”).read

We need to wrap the POST command in an exception block where the line:

OpenURI::HTTPError => the_error

is only executed IF the Http response status is > 399.  You can then do more fine grained responses to the error condition.  Specifically, if the_error.io.status[0]==409 you have attempted to POST the same document twice (at least two documents with the same ID).

That looks like a wrap for now.

5901067736_08fe849334_z

Posted 2011/07/22 by Cloud2013 in Aptana, couchdb, REST, Ruby

Tagged with , , , ,

Microsoft MIX 11 – Day Three – The OData Deal Goes Down   2 comments

If I had a gun for every ace I have drawn,

I could arm a town the size of Abilene.

And you know I’m only in it for the gold.

All that I am asking for is ten gold dollars

And I could pay you back with one good hand

You can look around about the wide world over

And you’ll never find another honest man.

Loser – Hunter/Garcia 1971

MIX 11 Day One

MIX 11 Day Two

Mix 11 Day Three

        Microsoft continued today to play the open source hand today on this the final day of MIX 11.  About fifty per cent of the attendees where absent today.  I don’t want to name names but the remaining attendees do not work for the Mephistopheles of Redmond. Open Source in the form of the very nicely developing NuGet was very much in on everyone’s minds.  The Hackers: Phil Haack and Scott Hanselman gave a well attended presentation, “NuGet In Depth: Emerging Open Source on the .Net Platform.”  DSC_0026 The session tape of this presentation needs to be seen to be believed.  The NuGet effort moves way beyond the effort in Codeplex as a open source store of open sourced Dot Net related development efforts.  Continuing on the theme in evidence yesterday the packaging strategy and access methods  for NuGet reminds us of the GEM package system for Ruby AND as embedded in VS2010 looks a lot like the GEM interface found in the late great Aptana IDE for RadRails.  Please do not misunderstand me. I love Ruby and Rudy on Rails and think that embedded the very powerful conventions and concepts into MVC3 and NuGet are a quantum leap forward in Microsoft’s approach to software development.  The fun in programming in Microsoft shops may well be back. We think the commitment to open source co-development of Dot Net Framework products is great.

       OData and the new improved WCF was another hot topic of the day with four sessions devoted to these topics.  The three sessions I attended and would recommend viewing the tapes of are:

Glenn Block: WCF Web APIs

Assad Khan and Maceleo Lopez Ruiz: Data In the HTML World

Jonathan Carter: OData Roadmap

       When first pushed out WCF was primarily a enterprise strength product which was XML and serverDSC_00021 to server or server to (non Web) client oriented product, and it was fine in that role.  But the AJAX services and JSON arrived and pretty much took over the real web where real people and companies actually work.  It has taken some time (and the development of LINQ among other technologies) for MS to catch the wave again.  WCF Data Services are not your older brother’s Web services any more.  With the death of the SVC extension, route mapping and the introduction of the light weight WebGet meta tags WCF services are back in the game as a rock solid Web AJAX source.  The newish API is very rich.  For example a very interesting  HttpResponseMessage object  has been added to the framework to provide us with low level (read header) control over the whole message exchange process which gives us fine granularity, low weight and a greater ability to work in the standards world of HTTP 1.1 raw communication without needed obtuse code.  Better REST and more restful.  Glenn Blocks presentation is a useful introduction to what is happening in this area.

       How does ODataDSC_0027 fit into all this? For a lot of reason too complex ( and mostly sociological) to go into in this post XML never had staying power among the Web first crowd.  JSON has fidelity with the consumer of AJAX calls (being JavaScript and all that) and XML processing in JavaScript is not nearly as fast or as fancy free as a lot of browser coders would like.  OData is based on an open standard (what that means is a little nebulous) and is structured and extensible (and supports namespaces but these are lost in the JSON transformation on the way from the server to the browser).  Microsoft, among other big players, have gotten behind the OData wagon. OData will be supported by Microsoft on the browser and the server side. Khan and Ruiz presentation, “Data In the HTML World” introduced a JavaScript HTML5 library for OData consumers called: DataJS.js (no this is not a typo but the department of name redundancy redundancy).   This open source script library is available via Codeplex(and I think NuGet) and allows a very structured way for (modern) browsers to consume OData shaped data messages.  When the browser requests and receives OData formatted messages via this script from servers not in the domain which originally served up the consuming page the Script uses JSONP – think browser mashups of OData here. All this seems to work cleanly and without undue bit twittering on the JavaScript coders part.  Interestingly and in keeping with the new approach by Microsoft to these problems the demo featured the use of JQuery’s new Template methodologies to display data consumed via DataJS.js.

     Some much for the browser side.  Jonathan Carter’s presentation, “OData Roadmap” brings this all together with the  server side code.  Carter introduced a very interesting open source Dot Net thingy:  The WCF Data Services Toolkit, this Framework Toolket is is open sourced, subject to on going development community development and is available via CodePlex.  The toolkit introduces an IQueryable based OData object which is extensible and who external call syntax is based on well known OData conconical uri’s and data query conventions.  The conventions are for ease of programming and consistency with current practices in the OData community (think Ruby on Rails again here) but can be over ridden as needed. The translation from traditional rectangular database tables to OData is straight forward (well a LOT of Entity Framework and LINQ is going on in the background but the API user does not see this).   Interestingly one can nest OData mappings to different data sources within a single output OData message.  Think server side mash ups here.  A simple clean programming model is provided which is extremely powerful.  Keeping in the Ruby On Rails colorization of the whole MVC3/OData/WCF Toolkit, the demo for this presentation featured a mapping of a Mongo DB data source into OData (with an assist from a NuGet community developed Mongo DB helper assembly).  Combine this with DataJS.js and you pretty much have what you want:  a light, fast, extensible AJAX/REST messaging system.  And who wouldn’t want that?   Now if we can get Web Socket support into the browsers…..

     OK, Folks.  I’ve gotta plane to catch.  See you next year!

 DSC_0001

 

Microsoft MIX 11 – Round Two   1 comment

Criminally IrresponsibleDouglas Crockford on the work of the HTML5 standards committee of the W3C.

MIX 11 Day One

MIX 11 Day Two

MIX 11 Day Three

On the second day of MIX 11,  noted standup comic and all around funny man, Douglas Crockford, brought his act back to Vegas.  His routine was supported by fellow Javascript lumniaries: Luke Hoban, Allen Wirfs-Brock and Tomasz Janczuk.  Crockford took issue with the W3C’s work on HTML5 for  (and we paraphrase) ‘adding features without dealing with the security implications of these features and failing to correct existing security features in the DOM.’  Such comic bits along with his well known ‘Javascript is a near perfect language’ riff bought the crowd of the almost empty auditorium to its feet.  The small turn out for this panel is inexplicable to your scribe who attended a standing room only audience of almost 2,000 for a similar panel on Javascript (which included Crockford, Wirfs-Brock, Brendon Eich and Mark Miller) at the great, lamented Ajax Experience Conference in Boston in 2009.   The panel is to be complemented also for the extremely well tuned sense of haute couture which they brought to the stage.  Douglas Crockford also gave a second presentation coving changes coming in ECMAScript 5

DSC_0029

Criminally Irresponsible

A high point of the Microsoft sessions on Day Two was “An Over view of the MS Web Stack of Love” as presented by the always entertaining Scott Hanselman.  Walking the high wire with a live demo featuring yesterday’s (!) build of the Visual Studio/Framework 4 MVC Scott presented a lively and very informative overview of MVC 3.  The session tape for this session is a much see.  Anyone who has any experience with RADRails will enjoy the familiar ‘embrace and extend’ strategy with Microsoft can play so well.   This is a monster effort from Microsoft and should grab back some of the mind share which MS lost in the Web community with its soon to be forgotten Web Forms approach.  Please have a look at what MS is doing in this area.  Insanely great and with an open embrace of the open community support (via NuGet). Entity Framwork which has some what of a confused history at Microsoft has finally found a home in this product.  MVC support (via EF) not only code first design (ala RADRails) but Model First and Schema First – dealers choice!  That and a little Linq, some JQuery and (soon) HTML5/CSS3 and this product is

Hot, Hot, Hot!

Get the bits, watch the session tape and code.

DSC_0018

Scott Wows The Crowd at the Tuesday Keynote.

(See boss I did attend at least some of at least one keynote)

And of course there were the mini sessions in the afternoon:

DSC_0038

%d bloggers like this: