Archive for the ‘Ruby’ Category

On Public Access to the Internet Archive’s Grateful Dead Collection   1 comment

Although I have covered this topic in a technical blog some time ago, the programmatic approach is not for everyone.  As one correspondent wrote:

Cloud,

I just want to listen to Dark Star on my IPOD, not get a computer science degree.

paul6

Well, if you just want to listen to Dark Star (or Fire On the Mountain for that matter) just go here or here.  Right Click on any track listed, select SAVE AS and Bear’s your Uncle.  But if your needs go deeper and you don’t want to write code; here dear reader is a simple explanation involving nothing more technical than a browser (Chrome will do fine but use Internet Explorer will work also) and a text editor (notepad for example).

Know Your Rights

The first thing we do, let’s kill all the lawyers…

                                              Shakespeare  ‘Henry VI,” Part II

Please read the statement from the Internet Archive on use of the Grateful Dead recordings stored on the Internet Archive here.  Understanding and interpretation of same is up to you, not me. I am not a lawyer, I don’t even play one on television, thank you.

PBS680505-01-FP

Doing a Single Track

Let’s say we like an early Dark Star, say:  Grateful Dead Live at Carousel Ballroom on 1968-01-17 for example.  Cursing to the Internet Archive we find the concert page for the Soundboard version we like the best:

image

Opps, No downloads on this concert. Let’s take a closer look at the browser screen, at the URL text box at the top:

Picture2

the “Https://archive.org/details” is the same on all concerts but the other part is the unique key for this particular version of this concert:

gd1968-01-17.sbd.jeff.3927.shnf

I will call this the IAKEY.  We will use this key to delve into the Internet Archive a little more deeply.  Copy this text string from the URL text box and copy it into a Notepad text document.  Now we need a list of all the mp3 files associated with this recording of this concert. Internet Archive stores these in an xml file uniquely defined for each recording. Copy the following template string into a second Notepad text document:

http://www.archive.org/download/{IAKEY}/{IAKEY}_files.xml

Now using text replace the string “{IAKEY} with the IAKey we got perviously.  When you are done the string will look like:

http://www.archive.org/download/gd1968-01-17.sbd.jeff.3927.shnf/gd1968-01-17.sbd.jeff.3927.shnf_files.xml

Open a new browser window and copy this new string into the URL text box and press enter.  Here is what you get back:

Picture3

Search for “Dark Star” on the screen and locate the mp3 file name I will call this the TrackMP3 (ignore any mp3 entries with the digits 64 in the mp3 file name).  In this case the mp3 for Dark Star is:

gd68-01-17d1t03_vbr.mp3

Open up your Notepad document as paste this string into it:

http://www.archive.org/download/{IAKEY}/{TrackMP3}

Now in this string replace {IAKEY} with gd1968-01-17.sbd.jeff.3927.shnf as we did before and replace {TrackMP3} with gd68-01-17d1t03_vbr.mp3

The final text string will now look like:

http://www.archive.org/download/gd1968-01-17.sbd.jeff.3927.shnf/gd68-01-17d1t03_vbr.mp3

Open up a new browser tab and paste this string into it and press enter.  There you are Dark Star will start playing and just RIGHT CLICK and Select “Save As” and Bear’s your uncle.

r733929_5943031

Doing a Group of Tracks

This gets tedious right away.  You can improve this process to skip the part where the file starts playing first by using a little HTML code.  I am not going to explain the whole process but it works like this:

follow the steps above and get a couple (or all) of the TrackMP3 strings.  “Turn on Your Love Light” has an TrackMP3 of gd68-01-17d1t01_vbr.mp3.  Cryptical Envelopment has a TrackMP3 of gd68-01-17d2t01_vbr.mp3 and  Dark Star we already know.

Open a text editor and enter the lines:

<html><body>

<a href=http://www.archive.org/download/{IAKEY}/{TrackMP3}> {title}</a>

<a href=http://www.archive.org/download/{IAKEY}/{TrackMP3}> {title}</a>

<a href=http://www.archive.org/download/{IAKEY}/{TrackMP3}> {title}</a> 

</body><html>

Now replace {IAKey} and {TrackMP3} and {title} in each line with the strings you got above and save the text file with the extension HTML (NOT as a txt file).  Load the HTML file into the browser (drag and drop will work) and you will see in the browser:

Turn on Your Love Light

Cryptical Envelopment

Dark Star

Right click on each of these in turn and click Save As… I hope you are getting the picture now.

tumblr_mg3mwmD25A1qabpu5o1_500

Microsoft MVC 3 and CouchDB – Low Level Get Calls   1 comment

I have written elsewhere on couchdb on Windows and using Ruby on Rails to interface to this system.  These posts can be found here:couchdb

Part 0 – REST, Ruby On Rails, CouchDB and Me

Part 1 – Ruby, The Command Line Version

Part 2 – Aptana IDE For Ruby

Part 3 CouchDB Up and Running on Windows

Part 4 – CouchDB, Curl and RUBY

Part 5 – Getting The Data Ready for CouchDB

Part 6 – Getting The Data Into And Out Of CouchDB

Part 7 – JQUERY,JPlayer and HTML5

In my work life I work in a Microsoft shop which for us means Microsoft servers for the back end and (mostly) pure HTML/AJAX frontends.  We are transitioning towards using Microsoft MVC 3 to provide HTTP end points for our AJAX calls.  Here are some notes from my POC work in this area.  My couch data consists of documents describing Grateful Dead concerts stored on the great site Internet Archive, if you have never visited the Internet Archive, please do so.  I back engineered the meta data of IA’s extensive collection of Dead concerts (over 2,000 concert recordings).  Visit the Grateful Dead Archive Home at the Internet Archive here.

CouchDB Documents and Views

I stored the meta data into a local couchdb (running on Windows XP).  The basic document I am storing is a master detail set for the ‘best’ recording for each Dead concert.  The Master part of the document contains the date, venue and other data of the concert and the detail set is an array of meta data on each song preformed during the concert.  As is traditional with couchdb, the documents are represented as JSON strings.  Here is what the document for the UR recording (1965-11-01) found on the IA:

{

“_id”: “1965-11-01”,tumblr_ld4jfoNw7F1qai6ym

“_rev”: “1-6ea272d20d7fc80e51c1ba53a5101ac1”,

“mx”: false,

“pubdate”: “2009-03-14”,

“sb”: true,

“venue”: “various”,

“tracks”: [

{

“uri”: “http://www.archive.org/download/gd1965-11- 01.sbd.bershaw.5417.sbeok.shnf/Acid4_01_vbr.mp3”,

“track”: “01”,

“title”: “Speed Limit”,

“time”: “09:48”

},

{

“uri”: “http://www.archive.org/download/gd1965-11-01.sbd.bershaw.5417.sbeok.shnf/Acid4_02_vbr.mp3&#8221;,

“track”: “02”,

“title”: “Neil Cassidy Raps”,

“time”: “02:19”

}

]

}

Couchdb allow the creation of views which are binary trees with user defined Keys and user defined sub sets of the document data.  If one wanted to return the venue and the tracks for each concert for a given Month and Day (across all years) the view created in couchdb would look like:

“MonthDay”: {

“map”: “function(doc){emit(doc._id.substr(5,2)+doc._id.substr(8,2),[doc.venue , doc.IAKey, doc.tracks ])}”

}

This view allows us to use and HTTP GET to pass in a monthday key (e.g. “1101”) and get back (as a JSON array)

the date (MMDDYY: doc._id.substr(5,2)+doc._id.substr(8,2))

the venue (doc.venue);

the AI URI of the concert (doc.IAKey); and

an array of track data (doc.tracks)

MVC URL Routing Maps

Although we could call couchdb directly from the browser, we normally work through a gateway system for security, so we will build a shim to sit between the browser and couchdb.  This allows us to flow the authentication / authorization stack separately from couchdb’s security system.  In MS MVC we can create a new HTTP endpoint for AJAX calls (our shim) is a very simple manner. Let’s create an endpoint which will look like:

http:\\{our server path}\DeadBase\MonthDay\{month}\{day}

where vacuum_routing

http:\\{our server path}\DeadBase\MonthDay\111

would request month:11 and day:01 concerts.  In MVC we can declare this routing as:

routes.MapRoute(

“MyMonthDay”,

“{controller}/{action}/{month}/{day}”, 

new { controller = “DeadBase”, action = “RestMonthDay”,null} );

Done.  Interestingly in MVC 3 this route definition will accept either the form:

http:\\{our server path}\DeadBase\MonthDay\{month}\{day} ; or

http:\\{our server path}\DeadBase\MonthDay?month=”??”&day=”??”

In the second form,  parameter order does not matter, but case does; quotation marks are optional and need to be dealt with internally by the action method.

either of these call will resolve to the same controller and method.

MVC Controller and Method HandlerMVC

We now need to create the shim which will be the target for the Http Endpoint.  In C# this looks like:

public class DeadBaseController : Controller

public string RestMonthDay( string month, string day )
{
//our shim code goes here

      }

    }

We able to use string as our return type because we will be calling couchdb which returns a string from of JSON by default.  As a side note if we wanted to use MVC 3 to return JSON from a native C# object our controller method takes a different form:

public JsonResult GetStateList()

{

List<ListItem> list = new List<ListItem>() {

new ListItem() { Value = “1”, Text = “VA” },

new ListItem() { Value = “2”, Text = “MD” },

new ListItem() { Value = “3”, Text = “DC” } };

return this.Json(list);

}

Our AJAX call from the browser does not need to know any of these details.  Here is one way to code the call in JavaScript using JQuery:

var url = urlBase + “?” + args;ajax

$.ajax({

url: url,

dataType: ‘json’,

success: okCallBack,

error: nookCallBack

});

function okCallBack(data) {

gdData = data;

//do something useful here

}

function nookCallBack(xhr, ajaxOptions, errorThrown) {

alert(“ErrorText:” + errorThrown + ” ” + “Error Code:” + xhr.status);

}

}

From Handler to CouchDB in C#

Here is the rest of the generic C# code to go from the Handler to CouchDB and back.

Clean the parameters and pass the call to a generic couchDB GET caller:mvc

image

Format the view name and parameter into couchdb format  and pass to the low level couchDB caller:

image

Classic Framework HTTP code to make the HTTP GET and return the results as a string back up the call stack:

image

We could (and did) take our Browser code from the Ruby on Rails project above and with minimum changes call our MVC shim.

Simple clean and fun.

Occupy your mind2

REST, Ruby On Rails, CouchDB and Me – Part 6 Getting The Data Into And Out Of CouchDB   1 comment

Part 0 – REST, Ruby On Rails, CouchDB and Me

Part 1 – Ruby, The Command Line Version

Part 2 – Aptana IDE For Ruby

Part 3 CouchDB Up and Running on Windows

Part 4 – CouchDB, Curl and RUBY

Part 5 – Getting The Data Ready for CouchDB

Part 7 – JQUERY,JPlayer and HTML5

 [This just in: couchDB and Microsoft MVC 3: here]

Our mission in today’s post: 

  • Serialize the Concert object to a JSON string (i.e. a JSON document);
  • Do POST requests to insert  a JSON document for each concert into the couchdb database;
  • Create couchDB views to allow optimized data retrieval; and
  • Create a couchDB view to optimize retrieval recordings for all years for an arbitrary Month and Day (this duplicates the data provided by the “Grateful Dead Shows on This Day In History” selection in the Internet Archive.

Our last post outlined the formal structure of our document and the JSON format of the document we intend to post to our couchdb database.  Also in that post we outlined the RUBY code used to collect and clean the data we accessed from the Internet Archive.  In order to place our concert information into our couchdb database we need to transform our RUBY objects into proper JSON format and then call an HTTP POST to place our resource (document) into the database.  Why a POST rather than a PUT command you ask?  Better to ask a true RESTafarian.  In REST PUT is used to create a resource (in couchdb terms this is the database) and POST is used to modify a resource (in our case create a document within the database – which modifies the database I guess).  Hmmmm.

Why JSON?  Ask These Guys:

dsc_0029

To transform a RUBY object into JSON format requires the use of a RUBY GEM: json.  This GEM comes in two flavors, a pure RUBY form (slower) and a C form (faster, but it requires that you have a working RUBY development stack, GNU C compiler etc. deployed on your local machine).  Since we are happy with the speed of the pure RUBY form.  The downside of this GEM is that it will not serialize pure RUBY objects as is.  To do that you need to code custom serialization handlers. By default, the JSON serializer will work directly on primitive (native) types (strings, ints, what have you), and the simple structures: arrays and hash tables.  So our first task will be to transform our data into hash table format.  Assume a RUBY array of concert objects (@selectList) and a parallel array of data for each track (tracks). In pseudo code we have:

@selectList.each do |recording|
tracks=GDConcertTracks.new(recording)
tracks.getTrackDetails

   jsonString=makeJSON(recording,tracks)
PostRecording(jsonString)
end

 

Our method makeJSON takes the RUBY objects and returns a JSON string as:

def makeJSONFull(recording,tracks)
tList=Hash.new()
tList[“_id”]=recording.date
tList[“IAKey”]=recording.uri
tList[“description”]=recording.description
tList[“venue”]=recording.title
tList[“pubdate”]=recording.pubdate
tList[“cm”]=recording.cm
tList[“sb”]=recording.sb
tList[“mx”]=recording.mx
tTracks=Array.new()
tracks.each do |t|
tItem=Hash.new()
tItem[“title”]=t.title
tItem[“time”]=t.time
tItem[“uri”]=t.trackURI
tItem[“track”]=t.track #this could be derived as offset – 1
tTracks[tTracks.count]=tItem
end
tList[“tracks”]=tTracks
return JSON.generate(tList)
end

where JSON.generate(tList) is the JSON GEM method generate (to serialize) and tList is a Hash containing only primitive types (strings) and a Hash list of track data.  The track data contains only primitive types (strings).  We pass the returned string (which contains our data in JSON format) to our routine PostRecording(jsonString).  This Routine looks like:

def PostRecording(jsonString)
uri=”
http://127.0.0.1:5984/deadbase/&#8221;
begin
responseBody=open(uri,:method=> :post, :body => jsonString,”Content-Type” => “application/json”).read
puts ‘POST Response Success: ‘ + responseBody
rescue OpenURI::HTTPError => the_error
puts ‘##########Post Response Error: ‘ + the_error.io.status[0]
end
end

HiREST-LowREST1

OK this routine depends on the RUBY GEM: rest-open-uri .  This GEM is a single file of RUBY code which depends on the RUBY components: uri and net/http (these are part of the base RUBY system).  “rest-open-uri” extends the base RUBY HTTP components for the HTTP GET verb  and extends them to include all of the HTTP verbs (PUT, POST, DELETE and HEAD)  necessary to implement any REST system.  The open method, as used here:

open(uri,:method=> :post, :body => jsonString,”Content-Type” => “application/json”).read

uri: the address of our couchdb database

Content-Type: our MIME type for this call (“application/json”)

body: our document

method: the HTTP verb POST
If this POST command fails we want to capture the error and (for this example) display the error:

rescue OpenURI::HTTPError => the_error
puts ‘##########Post Response Error: ‘ + the_error.io.status[0]

Actual error recovery is left as an exercise for the reader.

 

OK, we now have data in our database.  In our case this is is 2,014 documents (i.e. 2,014 separate Grateful Dead concerts).  Using Futon we see the database as:

clip_image001

In Futon, we can look at our data:

clip_image001[7]

And we can drill in on an individual document (for example the primordial recording on 1965-11-01):

clip_image001[9]

 

Of course what we need to able to return our document with a simple HTTP GET request.  Further we will want to return only those fields we will actually need from a document (i.e. we need to shape our data to minimize bandwidth).  We do that in couchdb with a view which is defined in a design document.  We can define multiple views in a single design document and can have multiple design documents in a given database. 

For our first view lets set the task to return all documents which occurs on a given month and day (regardless of year).  This is the filter. We want to shape our data to return only a subset of fields: the venue, the IAKey and the document tracks.  The key for this view will be the month+day (in MMDD format).  Our design document might look like:

 {
“_id”: “_design/basic”,
“_rev”: “19-fd2c9b34d2536ce1f187ab2d4e5413de”,
“views”: {
“MonthDay”: {
“map”: “function(doc){emit(doc._id.substr(5,2)+doc._id.substr(8,2),[doc.venue , doc.IAKey, doc.tracks ])}”
}
}
}

 

What this view (called MonthDay) does is maps each document (doc) into a key (doc._id.substr(5,2)+doc._id.substr(8,2)) and an array of return fields:

[doc.venue , doc.IAKey, doc.tracks ]

Note that doc.tracks itself returns an array of tracks.

Using Futon to test our work and find the results for all New Years Day Grateful Dead Concerts.  The first document looks like:

 

clip_image001[11]

(additional lines are omitted from this screen snap).

 

As an HTTP Get Verb we would write:

http://127.0.0.1:5984/deadbase/_design/basic/_view/MonthDay?startkey=%2201-01%22

This command users the

URI: http://127.0.0.1:5984/deadbase

The View: _design/basic/_view/MonthDay

startkey: this is a key word parameter field which tells couchdb to go to the b-tree which represents the MonthDay view and finds the first Key match and continues through the tree returning all Key matches.

If we wanted a range of Keys (say all holiday season Concerts) we can use the startkey and the endkey parameters:

http://127.0.0.1:5984/deadbase/_design/basic/_view/MonthDay?startkey=%221224%22&endkey=%221969-12-31%22

startkey” gives us a starting key in the b-tree and returns all documents until (and including) the b-tree key defined by endkey.

 

JSConf-eu-Alexander-Lang--Writing-apps-on-the-edge-with-CouchDB-e8451113

For more details see the Definitive Guide to CouchDB chapter: Finding Your Data With Views.  The key to simple view definition is defining an appropriate lookup key and defining what data to return.  We can define a simple listing view as:

"Jump": { "map": "function(doc){emit(doc._id,'['+doc.venue+'] ' +'http://www.archive.org/details/'+doc.IAKey)}" },
This returns a lookup key equal to the original key used in the database (doc._id) and the data as venue (doc.venue)
and a URL for the concert recording ('http://www.archive.org/details/'+doc.IAKey). Now if we want to return all
concerts in August, 1969 we can issue the GET:
http://127.0.0.1:5984/deadbase/_design/basic/_view/Jump?startkey=%221969-08-00%22&endkey=%221969-08-31%22

This will return the rows as:{“total_rows”:2013,”offset”:156,”rows”:[ {“id”:”1969-08-02″,”key”:”1969-08-02″,”value”:”[Family Dog at the Great Highway] http://www.archive.org/details/gd69-08-02.sbd.miller.30651.sbeok.flacf&#8221;}, {“id”:”1969-08-03″,”key”:”1969-08-03″,”value”:”[Family Dog at the Great Highway] http://www.archive.org/details/gd1969-08-03.sbd.miller.30652.sbeok.flac16&#8243;}, {“id”:”1969-08-16″,”key”:”1969-08-16″,”value”:”[Woodstock Music] http://www.archive.org/details/gd1969-08-16.sbd.gmb.fixed.95918.flac16&#8243;}, {“id”:”1969-08-21″,”key”:”1969-08-21″,”value”:”[Aqua Theater] http://www.archive.org/details/gd1969-08-21.sbd-part.tremblay.1170.sbeok.shnf&#8221;}, {“id”:”1969-08-23″,”key”:”1969-08-23″,”value”:”[Pelletier Farm] http://www.archive.org/details/gd1969-08-23.sbd.lai.6639.shnf&#8221;}, {“id”:”1969-08-28″,”key”:”1969-08-28″,”value”:”[Family Dog at the Great Highway] http://www.archive.org/details/gd69-08-28.sbd.lepley.4234.sbeok.shnf&#8221;}, {“id”:”1969-08-29″,”key”:”1969-08-29″,”value”:”[Family Dog at the Great Highway] http://www.archive.org/details/gd1969-08-29.sbd.lai.9179.sbefail.shnf&#8221;}, {“id”:”1969-08-30″,”key”:”1969-08-30″,”value”:”[Family Dog at the Great Highway]

]}

Get it? Good.

 
6011428946_f722b3cb71
What Would Dweelze Do?

Now let’s return to RUBY and write a web page which will display concert and track data for any given day in Grateful Dead History. Using the view:MonthView as defined above:

“MonthDay”: {
“map”: “function(doc){emit(doc._id.substr(5,2)+doc._id.substr(8,2),[doc.venue , doc.IAKey, doc.tracks ])}”
}

Lets return data using RUBY for concerts for all years for the Month and day of todays Date.  In Ruby we need to define a controller and a erb HTML file to accomplish this.  Lets say our page will be called player.  Our controller might look like:

require ‘rest-open-uri’

require ‘json’

class PlayerController < ApplicationController
def player
mmdd=Date.today.to_s[5..-1]
mmdd=mmdd.gsub(“-“,””)
base=’
http://127.0.0.1:5984/deadbase/
    view=’_design/basic/_view/MonthDay’
url=base+view+’?key=’
url=url+’%22’+mmdd+’%22′
jsonString=returnJSONString(url)
@parm=jsonString
end
def returnJSONString(url)
jsonString=”
open (url) do |x|
x.each_line do |y|
jsonString=jsonString+y
end
end
return jsonString;
end 

end

This returns the JSON returned from couchdb unaltered as a parameter to the erb page for player (player.html.erb).  If we wanted to work with the data within RUBY we would need to change the JSON back into a RUBY format by calling:

JSON.parse(jsonString)

For our purposes, however, we want to pass the data directly to the browser and will process the data using JavaScript this will in almost all cases be faster than processing the JSON back to RUBY and then formatting the data on the erb page.  In our first pass the route map for our page will be simple:

map.connect ‘player’, :controller => ‘player’, :action => ‘player’

and our first pass at an HTML page will look like this:

<!–DOCTYPE HTML PUBLIC “-//W3C//DTD HTML 4.01//EN” “http://www.w3.org/TR/html4/strict.dtd>
<html>
<head>
<meta http-equiv=”Content-Type” content=”text/html; charset=iso-8859-1″ />
<title>Player: On This Day In Grateful Dead History</title>
<script type=”text/javascript”>
gdData=<%=  @parm  %> ;
</script>
</head>
<body>
<h3>On This Day In Grateful Dead History</h3>
</body>
</html>

This page does absolutely NOTHING…  Except gets our JSON data to the Javascript on the browser page.  And that, dear reader is as far as we need to go with today’s post.

5719769521_0380972a03




	

REST, Ruby On Rails, CouchDB and Me – Part 4 – CURL on Windows And Ruby POST   Leave a comment

Part 0 – REST, Ruby On Rails, CouchDB and Me

Part 1 – Ruby, The Command Line Version

Part 2 – Aptana IDE For Ruby

Part 3 CouchDB Up and Running on Windows

Part 4 – CouchDB, Curl and RUBY

Part 5 – Getting The Data Ready for CouchDB

Part 6 – Getting The Data Into And Out Of CouchDB

Part 7 – JQUERY,JPlayer and HTML5

In The Post:

  • CURL and Couchdb
  • Documents Design and Otherwise
  • Posting Documents to couchDB Using Ruby

If you are like me you have spent some time with the free ebook: CouchDB The Definitive Guide.  If you are a windows user you may have run into some problems with the examples given in the chapter  on “Design Documents”.  Specifically they don’t work ‘out of the box’.  The examples in that chapter show us how to: create a database, to create and post a design document and to post a document to the database.  These examples use  CURL in a command shell.

OmniVortex

Since we are running Windows first we need to install CURL on our system.  Either set your system path to include the CURL executable. We can get a windows version here.  Use the version labeled DOS, Win32- MSVC or Win64 depending on your system. We assume here that couchDB has been installed successfully on your system. Now open a ‘command prompt’ on your system.  If you must have a UNIX type shell you need to install CYWIN or some other UNIX emulator for Windows.  If you are using the Aptana IDE like me you need to create an “external command” to open a command shell within Aptana.  This figure illustrates the setup within the Aptana IDE to do this:

image

In the command shell you can create a couchdb Database using a POST command and CURL.  Couchdb is RESTful so we use a PUT command for all actions which CREATE a resource, of which a database is one example.  The format of the command is:

curl -X PUT http://{couchdb}/{yourdatabasename}I want to create a database named deadbase so on my system this command and response looks like:

C:\Documents and Settings\dredfield\My Documents\Aptana Studio Workspace\couchDB01

>curl -X PUT http://127.0.0.1:5984/deadbase

{“ok”:true}

The where “{“ok”:true}” is the response body of the http response to my put command.  Confirm your work by starting a browser and navigating to Futon user interface to your couchdb installation.  On my system this url is:

http://127.0.0.1:5984/_utils/index.html

you should see something like this:

image

CURL and Documents

OK, now lets make a design document for this database and PUT that document to the new database.  With slight modifications to the example given in CouchDB The Definitive Guide my first cut at a design document looks like this:

{

     “_id” : “_design/example”,

     “views” : {

        “View00” : {

       “map” : “function(doc){emit(doc._id, doc.date)}”

        }

  }

}

This is a JSON formatted document.  Initial syntax checking is up to you.  Basically couchDB will accept anything within the outer brackets whether or not it is formatted as usable JSON or not.  We have several options for checking syntax.  There are free online syntax checkers like JSONLint.  The interface to JSONLint looks like:

clip_image001

An installable open source JSON checker and visualizing tool, JSON View is available here.  JSON View’s output looks like:

clip_image001[10]

Now that we know our syntax is correct (if not the logic of the design document – more on this in the next installment) we can PUT this document to our database.  We can have more than one design document in a given database.  The name (id) of this document is “_design/example”.  where “_design” tells couchdb this is indeed a design document and its name is “example”.   My document is named mydesign.json on my file system.  The CURL command to PUT this into the database looks like:

curl -X PUT http://127.0.0.1:5984/deadbase/_design/example -d @mydesign.json

couchdb will respond:

{“ok”:true,”id”:”_design/example”,”rev”:”1-45f081a3f681b28ce7a0bdc5db216e74″}

Note here that this is NOT the syntax shown in CouchDB The Definitive Guide.  The syntax there will not work in a windows shell (i.e. command prompt).  Even when you have syntax correct JSON document  and the correct format of the PUT statement on Windows you may recieve an error message from CURL complaining about UTF8 errors within the document and have a PUT failure.  The problem here is that the Windows file system supports several encoding schemes and various windows programs save documents in to different default encoding.  If you are using Notepad.exe to create your files be sure to save the files in ANSIformat.

 
Check your work using the FUTON interface locate the “_design/example document” in deadbase

clip_image001[12]

Double click on the document:

clip_image001[16]

Note that “views” is a “Field” within the document.  Select the “Source” tab  and take a look inside the document:

clip_image002[4]

Now lets POST a document into the database.  Since we have not defined any validation fields we can push anything into the database.  Even documents which consist of just “{}”.  CouchDB defines only one innate restriction:

If a document defines the id field (“_id”) then the value of _id must not conflict with an existing value of the ID field of ANY other document in the database.

If the document does not define an ID field, couchDB will generate an ID (as a UUID) and apply it to the document.  You can supply your own ID values.  If you can either generate your own value  (Ruby can generate a GUID for you) or you can request a GUID from couchdb with a GET command.  See this page for more information.  In the sample program I will be developing for this series I will be using a ‘natural key’ – that is a key whose value has an actual meaning (a Social Security is such a natural key for example, but please never use this).  If you try to POST a document and use a duplicate key you will get back a 409 status code for the error.

The document I will be using in the next post looks like this:

{

“_id” : “1972-07-22”,

“IAKey” : “gd1972-07-22.sbd.miller.94112.sbeok.flac16”,

“description” : “Set 1 Bertha Me And My Uncle You Win Again Jack Straw Bird Song Beat It On Down The Line Sugaree Black Throated …

“pubdate”: “2008-08-15”,

“sb”: true,

“cm”: true,

“mx”: false,

“venue”: “Paramount Northwest Theatre”,

}

If I save this document as ConcertRecord.json I can use CURL to POST this document as:

curl -H “Content-Type: application/json” -X POST http://127.0.0.1:5984/deadbase/ -d @ConcertRecord.json

and couchdb will reply with an HTTP status 200 and a response body of:

{“ok”:true,”id”:”1972-07-22″,”rev”:”1-01a182f329c40ba3bab4b13695d0a098″}

In couchDB Futon this document looks like:

clip_image001[20]

Note that the order of the fields is set by couchDB not the order in the first loaded document.

Ruby At Last

OK, enough of the command shell let’s do some couchDB work using RUBY.  I am going to access couchDB from a fairly low level within Ruby in these posts.  There are several ActiveRecord type GEMS which will interface with couchDB but my focus here will be on: (1)  speed of access and (2) transferability of knowledge between Ruby access and direct Javascript/Browser access to couchDB.

Here’s a minimum of what we need to POST a document to a couchdb using RUBY.

The GEMS for

JSON : This will always load the Ruby based version of the JSON module.  If you want to have ‘pure’ JSON (i.e. a C based module you will need to have the Ruby/Windows DEVKit installed on your system.  For our purposes the ‘pure’ version is not necessary.

REST-OPEN-URI:  This extends open-uri by using the net/http  and the uri GEMs to cover all of the REST verbs (GET, POST, PUT and DELETE).  This is a very light install and is only lightly documented.

Here is the basic plan:

Assume we have a RUBY object (call it “rec”) which includes, among other things the fields we want to POST into the deadbase as a deadbase document like the one developed above.  We first need to convert the fields into a JSON string and then to POST the JSON string into the deadbase.  The JSON GEM is used to achive the first goal and REST-Open-URI is used to accomplish the second.

JSON Strings:

The JSON GEM will only serialize Ruby base types (strings, numbers and bools and HASH objects).  The JSON GEM is quite limited in that it will not serialize a Ruby object derived from the base RUBY object  into a JSON string, even if that object consists only of base types and Hash objects.  Although you may extend JSON we did not choose to do so. Rather we will create a simple Hash object and populate it manually via Ruby code with the fields we want to use for a document. Simply this could look like:

def makeJSON(rec)

thing=Hash.new()  #we know that JSON can serialize this type of object

thing[“_id”]=rec.date

thing[“IAKey”]=rec.uri

thing[“description”]=rec.description

thing[“venue”]=rec.title

thing[“pubdate”]=rec.pubdate

thing[“cm”]=rec.cm

thing[“sb”]=rec.sb

thing[“mx”]=rec.mx

return JSON.generate(thing)  #this returns a JSON String

end

REST-OPEN_URI:

Our POST routine will use the output form makeJSON and POST the JSON string to the deadbase.  In simple for this routine looks like:

def PostRecording(jsonString)

uri=”http://127.0.0.1:5984/deadbase/”   #this is our database

begin

responseBody=open(uri,:method=> :post, :body => jsonString,”Content-Type” => “application/json”).read

puts ‘POST Response Success: ‘ + responseBody

end

rescue

OpenURI::HTTPError => the_error

puts ‘Post Response Error: ‘ + the_error.io.status[0]

end

end

The key line is, of course:

responseBody=open(uri,:method=> :post, :body => jsonString,”Content-Type” => “application/json”).read

If we ran this line as:

responseBody=open(uri,:method=> :post, :body => jsonString).read

we would get an http Status code for an “Invalid Media Type”.  That’s because the default “Content-Type” for POST commands is “application/xxx-form” which is the typical format of a HTML “form” involved in a POST from a web browser.  We are far from a browser here and our “Content-Type” needs to be “application/json”.  The way to add Headers to the POST is to provide one or more key/value pairs with the desired header information.  Hence:

“Content-Type” => “application/json”

and the correct Ruby line is:

responseBody=open(uri,:method=> :post, :body => jsonString,”Content-Type” => “application/json”).read

We need to wrap the POST command in an exception block where the line:

OpenURI::HTTPError => the_error

is only executed IF the Http response status is > 399.  You can then do more fine grained responses to the error condition.  Specifically, if the_error.io.status[0]==409 you have attempted to POST the same document twice (at least two documents with the same ID).

That looks like a wrap for now.

5901067736_08fe849334_z

Posted 2011/07/22 by Cloud2013 in Aptana, couchdb, REST, Ruby

Tagged with , , , ,

REST, Ruby On Rails, CouchDB and ME – Part 2 – Aptana IDE For Ruby   3 comments

Part 0 – Introduction to Ruby, RadRails and CouchDB

Part 1 – REST, Ruby On Rails, CouchDB and ME – Part 1 – Ruby

Part 2 – This Post

Part 3 CouchDB Up and Running on Windows

Part 4 – CouchDB, Curl and RUBY

Part 5 – Getting The Data Ready for CouchDB

Part 6 – Getting The Data Into And Out Of CouchDB

MaizeGod

Although not found too often on Windows programming system, the open source programmer’s editor Eclipse is one of the most popular in the world.  The Aptana Corporation (now part of Appcelerator) has developed a Ruby and a Ruby on Rails plug in for this popular browser.  The Aptana plug-in are fully open sourced.  Ruby for Rails version 2.0 is currently supported.  In this post we will not discuss The Ruby On Rails plug-in (that’s for a later post), herein we will look at installing Aptana community editor and getting simple Ruby projects up and running.  Start by downloading Aptana Studio 2.0 from the Aptana site. Install New Features We will add RadRails to Aptana Studio after we install the base Aptana system.  Although Aptana will allow you to down load just the RadRails Editor, if you do this you will not get the full development environment this way.  Trust me.   Be sure you select the Windows installer and proceed to install the Aptana Editor just as you would any other Windows product.  If you are on Vista (and perhaps Windows 7 – who knows) you will want to modify the shortcut for Aptana as outlined here.

Now that you have the base product installed, start the editor, select help and select “Install Aptana Features”.  This gets you to a selection screen where you can select (Under Web Application Platforms)  Aptana RadRails (this is the plug in for RadRails 2.0) and (Under JavaScript Libraries) JQUERY Support (trust me).  Click on Install and follow the prompts.  When you get done we will be ready to rock.  Assuming you have only ONE version of RUBY installed on your machine and that version is on the Path statement the Ruby and RADRails system can figure our how to install itself.  Otherwise you will need to do some manipulations of your Preferences for Ruby and Rails.  The preference screen can be found under the View menu item.

Ruby At LastGems

Aptana provides “Perspectives” for developing Ruby and RadRails Projects.  For each perspective there are multiple “Views” for working with each project type.  Only one perspective can be open at a time but multiple views can (and will) be in use within each perspective.  For example with the Ruby Perspective, we can select the GEM view.  This will give us a nice GUI with which to add, remove and update the GEMs associated with the current version of Ruby we have installed. Nice.Perspective

Let’s start with Ruby (not Ruby on Rails).  To start coding:  Select ‘”View”, select “Open Perspective” and Select “Ruby”.  Select “File”, Select “New”, Select “Ruby Project”.  Give your Project A Name: Let’s use “ClassTutorial”.  Right Click “ClassTutorial” in the tree view of the Ruby Perspective and select “New” and “Ruby File”.  Give this first file the name: “ClasTutorialMain.rb”.  A file editor view will open for entering text into this file.  Enter the following text into the file editor:

class Person   attr_accessor :fname, :lname   def initialize(fname, lname)          @fname = fname          @lname = lname   end   def to_s     @lname + ", " + @fname   end   end puts 'Create New Person' matz = Person.new("Yukihiro", "Matsumoto") puts matz puts 'exit' exit 

Now right click on the view editor.  RIght click on extreme right of line #17 and select “toggle breakpoint”.  Select “Run” from the main menu and “Debug As” and “Ruby Application”.  Now a couple of other ‘views’ will open (Console & Debug).  Your screen should look something like this. Untitled Note that your step debugger controls are in the top right of the Debug Window.  Your watch statements are on the right.  Console output will be displayed in the Console window on the bottom.  You can, of course, positions these windows as you see fit.  While not heaven it is ‘nice’.  Either step through the code or press the RED X to terminate the current run.  To run without the step debugger select Run/Debug As/Ruby Application.

In the next post we will look at some of this code (and more) as we get our feet wet with Ruby and this IDE.

MIX11_BB_SeeYouAt_1 tumblr_lha4s0MgCk1qzt4vjo1_500

Posted 2011/03/15 by Cloud2013 in Aptana, Ruby

Tagged with , , ,

REST, Ruby On Rails, CouchDB and ME – Part 1 – Ruby   Leave a comment

Part 0 – Introduction to Ruby, RadRails and CouchDB

Part 1 – This Post

Part 2 – Aptana IDE For Ruby

Part 3 –CouchDB Up and Running on Windows

Part 4 – CouchDB, Curl and RUBY

Part 5 – Getting The Data Ready for CouchDB

Part 6 – Getting The Data Into And Out Of CouchDB

Part 7 – JQUERY,JPlayer and HTML5

We installed Ruby for Windows from RubyInstaller (for Windows) MIX11_BB_SeeYouAt_1. We installed Ruby 1.8.7-p180 and its associated documentation.  The documentation is in p7zip format – so you might need 7zip imagefor windows to open up the documentation.  Download a 7Zip installer  here.  I allowed the windows installer for windows to place the Ruby binaries and helper files into it default location (c:\Ruby187).

Since Aptana (which will will use for Rails development) is touchy about changing Ruby versions this is a better option than placing Ruby in a generic location like C:\Ruby.  To use Ruby scripts or interactively you will need to have ruby on your path.  In my case the path value needs to include c:\Ruby187\bin.  Note that this is the executable path for Ruby and irb (interactive Ruby) and not the load path (aka $Load_Path).  I put the path into my local user environment set.  As you develop your Ruby chops you can write short Ruby programs in the text editor of your choice, save them with an extension of “rb”  and then run them from a command shell as:   image For quick learning experiences there is also a Ruby line interpreter (irb).   Run it as:clip_image001

Ruby.exe (and irb) can be run with an assortment of  command line options.  Find out about these by running:  Ruby –h and irb –h.  There are several ways to install support libraries to Ruby.  The most common method is to use Ruby GEMS.  GEMS support is pre-loaded with Ruby 1.8.7 and can be accessed through the command shell and (as usual) minimal help canclip_image001[9] be found using the –h command line option.  So there you have it for command line Ruby. A Ruby compiler and Why’s (poignant) Guide To Ruby is all you need to get started.   In my next installment to this series we will move on to a Ruby IDE – specifically the Aptana RadRails (and Ruby) IDE.

small

Posted 2011/03/11 by Cloud2013 in Ruby

Tagged with ,

REST, Ruby On Rails, CouchDB and Me – Part 0   4 comments

Part 0 – This Post

Part 1 – Ruby, The Command Line Version

Part 2 – Aptana IDE For Ruby

Part 3 – CouchDB Up and Running on Windows

Part 4 – CouchDB, Curl and RUBY

Part 5 – Getting The Data Ready for CouchDB

Part 7 – JQUERY,JPlayer and HTML5

The is the first in asmall series of blog posts concerning what is often known as as Open Source Stack for Web Development.  My interests here are in WEB 2.0 development.  We will be working towards using JQUERY and CouchDB to develop REST(ful) Web sites.  I will start with a focus on Rudy and Ruby on Rails (aka RadRails).   However, I will be using free and open source products not on UNIX but on Windows based systems.  Windows runs on fairly cheap hardware (compared to Apples UNIX based platforms) and Windows is much more accessible to new developers than Linux systems.  I will be working here with Windows XP and Windows Vista but you can play alone on Linux and MAC x86.  To get started let’s get together some software and supporting documentation and begin.

Ruby

Version 1.8.7 is the current stable release but since we are interested in using CouchDB I have installed the required ‘edge’ version 1.9.2.  Either of these version will support Ruby on Rails development. Ruby installations are available from several sites. Until you know exactly what you are doing stick with distributions from RubyInstaller.org since these are the baseline Ruby builds which everyone tests against first.  As you get deeper into it (or if  things get a little more sticky)  you try an integrated open stack like the one offered by BitNami.  At some point you may need to install the DevKit.  This is not as heavy as it sounds (but you will need to work with your system paths to get everything in place).  You can get the DevKit here and installation instructions are here.   Note:  the devkit download is now a Windows MSI file but the instructions are still based on a ZIP file extraction, but it all works trust me.  Don’t install either the BitNami Stack or the DevKit until you know you need it.   If you completely new to Ruby a nice introduction can be found in Why’s Poignant Guide To Ruby.

Ruby On Rails

There are a lot of different options to developing Ruby on Rails applications.  If I was not an IDE freak I could use  a simple text editor (I often use the non-open Primal Script) but being new to Ruby on Rails I wanted the support of and IDE.  For this option I selected the Aptana RadRails Development Environment.  This free development environment is based on the open Eclipse editor.  I downloaded the base Aptana IDE (Studio 2) and then added RADRails as a plug in.  These are available in Linux and Mac x86 installers in addition to windows.  You could install only RADRails but then you would have a crippled version of the Aptana product.  We will be noting Ruby on Rails training materials as we move along.

CouchDB

Although we will be using mySQL or SQLlite for some of our development, our real content manager will be the NOSQL couchDB.  This is our real goal in the project – testing the viability  of a REST(ful) HTTP addressable NOSQL database.  This project is from the Apache organization and is available for Linux, MAC x86 and Windows.  It runs as a service on the Windows OS.  The Windows installer is available here.  There is an excellent open source on line couchDB book available here.  For Ruby on Rails we will be using – CouchRest for our Ruby on Rails work with couchdb.  CouchRest is available as a ruby GEM and can be installed on Linux, MAC x86 and Windows versions of Ruby.

JQUERY

Web 2.0 is not possible with out modern, sophisticated JavaScript libraries.  One of the best of these is JQUERY and not surprisingly couchDB ships with a powerful jquery library (couchdb.js) to facilitate browser side manipulation of couchdb data.  For browser work you should be pulling your JQuery from CDN Host.  For use within Ruby for Rails project you will need to add JQUERY  support to your Aptana IDE.  JQuery works with all modern browsers (and even some which not).PacalII

%d bloggers like this: