Archive for the ‘JSON’ Category

My (Virtual) Year On Tour With the Grateful Dead   1 comment

 

From Tape to The Internet

Crimson flames tied through my ears
Rollin’ high and mighty traps
Pounced with fire on flaming roads
Using ideas as my maps
“We’ll meet on edges, soon,” said I
Proud ’neath heated brow
Ah, but I was so much older then
I’m younger than that now

My Back Pages by Bob Dylan

I first heard the Grateful Dead live in Albuquerque in 1971.  Like many, this was a seminal experience for me, changing my understanding of the meaning of performance and of Rock and Roll.  That over weight, black clad, Prankster with a halo of unmanageable black hair playing guitar was clearly the center of the band and the performance.  I was young and was unaware that this was Captain Trips.   Captain America was more like it.  Nor was I aware of the mythical aura that was even then growing around the band and Jerry and their fans.  Like many, my first experience of the Dead was prepared only by my youth and The Bear’s  purple haze of the night.  In the intervening years I was more attracted to Frank Zappa who produced a consistent recording experience that the Dead never would achieve. After the (limited) success of the Grateful Dead Movie in capturing what the Dead were, the world moved on. But the band played on.  Perhaps it is better that way.  The early taping of Dead shows from  the sound boards (thank you Owsley) and later by dedicated deadhead tapers left us with a rich vein of music and magic in the over 3000 individual live performances available in one form or another.  Beginning in the last decade of the last century, the Dead organization began to issue live sound board recordings from this corpus. 

Dick’s Picks and Me

Half-wracked prejudice leaped forth
“Rip down all hate,” I screamed
Lies that life is black and white
Spoke from my skull. I dreamed
Romantic facts of musketeers
Foundationed deep, somehow
Ah, but I was so much older then
I’m younger than that now

While a great resource for those of us not conversant in bit torrent, these releases where frequently expensive and (to my ears) over produced which subtracted from the raw energy of the original sound board tapes.  I am lucky to have access to a great music store with plentiful numbers of used Grateful Dead CD.  If you are in Maine visit a local Bull Moose Music store.   I was luck to be able to pick up many Live Dead concert CDs at a reasonable price.  Thank you Bull Moose.  Although the official Dead releases enhanced my life, there are some problems:

I) The list prices are quite high (try to get used copies)

II) Selections for the Dick’s Pick Series seem to be primarily based on best complete shows (The Dead were often hit and miss in the same night and limiting yourself to the best complete show skips a lot of great music. This problem has been reduced by the newer Road Trips Series and specialty releases like Ladies and Gentlemen… the Grateful Dead which cooks down the best of a four night stand at the Fillmore, 1971).

III) The processing of the raw tapes, IMHO, sometimes cook the life out of some the releases.

IV) Some of the specialty releases seem to be picked more for the historical importance than the quality of the performance (Closing the Winterland, for example).

Having said all that, if you can get the official releases used, some of them are great.

A Short Divergence in Our Story

Girls’ faces formed the forward path
From phony jealousy
To memorizing politics
Of ancient history
Flung down by corpse evangelists
Unthought of, though, somehow
Ah, but I was so much older then
I’m younger than that now

I started to cooked down my copies of the official live releases into playlist CDs (favorites of 1974, Dark Star releases, etc.).  Then I had open heart surgery, caught a post operative wound infection and almost died.  Stephen Gould wrote someplace that the greatest species in evolution are Bactria.  They are everywhere.  There are more Bactria in your body than body cells.  And I was in the three month war between the bugs and myself ( to be honest I had massive antibiotic infusions on my side).  My day was composed of pain medicine, James Joyce’s Ulysses, The Bible, and my CDs of live Dark Star performances.  Let’s just say that Dark Star and the Gospel of Mark were more significant than Tramadol in my recovery.   Rehab consisted of countless hours of treadmill work.  That, and an MP3 player packed with Scarlet Begonia and Fire on the Mountain.

BTW: Tom Constanten said somewhere that they didn’t play Dark Star, it was always going on,  they just joined in.  Although T.C. recommends ‘any East Coast Dark Star’ my favorites are early West Coast versions.

The Internet Archive Connection

In a soldier’s stance, I aimed my hand
At the mongrel dogs who teach
Fearing not that I’d become my enemy
In the instant that I preach
My pathway led by confusion boats
Mutiny from stern to bow
Ah, but I was so much older then
I’m younger than that now

The Internet Archive, in early 2000 began collecting, digitalizing and making available for re-distribution the large body of Grateful Dead concerts made by independent tapers and sound board recordings (SB) which were in circulation.  By policy SB recordings are available for playing on the web site and non-SB recordings are available for downloading.  There are multiple recordings available for most shows and these vary in quality from commercially releasable to barely audible.  There are over 8,000 individual recording of about 1,900 shows.  About 1,000 of these are SB. While vast the Internet Archive is not the most accessible site. Like most people I started with the feature of the Grateful Dead collection called:

Click Me:

 Grateful Dead Shows on This Day In History 

(If you have never been there – try the link right now).

For 08-30 (today while I am writing this) the Internet Archive will display 30 recordings (for shows of this date in 1985, 1983, 1981, 1980, 1970 and 1969).  There are limited sort options of these results.  Selecting a given recording brings one to a new web page containing an online player and (if the show is not an SB) download options.  I was hooked on the musical possibilities but trapped  by the limited user interface of the Internet Archive.   I wanted more.  Much more…

Hacking The Internet Archive

A self-ordained professor’s tongue Too serious to fool 
Spouted out that liberty
Is just equality in school
“Equality,” I spoke the word
As if a wedding vow
Ah, but I was so much older then
I’m younger than that now

My goal was to have the ability to listen to ALL of the Dead’s concerts but using only the best recordings and to be able to move through the collection using a better user interface which would allow me to decide where and when to go to any individual date. My goal was to spend a year and at least sample all 1,900 concerts and listen completely to all SB concert recordings.  I decided to complete this project in 12 months.  To do this I would first need to wrestle the Internet Archive (IA) to its knees.  Little did I know that this would take me on a programming journey involving three programing languages (Ruby, Javascript and C#), two data specifications (XML and JSON), two database engines (couchdb and SQL Server) as well as understanding the (somewhat loosely documented) search engine of IA, and more….  Readers interested in the technical details should see my series of postings on Ruby on Rails and CouchDB.  Please see Part 5 has the details of how to hack the Internet Archive to get at the data for the Grateful Dead recordings on IA.  Thus armed with the complete dataset from the Internet Archive of Grateful Dead recordings and a new front end I was ready to begin my listening  project.  There are over 8,000 recordings of over 2,000 concerts on the Internet Archive.  My first cut on the recordings in to use an algorithm to select ONE recording for each recording date for review.  This is a very simple selection based on the first of:

  • Was processed by Charlie Miller (IMHO the BEST processor of Grateful Dead Tapes)
  • Is a Sound Board Recording
  • Is  a Matrix Recording
  • Is the most recently posted tape for a given date.

Does this process miss some gems? Undoubtedly but it did give me 2,000 tapes to review rather than 8,000. With this criteria in place,  my local copy of the IA database and my own UI for IA I started listening in July, 2011.  I did not attempt to listen to all 2,000 recordings completely.  If a recording was of poor quality or the band was out of tune or Jerry was ‘uninspired’ I abandoned the tape have brief samples of my favorite tunes.  In the end I reviewed about 1,000 concerts by in thirteen months (I finished during the ‘days between’ period;[August 1 and  August 9]).  I ended up with about 475  concerts on my personal playlist of ‘greatest concerts’.  Along the way I wrote several reviews on this blog of concerts which I thought were particularly of note. and compiled hyperlinked list of shows by year (the series starts here) and hyperlinks to Dark Star concerts and Scarlet Begonia –> Fire on the Mountain concerts.  All of these blogs contain links to jump right into the concert within the Internet Archive (but you still need to use the IA music player however).  Do I have a favorite sequence of songs, a favorite concert, a favorite era.  Yes.  Am I going to tell you? No.  Dig in visit the Internet Archive and start listening.  It could save your life.

 

Days Between Grateful Dead

and there were days
and there were days I know
when all we ever wanted
was to learn and love and grow
Once we grew into our shoes
we told them where to go
walked halfway around the world
on promise of the glow
stood upon a mountain top

walked barefoot in the snow
gave the best we had to give
how much we’ll never know we’ll never know

Days Between by Garcia and Hunter

 

 

 

 

Was It Worth The Trip?

 

Yes!

 

                                                               To Bear and Captain Trips, we say Thank You and Rest In Peace.

All photos by cloud2013 except Bear and Captain Trips Credit: Rosi McGee 

 

PS: Stupid Grateful Dead Statistics From the Internet Archive Database

Top 12 Most Played By Era (excluding Space and Drums):

Title 1967-1971 1972-1978 1979-1990 1991-1995
Althea     *  
Big River   *    
Brown Eyed Women   *    
Casey Jones *      
Cassidy     *  
China Cat Sunflower *      
Corrina       *
Crazy Fingers       *
Cryptical Envelopment *      
Cumberland Blues *      
Dark Star *      
Deal   *    
El Paso   *    
Estimated Prophet     *  
Eyes Of ThWorld       *
Good Lovin *      
Hard to Handle *      
I Know You Rider *   *  
Jack Straw   *    
Lazy River Road       *
Little Red Rooster     *  
Looks Like Rain     *  
Me and My Uncle *      
Mexicali Blues   *    
         
Not Fade Away * * * *
Playing In ThBand   * *  
Sugar Magnolia   * * *
Sugaree   *    
Tennesse Jed   *    
Terrapin Station       *
The Other One     *  
Throwing Stones       *
Truckin   * *  
Turn On Your Lovelight *      
Uncle Johns Band *     *
Wang Dang Doodle       *
Way To Go Home       *
Wharf Rat     *  
When I Paint My Masterpiece       *

Internet Archive:  All Recordings and Sound Board Recordings

image

Concert Length

image

Song Counts By Year(Dark Star, Playin’ in the Band and Scarlet Begonia –> Fire On The Mountain)

ffimage

Microsoft MVC 3 and CouchDB – Low Level Get Calls   1 comment

I have written elsewhere on couchdb on Windows and using Ruby on Rails to interface to this system.  These posts can be found here:couchdb

Part 0 – REST, Ruby On Rails, CouchDB and Me

Part 1 – Ruby, The Command Line Version

Part 2 – Aptana IDE For Ruby

Part 3 CouchDB Up and Running on Windows

Part 4 – CouchDB, Curl and RUBY

Part 5 – Getting The Data Ready for CouchDB

Part 6 – Getting The Data Into And Out Of CouchDB

Part 7 – JQUERY,JPlayer and HTML5

In my work life I work in a Microsoft shop which for us means Microsoft servers for the back end and (mostly) pure HTML/AJAX frontends.  We are transitioning towards using Microsoft MVC 3 to provide HTTP end points for our AJAX calls.  Here are some notes from my POC work in this area.  My couch data consists of documents describing Grateful Dead concerts stored on the great site Internet Archive, if you have never visited the Internet Archive, please do so.  I back engineered the meta data of IA’s extensive collection of Dead concerts (over 2,000 concert recordings).  Visit the Grateful Dead Archive Home at the Internet Archive here.

CouchDB Documents and Views

I stored the meta data into a local couchdb (running on Windows XP).  The basic document I am storing is a master detail set for the ‘best’ recording for each Dead concert.  The Master part of the document contains the date, venue and other data of the concert and the detail set is an array of meta data on each song preformed during the concert.  As is traditional with couchdb, the documents are represented as JSON strings.  Here is what the document for the UR recording (1965-11-01) found on the IA:

{

“_id”: “1965-11-01”,tumblr_ld4jfoNw7F1qai6ym

“_rev”: “1-6ea272d20d7fc80e51c1ba53a5101ac1”,

“mx”: false,

“pubdate”: “2009-03-14”,

“sb”: true,

“venue”: “various”,

“tracks”: [

{

“uri”: “http://www.archive.org/download/gd1965-11- 01.sbd.bershaw.5417.sbeok.shnf/Acid4_01_vbr.mp3”,

“track”: “01”,

“title”: “Speed Limit”,

“time”: “09:48”

},

{

“uri”: “http://www.archive.org/download/gd1965-11-01.sbd.bershaw.5417.sbeok.shnf/Acid4_02_vbr.mp3”,

“track”: “02”,

“title”: “Neil Cassidy Raps”,

“time”: “02:19”

}

]

}

Couchdb allow the creation of views which are binary trees with user defined Keys and user defined sub sets of the document data.  If one wanted to return the venue and the tracks for each concert for a given Month and Day (across all years) the view created in couchdb would look like:

“MonthDay”: {

“map”: “function(doc){emit(doc._id.substr(5,2)+doc._id.substr(8,2),[doc.venue , doc.IAKey, doc.tracks ])}”

}

This view allows us to use and HTTP GET to pass in a monthday key (e.g. “1101”) and get back (as a JSON array)

the date (MMDDYY: doc._id.substr(5,2)+doc._id.substr(8,2))

the venue (doc.venue);

the AI URI of the concert (doc.IAKey); and

an array of track data (doc.tracks)

MVC URL Routing Maps

Although we could call couchdb directly from the browser, we normally work through a gateway system for security, so we will build a shim to sit between the browser and couchdb.  This allows us to flow the authentication / authorization stack separately from couchdb’s security system.  In MS MVC we can create a new HTTP endpoint for AJAX calls (our shim) is a very simple manner. Let’s create an endpoint which will look like:

http:\\{our server path}\DeadBase\MonthDay\{month}\{day}

where vacuum_routing

http:\\{our server path}\DeadBase\MonthDay\111

would request month:11 and day:01 concerts.  In MVC we can declare this routing as:

routes.MapRoute(

“MyMonthDay”,

“{controller}/{action}/{month}/{day}”, 

new { controller = “DeadBase”, action = “RestMonthDay”,null} );

Done.  Interestingly in MVC 3 this route definition will accept either the form:

http:\\{our server path}\DeadBase\MonthDay\{month}\{day} ; or

http:\\{our server path}\DeadBase\MonthDay?month=”??”&day=”??”

In the second form,  parameter order does not matter, but case does; quotation marks are optional and need to be dealt with internally by the action method.

either of these call will resolve to the same controller and method.

MVC Controller and Method HandlerMVC

We now need to create the shim which will be the target for the Http Endpoint.  In C# this looks like:

public class DeadBaseController : Controller

public string RestMonthDay( string month, string day )
{
//our shim code goes here

      }

    }

We able to use string as our return type because we will be calling couchdb which returns a string from of JSON by default.  As a side note if we wanted to use MVC 3 to return JSON from a native C# object our controller method takes a different form:

public JsonResult GetStateList()

{

List<ListItem> list = new List<ListItem>() {

new ListItem() { Value = “1”, Text = “VA” },

new ListItem() { Value = “2”, Text = “MD” },

new ListItem() { Value = “3”, Text = “DC” } };

return this.Json(list);

}

Our AJAX call from the browser does not need to know any of these details.  Here is one way to code the call in JavaScript using JQuery:

var url = urlBase + “?” + args;ajax

$.ajax({

url: url,

dataType: ‘json’,

success: okCallBack,

error: nookCallBack

});

function okCallBack(data) {

gdData = data;

//do something useful here

}

function nookCallBack(xhr, ajaxOptions, errorThrown) {

alert(“ErrorText:” + errorThrown + ” ” + “Error Code:” + xhr.status);

}

}

From Handler to CouchDB in C#

Here is the rest of the generic C# code to go from the Handler to CouchDB and back.

Clean the parameters and pass the call to a generic couchDB GET caller:mvc

image

Format the view name and parameter into couchdb format  and pass to the low level couchDB caller:

image

Classic Framework HTTP code to make the HTTP GET and return the results as a string back up the call stack:

image

We could (and did) take our Browser code from the Ruby on Rails project above and with minimum changes call our MVC shim.

Simple clean and fun.

Occupy your mind2

REST, Ruby On Rails, CouchDB and Me – Part 6 Getting The Data Into And Out Of CouchDB   1 comment

Part 0 – REST, Ruby On Rails, CouchDB and Me

Part 1 – Ruby, The Command Line Version

Part 2 – Aptana IDE For Ruby

Part 3 CouchDB Up and Running on Windows

Part 4 – CouchDB, Curl and RUBY

Part 5 – Getting The Data Ready for CouchDB

Part 7 – JQUERY,JPlayer and HTML5

 [This just in: couchDB and Microsoft MVC 3: here]

Our mission in today’s post: 

  • Serialize the Concert object to a JSON string (i.e. a JSON document);
  • Do POST requests to insert  a JSON document for each concert into the couchdb database;
  • Create couchDB views to allow optimized data retrieval; and
  • Create a couchDB view to optimize retrieval recordings for all years for an arbitrary Month and Day (this duplicates the data provided by the “Grateful Dead Shows on This Day In History” selection in the Internet Archive.

Our last post outlined the formal structure of our document and the JSON format of the document we intend to post to our couchdb database.  Also in that post we outlined the RUBY code used to collect and clean the data we accessed from the Internet Archive.  In order to place our concert information into our couchdb database we need to transform our RUBY objects into proper JSON format and then call an HTTP POST to place our resource (document) into the database.  Why a POST rather than a PUT command you ask?  Better to ask a true RESTafarian.  In REST PUT is used to create a resource (in couchdb terms this is the database) and POST is used to modify a resource (in our case create a document within the database – which modifies the database I guess).  Hmmmm.

Why JSON?  Ask These Guys:

dsc_0029

To transform a RUBY object into JSON format requires the use of a RUBY GEM: json.  This GEM comes in two flavors, a pure RUBY form (slower) and a C form (faster, but it requires that you have a working RUBY development stack, GNU C compiler etc. deployed on your local machine).  Since we are happy with the speed of the pure RUBY form.  The downside of this GEM is that it will not serialize pure RUBY objects as is.  To do that you need to code custom serialization handlers. By default, the JSON serializer will work directly on primitive (native) types (strings, ints, what have you), and the simple structures: arrays and hash tables.  So our first task will be to transform our data into hash table format.  Assume a RUBY array of concert objects (@selectList) and a parallel array of data for each track (tracks). In pseudo code we have:

@selectList.each do |recording|
tracks=GDConcertTracks.new(recording)
tracks.getTrackDetails

   jsonString=makeJSON(recording,tracks)
PostRecording(jsonString)
end

 

Our method makeJSON takes the RUBY objects and returns a JSON string as:

def makeJSONFull(recording,tracks)
tList=Hash.new()
tList[“_id”]=recording.date
tList[“IAKey”]=recording.uri
tList[“description”]=recording.description
tList[“venue”]=recording.title
tList[“pubdate”]=recording.pubdate
tList[“cm”]=recording.cm
tList[“sb”]=recording.sb
tList[“mx”]=recording.mx
tTracks=Array.new()
tracks.each do |t|
tItem=Hash.new()
tItem[“title”]=t.title
tItem[“time”]=t.time
tItem[“uri”]=t.trackURI
tItem[“track”]=t.track #this could be derived as offset – 1
tTracks[tTracks.count]=tItem
end
tList[“tracks”]=tTracks
return JSON.generate(tList)
end

where JSON.generate(tList) is the JSON GEM method generate (to serialize) and tList is a Hash containing only primitive types (strings) and a Hash list of track data.  The track data contains only primitive types (strings).  We pass the returned string (which contains our data in JSON format) to our routine PostRecording(jsonString).  This Routine looks like:

def PostRecording(jsonString)
uri=”
http://127.0.0.1:5984/deadbase/&#8221;
begin
responseBody=open(uri,:method=> :post, :body => jsonString,”Content-Type” => “application/json”).read
puts ‘POST Response Success: ‘ + responseBody
rescue OpenURI::HTTPError => the_error
puts ‘##########Post Response Error: ‘ + the_error.io.status[0]
end
end

HiREST-LowREST1

OK this routine depends on the RUBY GEM: rest-open-uri .  This GEM is a single file of RUBY code which depends on the RUBY components: uri and net/http (these are part of the base RUBY system).  “rest-open-uri” extends the base RUBY HTTP components for the HTTP GET verb  and extends them to include all of the HTTP verbs (PUT, POST, DELETE and HEAD)  necessary to implement any REST system.  The open method, as used here:

open(uri,:method=> :post, :body => jsonString,”Content-Type” => “application/json”).read

uri: the address of our couchdb database

Content-Type: our MIME type for this call (“application/json”)

body: our document

method: the HTTP verb POST
If this POST command fails we want to capture the error and (for this example) display the error:

rescue OpenURI::HTTPError => the_error
puts ‘##########Post Response Error: ‘ + the_error.io.status[0]

Actual error recovery is left as an exercise for the reader.

 

OK, we now have data in our database.  In our case this is is 2,014 documents (i.e. 2,014 separate Grateful Dead concerts).  Using Futon we see the database as:

clip_image001

In Futon, we can look at our data:

clip_image001[7]

And we can drill in on an individual document (for example the primordial recording on 1965-11-01):

clip_image001[9]

 

Of course what we need to able to return our document with a simple HTTP GET request.  Further we will want to return only those fields we will actually need from a document (i.e. we need to shape our data to minimize bandwidth).  We do that in couchdb with a view which is defined in a design document.  We can define multiple views in a single design document and can have multiple design documents in a given database. 

For our first view lets set the task to return all documents which occurs on a given month and day (regardless of year).  This is the filter. We want to shape our data to return only a subset of fields: the venue, the IAKey and the document tracks.  The key for this view will be the month+day (in MMDD format).  Our design document might look like:

 {
“_id”: “_design/basic”,
“_rev”: “19-fd2c9b34d2536ce1f187ab2d4e5413de”,
“views”: {
“MonthDay”: {
“map”: “function(doc){emit(doc._id.substr(5,2)+doc._id.substr(8,2),[doc.venue , doc.IAKey, doc.tracks ])}”
}
}
}

 

What this view (called MonthDay) does is maps each document (doc) into a key (doc._id.substr(5,2)+doc._id.substr(8,2)) and an array of return fields:

[doc.venue , doc.IAKey, doc.tracks ]

Note that doc.tracks itself returns an array of tracks.

Using Futon to test our work and find the results for all New Years Day Grateful Dead Concerts.  The first document looks like:

 

clip_image001[11]

(additional lines are omitted from this screen snap).

 

As an HTTP Get Verb we would write:

http://127.0.0.1:5984/deadbase/_design/basic/_view/MonthDay?startkey=%2201-01%22

This command users the

URI: http://127.0.0.1:5984/deadbase

The View: _design/basic/_view/MonthDay

startkey: this is a key word parameter field which tells couchdb to go to the b-tree which represents the MonthDay view and finds the first Key match and continues through the tree returning all Key matches.

If we wanted a range of Keys (say all holiday season Concerts) we can use the startkey and the endkey parameters:

http://127.0.0.1:5984/deadbase/_design/basic/_view/MonthDay?startkey=%221224%22&endkey=%221969-12-31%22

startkey” gives us a starting key in the b-tree and returns all documents until (and including) the b-tree key defined by endkey.

 

JSConf-eu-Alexander-Lang--Writing-apps-on-the-edge-with-CouchDB-e8451113

For more details see the Definitive Guide to CouchDB chapter: Finding Your Data With Views.  The key to simple view definition is defining an appropriate lookup key and defining what data to return.  We can define a simple listing view as:

"Jump": { "map": "function(doc){emit(doc._id,'['+doc.venue+'] ' +'http://www.archive.org/details/'+doc.IAKey)}" },
This returns a lookup key equal to the original key used in the database (doc._id) and the data as venue (doc.venue)
and a URL for the concert recording ('http://www.archive.org/details/'+doc.IAKey). Now if we want to return all
concerts in August, 1969 we can issue the GET:
http://127.0.0.1:5984/deadbase/_design/basic/_view/Jump?startkey=%221969-08-00%22&endkey=%221969-08-31%22

This will return the rows as:{“total_rows”:2013,”offset”:156,”rows”:[ {“id”:”1969-08-02″,”key”:”1969-08-02″,”value”:”[Family Dog at the Great Highway] http://www.archive.org/details/gd69-08-02.sbd.miller.30651.sbeok.flacf&#8221;}, {“id”:”1969-08-03″,”key”:”1969-08-03″,”value”:”[Family Dog at the Great Highway] http://www.archive.org/details/gd1969-08-03.sbd.miller.30652.sbeok.flac16&#8243;}, {“id”:”1969-08-16″,”key”:”1969-08-16″,”value”:”[Woodstock Music] http://www.archive.org/details/gd1969-08-16.sbd.gmb.fixed.95918.flac16&#8243;}, {“id”:”1969-08-21″,”key”:”1969-08-21″,”value”:”[Aqua Theater] http://www.archive.org/details/gd1969-08-21.sbd-part.tremblay.1170.sbeok.shnf&#8221;}, {“id”:”1969-08-23″,”key”:”1969-08-23″,”value”:”[Pelletier Farm] http://www.archive.org/details/gd1969-08-23.sbd.lai.6639.shnf&#8221;}, {“id”:”1969-08-28″,”key”:”1969-08-28″,”value”:”[Family Dog at the Great Highway] http://www.archive.org/details/gd69-08-28.sbd.lepley.4234.sbeok.shnf&#8221;}, {“id”:”1969-08-29″,”key”:”1969-08-29″,”value”:”[Family Dog at the Great Highway] http://www.archive.org/details/gd1969-08-29.sbd.lai.9179.sbefail.shnf&#8221;}, {“id”:”1969-08-30″,”key”:”1969-08-30″,”value”:”[Family Dog at the Great Highway]

]}

Get it? Good.

 
6011428946_f722b3cb71
What Would Dweelze Do?

Now let’s return to RUBY and write a web page which will display concert and track data for any given day in Grateful Dead History. Using the view:MonthView as defined above:

“MonthDay”: {
“map”: “function(doc){emit(doc._id.substr(5,2)+doc._id.substr(8,2),[doc.venue , doc.IAKey, doc.tracks ])}”
}

Lets return data using RUBY for concerts for all years for the Month and day of todays Date.  In Ruby we need to define a controller and a erb HTML file to accomplish this.  Lets say our page will be called player.  Our controller might look like:

require ‘rest-open-uri’

require ‘json’

class PlayerController < ApplicationController
def player
mmdd=Date.today.to_s[5..-1]
mmdd=mmdd.gsub(“-“,””)
base=’
http://127.0.0.1:5984/deadbase/
    view=’_design/basic/_view/MonthDay’
url=base+view+’?key=’
url=url+’%22’+mmdd+’%22′
jsonString=returnJSONString(url)
@parm=jsonString
end
def returnJSONString(url)
jsonString=”
open (url) do |x|
x.each_line do |y|
jsonString=jsonString+y
end
end
return jsonString;
end 

end

This returns the JSON returned from couchdb unaltered as a parameter to the erb page for player (player.html.erb).  If we wanted to work with the data within RUBY we would need to change the JSON back into a RUBY format by calling:

JSON.parse(jsonString)

For our purposes, however, we want to pass the data directly to the browser and will process the data using JavaScript this will in almost all cases be faster than processing the JSON back to RUBY and then formatting the data on the erb page.  In our first pass the route map for our page will be simple:

map.connect ‘player’, :controller => ‘player’, :action => ‘player’

and our first pass at an HTML page will look like this:

<!–DOCTYPE HTML PUBLIC “-//W3C//DTD HTML 4.01//EN” “http://www.w3.org/TR/html4/strict.dtd>
<html>
<head>
<meta http-equiv=”Content-Type” content=”text/html; charset=iso-8859-1″ />
<title>Player: On This Day In Grateful Dead History</title>
<script type=”text/javascript”>
gdData=<%=  @parm  %> ;
</script>
</head>
<body>
<h3>On This Day In Grateful Dead History</h3>
</body>
</html>

This page does absolutely NOTHING…  Except gets our JSON data to the Javascript on the browser page.  And that, dear reader is as far as we need to go with today’s post.

5719769521_0380972a03




	

REST, Ruby On Rails, CouchDB and Me – Part 5 Getting The Data Ready for CouchDB   3 comments

Part 0 – REST, Ruby On Rails, CouchDB and Me

Part 1 – Ruby, The Command Line Version

Part 2 – Aptana IDE For Ruby

Part 3 CouchDB Up and Running on Windows

Part 4 – CouchDB, Curl and RUBY

Part 5 – Getting The Data Ready for CouchDB

Part 6 – Getting The Data Into And Out Of CouchDB

Part 7 – JQUERY,JPlayer and HTML5

The Internet Archive and the Grateful Dead

The Internet Archive (IA)  is a 501( c ) 3 non-profit corporation dedicated to disseminating of public domain digital artifacts.  This includes such items as books, videos of imagesall kinds (TV Shows, shorts and feature films) audio recordings of all kinds (musical and spoken word).    One of their most popular projects is a truly huge collection of Grateful Dead concert recordings.  One 418910of the most visited pages of the Internet Archive’s web site is the “Grateful Dead Shows on This Day In History” page, which lists and allows play back of any Grateful Dead concerts they have in their collection of whatever day you visit the site.  Did I say that there IA has a large number of Grateful Dead concert recordings?   The are recordings of around 2,000 separate concert dates.  Any given concert may be represented by multiple recordings from a variety of sources: soundboard, audience recordings using professional and amateur equipment.  Original media ranges from cassette tapes, to 7 inch reel to reel and digital media.  If you are going to be working with the IA Grateful Dead collection please review the FAQ on the collections policy notes as well as the special notes here.

IA uses a very sophisticated data repository of meta data and an advanced query engine to allow retrieving both the meta data and the recordings.  Meta data can be retrieved directly using the “advanced search” engine.  On the day I started this post I visited IA and used the “Grateful Dead Shows on This Day In History”  The query returned data on 8  concerts (and 25 recordings of those 8 concerts).  A partial page image is given below:

image

Clicking on any of these entries moves us to a second screen in order to play the concert recording.  A screen shot of the playback screen looks like this:

image

Looking closer at the second screen we see the music player:

image

Can we design a faster, simpler and better looking interface into the Grateful Dead Archive?  Can couchDB help us? gratefuldead_20070108135140 The the first question will be addressed in a later post. This current post will look at how couchDB can  help us achieve a faster more efficient information system.  IA does a super job of serving up the music files on demand – there is no reason to duplicate their storage system.   However, IA is fairly slow to serve up meta data (such as the results of the “Grateful Dead Shows on This Day In History” query) Abstracting the IA metadata into a CouchDB database will allow us to serve up the meta data much faster than the IA query system.

Getting Data Into CouchDB

Our basic plan for using RUBY to get data from IA and into couchdb consists of:

  1. Prepare a URL query request to get the basic recording meta data (not the track meta data);
  2. Submit A GET request to IA using the URL query;
  3. Parse the XML returned to get at the individual Concert meta data fields;ruby
  4. Select the BEST recording for any given concert (more on this below);
  5. Prepare a URL to request track listing XML file based on the IA Primary Key of the selected concert recording;
  6. Submit a GET request to IA;
  7. Parse the XML returned to get at the individual track meta data fields;
  8. Create a ruby object which can be safely serialized to JSON;
  9. Serialize the object to a JSON string (i.e. a JSON document);
  10. Do a POST request to insert a JSON document for each concert into the couchdb database;
  11. Create couchDB views to allow optimized data retrieval; and
  12. Create a couchDB view to optimize retrieval recordings for all years for an arbitrary Month and Day (this duplicates the data provided by the “Grateful Dead Shows on This Day In History” selection in the Internet Archive.

Note we are not accessing nor storing the actual music files.  Before discussing how this plays out in practice lets define our JSON couchDB document.  We will cover items one through eight in this post.  We turn to items nine through twelve in the next post.

CouchDB Document Schema

CouchDB databases start with documents as the basic unit.  Typically a couchdb based application will have one database holding one or more variant document types.  There will be one or more design documents which provide multiple views, show functions and map functions as necessary to facilitate the application.  We will use a single document which will represent a abstract of the meta data contained in IA for individual recordings ( we are going to select the one ‘best’ recording per concert).  Our couchdb database will hold one document per concert.   The tracks (actually the track meta data will be stored as arrays within the  concert document).  We will populate the couchdb database in a single background session  pulling meta data (NOT THE MUSIC FILES) from IA and we will  include the IA publication date in the document so we can update our database when (if) new recordings are added to IA in the Grateful Dead collection.

Here are the document fields  which we will use:

Field

Notes

Typical Values

_id couchdb primary key.  We will use a natural key: a string representation of the concert date. 1969-07-04
_rev revision number provided by couchDB 1-6ea272d20d7fc80e51c1ba53a5101ac1
IAKey Internet Archive Key for this Recording gd1965-11-01.sbd.bershaw.5417.sbeok.shnf
pubdate Internet Archive Date When Recording was published to the web 2009-03-14
venue Wherethe concert took place Fillmore East Ballroom
description free text describing the concert – provided by the uploader Neal Cassady & The Warlocks 1965 1. Speed Limit studio recording/Prankster production tape circa late 1965
cm boolean – Recording by Charlie MIller  – used to select the ‘best’ recording true or false
sb boolean – Recording was made from a soundboard – used to select the ‘best’ recording true or false
mx boolean – A matrix style recording – used to select the ‘best’ recording true or false
tracks an array of meta data for each track of the recording see below

Each track in the tracks array  formally looks like:

Field Notes Typical value
IAKey The Internet Archive key for this track.  This key is unique within a given recording  (see the IAKey above) gd1965-11-01.sbd.bershaw.5417.sbeok.shnf/Acid4_01_vbr
track track number 02
title the song title Cold Rain and Snow
time the length of the track in minutes and seconds 09:48

Let call everything except the tracks our BASE data and the track data our TRACK data.

We insert documents to the database (using an HTTP post) as JSON so a typical document would look like this in JSON format:

{
“_id”: “1966-07-30”,
“IAKey”: “gd1966-07-30.sbd.GEMS.94631.flac16”,
“pubdate”: “2008-09-22”,
“venue”: “P.N.E. Garden Auditorium”,
“description”: “Set 1 Standing On The Corner I Know You Rider Next Time You See Me”,
“cm”: false,
“sb”: true,
“mx”: false,
“tracks”: [
{
“IAKey”: “gd1966-07-30.sbd.GEMS.94631.flac16/gd1966-07-30.d1t01_vbr”,
“track”: “01”,
“title”: “Standing On The Corner”,
“time”: “03:46”
},
{
“IAKey”: “gd1966-07-30.sbd.GEMS.94631.flac16/gd1966-07-30.d1t02_vbr”,
“track”: “02”,
“title”: “I Know You Rider”,
“time”: “03:18”
},
{
“IAKey”: “gd1966-07-30.sbd.GEMS.94631.flac16/gd1966-07-30.d1t03_vbr”,
“track”: “03”,
“title”: “Next Time You See Me”,
“time”: “04:00”
}
]
}

2654190796_c0a810ec44

Hacking The Internet Archive: Getting Data From IA and Into CouchDB:

Here is the URL to obtain the page for “Grateful Dead Shows on This Day In History”:

http://www.archive.org/search.php?query=collection:GratefulDead%20date:19??-08-04&sort=-/metadata/date

This is a simple GET request with a query for the IA “collection” of Grateful Dead items filtered on the date string: 19??-08-04 and sorted descending by the concert date.  This get returns an HTML page.  This type of interface is known as an HTTP RPC interface.  RPC (Remote Procedure Call) interfaces are not pure REST interfaces but they are somewhat RESTful inthat they allow us to make a data request using a late bound, loosely coupled HTTP call.  See here and here for more theoretic background on RPC calls.  IA provides an  “Advanced Search” function will allow us to return data for an arbitrarily complex query in one of several different data formats other than HTML.  We selected XML as the format  for our work here.  XML is the traditional format for HTTP RPC but other formats may be better for certain applications.  Unfortunaely IA does not directly document the format of the RPC data request but they do provide a QEB page to build the request.  The page looks like this:

image

Using this screen we can compose a HTTP RPC request which will mimic the URL produced by “Grateful Dead Shows on This Day In History” and with a little brain effort and experimentation we can  understand how to compose requests without using the QBE screen.  By feeding the RPC request query back into advanced search and selecting XML as an output format as shown here:

clip_image001

we produce both an example of the HTTP RPC request which will return our desired data in our desired format.  Thus we generate a HTMLEncoded RPC request like:

@uri=”http://www.archive.org/advancedsearch.php?q=collection%3AGratefulDead+date%3A#{_dateString}&fl%5B%5D=avg_rating&fl%5B%5D=date&fl%5B%5D=description&fl%5B%5D=downloads&fl%5B%5D=format&fl%5B%5D=identifier&fl%5B%5D=publicdate&fl%5B%5D=subject&fl%5B%5D=title&sort%5B%5D=date+desc&sort%5B%5D=&sort%5B%5D=&rows=2000&page=1&callback=callback&output=xml”

where we replace #{_dateString} with a date string like 19??-08-08.  Of course to one years worth of data we could use a data string like: 1968-??-??.  It is a simple extension of the query languge to replace the singular date request: date%3A#{_dateString} with a date range.

which returns Grateful Dead recording data for all years of the last century which were recorded on 08-08.  The XML output returned to the caller looks like:

clip_image001[4]

In a more graphic format the output looks like:

clip_image001[6]

Within Ruby we will need to make the HTTP Get request with a desired date range, transform the body of the return request into an XML document and use XPATH to parse the XML and retrieve the meta data values for each recording (see below).  The is NOTHING inherently wrong with this RPC interface.  It is flexible and allows us to select only the data fields we are interested in and return data only for the dates we wish.  Since RUBY supports neither native consumption of JSON nor XML. So the XML format of the data is as good as any other and numerous tools exist in RUBY to manipulate XML data.  I which RUBY had a more native interface for JSON but it does not.

At this point, we do not have meta-data about individual tracks in a given recording.  It turns out that we can get this data but not through an HTTP RPC request.  It turns our, dear reader, that if we have the IAKey for the recording we can obtain an xml file with track meta data by making the following call:

http://www.archive.org/download/{IAKEY}/{IAKEY}_files.xml.

This file contains assorted XML data, it varies by what formats IA makes available the individual tracks via a 309 (HTTP redirect).  This is not an RPC call so we are far from a RESTful interface here.  We do not have control over the fields or range of the data included in this call.  It is all or nothing.  But at least the XML format is simple to mainipulate.  With the IAKey in hand for an individual recording and making some reasonable guesses we can parse the XML file of track data and compose the TRACKS array for our couchDB document using XPATH. A single entry for the high bit rate mp3 track recording looks like:

<file name=”gd89-08-04d2t01_vbr.mp3″ source=”derivative”>
<creator>Grateful Dead</creator>
<title>Tuning</title>
<album>1989-08-04 – Cal Expo Amphitheatre</album>
<track>13</track>
<bitrate>194</bitrate>
<length>00:32</length>
<format>VBR MP3</format>
<original>gd89-08-04d2t01.flac</original>
<md5>91723df9ad8926180b855a0557fd9871</md5>
<mtime>1210562971</mtime>
<size>794943</size>
<crc32>2fb41312</crc32>
<sha1>80a1a78f909fedf2221fb281a11f89a250717e1d</sha1>
</file>

Note that we have the IAKey for the track (gd89-08-04d2t01 ) as part of the name attribute.

Garcia

Using a background Ruby Process to Read the Data

The following RUBY GEMS are required to complete this step:

rest-open-uri : This GEM extends open-uri to support POST, PUT and DELTE HTTP command

json : This GEM handles serialization and de-serialization of a limited subset of RUBY into JSON strings.

From the standard RUBY library we will also be using

rexml : This GEM creates XML documents from XML Strings and supports XPATH which we will use to read the XML documents from IA

Our first step is to extract the get the the data via HTTP and parse the XML file returned to find individual recordings.  There are  (in most cases) be multiple recordings per concert (per date) and we want to retain for the database only the “best”.

In pseudo Ruby code:

require ‘rest-open-uri’

require ‘rexml/document’

 def initialize(_dateString)

#HTTP GET, create a string of the response body and transform the string into an XML node tree
#mind the screen wrap and html Encoding:
@uri=http://www.archive.org/advancedsearch.php?q=collection%3AGratefulDead+date%3A#{_dateString}&fl%5B%5D=avg_rating&fl%5B%5D=date&fl%5B%5D=description&fl%5B%5D=downloads&fl%5B%5D=format&fl%5B%5D=identifier&fl%5B%5D=publicdate&fl%5B%5D=subject&fl%5B%5D=title&sort%5B%5D=date+desc&sort%5B%5D=&sort%5B%5D=&rows=2000&page=1&callback=callback&output=xml

  xmlString=”
open (@uri) do |x|       #build a representation of the response body as a string
x.each_line do |y|
xmlString=xmlString+y
end
if xmlString==”
puts ‘No String Returned From Internet Archive’
quit
end
end
@IAXMLDocument= REXML::Document.new(xmlString)  #turn the string into an XML document
end #open

Now we need  to loop through the XML document and pull out each ‘doc’ section using XPATH and read each doc section for the meta data for that recording.

#use XPATH and find each response/result/doc node and yield

def get_recordings(document)

document.elements.each(‘response/result/doc’)do |doc|

yield doc
end
end

#get the XML document and yield

def get_record(xmldoc)

get_recordings(xmldoc) do |doc|
yield doc
end
end

#general purpose XPATH method to extract element.text (the metadata values) for arbitrary XPATH expressions

def extract_ElmText(doc,xpath)

doc.elements.each(xpath) { |element|  return element.text }
end

def worker(xmldoc)

#main loop

_docCount=0

get_recordings(xmldoc) do |doc|
_docCount+=1
_date=extract_ElmText(doc,’date[@name=”date”]’)[0..9]
_description=extract_ElmText(doc,”str[@name=’description’]”)
_title=extract_ElmText(doc,”str[@name=’title’]”)
_title=clean_Title(_title)
_keylist=pull_keys(doc)
_pubdate=extract_ElmText(doc,’date[@name=”publicdate”]’)[0..9]  #there is a bug here , corrected by lines below

if (_pubdate.length==0)
_pubdate=’1999-01-01′
puts “#No Publication Date: {_date} #{_title}”
end
_uri=extract_ElmText(doc,’str[@name=”identifier”]’)

#make a RUBY class object to hold one recording
_record=GDRecording.new _date, _description, _tracklist, _title, _keylist, _pubdate,_uri

#save the recording class objects in an array

@list[@list.count]=_record

end

In this code the ‘worker’ method calls the helper methods to:

0) Do the HTTP  get to make the RPC request and read the response body one line at a time and

1) transform the lines into a single string and convert (REXML::Document.new) the string into an XML document for processing by XPATH

2) loop through the doc nodes of the xml tree and extract the values of the  meta data fields

3) the meta data values are passed to an RUBY class ( GDRecording) which holds this meta data for later processing,

4 finally we temporarily store the recordings in an array for the next processing step.

Note that these routines work  whether the query returns a single day (with multiple recordings) or multiple days or even the whole dataset!  What is essencial is that we process the file as N ‘doc’ sub trees (which represent single recordings) and have recording date (at least) to group our data and extract the ‘best’ recording within each date group.

Our next step will be group the recordings by day (i.e. concert) and provide our own filter to select a single ‘best’ recording for each concert.

Shake and Bake:  Finding A ‘Best’ Recording.

best

What is the best Grateful Dead concert.  Why the first one I went to of course.  Just ask any Deadhead and you will probably get the same answer.  But what is the best recording of any given GD concert? My approach is very simple.

  • Most recent posted recordings are better than older recordings. (least important criteria)
  • Soundboard recordings are better than audience recordings.
  • Matrix recordings are even better.
  • Recordings mixed by Charlie Miller are best of all. (most important criteria)

Well these are MY criteria.  What ever criteria as long as they are hieratical  you can code the select in a very trivial manner.  If we have a field in each recording for the concert date and a field for each selection criteria (we derive these from the keywords field in IA) we sort the recordings by date and then by each of the criteria from most important (Charlie Miller in may case) to least important (date posted) and then select the first recording in sort order within each date group. On Ruby the sort of our list of recordings is trivial to code and easy to maniuplate (add new criteria or change the priority of criteria). The sort statement looks like this:

@list.sort! { |a,b| (a.date+a.cm.to_s+a.sb.to_s+a.mx.to_s+a.pubdate ) <=> (b.date+b.cm.to_s+b.sb.to_s+b.mx.to_s+b.pubdate )   }

Once sorted we create a list of best recordings as:

def newSelect
_dateGroup=nil
_list=Array.new
if  @list==nil or @list.count==0
puts ‘No Recordings.’
return;
end
foreach do |rec|
if _dateGroup!=rec.date
if dateGroup!=nil
@selectList[@selectList.count]=list[0]
end
_dateGroup=rec.date
_list=Array.new
end
_list[_list.count]=rec
end
if dateGroup!=nil
@selectList[@selectList.count]=list[0]
end
end

Note that is code is not only simple but it is independent of the selection criteria we are using.

Now that we have a list of recordings we are interested in,  we can get the XML file of track meta data using the IAKey discussed above and making a simple GET call and parsing the XML file for the meta data for each.  Much of the code used duplicates the XML code presented above so we need not reproduce all the code except to show a short section which uses a slightly different XML XPATH syntax:

open (filesURI) do |x| x.each_line do |y| xmlString=xmlString+y end end
REXML::Document.new(xmlString).elements.each(‘files’) do |doc|

doc.elements.each(‘file’) {|file_elm|
obj=nil
title=nil
trackString=nil
lengthString=nil
obj=file_elm.attributes[“name”]
file_elm.elements.each(‘title’) { |element| title=element.text }
file_elm.elements.each(‘track’) { |element| trackString=element.text}
file_elm.elements.each(‘length’) { |element| lengthString=element.text}

{omitted code}

end

Okay now we have a (hash) list of recording meta data,  each item of which contains a (hash) list of track meta data for that recording.  In our next post we will leave this unRestful world behind and move into the RESTful world of couchDB when we:

  • Serialize the object to a JSON string (i.e. a JSON document);
  • Do POST requests to insert  a JSON document for each concert into the couchdb database;
  • Create couchDB views to allow optimized data retrieval; and
  • Create a couchDB view to optimize retrieval recordings for all years for an arbitrary Month and Day (this duplicates the data provided by the “Grateful Dead Shows on This Day In History” selection in the Internet Archive.

cat on fancy couch

JQuery 2009 & AJAX Experience 2009   Leave a comment


John Resig
Photo By dlr2008 On FlickR.com

JQuery Conference 2009

This conference took place at the Microsoft Research Center in Cambridge Massachusetts on September 12 & 13, 2009. This was a small but intense gathering of JQuery and JQuery UI core developers and assorted developers working on extensions to JQuery (aka plug-ins). There were about 300 participants most of whom were intensely involved with JQuery. Your humble scribe was one of the least informed participants. JQuery has never looked stronger, the development team is cohesive and motivated, and community support is extensive. John Resig’s numbers indicate that something in excess of 35% of all public sites running JavaScript are using JQuery. The library wars are over. Let the framework wars begin. The conference agenda and speakers list and slides can be found by starting: here.

JQuery core is stable at release 1.3.2 but 1.3.3 is coming “real soon now.” The 1.3 versions have focused on optimization, simplification and (proactive) HTML5 support.John’s talk about recent changes to JQuery Internals can be found here. Next year will see moving JQuery core and open source licenses will be transferred to Software Freedom Law Center. Projects for the next year (think version 1.4) include move JQuery\UI infrastructure into JQuery proper. Significant work has been done within JQuery core to streamline and simplify plug-in development via the Widget Factory ($.widget(…)) (thanks to Scott González for this excellent presentation). For the hard core, Paul Irish gave an excellent presentation on JQuery Anti-Patterns. This was bookended by Yehuda Katz’s excellent Best Practices presentation. Aaron Quint was among the Rubists who are advancing the state of JQuery Art. His Sammy.js project attempts to use JQuery to create a browser side MVC/REST framework. John Nunemaker is also working in this basic area and his presentation can be found here.


The Jupiter Room


Photo By dlr2008 On FlickR.com

The walk away component demonstrated at the conference was the work done by Filament Group employees, Todd Parker and Scott Jehl who developed and maintain the new ThemeRoller CSS generator for use with JQuery/UI components. Outstanding Work!

This year’s conference was sold out at 300 participants and was a mind blowing experience. Two days of the sort of “deep dive” Microsoft presenters can only dream of. All this, plus great food and a T-shirt, for one hundred bucks American. We won’t see this sort of thing until the next big thing comes along. Look for the following event during 2010: one virtual (online) developer conference (similar to JQuery 2009 but without the food) and three ‘bigger’ user conferences (London, Los Angles and Boston). Splendid!

The AJAX Experience 2009

This conference took place at the Boston Airport Hilton on September 14 – 16, 2009. What an ugly hotel. Isolated, bad restaurant, overpriced breakfast, cold design, the hotel truly sucks. The conference itself was much better. If at JQuery 2009 we saw significant evidence of what the web will be in the next two years, The AJAX Experience showed us some of what will not happen:

  • The HTML5 specification will be released too late to mater,
  • ES5 will not change the world, or JavaScript.
  • Browser vendors will implement HTML5 using different API’s and approaches,
  • Conflicts between Security and JavaScript will NOT be resolved anytime soon,
  • JSON is not going away but XML ,
  • JavaScript is not going to be changed in any fundamental way,
  • Page Refreshes are not.

The AJAX Experience is a developer driven conference and uniquely includes presenters from both the standards community (W3C, ES5) and major players (Google, Facebook, Yahoo and Microsoft).


Douglas Crockford


Photo By dlr2008 On FlickR.com

The conference is well worth attending to take the pulse of the AJAX/REST world from a developer perspective. It is a conference with does not have an edge to it.  To be fair, this is a conference with a large number of big faces present and the structure of the conference is oriented towards making these faces easily available for one on one’s.  And for that we thank the folks at AJAX Experience.

On the plus side AJAX and REST continue to hold an enduring fascination for web developers looking for the edge. One of the best attended sessions was Facebook’s overview of how AJAX, careful architectural design, client side caching and REST principles, can be brought together into a workable whole. The presentation slides can be found here. This is an important presentation and should be viewed by anyone who want to ‘go all the way’ with AJAX/REST. If nothing else this presentation shows how little native support there is in the current set of browsers for this effort and how clumsy the basic design of JavaScript is when this advanced work is attempted. Please understand dear reader that the best AJAX/REST web site would have only ONE page load all other data exchange and UI work is done on this canvas. The Facebook folks found that after years of effort they still force unnecessary page loads as the only way to clean up events, timers, namespace debris and memory leaks.

Security

For me the most interesting, and the most depressing presentation was the much anticipated panel discussion: Secure Mashups: Getting to Safe Web Plug-ins

The panelists where Marcel Laverdet (Facebook – FBJS), Mike Samuel (Google – Caja), Scott Issacs (Microsoft – The Open Source Web Sandbox) and Douglas Crockford (adSafe). And the discussion was spirited. The goal here is to develop a method by which a web site (think Facebook for example) can accept browser widgets (think ads or user gadgets) from multiple sources and assure “secure cooperation” between all widgets and with the vendor’s (the page owner) HTML and JavaScript code. Although there are nuances between the different approaches and differences in scope, each of these attempts to mash-up ‘security’ follow the same pattern:

  • Clean incoming HTML for “evil parts”
  • Clean incoming JavaScript for “evil parts”
  • Encapsulate the remaining JavaScript code in wrapper functions which ultimately call methods in:
  • The Vendor’s Security Library

The Security Library purpose is to prevent, at run time, manipulation of HTML belonging to other parts of the system and preventing JavaScript exploits. Each solution provides its own definition of what the “evil parts” are, what constitutes ‘secure’ behavior and what are the limits of the security library. None of the solutions currently support JavaScript libraries. Although, Google and Microsoft made noises like they will attempt to bring third party libraries (read JQuery here) into the tent. There was a lot of discussion around this point. My notebook shows that John Resig’s name was mentioned by panelists eight times during this discussion. The overall goals of the projected solutions vary from forcing no code changes (Web Sandbox) to forcing complete re-writes (adSafe is alone is requiring safe widgets to being written exclusively using ONLY the adSafe library). All four projects are in early beta.
Significantly, there were no presentations which addressed Secure Browser OS projects like Microsoft Gazelle or Google’s Chrome OS.

PS: On the UI side of the house Bill Scott gave a delightful presentation on UI methods. For UI designers his presentations (and books) are not be to be missed.

JQuery And AJAX Experience 2009 – The Movie:



Ninja JavaScript: (function(){})()   Leave a comment

dlr2008

We interrupt our discussion of structured JavaScript to discuss John Resig’s Ninja approach to Javascript. John’s approach to JavaScript places the accent on

  • Leveraging JQuery; and
  • Minimizing the use of global variables; Using
    • Anonymous Functions
    • Closures; and nest everything in the anonymous construct:
    • (function(){})()

Let’s define a real world problem:
Make an AJAX call to a data source every ‘X’ milliseconds (for the life of the page) and update the page UI on
each AJAX
return. (Think Stock Ticker for example). We will use the JQuery to manage

  • the AJAX call;
  • bind and signal events; and
  • UI manipulation.

We will use the JavaScript function setInterval to manage when the AJAX call is made. Along the way we will use

some JSON to help us manage passing arguments to the (JQUERY based) AJAX call. This combination will produce
some very tight code with a minimum of global variables (well no global functions if we want to).
First Here is the SHELL of the code, then I will present the final code and then break it down in detail.

SHELL: Based On Model: (function(){})()

(
function(pTarget, timeout, rootID, successEvent, failureEvent) {
$(‘#’ + rootID).bind(failureEvent, function(e, jsObject) {});
$(‘#’ + rootID).bind(successEvent, function(e, jsObject) {});
var _T = window.setInterval( function() {//setInterval anonymous function
$.ajax(//JQUERY AJAX Call
{//JQUERY AJAX Arguments
AJAXOPTIONS : values,//in this format
error: function(XMLHttpRequest, textStatus, errorThrown) {},//AJAX error function option
success: function(data) {}//AJAX success function option
}//end of arguments to $.ajax
)//end of ajax call
}//end setInterval anonymous function
, timeout //call this every ‘timeout’ milliseconds
); //end setInterval
function x(){}//local inner, private functions go here
} //end outer anonymous function
)//end of outer parentheses
(‘DatapageSecure.aspx’, 5, ‘KeepAlive’, ‘AICAJAXSUCCESS’, ‘AICAJAXFAILURE’ );//constructor arguments

Shell Discussion

OK. We start with our old friend:

(function(){})()

This time we will not create a return interface and we will not assign the output of this construction to
a (global) variable (or any other namespace).  We try for a pure Ninja approach.  Get in, set up the functionality and leave behind no trace in the JavaScript namespace.

Brief Overview:

  • (function(arguments) {})(arguments);
    • create an anonymous function (a closure) and pass the values of the constructor arguments to this function
    • Passing the (outer) argument list into the anonymous function preserves these fields (and there values) locally within the function closure
  • Now let’s focus on the anonymous function within the parentheses:
    • (function(arguments) {})(arguments)
  • Within this function we set up two event sinks (bind) to be used by the AJAX calls which will fire (trigger) these events on success or failure
    • $(selector).bind(eventName, function(e,argument) {});
    • These event sinks create anonymous functions (closures) to handle the work preformed when an event is caught.
    • See my notes below on how these functions leverage the properties of these closures.
  • Now we set up the repeating calls to the AJAX function using the (standard) JavaScript setInterval function
    • In abstract form call setInterval call looks like:
    • var t=context.setInterval(function(){},timeperiod);
    • See my notes below for additional comments on the setInterval function
  • Note that I am using another anonymous function as the target for the setInterval call.
    • By doing this the function has access, (if it needs them) to all of the local fields of the enclosing anonymous function
    • We are going to place the JQuery AJAX function within the anonymous function called by setInterval.
  • Take a moment to ponder that at this point we have created four anonymous functions
    • One each for the two bind events;
    • An outer anonymous function; and
    • an additional anonymous function nested inside of that.
  • We are going to nest two additional anonymous functions (closures) within.
    • In symbolic form the JQuery AJAX function takes the form:
    • $.Ajax({JSON FORMATTED Options})
    • See my comments below for additional comments on the $.Ajax function.
    • The JSON Formatted options take the form of optional arguments
      • such as type: ‘GET’ which determine the type of call to make (Get vs Post)
    • For our purposes here the most important options to discuss are the two (optional) functions which are called when the AJAX call returns
      • these are not anonymous functions
      • they have names: success and error
      • Since these functions are expressed in JSON format they act as nested closures in their own right
        • With access to any local fields
          • Local to $.Ajax and to the outer anonymous functions in our example.
    • The essential work of these functions (in this design) is to trigger events we have already defined, and
    • to pass data from the AJAX call to these event sinks
    • In symbolic form the success function looks like:
      • success: function(data) {
      • //do any necessary internal work then call the appropriate event
      • $(selector).trigger(event name, argument);
      • }
  • Ok at this point our work is done and all we need to is add the correct number of brackets and parenthesis
  • to close all of this work.
Take a break then come back and look at the full version of the code and my additional usage notes.
Mississippi River Home - New Orleans

Final Code

01            //local variables added to make this code read more clearly for this blog post
02            var rootID = ‘KeepAlive’; //parameter, major UI DIV ID
03            var targetURL = ‘DatapageSecure.aspx’; //parameter
04            var timeoutInSeconds = 1140; //parameter
05            var AJAXSuccessEvent = ‘AICAJAXSUCCESS’; //EVENTNAME
06            var AJAXFailureEvent = ‘AICAJAXFAILURE’; //EVENTNAME
07          (function(pTarget, timeout, rootID, successEvent, failureEvent) {
08              $(‘#’ + rootID).bind(AJAXFailureEvent, function(e, jsObject) {
09                    var tmp=’ Failure. ‘+new Date() +” HTTPRequest Status: “+jsObject.status;
10                    $(this).find(‘#’ + ‘KeepAliveFalure’).html(tmp);
11              });
12                $(‘#’ + rootID).bind(AJAXSuccessEvent, function(e, jsObject) {
13                    var stmp = ‘User: ‘ + jsObject.UserName + ” Calls: ” + jsObject.Calls ;
14                    $(this).find(‘#’ + ‘KeepAliveChat’).html(stmp);
15                });
16                timeout = timeout * 1000; //upgrade to miliseconds
17                var _Hits = 0;
18                var _T = window.setInterval(
19                            function() {
20                                $.ajax(
21                                            {
22                                                url: pTarget,
23                                                type: ‘GET’,
24                                                data: ”,
25                                                dataType: ‘HTML’,
26                                                timeout: timeout / 2,
27                                                error: function(XMLHttpRequest, textStatus, errorThrown) {
28                                                    window.clearInterval(_T);
29                                                    var x={};
30                                                    x[“status”]=XMLHttpRequest.status;
31                                                    x[“textStatus”]=textStatus;
32                                                    $(‘#’ + rootID).trigger(failureEvent,XMLHttpRequest);
33                                                },
34                                                success: function(data) {
35                                                    _Hits++; //for debuging
36                                                    var jsObject = ExtractData(data);
37                                                    jsObject[“Calls”] = _Hits;
38                                                    jsObject[“TimeStamp”] = new Date();
39                                                    $(‘#’ + rootID).trigger(successEvent, jsObject);
40                                                }
41                                            }//end of arguments to $.ajax
42                                        )//end of ajax call
43                            } //end setInterval anonymous function
44                              , timeout); //end setInterval
45              function ExtractData(data){
46                  //code omited as not relavent to blog post
47                  }//end of local function
48            } //end outer anonymous function
49                )(targetURL, timeoutInSeconds, rootID, AJAXSuccessEvent, AJAXFailureEvent);   //DO IT

Notes On Specific Features

Binding events:

1.     $(‘#’ + rootID).bind(failureEvent, function(e, jsObject) {
2.     var tmp=' Failure. '+tmp1 +" HTTPRequest Status: "+jsObject.status;
3.     $(this).find('#KeepAliveFalure').html(tmp);
4.     });

Line 1: JQuery is built up using selectors which simulate CSS selectors. Here $(‘#’+rootID) seeks within the DOM all
HTML elements with an ID equal to the value of the variable ‘rootID’. The “bind” JQuery verb binds an event (here a
custom event with a name contained in the variable ‘failureEvent’) with an anonymous function:
function(e, jsObject){}. JQuery normalizes the binding of events between major browsers so we do not need to
concern our code with what vendors agent we are running within. The bind verb will pass two arguments from
the signal event: ‘e’ which is the standard event object and any optional arguments provided by the signaler. In this
case the second argument is a JavaScript object with a (user defined) set of properties.
Line 3: This is a very nice example of terse JQuery script. The selector: $(this) refers to whatever was defined as the
selector in Line 1. The ‘find’ verb applies a selector (in this case looking for the ID ‘KeepAliveFalure) within the children
of the primary selector (in this case ‘this’). The ‘ html’ verb replaces the innerHTML of the find selector with the value of the variable ‘tmp’.
This is tight, no?

Calling AJAX with JQUERY


JQuery defines several ways to make AJAX calls I am using one of these ( $.ajax) here. The structure of this call is:
Line 1. $.ajax(
Line 2. {}
Line 3. );
Line 2 is of course the meat it is a JSON formatted argument list containing, in our case:
Line 1. url: pTarget, //the url to call
Line 2. type: 'GET', //call type
Line 3. data: '', //Query String Arguments to pass to the target URL
Line 4. dataType: 'HTML', //return datatype
Line 5. timeout: timeout / 2,//how long to wait for a response
Line 6. error: function(XMLHttpRequest, textStatus, errorThrown) {...},
Line 7. success: function(data) {...}
Note: Line 4 defines the type of data returned from the AJAX call. In this case use HTML since we do not want any post
Processing of the data to occur . We could use the key word ‘JSON’ if we expect a JSON response and we want
JQUERY to hydrate the JSON string back into a JavaScript object prior to passing the data to the success anonymous function.
Line 6 defines an anonymous function to be called when the call times out or if the call returns an HTTP code => 400.
JQuery defines and handles the three arguments passed to the error function.
Line 7 defines an anonymous function to be called on a successful AJAX call and the argument ‘data’ is the
actual text returned (subject to the constraint imposed in Line 4).

Processing the AJAX Return

Let’s look at the anonymous success function (given as Line 6 above). In the actual code example we are using
Line 1. success: function(data) {
Line 2. _Hits++; //for debuging
Line 3. var jsObject = ExtractData(data)
Line 4. jsObject["Calls"] = _Hits;
Line 5. jsObject["TimeStamp"] = new Date();
Line 6. $('#' + rootID).trigger(successEvent, jsObject);
Line 8. }
Line 2. ” _Hits” is a local variable defined in the full code example. This is a local variable. Local to the closure of the outer
Anonymous function. It is visible there and to both the success and error anonymous functions but is not visible globally.
I am using it here as a simple debugging device to count the number of the AJAX calls made.

Line 3 takes us Outside of the scope of the current blog and is used to extract some of the data from the return stream and
to create a Javascript object to hold that data. In the working code it is defined as a local non-anonymous function The code has been omitted from our example code since this takes us way outside of the scope of this blog.
Lines 4 and 5 define additional properties of jsObject (used for debugging in our example).
Finally, in Line 6, we use JQUERY to trigger an event and pass the locally created JavaScript object to it. Some things to note about this
Very terse line of code:

  • $(‘#’+rootID) The JQuery selector for the DOM element to trigger to event against. AKA the event sink. See the section on Binding Event Above.
  • .trigger The JQuery trigger event verb, this is a browser neutral event trigger
  • successEvent A variable which contains the name of the event being triggered.
  • jsObject The JavaScript Object containing the local data to pass to the event sink.

Setting Up the setInterval Function Call

The JavaScript function “setInterval” controls calling a function repeatedly every “X” milliseconds. Its form is:
var _T = owner.setInterval(function(){},timeout);
where:
owner is the context which ‘owns’ the setInterval
_T is the handle for the setInterval.
Function(){} is the anonymous function to call; and
Timeout is the milliseconds between invocations of the anonymous function.
Once launched, setInterval will continue to be called for the life of the page or until a call is made to cancel the setInterval cycle:
owner.clearInterval(_T);
Note: the handle (_T here) returned by the setInterval function is a simple number if we have
Var _T a.setInterval(function(){},timeout);
and
b.clearInterval(_T);
the original setInterval will not be cleared and no error is generated with the call.
One could write
var _T = setInterval(function(){},timeout);
and
clearInterval(_T);
and HOPE that the context is the same for each call. But better to assure success by specifying the context of both
the setInterval and the clearInterval call. In our example code we assign each call to the “window” context:
var _T = window.setInterval(function(){},timeout);
and
window.clearInterval(_T);
so our context and intent is clear.

So What’s The Down Side?

We get some very compact code here with NO side effects and a very minimal global namespace impact when we use John’s Ninja approach. The downside is the syntax is very complex and very fragile. A typo anywhere in the code is likely to get you a general syntax error message, generally in the last line of the (function(){})(); construct. Not very helpful that. We are mixing standard JavaScript syntax with JSON syntax together and it is easy to get lost without good planning. Personally we never make syntax errors, but for others this may be a problem! An additional downside is debugging anonymous functions by setting break points in your running code will not work. I work in a team environment and when I move this solution into production, I will likely break out some of the anonymous functions into normal (inner functions) to make it easier for other team members see the intent of the code, to ease making code changes and to simplify debugging. If I was shipping code to be worked on only by myself or by a (very) limited sized team I would leave it just as it and watch it fly.

Home
%d bloggers like this: