Archive for the ‘REST’ Tag

Sharepoint 2013 REST API: Retrieving List Item Data   1 comment

I first discussed retrieving List Item data here: Sharepoint 2013 REST API: The C# Connection: Part 3 Working With List Item Data.  The simple REST API call:

Web/list(guid’850fae4-0cce-8c30-c2a85001e215’)/Items

functions like

select * from Table1

in a SQL data base.  However Sharepoint, although resting on SQL Server for everything, will only return the a limited number of rows from the underlying list.  The maximum number of items (rows) returned in a single call is governed by a system constraint.  On our out of the box Sharepoint 2013 installation the limit was 100 rows!  Supposable Sharepoint Sever Admins can change the maximum value but they can not eliminate the constraint.  The naïve coder might think that this can be gotten around using the $SKIP parameter in the ODATA extensions to the Sharepoint REST API.  Alas $SKIP is not implemented, instead M$ implemented it own undocumented skip parameters.  No to worry you don’t need to provide your own parameter values.  The initial call using

…/Items

returns X rows of the table (in primary key index order) where X is equal to the page constraint discussed above.  If there are additional rows the metadata in the returned data structure will return a “next” hypermedia link whose value is fully qualified Sharepoint Rest API call for the next X rows (i.e. the next page). 

google-barge-sinking

                               Google Barge Sinking Slowly in San Francisco Harbor after being infected with the Sharepoint 2013 Virus

We need then to setup a typical paging object to read items by page until all items are read.  I will be leveraging the code patterns given in the series referenced above the page calls (…/items) return results in XML format and is cast as an XDocument type.  Then we can probe for a “next” link as:

static string _GetLinkToNextPage(XDocument xDoc)
{
const string matchString = “/Web/Lists”;
var links = from g in xDoc.Descendants().Elements(CSPNamespace.baseNS + “link”)
select g;
const string NEXT = “next”;
foreach (XElement link in links)
{
string rel = string.Empty;
string href = string.Empty;
foreach (XAttribute att in link.Attributes())
{
if (att.Name.LocalName == “rel”)
{
rel = att.Value;
}
if (att.Name.LocalName == “href”)
{
href = att.Value;
}

               }
if (rel == NEXT)
{

                   //return just the relative path
return href.Substring(href.IndexOf(matchString));
}
}

           //return null if no link is found
return null;
}

The complete object template looks like:

public static class CItemPage
{
public static List<CEntry> _Worker(HttpClient client, string next)
public static List Exec(HttpClient client, CEntry listMetaData)
       static void _Accumulate(XDocument items, ref List allItems)
static string _GetLinkToNextPage(XDocument xDoc)
static XDocument _GetListMetaData(HttpClient client)

}

The entry point for the paging activity is CItemPage.Exec which turns around and calls worker where we loop through the data and (for my own purposes) I then accumulate the items from all pages in a single list<CEntry>  using my method _Accumlate.  calls Worker

       public static List _Worker(HttpClient client, string next)
{
try
{
List allItems = new List();
XDocument items = null;
while (next != null)
{
items = _GetItems(client, next);
next = _GetLinkToNextPage(items);
_Accumulate(items, ref allItems);
}
return allItems;
}
catch (Exception x)
{
var b = x.Message;
return null;
}
}

 

Easy and fun.  Should there be an easier, faster way? yes.

DLRSharePoint-On-Prem-isnt-Dead-Yet

This madness must stop.

Sharepoint 2013 REST API: The C# Connection: Part 2 Query List or Item and Decoding The Meta-Data   4 comments

Let’s assume that we have an HTTPClient object as described in Part 1.  That is we have

  • created the HttpClient objected,
  • set the base URL to our Sharepoint site, 
  • set the Authentication Header; and
  • set the Accept Header for  “application/atom+xml”. 

Now we want to query the REST API for the metadata of a Sharepoint List.  Sharepoint List Titles are unique by design we can use the List Title (rather than the GUID) to locate the item and return its metadata. The format of this call as a uri fragment is:

web/lists/GetByTitle(‘listname’)

Now our code looks like:

string uri=”web/lists/GetByTitle(‘Master Document Library’)”;

//Note that spaces are allowed in most calls but there are other situations where spaces are escaped!

HttpResponseMessage resp = client.GetAsync(uri).Result;

string respString = resp.Content.ReadAsStringAsync().Result;

if (resp.StatusCode != HttpStatusCode.OK)

{

throw new ApplicationExcepiton(

                    “HTTP Error. Status: {0} Reason: {1}”,resp.StatusCode,    resp.ReasonPhrase );         

           }

This will put a string representation of the XML formatted metadata about the list into the HttpResponseMessage.  Please note that the call to extract the XML from the response body:

string respString = resp.Content.ReadAsStringAsync().Result;

is only appropriate for string results (XML or JSON as specified in our Accept Header) and is not correct if the return is binary data.  I will copy binary data in a subsequent blog when I discuss file upload and download.

Few things are less documented than the exact contents of the metadata feed returned by calls like this from the REST API.  On a high level it is an Atom feed which allows for a combination of collections of  Property nodes (key/value pairs) and collections of Link nodes. The Property Nodes are the meta data fields related to the list and the Link Nodes are uri segments to guide additional REST API calls concerning what ever item is currently being queried.  Neither the properties nor the links are a fixed are vary from Sharepoint object to object and even vary between objects of the same type if those object depending on the values of the property fields (for example a list item on a list with versioning will contain a Link node for access to the versions.  If the list items are not versioned then the link item of versions will not be emitted.

Rather than list XML directly I will use the visualization tool XMLSPY to display the XML in a “grid view”.  On a high level the entry for a list would look like:

Untitled picture

The Links are on the root of the entry node and the properties are nested as entry/content/properties.  Note that the XML makes heavy use of XML namespaces and any creative manipulation of the entry requires some knowledge of XML, namespaces,  XPath or LINQ for XML.  I use LINQ for XML at my desk so I will use that idiom rather than XPATH to manipulate these objects.  If we expand the properties node it will look something like this:

Untitled picture2

There is a lot of data here about the list most of it only lightly documented.  We can see however that the property key: d:title contains the name of our list which we queried on and d:iD contains the GUID for the list.  The later never changes but the former can be renamed.

If we expand the Links collection  it would look something like this:

Untitled picture3

Note item 10, the Items link. The href attribute contains the uri for the REST API to retrieve all the items in this list, while the Fields link (item 7) is an uri for the Fields currently defined for this list. If we know the d:Id of a particular item (item IDs are not GUIDS for simple integers), say 6, we can retrieve a single item with the uri of the Items uri and post pending in the form of:

Web/list(guid’850fae4-0cce-8c30-c2a85001e215’)/Items(6)

What about the link with a blank title? For historical reasons this is blank but it represents the EDIT link.  To make my life simpler I translate the XML property and link collections into C# Dictionary objects and place them in a C# class with two supporting methods:

public class CEntry

{

public Dictionary<string, string> Links;

public Dictionary<string, string> Properties;

      public string  GetLink(string key){

string value = string.Empty;

Links.TryGetValue(key, out value);

return value;

}

      public string GetProperty(string key)

{

string value = string.Empty;

Properties.TryGetValue(key, out value);

return value;

}

}

At this time I am not using any of the root nodes so I just discard them. I get to the Dictionary objects from the XML using LINQ for XML.  I learned what little I know about LINQ for XML from this book.  To brush up on your XML try this book.   For a XML tree containing entry node(s) my LINQ looks like this:

public static class CSPNamespace

{

public static XNamespace metaDataNS = @”http://schemas.microsoft.com/ado/2007/08/dataservices/metadata”;

       public static XNamespace baseNS = @”http://www.w3.org/2005/Atom”;

       public static XNamespace dataServicesNS = @”http://schemas.microsoft.com/ado/2007/08/dataservices”;

   }

XDocument xList =XDocument.Parse(respString);

IEnumerable<CEntry> group = from g in xList.Descendants(CSPNamespace.baseNS + “entry”)

select new CEntry

{

           Links = MakeLinkDictionary(g),

Properties = MakePropertyDictionary(g)

};

The IEnumerable collection needs special processing before it is accessed.  The following test can help:

To see if the collection contains one or more entries:

group != null && group.Any();

Having passed that test, we can then use the simple Count function to see how many entries are in the collection.

group.Count()

To get the first (or only) entry from the collection

group.First()

These last two test will fails if the collection fails the test above

        CEntry cell = group.First();  //Assumes one and one only

Where MakeLinkDicitionary and MakePropertyDictionary look like:

public  static Dictionary<string, string=””> MakePropertyDictionary(XElement xs)

{

Dictionary<string, string=””> pList = new Dictionary<string, string=””>();

var group = from g in xs.Elements(CSPNamespace.baseNS + “content”).Descendants(CSPNamespace.metaDataNS + “properties”)

select g;

foreach (XElement property in group.Elements())

{

pList.Add(property.Name.LocalName, property.Value);

}

return pList;

}

public  static Dictionary<string, string=””> MakeLinkDictionary(XElement xs)

{

       Dictionary<string, string=””> lList = new Dictionary<string, string=””>();    IEnumerable links = from g in

                      xs.Elements(CSPNamespace.baseNS + “link”)

select g;

foreach (XElement link in links)

{

string rel = string.Empty;

string href = string.Empty;

foreach (XAttribute att in link.Attributes())

{

if (att.Name.LocalName == “title”)

{

                   if (string.IsNulOrEmpty(att.Value)){

                      rel = “Edit”;

                   }else{

rel = att.Value;

                        }

}

if (att.Name.LocalName == “href”)

{

href = att.Value;

}

}

lList.Add(rel, href);

       }

return lList;

}

After this pre-processing the meta data can be accessed in a fairly straight forward manner.

var listMetaData=group.First();

string uri=listMetaData.GetLink(“Fields”);

string iD=listMetaData.GetProperty(“Title”);

We will turn to what to actual do with the meta data in the next post.

Sharepoint 2013 REST API: The C# Connection: Part 1 Using System.Net.Http.HttpClient

Sharepoint 2013 REST API: The C# Connection: Part 2 Query List or Item and Decoding The Meta-Data

Sharepoint 2013 REST API: The C# Connection: Part 3 Working With List Item Data

Sharepoint 2013 REST API: The C# Connection: Part 4 Document Libraries, Folders And Files
Sharepoint 2013 REST API: The C# Connection: Part 5 REST API More on Folders and Other Odds & Ends

Sharepoint 2013, The REST API and The C# HTTPClient   4 comments

This is a short post to highlight two issues with how the C# HTTPClient is implemented in Framework 4.0 and how to work around these issues.  The issues are the Content Header and the Accept Headers for JSON data.  This post is NOT a full discussion of using the HTTPClient and the Sharepoint 2013 REST API but is limited solely to properly code the Content Header and the Accept Header for this interface.

The new Asynchronous web client   System.Net.Http.HttpClient is a joy to work with.  Recently I was tasked with interfacing with the Sharepoint 2013 REST API using C# (don’t ask why). Most example code for the REST interface are written using JQUERY on the browser.  Since we needed to call the API from within a C# program we attempted to use the HttpClient for this purpose.  Sharepoint Data in the REST interface is in ODATA format.  To use the API we need to declare the Accept Headers for the format of the data we want the ODATA to be formatted by Sharepoint.  If we are using POSTS, PUTS or DELETE with the API we need to declare the Content Type Header to describe the format of the data we are sending to the Sharepoint REST API.  Our choices are:

Data Format Header Value
XML application/atom+xml
JSON application/json;odata=verbose

Setting up the HTTPClient, for all verbs looks like this:

1)System.Net.Http.HttpClient _Client = new System.Net.Http.HttpClient();
2)  _Client.BaseAddress = new Uri(baseURL);

// When the format we are using for incoming data is XML we add this line:

3) _Client.DefaultRequestHeaders.Accept.Add(new MediaTypeWithQualityHeaderValue(“application/atom+xml”));

When the format we are using for incoming data is JSON if we replace, in line #3 “application/atom+xml” with the required “application/json;odata=verbose”, Line #3 will thrown an Exception.  The work around is to replace line #3 with:

_Client.DefaultRequestHeaders.Add(“Accept”, “application/json;odata=verbose”);

//and off we go

4) HttpResponseMessage resp = client.GetAsync(uri).Result;
5) string respString = resp.Content.ReadAsStringAsync().Result;

When we are using the HTTP verbs POST, PUT or DELETE we need to send a Request Body with the data we want to send to the server and set a Content Type Header to tell server what format data in the body contains.  The HTTPClient holds the request body in its own object (System.Net.Http.HttpContent):

1) string myData=”your data XML or JSON goes here”;

2) System.Net.Http.HttpContent reqContent = new StringContent(myData);

//We set the Content Type header on this object, NOT on the HttpClient object as:

3) reqContent.Headers.ContentType = new System.Net.Http.Headers.MediaTypeHeaderValue(“application/atom+xml”);

When the format for HTTPContent is JSON, if we replace “application/atom+xml” with the required “application/json;odata=verbose”, Line #3 will thrown an Exception. The work around is to replace line #3 with:

reqContent.Headers.ContentType = System.Net.Http.Headers.MediaTypeHeaderValue.Parse(“application/json;odata=verbose”);

//and off we go

4) var resp = _ClientAddItem.PostAsync(addItemURL, reqContent).Result;

5) string respStringOut = resp.Content.ReadAsStringAsync().Result;

Strange but true.  You are welcome.

REST, WEB API And CORS   2 comments

Introduction

Cross Domain AJAX calls (CORS) on desktop browsers require special processing on both the server side and in the way we call AJAX from within the browser. A general overview of CORS can be found hereASP.Net WEB API allows a couple of fairly straight forward ways to implement REST HTTP endpoints with CORS support.  Using the current release build of WEB API we can code our CORS handlers directly or if you want to use the nightly builds of the WEB API you can use an attribute approach.  This post will concentrate on how to write CORS handlers directly since this is the approach I have this in test right now and this approach allows you more flexibility in implementation and improved debugging options.  I will be concentrating on implementation details and assume you have read the background blogs listed above before we start.  I will also be looking at the browser side implementation of the CORS call and some issues with IE browsers (IE 9 in particular).  We are testing with Windows Server 2012 and are using Firefox, Chrome and IE as our test browsers.

Voice from the future: Brock Allen’s great work on CORS, CORS based CORS Attribute support has now been incorporated into Web API 2.  See here and here for details.

So What’s the Problem.

The W3C defines special procedures required if a browser is going to make an AJAX call to a server which is not in the domain of the page which served the page which is making the call (hence Cross Domain).  To enable CORS the server must implement CORS and the browser must make the AJAX call following some conventions.  In the WEB API framework CORS can be implemented on the method or site level.  We will focus on site level CORS in this post.  The WEB API pipeline allows us to hook in message handlers at several places.  The canonical CORS handler, given by the links listed above looks like this:

using System;
using System.Collections.Generic;
using System.Linq;
using System.Net;
using System.Net.Http;
using System.Security.Claims;
using System.Threading;
using System.Threading.Tasks;
using System.Web.Http;
using System.Web.Http.Tracing;

public class CorsHandler : DelegatingHandler
{
const string AccessControlRequestMethod = “Access-Control-Request-Method”;
const string AccessControlRequestHeaders = “Access-Control-Request-Headers”;
const string AccessControlAllowOrigin = “Access-Control-Allow-Origin”;
const string AccessControlAllowMethods = “Access-Control-Allow-Methods”;
const string AccessControlAllowHeaders = “Access-Control-Allow-Headers”;

       protected override Task<HttpResponseMessage> SendAsync( HttpRequestMessage request, CancellationToken cancellationToken )
{
const string Origin = “Origin”;
bool isCorsRequest = request.Headers.Contains( Origin );
bool isPreflightRequest = request.Method == HttpMethod.Options;
if ( isCorsRequest )
{
//HTTP CORS OPTIONS
if ( isPreflightRequest )
{
HttpResponseMessage response = new HttpResponseMessage( HttpStatusCode.OK );
response.Headers.Add( AccessControlAllowOrigin, request.Headers.GetValues( Origin ).First( ) );
                   string accessControlRequestMethod = request.Headers.GetValues( AccessControlRequestMethod ).FirstOrDefault( );
if ( accessControlRequestMethod != null )
{
response.Headers.Add( AccessControlAllowMethods, accessControlRequestMethod );
}

                   string requestedHeaders = string.Join( “, “, request.Headers.GetValues( AccessControlRequestHeaders ) );
if ( !string.IsNullOrEmpty( requestedHeaders ) )
{
response.Headers.Add( AccessControlAllowHeaders, requestedHeaders+”, AICJWT” );
}

                   TaskCompletionSource<HttpResponseMessage> tcs = new TaskCompletionSource<HttpResponseMessage>( );
tcs.SetResult( response );
return tcs.Task;
}
else
{
//HTTP CORS GET
return base.SendAsync( request, cancellationToken ).ContinueWith<HttpResponseMessage>( t =>
{
HttpResponseMessage resp = t.Result;
resp.Headers.Add( AccessControlAllowOrigin, request.Headers.GetValues( Origin ).First( ) );
return resp;
} );
}
}
else
{
//NOT A CORS CALL
return base.SendAsync( request, cancellationToken );
}
}
}

Lets break this down from the simplest part first.  We create a class derived from DelegatingHandler (since we are implementing at the site level).  We hook this handler into the system within the framework generated class WebApiConfig as

public static class WebApiConfig
{
public static void Register( HttpConfiguration config )
{
//your route code here

           );
config.MessageHandlers.Add( new WebAPI.Handler.CorsHandler( ) );

//other handlers are included here.
}
}

If you have other classes based on DelegatingHandler the order in which they are added in WebApiConfig matters.

In the simplest case where we are not making a CORS call we can simply return the handler without action as:

return base.SendAsync( request, cancellationToken );

When the CORS call is made by the browser the caller should include the standard HTTP header: Origin with a value of the calling pages domain.  The canonical code assumes this and uses the presence of this header to detect a CORS call. Hence the code:

const string Origin = “Origin”;
bool isCorsRequest = request.Headers.Contains( Origin );

If the CORS call is not an OPTIONS call (which the canonical code call preFlight) we see the code:

return base.SendAsync( request, cancellationToken ).ContinueWith<HttpResponseMessage>( t =>
{
HttpResponseMessage resp = t.Result;
resp.Headers.Add( AccessControlAllowOrigin, request.Headers.GetValues( Origin ).First( ) );
return resp;
} );

Here the code returns a required header for the Browser: Access-Control-Allow-Origin with the value taken from the Origin Header of the caller.

We could, if we choice to, have set the value to  the wild card value ( * ) but this openness may make your system administrator nervous.  Notice here that nothing in the W3C specification restricts what other Headers the sender can include in the CORS call.  However certain browsers (IE) and certain Javascript packages (jQuery) restrict the call to standard HTTP request Headers.  In our implementation this gave us some problems but more on this later. The browser code (User-Agent), not the user code, will refuse to accept the return if the Origin Header is missing or does not contain either the wild card or the calling page’s domain in the value for the Origin header.

So What is the Rest of the Handler Code Doing?

Following this document from Mozilla.org, the browser making the call may make an optional CORS OPTIONS call (see here for HTTP verbs if this one is new to you).  This preflight call (as the canonical code names it) has asks the server for details about what may be in the CORS request when it is actually call.  Following the Mozilla explanation here is what needs to happen:

    • 1a. The User-Agent, rather than doing the call directly, asks the server, the API, the permission to do the request. It does so with the following headers:
      • Access-Control-Request-Headers, contains the headers the User-Agent want to access.
      • Access-Control-Request-Method contains the method the User-Agent want to access.
    • 1b. The API answers what is authorized:
      • Access-Control-Allow-Origin the origin that’s accepted. Can be * or the domain name.
      • Access-Control-Allow-Methods a list of allowed methods. This can be cached. Note than the request asks permission for one method and the
        server should return a list of accepted methods.
      • Access-Allow-Headers a list of allowed headers, for all of the methods, since this can be cached as well.

In the canonical code given above here is what happens in the CORS OPTIONS call:

//( 0 )create a response object

HttpResponseMessage response = new HttpResponseMessage( HttpStatusCode.OK );
//( 1 ) build the value string for the Access-Control-Allow-Origin header from the ORIGIN header value of the request

response.Headers.Add( AccessControlAllowOrigin, request.Headers.GetValues( Origin ).First( ) );

//( 3 )build the value string for the Access-Control-Request-Headers from the values in the request
string accessControlRequestMethod = request.Headers.GetValues( AccessControlRequestMethod ).FirstOrDefault( );
if ( accessControlRequestMethod != null )
{
response.Headers.Add( AccessControlAllowMethods, accessControlRequestMethod );
}

//( 4 ) build the value string for the Access-Control-Allow-Headers header from the ORIGIN headers value of the request

string requestedHeaders = string.Join( “, “, request.Headers.GetValues( AccessControlRequestHeaders ) );
if ( !string.IsNullOrEmpty( requestedHeaders ) )
{
response.Headers.Add( AccessControlAllowHeaders, requestedHeaders);
}

//( 5 ) interrupt the pipeline and return the response object to the caller.

TaskCompletionSource<HttpResponseMessage> tcs = new TaskCompletionSource<HttpResponseMessage>( );
tcs.SetResult( response );
return tcs.Task;

Please note that we can put whatever we need into the Header values.  For example if we wanted to limit CORS calls to GET request only we could replace ( 3) with the simple:

response.Headers.Add( AccessControlAllowMethods, “GET” );

To allow a specific domain only to make the CORS call we could replace ( 1 ) with:

response.Headers.Add( AccessControlAllowOrigin, “www.special.com” );

In our case we wanted to allow a specific non-standard Header into the CORS request.  We called this Header AICJWT. So we expanded the key line in ( 4 ) to be:

response.Headers.Add( AccessControlAllowHeaders, requestedHeaders+”, AICJWT” );

The reason we added it explicitly here is due to problems in both JQuery and in IE.  Please note again that the CORS OPTIONS call is optional.  At this point in our development we were using the awesome async Framework 4.5  object: System.Net.Http.HttpClient.  This is a get object and very useful during development BUT there is no User-Agent (browser side code) involved.

The Trouble Begins: Browser Side Code

All seemed swell, till the JavaScript coders tried to call into the system.  JQuery forces a CORS OPTIONS call when it detects a Cross Domain AJAX call.  For reasons which remain unclear JQUERY does not include custom headers in the OPTIONS request.  Some people think this is in the W3C spec for CORS but I don’t see it there, do you?  Some folks out there indicate that the user-agent is forcing the OPTIONS request but this is not true.  If we use a direct AJAX call, not using JQUERY we can make our own CORS OPTIONS request or skip the OPTIONS call completely.  Here is the code to make the call using JavaScript in IE:

function callHttpReq() {
var invocation = new XMLHttpRequest();
var url = ‘
http://whereever/api/Concert?Year=1980′;
    var body = ”;

var token = “myspecialstuff”;
function callOtherDomain(){
if(invocation)
{
invocation.open(‘GET’, url, true);
invocation.setRequestHeader(‘AICJWT’, token);
invocation.setRequestHeader(‘Accept’, ‘application/json’);
invocation.onreadystatechange = httpHandler;
invocation.send(body);
}
}
callOtherDomain();

    function httpHandler() {
if(invocation.readyState == 4) {
var jsonObj = $.parseJSON(invocation.responseText);
if(jsonObj.length > 0) {
htmlStr = “<ul>”;
$.each(jsonObj, function(i,row) {
htmlStr += “<li>” + row.Date + ‘—-‘ + ” ” + row.Venue +”</li>”;
});
htmlStr += “</ul>”;
$(“#responeBody”).append(htmlStr);
}
}
}

}

Note we are skipping JQUERY because we require a custom header in our use case.  This step is not necessary if you are NOT using custom headers in the CORS call.  Note also that if you are not using JQUERY you need to use different AJAX object other than IE’s XMLHttpRequest.  If you can use JQUERY there is a masive amount of documentation about how to make AJAX calls and JQUERY will handle CORS and differences between the IE and other AJAX objects automatically.

IE Blues

OK all is good but when we test with IE 8 or 9 we get back the data from the CORS get BUT the user also gets the dialog box:

image

Microsoft tells us the USER can suppress this IN IE8 and IE9 by following this procedure:

You can check your Security Zone level as well as change this setting by going to Tools, Internet Options, and click Security tab. Your Security Zone level will be listed on this screen, by default this is set to Medium-high. To change the setting for your message click Custom Level , locate the Access data sources across domains under the Miscellaneous section and change the setting from Prompt to a desired setting.

image

We do not have this problem in Chrome or Firefox. Live Free or Die.

One Last Server Side Issue

During our testing, using Windows Server 2012 we ran into one additional problem.  Our CORS OPTIONS calls were not getting to our site but where being intercepted by the HTTP Module prior to the site Delegate Handler.  Without getting into it too deeply we needed to modify the web.config for our CORS site and disable WebDAV (don’t ask) and allow OPTIONS for the ExtensionlessUrlHandler.  See here for details.  As far as we know this is a pure Windows 2012 server issue.

Web API: HTTP Restful Endpoints, The WCF Way   3 comments

Part I: Overview and Serialization (this post)

Part II: WCF Endpoints

Part III: Implementing WCF EndPoints

Restful HTTP Endpoints

I love QCON, the technical web conference sponsored by InfoQ.  This year I attended the conference in San Francisco.  Some nice things about QCON is that it is not a a trade conference for a major powerhouse like Microsoft or Google.  Although I am a regular attendee of Microsoft’s MIX conferences, but there is something to be said for QCON focus on emergent web technologies and its focus on ideas and not products. My conference report is can be found here. This year I attended the tutorial “REST in Practice” which was presented by the authors of the book “REST in Practice”, Ian Robinson & Jim Webber.  The presenters and the audience were very well informed on both the web in the real world and the minutia of HTTP programming.  For a nerd like me (NLM), nothing is more fun than getting into  the finer points of a Fiddler dump. Untitled picture One thing bumping around QCON this year was the topic “WEB API”, what in the world I though was a WEB API.  Don’t be unhip like me, what is meant by a WEB API is exposing a full API for a distinct system solely through a coordinated set of RESTful HTTP endpoints (think Netflix and Flickr).  Somehow this wants to be a TLA like WEP (Web End Point but WEP is already taken).  This is the concept of AJAX matured into a true systematic technology.   In my day job I work on the development of secure Web Portals, armed with the background provided by the tutorials and sessions at QCON I returned to work resolved to cleanup our HTTP end points.  A couple of references may help.  On RESTful development see the Richardson Maturity Model and the InfoQ classic: How To GET a Cup Of Coffee.

WCF: The Microsoft Connection

I work in a Microsoft shop and so the Microsoft’s WCF is the tool of choice to develop HTTP Endpoints.  Because of our particular product goals HTTP Get’s provide the bulk of our services.  Starting with Framework 3.5, WCF has provided a rich set of tools to create RESTful dynamic HTTP Endpoints.   I started out programming XML Web Service for intra-server communication.  Although this is a very mature and powerful technology it is not optimized for Browser to Server communication, particularly  Javascript is difficult to use to develop Web Service proxies (although it can be done) and XML is rarely the tool of choice to communicate object graphs to Javascript, JSON is preferred for this purpose.  WCF For Rest (see this and this reference).  Here is a chart I developed from my notes during QCON comparing Web Services, REST and WCF for Rest:

Web Service Standard REST Service WCF For REST (Framework 3.5)
1 TCP/IP + others TCP/IP TCP/IP
2 SOAP Wrapper HTTP HTTP
3 SOAP Headers HTTP Headers HTTP Headers
4 WS*Security Basic Auth/SSL Basic Auth/SSL or WS*Security
5 Early Binding Late Binding Late Binding
6 XSD WADL XSD
7 XML Media Negotiation Media Negotiation
8 SOAP FAULTS HTTP Response Codes HTTP Response Codes
9 Single Endpoint Multiple Endpoints, URI Templates Multiple Endpoints, URI Templates
10 Client Proxy Custom auto-generated Javascript proxy

The core text I used in developing WCF for Rest is the excellent and insightful RESTful .NET by Jon Flanders (see also and Jon’s Blog).

What We Are Trying to Achieve: The Problem Domain

In this blog post we will focus on a limited problem I need to solve.  Here is the overview of the problem.  At fixed points in time our code needs to collect a complex set of information on a particular subject (i.e. a graph) and persist that information for later recall. At irregular times browsers will request sub sets of the information in JSON format. The data object is quite complex.  Entity Framework would happily generate the hundreds (thousands?) of lines of code to persist and retrieve the data from a SQL database but this task cries out for a key value store (like couchDB ) so rather than looking at:

Picture2

We can have something like:

Untitled picture2Where User_Id and Date are meta data fields for the object graph represented in the column AlertsUser.  This seems like a good start, before developing the actual HTTP Endpoint we need to deal with serialization issues.  How and in what format to serialize the C# object into a nvarchar column in the database, how to desterilize that data back into a C# object and then to serialize the object into JSON for delivery to the user.  Note that in some use cases you could simplify this to:  Object serialized to JSON; and then inserted into the database  and then retrieved and passed directly to the caller.  In our use case however a single HTTP Get will request one or more rows from the data base so we require the more complex: Object serialized to nvarchar compatible object then deserialized into an array of C# object and then serialized into a JSON array.  In this use case we will use XML as our format for storing data into the database, so we have

Object –> database rows of XML in string format

XML rows – > An Array of C# Objects

Array of C# Object –> JSON Array.

In the next section of this post I will cover how to do this using C# and when this is accomplished I will in the subsequent section turn to how I developed a generic HTTP Get endpoint to serialize the data into JSON and return data in that representation to the browser.

Serialization: Objects &  XML

For purposes of this post lets define a basic C# object which we want to work with.  Here is a minimal object for serialization:

    public class CObject{
public string ID { get; set; }
public string Name { get; set; }
public List<string> Data { get; set; }
public CObject( ){
Data = new List<string>( );
}
}


Note that I have defined an an explicit class initializer. If you are going to be doing serialization a parameterless initializer is required.  To make fields available for WCF type serialization we need to add attributes: DataContract and DataMember.  Use the DataMember attribute to identify those fields you whish to serialize.

    [DataContract]
    public class CObject{
[DataMember]
public string ID { get; set; }
[DataMember]
public string Name { get; set; }
[DataMember]
public List<string> Data { get; set; }
public CObject( ){
Data = new List<string>( );
}
}

For XML serialization we may further shape the XML by using the XML attributes as follows:

XmlRoot                                            Defines The XML Namespace used by the object

XmlElement                                     Causes the XML serializer to output a field as an element and optionally names the element

XmlAttribute                                   Causes the XML serializer to output a field as an attribute to the enclosing element and optionally names the attribute

XmlArray and XmlArrayItem             Names The Root Element of an Array and separately the names of each array member

Our full adorned object might then look like:

    [DataContract]
[XmlRoot(Namespace=”
http://cloud2013/2012/01/”)]
    public class CObject{
[DataMember]
[XmlElement(“Key”)]
public string ID { get; set; }
[DataMember]
[XmlAttribute]
public string Name { get; set; }
[DataMember]
[XmlArray(“Points”)]
[XmlArrayItem(“Point”)]
public List<string> Data { get; set; }
public CObject( ){
Data = new List<string>( );
}
}
We can use the WCF Data Contract Serializer to serialize an object to an XML string using the generic method ToXMLString and serialize from and XML string back into a C# Object using the generic method XMLToObject as:

public class XMLMethods<T>
{
const string _NameSpaceBase=”
http://schemas.datacontract.org/2004/07/”;
       public static string ToXMLString( T myObject )
{
MemoryStream stream1 = new MemoryStream( );
DataContractSerializer ser = new DataContractSerializer( typeof( T ) );
ser.WriteObject( stream1, myObject );
return Convert.ToString( Encoding.UTF8.GetString( stream1.GetBuffer( ), 0, ( int ) stream1.Length ) );
}
public static T XMLToObject( string xmlString )
{
TextReader sr=new StringReader( xmlString );
XmlRootAttribute root=new XmlRootAttribute( typeof( T ).Name );
root.Namespace = _NameSpaceBase + typeof( T ).Namespace;
XmlSerializer xmlSerializer = new XmlSerializer( typeof( T ), root );
return ( T ) xmlSerializer.Deserialize( sr );
}
}

calling these as:

CObject myObject=new CObject();

string xmlString=XMLMethods<CObject>.ToXMLString(myObject);

myObject=XMLMethods<CObject>.XMLToObject(xmlString);

Easy and Fun, no?

If we insert or update a SQL server column with an XML String we will need to escape the XML so we don’t run into conflicts with the string syntax of what ever SQL server type you are losing.

In my case with SQL server I need to escape single and double quote marks.  One simple way to do this, but not necessarily the fastest way,  is to 64 bit encode the string. Example code to do this is:

public static class CStaticFunctions
{
static public string EncodeTo64( string toEncode )
{
byte[] toEncodeAsBytes = System.Text.ASCIIEncoding.ASCII.GetBytes( toEncode );
return System.Convert.ToBase64String( toEncodeAsBytes );
}
static public string DecodeFrom64( string encodedData )
{
byte[] encodedDataAsBytes = System.Convert.FromBase64String( encodedData );
return System.Text.ASCIIEncoding.ASCII.GetString( encodedDataAsBytes );
}
}

[I have added a programmer’s note on using SQL Server as an object serialization store.  See this Post]

OK.  Enough for Part I. To be continued…

QCON 2011 San Francisco and Occupy California   2 comments

Let me say write off that I do not pay for my own ticket to QCON, my boss picks up the tag.  I love QCON.  It is definitely not MIX. I go there to see what is happening in the world which 6439629043_9a7e84a2bd_z is NOT Oracle and Not Microsoft.  That’s the same reason I read their online Zine: InfoQ.   QCon always provides a look at what is current and recent in the open stack world.  This year we looked closely at REST, Mobile development, Web API and NOSQL. As they did last  year QCON provides a nice look at what is open and emerging.  Big metal with always be with us but the desk top is looking6373613127_9780c7d60f very weak during the next few years while Mobile devices of all kinds and makers are exploding.  The biggest fall out is that while HTML5 is only slowly emerging on desktops in place, all new Mobile devices (which is to say most new systems) will be fully HTML5 compliant.  Not only that but with the exception of Windows Phones, the rendering engine for all mobile devices is based on WebKit.  What this mean for those of us in the cubes is that worrying about how to bridge to pre-HTML5 browsers with HTML5 code is a non-issue.  Mobile development is HTML5 development.  The big metal end of the supply chain is being segmented into Web API servers (which service JSON XHR2 data calls) and the NOSQL engines which serve the WEB API farms.  Remember a native mobile app     ideally has pre-loaded all of its pages its interactions are solely over JSON XHR2 for data (be it documents, data or HTML fragments).  The traditional JSP or ASPX web server is not really in play with native mobile apps and has and increasingly small role to play in “native like” or browser based mobile apps.  Let’s move on.

“IPad Light by cloud2013”

Speaking of moving on: There is an occupation going on in this country.  I visited occupations sites in San Francisco, UCal Berkeley and  Berkeley “Tent City”.  These are all very active and inspiring occupy sites.  Now if we can only get to Occupy Silicon Valley! 

I attended the REST in Practice tutorial this year and it was a very nice.  The authors were well informed and the agenda comprehensive.  I personally like the Richardson maturity model but think that people are not facing up to the fact that level three is rarely achieved in practice and the rules of web semantics necessary to interoperate at level 3 are almost non-existent. Remember the original REST model is client/server.  The basic model is a finite state machine and the browser (and the user) are in this model required to be dumb as fish.  Whether Javascript is a strong enough model and late binding semantics can be made clear enough to pull off level three is really an open question which no one has an answer to.  If we forget about interoperability (except for OAuth) things start to fall into place but we thought OPENNESS was important to REST.

Workshop: REST In Practice by the Authors: Ian Robinson & Jim Webber

Why REST? The claims:

· Scalable

· Fault Tolerant

· Recoverable

· Secure

· Loosely coupled6439625819_5705585c80

Questions / Comment:6380018433_9172323197

Do we agree with these goals?

Does REST achieve them?

Are there other ways to achieve the same goals?

REST design is important for serving AJAX requests and AJAX requests are becoming central to Mobile device development, as opposed to intra-corporate communication. See Web API section below.

Occupy Market Street (San Francisco)            

The new basic Document for REST: Richardson Maturity Model (with DLR modifications)

Level 0:

One URI endpoint

One HTTP method [Get]

SOAP, RPC

Level 1:

Multiple URI,

One HTTP Method [Get]

Century Level HTTP Codes (200,300,400,500)

Level 2:

Multiple URI,

Multiple HTTP Methods

Fine Grain HTTP Codes (“Any code below 500 is not an error, it’s an event”)

URI Templates

Media Format Negotiation (Accept request-header)

Headers become major players in the interaction between client and server

Level 3:  The Semantic Web

Level 2 plus

Links and Forms Tags (Hypermedia as the engine of state)

Plus emergent semantics

<shop xmlns=”http://schemas.restbucks.com/shop&#8221;

xmlns:rb=”http://relations.restbucks.com/”&gt;

<items>

<item>…</item>

<item>…</item>

</items>

<link rel=”self” href=http://restbucks.com/quotes/1234 type=”application/restbucks+xml”/>

<link rel=”rb:order-form” href=”http://restbucks.com/order-forms/1234″ type=”application/restbucks+xml”/&gt;

</shop>

6439622787_7b614f312c

Think of the browser (user) as a finite State Machine where the workflow is driven by link tags which direct the client as to which states it may transition to and the URI associated with each state transition.6380028389_e64c6a826f

The classic design paper on applied REST architecture is here: How To GET a Cup Of Coffee. Moving beyond level 1 requires fine grain usage of HTTP Status Codes, Link tags, the change headers and media type negotiation. Media formats beyond POX and JSON are required to use level 3 efficiently (OData and ATOM.PUB for example).

Dude, where’s my two phase commit? Not supported directly, use the change headers (if-modified, if-non-match, etag headers) or architectural redesign (redefine resources or workflow). Strategic choice is design of the finite state machine and defining resource granularity.

clip_image002

(Slide from Rest in Practice)

Architectural Choices:

The Bad Old Days: One resource many, many ‘verbs’.

The Happy Future: Many, many resources, few verbs.

The Hand Cuff Era: Few Resources, Few verbs.

The Greater Verbs:

GET: Retrieve a representation of a resource

POST: Create a new resource (Server sets the key)

PUT: Create new resource (Client sets the key); ( or Update an existing resource ?)

DELETE: Delete an existing resource

Comment: The proper use of PUT vs. POST is still subject to controversy and indicates (to me) that level 3 is still not well defined.

Typically they say POST to create a blog entry and PUT at append a comment to a blog. In Couchdb we POST to create a document and PUT to add a revision (not a delta) and get back a new version number. The difference here is how the resource is being defined, which is an architectural choice.

6439621853_3275941633

The Lesser Verbs:

OPTIONS: See which verbs a resource understands

HEAD: Return only the header (no response body)

PATCH: Does not exist in HTML5. This would be a delta Verb but no one could agree on a specification for the content.  Microsoft did some early work on this with their XML Diffgram but no one else followed suit.

Security

Authentication (in order of increased security)

Basic Auth

Basic Auth + SSL

Digest

WSSE Authentication (ATOM uses this)

Message Security:

Message Level Encrypt (WS-SEC)

For the Microsoft coders I highly recommend

RESTful .Net (WCF For REST (Framework 3.5) Jon Flanders

There are significant advantages to building your RESTful services using .Net.  Here is a comparison table to get you oriented:

DLR’s Cross Reference:
Web Service Standard REST Service WCF For REST (Framework 3.5)
1 TCP/IP + others TCP/IP TCP/IP
2 SOAP Wrapper HTTP HTTP
3 SOAP Headers HTTP Headers HTTP Headers
4 WS*Security Basic Auth/SSL Basic Auth/SSL or WS*Security
5 Early Binding Late Binding Late Binding
6 XSD WADL XSD, WADL
7 XML Media Negotiation Media Negotiation
8 SOAP FAULTS HTTP Response Codes HTTP Response Codes
9 Single Endpoint Multiple Endpoints, URI Templates Multiple Endpoints, URI Templates
10 Client Proxy Custom auto-generated Javascript proxy

6439623457_d2599a85eb_m

The REST of the Week

Wednesday is more or less vendor day at QCON and the sessions are a step down from the tutorials but the session quality6373577519_b3a8be078c picked up again on Thursday and Friday.  XXX XXXX who gave an excellent tutorial last year gave an informative talk on ‘good code’.  The Mobile Development and HTML5 tracks were well attended and quite informative.  The fie   ld is wild open with many supporting systems being free to the developer (support will cost you extra) and the choices are broad: from browser ‘responsive design’ application to native appearing applications to native apps ( and someone threw in “hybrid app” into the mix).  The Mobile panel of IBM DOJO, JQuery.Mobil and Sencha was hot.  I am new (to say the least) to Mobile development but here are my (somewhat) random notes on these sessions:

MOBILE Development is HTML5 Development

HTML5 is the stack. Phone and Tablet applications use WebKit based rendering engines and HTML5 conformant browsers only (Windows Phone 7 is the exception here). HTML5 has its own new security concerns ( New Security Concerns)

Three major application development approaches are:

· Browser Applications;

· Native like Applications;

· Hybrid Applications; and

· Native Applications.

Browser applications may emulate the screens seen on the parallel desk top browser versions on the front end but in practice the major players (Facebook, YouTube, Gmail) make substantial modifications to at least the non-visual parts of the Mobile experience making extensive use of local storage and the HTML5 manifest standard for performance and to allow for a reasonable off line experience. Browser applications fall under the guidelines of Responsive Design (aka adaptive Design) and tend to be used when content will appear similarly between desktop and Mobile devices.

“Native like” applications use:

· The Browser in full screen Mode with no browser ‘chrome’; and

· Widgets are created using CSS, JS and HTML5 which simulate the ‘look and feel’ of a native application;

· No Access to Native Functionality (GPS, Camera, etc)6380026599_db3ba709db

· Tend to use, but does not require use of HTML5 manifest or local storage but it is strongly encouraged. 6439624411_22b452613f

A Native application is still an HTML5 application with the following characteristics:

· All JS Libraries, CSS and HTML are packaged and pre-loaded using a vendor specific MSI/Setup package;

· AJAX type calls for data are allowed;

· Access to Native Widgets and/or Widgets are created using CSS, JS and HTML5

· Access to Native Functionality (GPS, Camera, etc)

· Standard HTTP GET or POST are NOT allowed

A Hybrid Application is a “Native Like” Application” placed within a wrapper which allows access to device hardware and software (like the camera) via a special JavaScript interface and, with additional special coding, can be packaged within a MSI/Setup and distributed as a pure Native application.

AJAX calls are made via XHR2 (aka XMLHttpRequest Level 2) which among other things relaxes the single domain requirement of XHR and processing Blob and File interfaces.

The following major vendors offer free libraries and IDE for development:

Native Apps: PhoneGap, Appcelerator

Native App Like: Sencha, PhoneGap, IBM Dojo

Browser App: JQuery.Mobile

PhoneGap does NOT require replacement of Sencha, JQuery.Mobil, Dojo.Mobile JQuery libraries.

PhoneGap allows JavaScript to call PhoneGap JavaScript libraries which abstract access to device hardware (camera, GPS, etc).

Sencha does not require replacement of the JQuery.Mobil, Dojo.Mobile JQuery libraries.

Although it is theoretically possible to create “Native like” applications with only JQuery.Mobile this is NOT encouraged.6439625143_caa6996f39 6337926187_91ca36793d

Local Storage

This is a major area of performance efforts and is still very much open in terms of how best to approach the problem:

The major elements are:

App Cache (for pre-fetch. and Native App Approach)

DOM Storage (aka Web Storage)

IndexedDB (vs. Web SQL)

File API (this is really part of XHR2)

Storing Large Amounts of Data Locally

If you are looking to store many Megabytes – or more, beware that there are limits in place, which are handled in different ways depending on the browser and the particular API we’re talking about. In most cases, there is a magic number of 5MB. For Application Cache and the various offline stores, there will be no problem if your domain stores under 5MB. When you go above that, various things can happen: (a) it won’t work; (b) the browser will request the user for more space; (c) the browser will check for special configuration (as with the “unlimited_storage” permission in the Chrome extension manifest).

IndexedDB:

clip_image002

Web SQL Database is a web page API for storing data in databases that can be queried using a variant of SQL.

Storage Non-Support as of two weeks ago.

IE Chrome Safari Firefox iOS BBX[RIM] Android
IndexedDB Supported Supported No Support Supported No Support No Support No Support
WEB SQL No Support Supported Supported No Support Supported Supported Supported

6439626605_ee7b664332

Doing HTML5 on non-HTML5 Browsers: If you are doing responsive design and need to work with Desktop and6380016957_4c6b5e7345_z Mobil using the same code base: JQuery.Mobile, DOJO and , Modernizr(strong Microsoft support for this JavaScript library).

WEB API

What is it? Just a name for breaking out the AJAX servers from the web server. This is an expansion of REST into just serving data for XHR. It is a helpful way to specialize our design discussions by separating serving pages (with MVC or whatever) from serving data calls from the web page. Except for security the two can be architecturally separated.

Web APIs Technology Stack

clip_image002[6]

Look familiarr? Looks like our old web server stack to me.

NOSQL

The CAP Theorem  (and Here)

  • Consistency: (all nodes have the same data at the same time)
  • Availability: (every request receives a response – no timeouts, offline)
  • Partition tolerance: (the system continues to operate despite arbitrary message loss)

Pick Any Two6439627917_7f88626477_z

If some of the data you are serving can tolerate Eventual Consistency then NOSQL is much faster.6380029445_0e0ecf7d53

If you need two phase commit, either use a SQL database OR redefine your resource to eliminate the need for the 2Phase Commit.

NoSQL databases come in two basic flavors:

Key/Value: This are popular with content management and where response time must be minimal. In general you define what btrees you want to use before the fact. There are no on the fly Joins or projects. MongoDB and CouchDB are typical leaders in this area.

Column Map: This is what Google calls Big Table. This is better for delivering groups of records based on criteria which may be defined ‘on the fly’. Cassandra is the leader in this group.

Web Sockets:

6439628517_6c7955df1f_zSad to say this is still not standardized and preliminary support libraries are still a little rough.  Things do not seem to have moved along much since the Microsoft sessions I attended at MIX 11.

Photos: All Photos by Cloud2013

REST, Ruby On Rails, CouchDB and Me – Part 4 – CURL on Windows And Ruby POST   Leave a comment

Part 0 – REST, Ruby On Rails, CouchDB and Me

Part 1 – Ruby, The Command Line Version

Part 2 – Aptana IDE For Ruby

Part 3 CouchDB Up and Running on Windows

Part 4 – CouchDB, Curl and RUBY

Part 5 – Getting The Data Ready for CouchDB

Part 6 – Getting The Data Into And Out Of CouchDB

Part 7 – JQUERY,JPlayer and HTML5

In The Post:

  • CURL and Couchdb
  • Documents Design and Otherwise
  • Posting Documents to couchDB Using Ruby

If you are like me you have spent some time with the free ebook: CouchDB The Definitive Guide.  If you are a windows user you may have run into some problems with the examples given in the chapter  on “Design Documents”.  Specifically they don’t work ‘out of the box’.  The examples in that chapter show us how to: create a database, to create and post a design document and to post a document to the database.  These examples use  CURL in a command shell.

OmniVortex

Since we are running Windows first we need to install CURL on our system.  Either set your system path to include the CURL executable. We can get a windows version here.  Use the version labeled DOS, Win32- MSVC or Win64 depending on your system. We assume here that couchDB has been installed successfully on your system. Now open a ‘command prompt’ on your system.  If you must have a UNIX type shell you need to install CYWIN or some other UNIX emulator for Windows.  If you are using the Aptana IDE like me you need to create an “external command” to open a command shell within Aptana.  This figure illustrates the setup within the Aptana IDE to do this:

image

In the command shell you can create a couchdb Database using a POST command and CURL.  Couchdb is RESTful so we use a PUT command for all actions which CREATE a resource, of which a database is one example.  The format of the command is:

curl -X PUT http://{couchdb}/{yourdatabasename}I want to create a database named deadbase so on my system this command and response looks like:

C:\Documents and Settings\dredfield\My Documents\Aptana Studio Workspace\couchDB01

>curl -X PUT http://127.0.0.1:5984/deadbase

{“ok”:true}

The where “{“ok”:true}” is the response body of the http response to my put command.  Confirm your work by starting a browser and navigating to Futon user interface to your couchdb installation.  On my system this url is:

http://127.0.0.1:5984/_utils/index.html

you should see something like this:

image

CURL and Documents

OK, now lets make a design document for this database and PUT that document to the new database.  With slight modifications to the example given in CouchDB The Definitive Guide my first cut at a design document looks like this:

{

     “_id” : “_design/example”,

     “views” : {

        “View00” : {

       “map” : “function(doc){emit(doc._id, doc.date)}”

        }

  }

}

This is a JSON formatted document.  Initial syntax checking is up to you.  Basically couchDB will accept anything within the outer brackets whether or not it is formatted as usable JSON or not.  We have several options for checking syntax.  There are free online syntax checkers like JSONLint.  The interface to JSONLint looks like:

clip_image001

An installable open source JSON checker and visualizing tool, JSON View is available here.  JSON View’s output looks like:

clip_image001[10]

Now that we know our syntax is correct (if not the logic of the design document – more on this in the next installment) we can PUT this document to our database.  We can have more than one design document in a given database.  The name (id) of this document is “_design/example”.  where “_design” tells couchdb this is indeed a design document and its name is “example”.   My document is named mydesign.json on my file system.  The CURL command to PUT this into the database looks like:

curl -X PUT http://127.0.0.1:5984/deadbase/_design/example -d @mydesign.json

couchdb will respond:

{“ok”:true,”id”:”_design/example”,”rev”:”1-45f081a3f681b28ce7a0bdc5db216e74″}

Note here that this is NOT the syntax shown in CouchDB The Definitive Guide.  The syntax there will not work in a windows shell (i.e. command prompt).  Even when you have syntax correct JSON document  and the correct format of the PUT statement on Windows you may recieve an error message from CURL complaining about UTF8 errors within the document and have a PUT failure.  The problem here is that the Windows file system supports several encoding schemes and various windows programs save documents in to different default encoding.  If you are using Notepad.exe to create your files be sure to save the files in ANSIformat.

 
Check your work using the FUTON interface locate the “_design/example document” in deadbase

clip_image001[12]

Double click on the document:

clip_image001[16]

Note that “views” is a “Field” within the document.  Select the “Source” tab  and take a look inside the document:

clip_image002[4]

Now lets POST a document into the database.  Since we have not defined any validation fields we can push anything into the database.  Even documents which consist of just “{}”.  CouchDB defines only one innate restriction:

If a document defines the id field (“_id”) then the value of _id must not conflict with an existing value of the ID field of ANY other document in the database.

If the document does not define an ID field, couchDB will generate an ID (as a UUID) and apply it to the document.  You can supply your own ID values.  If you can either generate your own value  (Ruby can generate a GUID for you) or you can request a GUID from couchdb with a GET command.  See this page for more information.  In the sample program I will be developing for this series I will be using a ‘natural key’ – that is a key whose value has an actual meaning (a Social Security is such a natural key for example, but please never use this).  If you try to POST a document and use a duplicate key you will get back a 409 status code for the error.

The document I will be using in the next post looks like this:

{

“_id” : “1972-07-22”,

“IAKey” : “gd1972-07-22.sbd.miller.94112.sbeok.flac16”,

“description” : “Set 1 Bertha Me And My Uncle You Win Again Jack Straw Bird Song Beat It On Down The Line Sugaree Black Throated …

“pubdate”: “2008-08-15”,

“sb”: true,

“cm”: true,

“mx”: false,

“venue”: “Paramount Northwest Theatre”,

}

If I save this document as ConcertRecord.json I can use CURL to POST this document as:

curl -H “Content-Type: application/json” -X POST http://127.0.0.1:5984/deadbase/ -d @ConcertRecord.json

and couchdb will reply with an HTTP status 200 and a response body of:

{“ok”:true,”id”:”1972-07-22″,”rev”:”1-01a182f329c40ba3bab4b13695d0a098″}

In couchDB Futon this document looks like:

clip_image001[20]

Note that the order of the fields is set by couchDB not the order in the first loaded document.

Ruby At Last

OK, enough of the command shell let’s do some couchDB work using RUBY.  I am going to access couchDB from a fairly low level within Ruby in these posts.  There are several ActiveRecord type GEMS which will interface with couchDB but my focus here will be on: (1)  speed of access and (2) transferability of knowledge between Ruby access and direct Javascript/Browser access to couchDB.

Here’s a minimum of what we need to POST a document to a couchdb using RUBY.

The GEMS for

JSON : This will always load the Ruby based version of the JSON module.  If you want to have ‘pure’ JSON (i.e. a C based module you will need to have the Ruby/Windows DEVKit installed on your system.  For our purposes the ‘pure’ version is not necessary.

REST-OPEN-URI:  This extends open-uri by using the net/http  and the uri GEMs to cover all of the REST verbs (GET, POST, PUT and DELETE).  This is a very light install and is only lightly documented.

Here is the basic plan:

Assume we have a RUBY object (call it “rec”) which includes, among other things the fields we want to POST into the deadbase as a deadbase document like the one developed above.  We first need to convert the fields into a JSON string and then to POST the JSON string into the deadbase.  The JSON GEM is used to achive the first goal and REST-Open-URI is used to accomplish the second.

JSON Strings:

The JSON GEM will only serialize Ruby base types (strings, numbers and bools and HASH objects).  The JSON GEM is quite limited in that it will not serialize a Ruby object derived from the base RUBY object  into a JSON string, even if that object consists only of base types and Hash objects.  Although you may extend JSON we did not choose to do so. Rather we will create a simple Hash object and populate it manually via Ruby code with the fields we want to use for a document. Simply this could look like:

def makeJSON(rec)

thing=Hash.new()  #we know that JSON can serialize this type of object

thing[“_id”]=rec.date

thing[“IAKey”]=rec.uri

thing[“description”]=rec.description

thing[“venue”]=rec.title

thing[“pubdate”]=rec.pubdate

thing[“cm”]=rec.cm

thing[“sb”]=rec.sb

thing[“mx”]=rec.mx

return JSON.generate(thing)  #this returns a JSON String

end

REST-OPEN_URI:

Our POST routine will use the output form makeJSON and POST the JSON string to the deadbase.  In simple for this routine looks like:

def PostRecording(jsonString)

uri=”http://127.0.0.1:5984/deadbase/”   #this is our database

begin

responseBody=open(uri,:method=> :post, :body => jsonString,”Content-Type” => “application/json”).read

puts ‘POST Response Success: ‘ + responseBody

end

rescue

OpenURI::HTTPError => the_error

puts ‘Post Response Error: ‘ + the_error.io.status[0]

end

end

The key line is, of course:

responseBody=open(uri,:method=> :post, :body => jsonString,”Content-Type” => “application/json”).read

If we ran this line as:

responseBody=open(uri,:method=> :post, :body => jsonString).read

we would get an http Status code for an “Invalid Media Type”.  That’s because the default “Content-Type” for POST commands is “application/xxx-form” which is the typical format of a HTML “form” involved in a POST from a web browser.  We are far from a browser here and our “Content-Type” needs to be “application/json”.  The way to add Headers to the POST is to provide one or more key/value pairs with the desired header information.  Hence:

“Content-Type” => “application/json”

and the correct Ruby line is:

responseBody=open(uri,:method=> :post, :body => jsonString,”Content-Type” => “application/json”).read

We need to wrap the POST command in an exception block where the line:

OpenURI::HTTPError => the_error

is only executed IF the Http response status is > 399.  You can then do more fine grained responses to the error condition.  Specifically, if the_error.io.status[0]==409 you have attempted to POST the same document twice (at least two documents with the same ID).

That looks like a wrap for now.

5901067736_08fe849334_z

Posted 2011/07/22 by Cloud2013 in Aptana, couchdb, REST, Ruby

Tagged with , , , ,

%d bloggers like this: