Archive for the ‘Open source’ Tag

Hey Flickr, Where Did My Statistics Go? The CouchBase Connection. Part III   1 comment

This is the third post in this series on how to harvest statistical data from your (or a friend’s) Flickr Picture View
data.  The series begins
here.  Today we are looking at CouchBase as a noSQL database to store our Flickr data.  This post will get as far as getting the shell of a console application up and will defer example code samples for the next blog post.


CouchBase  iscouchbase a commercialized version of the public domain project
Apache CouchDB.  CouchDB is open source and CouchBase is not.  Both support API libraries for .Net and Java.  Commercial development with CouchBase is NOT free.  The CouchDB wiki lists five active C# libraries for CouchDB.  CouchBase supports a many API libraries including .Net and Java.  I have written about CouchDB and Ruby in a prior series of posts which can be found here. Both systems support multi-server nodes and concurrency controls.  Neither of these features will be touched on in the post.  Our focus here will be on an introduction to the minimum necessary administration skills and API coding to help us with our goal of storing information about Users, Photos and View Counts through time.  Along the way we will also discuss JSON Serialization / Deserialization using Newtonsoft.JSON, open source application Logging with NLog.  I will defer the discussion of CouchBase views for a subsequent post.

Data Model Overview.

Ultimately we want to store information about each User. For each user we will store information for one or more Photo and for each Photo, information on one or more View Counts.  Users and Photos have their own Primary Key, supplied as an ID field from Flickr.  Our view counts will be collected each day and the Primary Key of the Counts is represented by the date the view count data was collected.  This could be modeled into a traditional RDBMS in third normal form, but this pattern is also most naturally represented as a nesting of lists of objects within container objects.  Rather than say we are storing nested objects it is more typical today to say that this data can be thought of as a structured Document.  The most natural way to store and access this data is by simple (or compounds of ) primary keys.  When we get to the point where we are doing manipulation of the data for statistical analysis and summary our most natural mode of access will be by a key composed of the User ID and Photo ID and iterating there view counts by Date ID (or Date ID range).  A very simple way to model this is with a Key / Value noSQL database based on document storage (aka a Document Store).  We could call this design an object oriented database model but that would be old fashion.  Here is the visual of the data model:image

The full Document could be represented as a compound C# object:

   class CObject
{
public CUser User { get; set; }
public List<CPhoto> Photo { get; set; }
}

public class CUser
{
public string FullName { get; set; }
public string Username { get; set; }
public string UserId { get; set; }
public string APIKey { get; set; }
public string SharedSecret { get; set; }
public string Token { get; set; }
public string TokenSecret { get; set; }

}

public class CPhoto
{
public string ID { get; set; }
public string Title { get; set; }
public string ThumbnailURL { get; set; }
public List<CView> Views { get; set; }
}

public class CView
{
public string Date { get; set; }
public int Views { get; set; }
}

In this post we will setup a single server CouchBase instance and develop a single user application to manipulate documents in a CouchBase “bucket”.  We will not model the complete object in this post but deal with a simplified version of Photo Object while we get our feet wet on CouchBase CRUD operations and simple CouchBase server Administration.  To make things as simple as possible, for this post, we will be working only with a modified version of the CPhoto object (Document).

cropped-2001spaceodyssey025

Getting The Stuff You Need.

Shopping List

Setting up a single node Windows CouchBase Server simple and basic administration is easy and fun. Download and run the installation of CouchBase from here. Fred willing all will go well and you will be placed at local page in your default browser. Bookmark this page and note the Port Number that has been assigned to the default instance of CouchBase.  On first use you will need to create an administrator Username and Password. I left the defaults alone for the Cluster and Server Nodes. Select the Tab Data Buckets.  You will need to decrease the Quota Usage limits for the default Bucket.  With the space you freed up, create a new Bucket called “DLR.Flickr.Example1” .  Here is what mine looks like:

 image

And Here is the Bucket Settings Page:

image

OK.  Now take some time and review the documentation for the .Net SDK here.  You can read through or code along with the examples given there. Done? Now let’s get to work.

Starting to Code CouchBase

Open Visual Studio and select Tools/NuGet Package Manager/Package Manager Console and enter the command:

Install-Package CouchbaseNetClient

Create a new Visual Studio Console application.  I called mine:

DLR.CCDB.ConsoleApp and set the default namespace to DLR.CCDB.  Add references to:

Couchbase

Enyim.Memcached

Newtonsoft.Json

[ If you can not resolve Newtonsoft.Json:  Right click on the root of the project and select: Manage NuGet Packages.  Search on Newtonsoft.Json.  Select Install on JSON.Net.  Now try adding the Newtonsoft reference again.]

Now is a good time to add the open source Logging solution to your project.  Select: Manage NuGet Packages.  Search on NLOG. Install both  NLog and NLog Configuration.

Open your App.Config project file.  You will need to make several changes.  Here is what mine looks like after the changes.

Red items are added manually by me (you) and the Blue entries are added by the NuGet Package manager during the sets you followed above.

<!–?xml version=”1.0″ encoding=”utf-8″?>
<configuration>
<configSections>
Couchbase.Configuration.CouchbaseClientSection, Couchbase” />
</configSections>
<couchbase>
<servers bucket=”DLR.Flickr.Example1″ bucketPassword=””>
uri=”
http://127.0.0.1:8091/pools” />
</servers>
</couchbase>
    <startup>
<supportedRuntime version=”v4.0″ sku=”.NETFramework,Version=v4.5″ />
</startup>
<runtime>
<assemblyBinding xmlns=”urn:schemas-microsoft-com:asm.v1″>
<dependentAssembly>
<assemblyIdentity name=”Newtonsoft.Json” publicKeyToken=”30ad4fe6b2a6aeed” culture=”neutral” />
<bindingRedirect oldVersion=”0.0.0.0-6.0.0.0″ newVersion=”6.0.0.0″ />
</dependentAssembly>
<dependentAssembly>
<assemblyIdentity name=”Enyim.Caching” publicKeyToken=”05e9c6b5a9ec94c2″ culture=”neutral” />
<bindingRedirect oldVersion=”0.0.0.0-1.3.7.0″ newVersion=”1.3.7.0″ />
</dependentAssembly>
<dependentAssembly>
<assemblyIdentity name=”NLog” publicKeyToken=”5120e14c03d0593c” culture=”neutral” />
<bindingRedirect oldVersion=”0.0.0.0-3.1.0.0″ newVersion=”3.1.0.0″ />
</dependentAssembly>
</assemblyBinding>
</runtime>
</configuration>

We are most interested in this section:

<servers bucket=”DLR.Flickr.Example1″ bucketPassword=””>
uri=”
http://127.0.0.1:8091/pools” />
</servers>

 

bucket=”DLR.Flickr.Example1″

This sets your default API calls to the bucket “DLR.Flickr.Example1” which you created above.  Although we will not develop the theme here you can override the default bucket during runtime to deal with calls to multiple buckets in the same program.

uri=”http://127.0.0.1:8091/pools

This sets your local node. the http://127.0.0.1 is a constant for development projects (localhost) and the 8091 is the port assigned to CouchBase during installation (double check this value on your system by navigating to the CouchBase Console page you added to your favorites list above.

While we are here let’s make some changes (without explanation why at this point) in NLog.Config (which was created when you installed NLog above).  Replace the entire contents of the file with (mind the wrap):

<!–?xml version=”1.0″ encoding=”utf-8″ ?>
<nlog xmlns=”
http://www.nlog-project.org/schemas/NLog.xsd”
      xmlns:xsi=”http://www.w3.org/2001/XMLSchema-instance”
      throwExceptions=”true”
internalLogFile=”C:/temp/NLog/WEBAPI/Internal.txt”
internalLogLevel=”Info”
>
<targets>
<target name=”debugfile” xsi:type=”File” layout=”${message}”  fileName=”C:/temp/DLR.Flickr/Debug.txt” archiveNumbering=”Rolling”  archiveEvery=”Day” maxArchiveFiles=”7″ ConcurrentWrites=”true”                        />
<target name=”logfile” xsi:type=”File” layout=”${message}”    fileName=”C:/temp/DLR.Flickr/Info.txt”  archiveNumbering=”Rolling”  archiveEvery=”Day” maxArchiveFiles=”7″ ConcurrentWrites=”true”                      />
<target name=”Errorsfile” xsi:type=”File” layout=”${message}” fileName=”C:/temp/DLR.Flickr/Error.txt” archiveNumbering=”Rolling”  archiveEvery=”Day” maxArchiveFiles=”7″ ConcurrentWrites=”true”                   />
<target name=”Fatalfile” xsi:type=”File” layout=”${message}”  fileName=”C:/temp/DLR.Flickr/Fatal.txt” archiveNumbering=”Rolling”  archiveEvery=”Day” maxArchiveFiles=”7″ ConcurrentWrites=”true”                    />
</targets>
<rules>
<logger name=”*” minlevel=”Debug” maxlevel=”Debug” writeTo=”debugfile” />
<logger name=”*” minlevel=”Info” maxlevel= “Fatal” writeTo=”logfile” />
<logger name=”*” minlevel=”Error” maxlevel=”Fatal” writeTo=”Errorsfile” />
<logger name=”*” minlevel=”Fatal” maxlevel=”Fatal” writeTo=”Fatalfile” />
</rules>
</nlog>

We will get back to the details of this configuration file in the next post.

Write the minimum test code possible. 

Replace the contents of Program.cs with

using System;

using Couchbase;

using Enyim.Caching.Memcached;

using Newtonsoft.Json;

using NLog;

namespace DLR.CCDB.ConsoleApp
{

class Program
{
static void Main(string[] args)
{
CouchbaseClient client=new CouchbaseClient();

        }

    }

}

Build and run.  You should have no errors and client should not equal null after the call

CouchbaseClient client=new CouchbaseClient();

Success?  Take a break and we will continue next week.

 

REST, Ruby On Rails, CouchDB and Me – Part 0   4 comments

Part 0 – This Post

Part 1 – Ruby, The Command Line Version

Part 2 – Aptana IDE For Ruby

Part 3 – CouchDB Up and Running on Windows

Part 4 – CouchDB, Curl and RUBY

Part 5 – Getting The Data Ready for CouchDB

Part 7 – JQUERY,JPlayer and HTML5

The is the first in asmall series of blog posts concerning what is often known as as Open Source Stack for Web Development.  My interests here are in WEB 2.0 development.  We will be working towards using JQUERY and CouchDB to develop REST(ful) Web sites.  I will start with a focus on Rudy and Ruby on Rails (aka RadRails).   However, I will be using free and open source products not on UNIX but on Windows based systems.  Windows runs on fairly cheap hardware (compared to Apples UNIX based platforms) and Windows is much more accessible to new developers than Linux systems.  I will be working here with Windows XP and Windows Vista but you can play alone on Linux and MAC x86.  To get started let’s get together some software and supporting documentation and begin.

Ruby

Version 1.8.7 is the current stable release but since we are interested in using CouchDB I have installed the required ‘edge’ version 1.9.2.  Either of these version will support Ruby on Rails development. Ruby installations are available from several sites. Until you know exactly what you are doing stick with distributions from RubyInstaller.org since these are the baseline Ruby builds which everyone tests against first.  As you get deeper into it (or if  things get a little more sticky)  you try an integrated open stack like the one offered by BitNami.  At some point you may need to install the DevKit.  This is not as heavy as it sounds (but you will need to work with your system paths to get everything in place).  You can get the DevKit here and installation instructions are here.   Note:  the devkit download is now a Windows MSI file but the instructions are still based on a ZIP file extraction, but it all works trust me.  Don’t install either the BitNami Stack or the DevKit until you know you need it.   If you completely new to Ruby a nice introduction can be found in Why’s Poignant Guide To Ruby.

Ruby On Rails

There are a lot of different options to developing Ruby on Rails applications.  If I was not an IDE freak I could use  a simple text editor (I often use the non-open Primal Script) but being new to Ruby on Rails I wanted the support of and IDE.  For this option I selected the Aptana RadRails Development Environment.  This free development environment is based on the open Eclipse editor.  I downloaded the base Aptana IDE (Studio 2) and then added RADRails as a plug in.  These are available in Linux and Mac x86 installers in addition to windows.  You could install only RADRails but then you would have a crippled version of the Aptana product.  We will be noting Ruby on Rails training materials as we move along.

CouchDB

Although we will be using mySQL or SQLlite for some of our development, our real content manager will be the NOSQL couchDB.  This is our real goal in the project – testing the viability  of a REST(ful) HTTP addressable NOSQL database.  This project is from the Apache organization and is available for Linux, MAC x86 and Windows.  It runs as a service on the Windows OS.  The Windows installer is available here.  There is an excellent open source on line couchDB book available here.  For Ruby on Rails we will be using – CouchRest for our Ruby on Rails work with couchdb.  CouchRest is available as a ruby GEM and can be installed on Linux, MAC x86 and Windows versions of Ruby.

JQUERY

Web 2.0 is not possible with out modern, sophisticated JavaScript libraries.  One of the best of these is JQUERY and not surprisingly couchDB ships with a powerful jquery library (couchdb.js) to facilitate browser side manipulation of couchdb data.  For browser work you should be pulling your JQuery from CDN Host.  For use within Ruby for Rails project you will need to add JQUERY  support to your Aptana IDE.  JQuery works with all modern browsers (and even some which not).PacalII

%d bloggers like this: