Archive for the ‘NLog’ Tag

Hey Flickr, Where Did My Statistics Go? The CouchBase Connection. Part IV   Leave a comment

We interrupt this series to take a side trip concerning application logging.  The series begins here. NLog is an excellent open source logging project available from NuGet and other sources.   The sample code for this blog post can be found HERE. Although this is a kitchen sink implementation (Log to files, event logs, database, SMTP whatever) I will be using it as a simple way to log text information to files.  Once you have created a Visual Studio Project open Tools / NuGet Package  Manager/Package Manager Console.  From Here you can add NLog to your object with the command:

PM> Install-Package NLog

This will install NLog, modify your project and add a project reference for NLog.  Although NLog targets and rules can be managed programmatically, I 
normally user the configuration file: 

NLog.Config

You can set this up using the Package Manager Console with the command:

PM> Install-Package NLog.Config

Configuration File Setup

The NLog config file is then modified to define “targets” and “rules”.  The former defines where log entries are written and the latter define which log 
levels are written to which targets.  A file based target section might look like:

<targets>

   DLR.Flickr/Debug.txt” archiveNumbering=”Rolling”   archiveEvery=”Day” maxArchiveFiles=”7″ ConcurrentWrites=”true”/>

<target name=”logfile” xsi:type=”File” layout=”${message}”    fileName=”C:/temp/DLR.Flickr/Info.txt”  archiveNumbering=”Rolling”  archiveEvery=”Day” maxArchiveFiles=”7″ ConcurrentWrites=”true”/>

<target name=”Errorsfile” xsi:type=”File” layout=”${message}” fileName=”C:/temp/DLR.Flickr/Error.txt” archiveNumbering=”Rolling”  archiveEvery=”Day” maxArchiveFiles=”7″ ConcurrentWrites=”true”/>

<target name=”Fatalfile” xsi:type=”File” layout=”${message}”  fileName=”C:/temp/DLR.Flickr/Fatal.txt” archiveNumbering=”Rolling”  archiveEvery=”Day” maxArchiveFiles=”7″ ConcurrentWrites=”true”/>

</targets>

where name is the symbolic name of the target xsi:type defines this as a file target.  If you are controlling the layout of the log entry set layout to “${message}”.  Given that we are using xsi:type as File we can use fileName to set the physical location of the log file.  The value of fileName can be changed programmatically at runtime but I will not give examples here.

NLog defines five Log levels:  Debug, Info, Warn, Error and Fatal.  These levels are defined in an enum and have the names have no special significance except as you define them.  The Rules section of the config file defines which Log Levels are written to which targets. A given level can be written to zero to many targets.  My Rules section typically looks like:

<rules>

<logger name=”*” minlevel=”Debug” maxlevel=”Debug” writeTo=”debugfile” />

<logger name=”*” minlevel=”Info” maxlevel= “Info” writeTo=”logfile” />

<logger name=”*” minlevel=”Warn” maxlevel=”Warn” writeTo=”Warnfile” />

<logger name=”*” minlevel=”Error” maxlevel=”Error” writeTo=”Errorfile” />

<logger name=”*” minlevel=”Fatal” maxlevel=”Fatal” writeTo=”Fatalfile” />
  </rules>

More complex rules like the following are possible:

    <logger name=”*” minlevel=”Error” maxlevel=”Error” writeTo=”Errorfile” />

       <logger name=”*” minlevel=”Error” maxlevel=”Fatal” writeTo=”Fatalfile” />

NLog initialization at runtime is very simple.  Typically you can you a single line like:

using NLog;

static Logger _LogEngine = LogManager.GetLogger(“Log Name”);

this need only be called once.

The simplest NLog log call (given the definition layout=”${message}”  ) would look like:

_LogEngine.Log(NLog.LogLevel.Info, “Info Message”);

We can extend this quite simply.  I have a single class extension providing a simple extension of NLog on Git Hub.  You can find it here.  Specifically I have provided wrapper methods for each NLog.LogLevel and support for Exception Stack Dumps.  Include this file in your project (after installing NLog and NLog config) then you can write:

using DLR.Util;

namespace DLR.CCDB.ConsoleApp

{

    class Program

{

static void Main(string[] args)

{

string _CorrelationID=System.Guid.NewGuid().ToString();

CCDB cbase = new CCDB { CorrelationID = _CorrelationID };

cbase.Client = CouchbaseManager.Instance;

NLS.Info(_CorrelationID, “Helllo, CouchBase”);

try{

throw new ApplicationException(“My Exception”);

}catch(Exception x){

NLS.Error(_CorrelationID,”Error”,x.Message);

//OR

NLS.Error(_CorrelationID,”Error”,x);

}

_CorrelationID is supported here so in multiuser situations (like WebAPI) we can identify which messages where written by which task.  In a console app this is not strictly necessary.  The call to NLS.Info results in an output log line like:

DLR|20140909-152031037|2f8f89ce-51de-4269-9ae0-9313ad2a0243|Helllo, CouchBase|

where:

  • DLR is the Log Engine name (more than one engine can write to a given log file);
  • 20140909-152031037 is the terse timestamp of the form: YYYYMMDD-HHMMSSmmm; and
  • Hello, CouchBase is our text message

My call:

NLS.Error(_CorrelationID,”Error”,x);

would result in a log line like:

DLR|20140909-152544801|46e656cd-4e17-4285-a5f3-e1484dad2995|Error|Error Data. Message: [My Exception]Stack Trace:  DLR.CCDB.ConsoleApp.Program.MainString args|

where Error is my message;

Error Data. Message: [My Exception] is the Message in ApplicationException; and

Stack Trace:  DLR.CCDB.ConsoleApp.Program.MainString args| is the stack dump.

NLS will handle nested exceptions and stack dumps but we are only showing a single un-nested exception in this example.

OK! That’s it for this post.  We will, hopefully return to couchBase and the Flickr API in the next post.

Posted 2014/09/09 by Cloud2013 in GitHub, Microsoft, NLog, NuGet

Tagged with , , ,

Hey Flickr, Where Did My Statistics Go? The CouchBase Connection. Part III   1 comment

This is the third post in this series on how to harvest statistical data from your (or a friend’s) Flickr Picture View
data.  The series begins
here.  Today we are looking at CouchBase as a noSQL database to store our Flickr data.  This post will get as far as getting the shell of a console application up and will defer example code samples for the next blog post.


CouchBase  iscouchbase a commercialized version of the public domain project
Apache CouchDB.  CouchDB is open source and CouchBase is not.  Both support API libraries for .Net and Java.  Commercial development with CouchBase is NOT free.  The CouchDB wiki lists five active C# libraries for CouchDB.  CouchBase supports a many API libraries including .Net and Java.  I have written about CouchDB and Ruby in a prior series of posts which can be found here. Both systems support multi-server nodes and concurrency controls.  Neither of these features will be touched on in the post.  Our focus here will be on an introduction to the minimum necessary administration skills and API coding to help us with our goal of storing information about Users, Photos and View Counts through time.  Along the way we will also discuss JSON Serialization / Deserialization using Newtonsoft.JSON, open source application Logging with NLog.  I will defer the discussion of CouchBase views for a subsequent post.

Data Model Overview.

Ultimately we want to store information about each User. For each user we will store information for one or more Photo and for each Photo, information on one or more View Counts.  Users and Photos have their own Primary Key, supplied as an ID field from Flickr.  Our view counts will be collected each day and the Primary Key of the Counts is represented by the date the view count data was collected.  This could be modeled into a traditional RDBMS in third normal form, but this pattern is also most naturally represented as a nesting of lists of objects within container objects.  Rather than say we are storing nested objects it is more typical today to say that this data can be thought of as a structured Document.  The most natural way to store and access this data is by simple (or compounds of ) primary keys.  When we get to the point where we are doing manipulation of the data for statistical analysis and summary our most natural mode of access will be by a key composed of the User ID and Photo ID and iterating there view counts by Date ID (or Date ID range).  A very simple way to model this is with a Key / Value noSQL database based on document storage (aka a Document Store).  We could call this design an object oriented database model but that would be old fashion.  Here is the visual of the data model:image

The full Document could be represented as a compound C# object:

   class CObject
{
public CUser User { get; set; }
public List<CPhoto> Photo { get; set; }
}

public class CUser
{
public string FullName { get; set; }
public string Username { get; set; }
public string UserId { get; set; }
public string APIKey { get; set; }
public string SharedSecret { get; set; }
public string Token { get; set; }
public string TokenSecret { get; set; }

}

public class CPhoto
{
public string ID { get; set; }
public string Title { get; set; }
public string ThumbnailURL { get; set; }
public List<CView> Views { get; set; }
}

public class CView
{
public string Date { get; set; }
public int Views { get; set; }
}

In this post we will setup a single server CouchBase instance and develop a single user application to manipulate documents in a CouchBase “bucket”.  We will not model the complete object in this post but deal with a simplified version of Photo Object while we get our feet wet on CouchBase CRUD operations and simple CouchBase server Administration.  To make things as simple as possible, for this post, we will be working only with a modified version of the CPhoto object (Document).

cropped-2001spaceodyssey025

Getting The Stuff You Need.

Shopping List

Setting up a single node Windows CouchBase Server simple and basic administration is easy and fun. Download and run the installation of CouchBase from here. Fred willing all will go well and you will be placed at local page in your default browser. Bookmark this page and note the Port Number that has been assigned to the default instance of CouchBase.  On first use you will need to create an administrator Username and Password. I left the defaults alone for the Cluster and Server Nodes. Select the Tab Data Buckets.  You will need to decrease the Quota Usage limits for the default Bucket.  With the space you freed up, create a new Bucket called “DLR.Flickr.Example1” .  Here is what mine looks like:

 image

And Here is the Bucket Settings Page:

image

OK.  Now take some time and review the documentation for the .Net SDK here.  You can read through or code along with the examples given there. Done? Now let’s get to work.

Starting to Code CouchBase

Open Visual Studio and select Tools/NuGet Package Manager/Package Manager Console and enter the command:

Install-Package CouchbaseNetClient

Create a new Visual Studio Console application.  I called mine:

DLR.CCDB.ConsoleApp and set the default namespace to DLR.CCDB.  Add references to:

Couchbase

Enyim.Memcached

Newtonsoft.Json

[ If you can not resolve Newtonsoft.Json:  Right click on the root of the project and select: Manage NuGet Packages.  Search on Newtonsoft.Json.  Select Install on JSON.Net.  Now try adding the Newtonsoft reference again.]

Now is a good time to add the open source Logging solution to your project.  Select: Manage NuGet Packages.  Search on NLOG. Install both  NLog and NLog Configuration.

Open your App.Config project file.  You will need to make several changes.  Here is what mine looks like after the changes.

Red items are added manually by me (you) and the Blue entries are added by the NuGet Package manager during the sets you followed above.

<!–?xml version=”1.0″ encoding=”utf-8″?>
<configuration>
<configSections>
Couchbase.Configuration.CouchbaseClientSection, Couchbase” />
</configSections>
<couchbase>
<servers bucket=”DLR.Flickr.Example1″ bucketPassword=””>
uri=”
http://127.0.0.1:8091/pools” />
</servers>
</couchbase>
    <startup>
<supportedRuntime version=”v4.0″ sku=”.NETFramework,Version=v4.5″ />
</startup>
<runtime>
<assemblyBinding xmlns=”urn:schemas-microsoft-com:asm.v1″>
<dependentAssembly>
<assemblyIdentity name=”Newtonsoft.Json” publicKeyToken=”30ad4fe6b2a6aeed” culture=”neutral” />
<bindingRedirect oldVersion=”0.0.0.0-6.0.0.0″ newVersion=”6.0.0.0″ />
</dependentAssembly>
<dependentAssembly>
<assemblyIdentity name=”Enyim.Caching” publicKeyToken=”05e9c6b5a9ec94c2″ culture=”neutral” />
<bindingRedirect oldVersion=”0.0.0.0-1.3.7.0″ newVersion=”1.3.7.0″ />
</dependentAssembly>
<dependentAssembly>
<assemblyIdentity name=”NLog” publicKeyToken=”5120e14c03d0593c” culture=”neutral” />
<bindingRedirect oldVersion=”0.0.0.0-3.1.0.0″ newVersion=”3.1.0.0″ />
</dependentAssembly>
</assemblyBinding>
</runtime>
</configuration>

We are most interested in this section:

<servers bucket=”DLR.Flickr.Example1″ bucketPassword=””>
uri=”
http://127.0.0.1:8091/pools” />
</servers>

 

bucket=”DLR.Flickr.Example1″

This sets your default API calls to the bucket “DLR.Flickr.Example1” which you created above.  Although we will not develop the theme here you can override the default bucket during runtime to deal with calls to multiple buckets in the same program.

uri=”http://127.0.0.1:8091/pools

This sets your local node. the http://127.0.0.1 is a constant for development projects (localhost) and the 8091 is the port assigned to CouchBase during installation (double check this value on your system by navigating to the CouchBase Console page you added to your favorites list above.

While we are here let’s make some changes (without explanation why at this point) in NLog.Config (which was created when you installed NLog above).  Replace the entire contents of the file with (mind the wrap):

<!–?xml version=”1.0″ encoding=”utf-8″ ?>
<nlog xmlns=”
http://www.nlog-project.org/schemas/NLog.xsd”
      xmlns:xsi=”http://www.w3.org/2001/XMLSchema-instance”
      throwExceptions=”true”
internalLogFile=”C:/temp/NLog/WEBAPI/Internal.txt”
internalLogLevel=”Info”
>
<targets>
<target name=”debugfile” xsi:type=”File” layout=”${message}”  fileName=”C:/temp/DLR.Flickr/Debug.txt” archiveNumbering=”Rolling”  archiveEvery=”Day” maxArchiveFiles=”7″ ConcurrentWrites=”true”                        />
<target name=”logfile” xsi:type=”File” layout=”${message}”    fileName=”C:/temp/DLR.Flickr/Info.txt”  archiveNumbering=”Rolling”  archiveEvery=”Day” maxArchiveFiles=”7″ ConcurrentWrites=”true”                      />
<target name=”Errorsfile” xsi:type=”File” layout=”${message}” fileName=”C:/temp/DLR.Flickr/Error.txt” archiveNumbering=”Rolling”  archiveEvery=”Day” maxArchiveFiles=”7″ ConcurrentWrites=”true”                   />
<target name=”Fatalfile” xsi:type=”File” layout=”${message}”  fileName=”C:/temp/DLR.Flickr/Fatal.txt” archiveNumbering=”Rolling”  archiveEvery=”Day” maxArchiveFiles=”7″ ConcurrentWrites=”true”                    />
</targets>
<rules>
<logger name=”*” minlevel=”Debug” maxlevel=”Debug” writeTo=”debugfile” />
<logger name=”*” minlevel=”Info” maxlevel= “Fatal” writeTo=”logfile” />
<logger name=”*” minlevel=”Error” maxlevel=”Fatal” writeTo=”Errorsfile” />
<logger name=”*” minlevel=”Fatal” maxlevel=”Fatal” writeTo=”Fatalfile” />
</rules>
</nlog>

We will get back to the details of this configuration file in the next post.

Write the minimum test code possible. 

Replace the contents of Program.cs with

using System;

using Couchbase;

using Enyim.Caching.Memcached;

using Newtonsoft.Json;

using NLog;

namespace DLR.CCDB.ConsoleApp
{

class Program
{
static void Main(string[] args)
{
CouchbaseClient client=new CouchbaseClient();

        }

    }

}

Build and run.  You should have no errors and client should not equal null after the call

CouchbaseClient client=new CouchbaseClient();

Success?  Take a break and we will continue next week.

 

%d bloggers like this: