Archive for the ‘Photography’ Category

Hey Flickr, Where Did My Statistics Go? The CouchBase Connection. Part III   1 comment

This is the third post in this series on how to harvest statistical data from your (or a friend’s) Flickr Picture View
data.  The series begins
here.  Today we are looking at CouchBase as a noSQL database to store our Flickr data.  This post will get as far as getting the shell of a console application up and will defer example code samples for the next blog post.


CouchBase  iscouchbase a commercialized version of the public domain project
Apache CouchDB.  CouchDB is open source and CouchBase is not.  Both support API libraries for .Net and Java.  Commercial development with CouchBase is NOT free.  The CouchDB wiki lists five active C# libraries for CouchDB.  CouchBase supports a many API libraries including .Net and Java.  I have written about CouchDB and Ruby in a prior series of posts which can be found here. Both systems support multi-server nodes and concurrency controls.  Neither of these features will be touched on in the post.  Our focus here will be on an introduction to the minimum necessary administration skills and API coding to help us with our goal of storing information about Users, Photos and View Counts through time.  Along the way we will also discuss JSON Serialization / Deserialization using Newtonsoft.JSON, open source application Logging with NLog.  I will defer the discussion of CouchBase views for a subsequent post.

Data Model Overview.

Ultimately we want to store information about each User. For each user we will store information for one or more Photo and for each Photo, information on one or more View Counts.  Users and Photos have their own Primary Key, supplied as an ID field from Flickr.  Our view counts will be collected each day and the Primary Key of the Counts is represented by the date the view count data was collected.  This could be modeled into a traditional RDBMS in third normal form, but this pattern is also most naturally represented as a nesting of lists of objects within container objects.  Rather than say we are storing nested objects it is more typical today to say that this data can be thought of as a structured Document.  The most natural way to store and access this data is by simple (or compounds of ) primary keys.  When we get to the point where we are doing manipulation of the data for statistical analysis and summary our most natural mode of access will be by a key composed of the User ID and Photo ID and iterating there view counts by Date ID (or Date ID range).  A very simple way to model this is with a Key / Value noSQL database based on document storage (aka a Document Store).  We could call this design an object oriented database model but that would be old fashion.  Here is the visual of the data model:image

The full Document could be represented as a compound C# object:

   class CObject
{
public CUser User { get; set; }
public List<CPhoto> Photo { get; set; }
}

public class CUser
{
public string FullName { get; set; }
public string Username { get; set; }
public string UserId { get; set; }
public string APIKey { get; set; }
public string SharedSecret { get; set; }
public string Token { get; set; }
public string TokenSecret { get; set; }

}

public class CPhoto
{
public string ID { get; set; }
public string Title { get; set; }
public string ThumbnailURL { get; set; }
public List<CView> Views { get; set; }
}

public class CView
{
public string Date { get; set; }
public int Views { get; set; }
}

In this post we will setup a single server CouchBase instance and develop a single user application to manipulate documents in a CouchBase “bucket”.  We will not model the complete object in this post but deal with a simplified version of Photo Object while we get our feet wet on CouchBase CRUD operations and simple CouchBase server Administration.  To make things as simple as possible, for this post, we will be working only with a modified version of the CPhoto object (Document).

cropped-2001spaceodyssey025

Getting The Stuff You Need.

Shopping List

Setting up a single node Windows CouchBase Server simple and basic administration is easy and fun. Download and run the installation of CouchBase from here. Fred willing all will go well and you will be placed at local page in your default browser. Bookmark this page and note the Port Number that has been assigned to the default instance of CouchBase.  On first use you will need to create an administrator Username and Password. I left the defaults alone for the Cluster and Server Nodes. Select the Tab Data Buckets.  You will need to decrease the Quota Usage limits for the default Bucket.  With the space you freed up, create a new Bucket called “DLR.Flickr.Example1” .  Here is what mine looks like:

 image

And Here is the Bucket Settings Page:

image

OK.  Now take some time and review the documentation for the .Net SDK here.  You can read through or code along with the examples given there. Done? Now let’s get to work.

Starting to Code CouchBase

Open Visual Studio and select Tools/NuGet Package Manager/Package Manager Console and enter the command:

Install-Package CouchbaseNetClient

Create a new Visual Studio Console application.  I called mine:

DLR.CCDB.ConsoleApp and set the default namespace to DLR.CCDB.  Add references to:

Couchbase

Enyim.Memcached

Newtonsoft.Json

[ If you can not resolve Newtonsoft.Json:  Right click on the root of the project and select: Manage NuGet Packages.  Search on Newtonsoft.Json.  Select Install on JSON.Net.  Now try adding the Newtonsoft reference again.]

Now is a good time to add the open source Logging solution to your project.  Select: Manage NuGet Packages.  Search on NLOG. Install both  NLog and NLog Configuration.

Open your App.Config project file.  You will need to make several changes.  Here is what mine looks like after the changes.

Red items are added manually by me (you) and the Blue entries are added by the NuGet Package manager during the sets you followed above.

<!–?xml version=”1.0″ encoding=”utf-8″?>
<configuration>
<configSections>
Couchbase.Configuration.CouchbaseClientSection, Couchbase” />
</configSections>
<couchbase>
<servers bucket=”DLR.Flickr.Example1″ bucketPassword=””>
uri=”
http://127.0.0.1:8091/pools” />
</servers>
</couchbase>
    <startup>
<supportedRuntime version=”v4.0″ sku=”.NETFramework,Version=v4.5″ />
</startup>
<runtime>
<assemblyBinding xmlns=”urn:schemas-microsoft-com:asm.v1″>
<dependentAssembly>
<assemblyIdentity name=”Newtonsoft.Json” publicKeyToken=”30ad4fe6b2a6aeed” culture=”neutral” />
<bindingRedirect oldVersion=”0.0.0.0-6.0.0.0″ newVersion=”6.0.0.0″ />
</dependentAssembly>
<dependentAssembly>
<assemblyIdentity name=”Enyim.Caching” publicKeyToken=”05e9c6b5a9ec94c2″ culture=”neutral” />
<bindingRedirect oldVersion=”0.0.0.0-1.3.7.0″ newVersion=”1.3.7.0″ />
</dependentAssembly>
<dependentAssembly>
<assemblyIdentity name=”NLog” publicKeyToken=”5120e14c03d0593c” culture=”neutral” />
<bindingRedirect oldVersion=”0.0.0.0-3.1.0.0″ newVersion=”3.1.0.0″ />
</dependentAssembly>
</assemblyBinding>
</runtime>
</configuration>

We are most interested in this section:

<servers bucket=”DLR.Flickr.Example1″ bucketPassword=””>
uri=”
http://127.0.0.1:8091/pools” />
</servers>

 

bucket=”DLR.Flickr.Example1″

This sets your default API calls to the bucket “DLR.Flickr.Example1” which you created above.  Although we will not develop the theme here you can override the default bucket during runtime to deal with calls to multiple buckets in the same program.

uri=”http://127.0.0.1:8091/pools

This sets your local node. the http://127.0.0.1 is a constant for development projects (localhost) and the 8091 is the port assigned to CouchBase during installation (double check this value on your system by navigating to the CouchBase Console page you added to your favorites list above.

While we are here let’s make some changes (without explanation why at this point) in NLog.Config (which was created when you installed NLog above).  Replace the entire contents of the file with (mind the wrap):

<!–?xml version=”1.0″ encoding=”utf-8″ ?>
<nlog xmlns=”
http://www.nlog-project.org/schemas/NLog.xsd”
      xmlns:xsi=”http://www.w3.org/2001/XMLSchema-instance”
      throwExceptions=”true”
internalLogFile=”C:/temp/NLog/WEBAPI/Internal.txt”
internalLogLevel=”Info”
>
<targets>
<target name=”debugfile” xsi:type=”File” layout=”${message}”  fileName=”C:/temp/DLR.Flickr/Debug.txt” archiveNumbering=”Rolling”  archiveEvery=”Day” maxArchiveFiles=”7″ ConcurrentWrites=”true”                        />
<target name=”logfile” xsi:type=”File” layout=”${message}”    fileName=”C:/temp/DLR.Flickr/Info.txt”  archiveNumbering=”Rolling”  archiveEvery=”Day” maxArchiveFiles=”7″ ConcurrentWrites=”true”                      />
<target name=”Errorsfile” xsi:type=”File” layout=”${message}” fileName=”C:/temp/DLR.Flickr/Error.txt” archiveNumbering=”Rolling”  archiveEvery=”Day” maxArchiveFiles=”7″ ConcurrentWrites=”true”                   />
<target name=”Fatalfile” xsi:type=”File” layout=”${message}”  fileName=”C:/temp/DLR.Flickr/Fatal.txt” archiveNumbering=”Rolling”  archiveEvery=”Day” maxArchiveFiles=”7″ ConcurrentWrites=”true”                    />
</targets>
<rules>
<logger name=”*” minlevel=”Debug” maxlevel=”Debug” writeTo=”debugfile” />
<logger name=”*” minlevel=”Info” maxlevel= “Fatal” writeTo=”logfile” />
<logger name=”*” minlevel=”Error” maxlevel=”Fatal” writeTo=”Errorsfile” />
<logger name=”*” minlevel=”Fatal” maxlevel=”Fatal” writeTo=”Fatalfile” />
</rules>
</nlog>

We will get back to the details of this configuration file in the next post.

Write the minimum test code possible. 

Replace the contents of Program.cs with

using System;

using Couchbase;

using Enyim.Caching.Memcached;

using Newtonsoft.Json;

using NLog;

namespace DLR.CCDB.ConsoleApp
{

class Program
{
static void Main(string[] args)
{
CouchbaseClient client=new CouchbaseClient();

        }

    }

}

Build and run.  You should have no errors and client should not equal null after the call

CouchbaseClient client=new CouchbaseClient();

Success?  Take a break and we will continue next week.

 

Hey Flickr, Where Did My Statistics Go? OAuth, the Flickr API and You. (Part II)   1 comment

This is Part II of my series on the Flickr API and View Statistics.  You can find Part 1 here.

Assume a C# static class containing the critical information you need to identify yourself to Flickr:

public static class User{

public static string FullName {get;set;}

public static string Username {get;set;}

public static string UserId {get;set;}

public static string APIKey {get;set;}

public static string SharedSecret {get;set;}

public static string Token {get;set;}

public static string TokenSecret {get;set;}

}

Where APIKey and SharedSecret identify you (as the developer) to Flickr and Token and TokenSecret identify your relationship with the user which encodes what rights the user has granted to you.

14388202106_613a1b18f2_o

 

FlickrNet defines a C# class OAuthAccessToken as:

 public class OAuthAccessToken : IFlickrParsable
{

        public string UserId {get;set;}

        public string USername {get;set;}
public string Token { get; set; }
public string TokenSecret { get; set; }
}
}

14601596573_4bbaa2477e_o

Given the above, to get things rolling we need to populate the FlickrNet OAuthAccessToken fields from our User class, and then instantiate FlickrNet’s FlickrManager:

OAuthAccessToken o = new OAuthAccessToken();

o.Token=User.Token;

o.TokenSecret=User.TokenSecret;

o.UserId=User.UserId;

o.Username=User.Username;

FlickrNet.Flickr _F = FlickrManager.GetAuthInstance(User.APIKey, User.SharedSecret);

Now we can loop through our photos getting the number of views for each photo on the datetime of the call.  We do these with paged calls to FlickrNet’s PeopleGetPhotos:

public PhotoCollection PeopleGetPhotos(PhotoSearchExtras extras, int page, int perPage);

14528978934_61edc51d2c_o

Here is a simple loop to do so (without error checking):

int page=0;

while (true){

PhotoCollection photo = _F.PeopleGetPhotos(PhotoSearchExtras.Views, page++, 500);

     if (photo.Count == 0){ break; }

//Loop through each photo on the page:

foreach (FlickrNet.Photo p in photo)
{
//Save The Results

    SaveByID(User.UserID,p.PhotoID, p.Title, p.ThumbnailURL, p.Views); 

}

}

OK so far?  Note that in FlickrNet the Views field is defined as int? (which means that Views may contain a null) For our purposes here we will regularly convert those nulls to zeros.  The Views is not a delta but the total number of views (clicks since the photo was first posted up to the time in which the call is made).  Most of our next post will discuss one approach to collect and store views per day (or some other sample period).

ThumbnailURL can be used to retrieve the thumbnail of the image.  Note here that starting last month Flickr increased the default size of the thumbnails on new photos.  We will come back to this issue in a later post which deals with displays.  Speaking of display there is a field called LargeURL associated with each Flickr Photo object but we have had trouble working with.  Instead I will display full size images within Flickr using the formula:

https://www.flickr.com/photos/{full user name no spaces}/{photoID/in/photostream/

Example:

https://www.flickr.com/photos/dennisredfield/14588760519/

display the Flickr page for this photo.

14588760519_b7ce0afceb_o

 

Series continues here.

 

Posted 2014/07/29 by Cloud2013 in Flickr, FlickrNet, Photography

Tagged with , ,

Hey Flickr, Where Did My Statistics Go? OAuth, the Flickr API and You.   3 comments

In Yahoo’s misguided attempts to make Flickr a Phone photo sharing app like Instragram has radically redesigned the interfaces on both the desk top and on phones. Along the way they give you the option of uploading all of your phone photos directly to Flickr from your phone.  Which is nice, I guess, but Google and Microsoft and DropBox also offer the same service. I mean why would I want to dump my raw photos to Flickr when I, like most Flickr users, do our processing offline (in Photoshop for example) and post our best work to Flickr.  It would seem they are abandoning their core users (amateur photographers) to make way for the Instragram raw sewage stream of sunsets, meals and selfies.  Strange.  They are going to lose out to the Instragram / Facebook for that market and alienate the current group of core users. Is this what you want to see when you go to Flickr?  This:

selfie  Or this?  9475221248_2009bb3d34_o[1]

In order to accommodate the change Flickr, like all the big players offers a huge amount of cloud storage ‘for free’.  Something like two Gigabyte of photo storage.   Before the big changes at Flickr there was a special Flickr service “Flickr Pro” which for $29 a year got you, among other things, daily usage statistics (going back 30 days) of your photos.  Statistics were also available via the Flickr API (more on this in a moment). With the new Flickr statistics are gone and the Flickr API for statistics are also gone.

So I started digging into the existing, and still supported, Flickr API to see what was necessary to get usage statistics from Flickr for my photos using the Flickr APP.  I a C# programmer by profession so I started by looking at existing Flickr API libraries to leverage in this quest.  The godfather of C# Flickr API libraries is the FlickrNet API Library which is available in source under both the LGPL 2.1 license, and the Apache 2.0 license.

But First

This is a great and fully comprehensive library but to use it you need a couple of things to get started.  You must obtain a non-commercial personal API Key from Flickr (an application key). Apply for one here. But wait there is more.  Although the Flickr API is fairly straight forward you must, must, must, use Flickr’s implementation of OAuth with the application key to access Flickr data.  This post will be devoted to getting up and running with the Flickr API and Oauth.  Subsequent posts will discuss how to read photo view data and present a simple file database to store statistics.  Finally I will turn to using CouchDB and a Web front end to do the same work.  For this part we will switch over to Ruby and write our own simplified FLickrNet library.

OAuth

OAuth, in its many variations, is all over the place on the web these days.  Every time you see a “Sign In with…” Facebook, Google, whatever you are using OAuth in action.  You must use the Flickr OAuth implementation  to access your own data on Flickr.  It is really not all that hard although at first blush it looks impossible.  Here is Flickr’s own overview:

Simple and fun you say!

32607_alice_madness_returns

It’s not that bad when you get into it.

Using code samples provided by the folks at FlickrNet here is a simple walk through of how to setup OAuth to access your own data for a desktop application.  First we will look at it as a simple cook book approach (in C#) and then give a much simpler view of OAuth than the above diagram.  And that will be it for this post.

  • Application Key

Get an application Key and a “shared” secret from Flickr (start here).

  • Grant Access Writes for this Application Key

Get the user (that’s you to grant you access to their (your) photos.

Prepare a special HTTP  Request to Yahoo which looks like

http://www.flickr.com/services/oauth/authorize?oauth_token={hash}&perms={permsValue}

where:

{permsValue} is the access writes you are requesting to have granted to the caller

{hash} is a   Base64 encoded SHA-1 hash based on your Application Key and shared secret.  Not to worry FlickrNet handles the details for you.

Assuming you have created a FlickrManager class object (following the examples in the FlickrNet Samples Site) which includes your Application Key and shared secret the request URL is created as:

Flickr f = FlickrManager.GetInstance();  //you create this and encode your Application Key and Shared Secret into this class

requestToken = f.OAuthGetRequestToken(“oob”); //”oob” indicates that you are making a desktop request not a request from a web page

string url = f.OAuthCalculateAuthorizationUrl(requestToken.Token, AuthLevel.Read);//Request access to only what you need, in this case READ

opening this url will direct you the Yahoo login page and once logged in ask if you which to grant access for this application key to your data (photos and stuff) on Flickr.  When you answer YES Flickr will present to you on the screen (no call back as requested) an grant access key which is unique to your data and your application key.  This key is also called the“verifier” Write this down, we will use it below.

Here is what is happening in principle.  Flickr and you as the developer have a trust relationship where they grant you and application key for your use as a non-commercial developer.  The shared secret  is used as a key to encrypt messages between you (the developer) and Flickr.  This is used to request an access grant key between this application key (as a developer) and your data (as a Flickr user).  This is a one time process.

Get The OAuth Token

Each time we want to get  data back from Flickr we need to combine our Application Key, Shared secret and the access grant into a second specialized URL Get request to receive an OAUTH token which is time limited. In simple FlickrNet Code this looks like:

Flickr f = FlickrManager.GetInstance();
var accessToken = f.OAuthGetAccessToken(requestToken, verifier);
FlickrManager.OAuthToken = accessToken;

Here requestToken is a class object containing the Application Key and shared secret.  The field verifier contains the Grant Access Key we obtained above. The call to OAuthGetAccessToken builds a HTTP Get request to:

https://www.flickr.com/services/oauth/request_token?oauth_signiature={sig}

where:

seg is a SHA-1 Hash of the input parameters. Let FlickrNet handle the details for you.

Ok So Far? Here is a picture:

Picture1

 

Each data call to Flickr for data is an HTTP call and should be signed with the Application Key and the Access Object Token. Responses are returned in any of several formats of which XML and JSON are the most important.  Here is what a simple call to get photo data for a specific user (and returning XML) looks like:

https://api.flickr.com/services/rest/?method=flickr.people.getPhotos&api_key=123456789&user_id=12345&format=rest&auth_token=12345-6789&api_sig=98765

where:

api_sig is a hash of this call based on our Token Secret.  Not to worry we will use the FlickrNet assembly to generate this value.

Ok.  That’s enough for one post.  Get your Application Key from Flickr, Run the samples for FlickrNet to get your access key and generate an Access Object (OAuth Token), and in the next post we will dig in with using the Flickr API.

 

Part 2 continues here.

 

 

 

 

 

Posted 2014/07/03 by Cloud2013 in Flickr, Photography, Random Notes

Tagged with , ,

P. K. Dick was Right: The Empire Never Died!   Leave a comment

8032917432_df881f6db8_b

Posted 2013/08/06 by Cloud2013 in P.K. Dick, Photography, Photoshop

Tagged with , , ,

Photo Processing on the Nexus 7   1 comment

We ARE excited to be in San Francisco for SPTechCon.  If nothing else road trips give us an opportunity to play with some of our devices.  The last time I was in San Francisco I was using a a Nexus 7  and Adobe Photoshop Explorer to post process photos.  The results can be found here and here.  Six months later finds me back in San Francisco still using the Nexus 7 as a platform but using Adobe’s PS Touch and the Snapspeed App for post processing.  My process to gets photos from The Nikon 7 to the Nexus has been simplified but is still unacceptable.

Getting The photos to the Nexus

Moving photos from the Nikon D50 to the Nexus’s file system is a pain. I could simplify this process by

  • buying a new camera and using the Raw Vision App and a cable; or
  • Using a Bluetooth enabled SD card; or
  • Upgrading me Nexus to something with a front facing camera.

I love my Nikon and am too cheap for the cost of the Bluetooth SD card so I went the hardware root. Here is the setup:

Nexus Media Installer App (~$3)

Nexus compatible To Go cable (~$3)

A USB SD card reader (~$2)

Step One: The Physical World

SD to card reader to To Go Cable to Nexus port  

_IMG_0579

Our Topology: Cheapness Electronics, Cambridge Mass

Step Two: Copy the files from the card to your system

When the card is inserted into  “the cable system” Nexus Media Installer App will start and attempt to read the SD card directory.  I needed to be sure not to be running my file manager (File Manager HD App – free) at the same time.  You can preview all photos on the card.  No photo objects on the cards are available as documents. Select the objects you want to copy  and hit the save icon.  This will start a background job, one for each object, to transfer the photos to your file system into a  “pictures” folder.  Multiplexed and asynchronous request fulfillment . oh my. Nice software.  This folder is visible to Gallery, PS Touch and Snapspeed.  Other Apps my not make this folder accessible.  But you can always move files around with a File Manager like File Manager HD.

Post Processing the Photos

All of the software I have seen on the Nexus allow only jpeg processing.  Even Adobe PS Touch supports only JPEG. There is no RAW file type support available on the Nexus platform.  I looked at post processing apps from three board groups:

Editors with Theme filters only

Editors with adjustable photo enhancement filters; and 

Editors with spot adjustment filters and other features.

In my last blog post on this topic I was using Photoshop Express, a nice free editor with adjustable photo enhancement.  I liked the results but the app can only process files in the camera’s DCIM folder (which requires us to do file copies of imported photos twice!)

San Francisco Bay – Photoshop Express

Feeling plush I spend $2.99 and bought the next step Photoshop App: PS Touch this provides a rich set of editing tools, layers (if you use Photoshop on a laptop you know what that means), a nice history stack and more.  The results are fine but, the interface is difficult to work with but some of that may be that its UI is exposed using primarily  Surface Touch  \ Windows 8 conventions. This will flow much better on a ten inch, preferably a windows, device. (I am working on a 7 inched Nexus not a 10 inch anything).  It’s not a lot of fun to use and it may be overkill unless you do photo post processing and you have no lap top or desk top version of Photoshop to use.  But if your Surface or 10 inch iPad are your primary device this is a good choice.  Not a fun or quick or easy, choice.

SPTechCon Speaker Andrew Connel – PS Touch

A nice free alternative to Photoshop Express is the very simple but effective Snapspeed.  This is a lot of fun to use and combines theme filters and photo adjustment filters.  I did a bunch of post processing with this app and was very happy with the results. Try it.

Art Gallery – Snapspeed

A Nexus Gallery (PS Touch and Snapspeed Post Processing On the Nexus 7)

(This last one has been processed using Photoshop C5 on a laptop)

Swamp Fox: Some Photoshop Fun   Leave a comment

Three Views Of A Louisiana Swamp

8451075219_48c52cac1a_b8452166310_d17b1769cb_b

8377551283_f18d931e04_b

Cloud2013 And Mary Jo In One Photo

8413253019_079a7d0091_b

QCon San Francisco Day Two In Pictures   2 comments

QCon. Day Two Tutorials.

Professional JavaScript Development (Peter Bell)

image

image

Microsoft Asynchronous RT (Erik Meijer,Dr. Mystical ,Paul Betts).

image

How is mathematical Duality like a Dutch date?

image

image

Dr. Mystical Explains Standard Deviations.

image

Paul Betts: It’s alive!  ALIVE!!

Views Around San Francisco on Day Two.

image

image

image

image

Technical Details:
Nikon D50

I used Picasa on my laptop to import the JPEG files to my hard drive.

Picasa automatically sync the photo to my Goggle+ account ‘in the cloud’.

I transfered the photos to be edited to my Nexus 7 via the Picasa Tools applet.

Edited these photos using the Adobe Photoshop Explorer applet.

This  was composed on the WordPress applet.

I followed this painful path since I wanted to exercise three applets so I could take the Nexus 7 out for a spin around the Google ecosystem.  Life would have been simple if the Nexus 7 could read the SD in my camera via the USB mount on the Camera.  But alas this will not work with the Nexus 7.  There are rumors that if I have an SD reader I can read the SD card directly via USB.

Day One Photos can be found here.

%d bloggers like this: