You can listen to the show here.

When approaching bits of performance tuning, I work like this:  the general approach I take is to put input / output trace-logging on all methods – this gives you a true picture of the application’s execution stack – the amount of calls it has to make to do something. Next I perform an operation in the application that will cause the code being monitored to execute, and then copy and paste the tracelog generated into Notepad++ and use the nesting of the log entries to see which methods are doing the most work and which methods are called the most times. These are the first two indicators of where your effort should go. Next apply caching, re-perform the operation, paste debug output into a new window and count how many less lines of output have been generated. It’s a very crude method – but work very well.

Before I started this exercise, StyleCop for ReSharper would generate something like 220 lines of debug output per keystroke within Visual Studio, this included 10s of I/O operations and expensive object creation. After doing the performance tuning spike this went down to less than 30 lines of debug output with no I/O and no expensive object creation.

The upside is that the feeling of “lag” you get while typing has gone and also the amount of managed memory used inside ReSharper does not balloon as it always has before. You should now be able to do a whole days coding without noticing any significant performance degradation.

As always you can find the latest release at: http://stylecopforresharper.codeplex.com/

When the StyleCop API was made public – to allow the community to build extensions and plugins – the first thing I did was to figure out how I could get StyleCop to run with every keystroke and translate this into ReSharper Highlighting info which would automatically display the location of any StyleCop violation – as my vision was to create a plugin that would give developers a MS Word Grammar / Spell Checker experience for Coding Standards violations.

StyleCop works by analysing a physical code file. If you run StyleCop from the Visual Studio context menu – it will save your current file and then read the contents and analyse it. In order for StyleCop for ReSharper to work on every keystroke – I had to add the ability for ReSharper to create a clone of the current file, in a temporary location and then get StyleCop to analyze it.

The main problem has been that with every keystroke in Visual Studio there is a small IO performance hit as this temporary file is written and then StyleCop performs another IO hit to read the file before analysis.

It was always our dream to be able to pass StyleCop a string representation of the the current file – to remove the IO performance hits. It’s taken several months and lots of hard work by Andy Reeves to implement an in memory analyzer feature for the now Open Source StyleCop project – which then allows StyleCop for ReSharper to pass in a string representation of the current document.

The performance improvement is quite staggering – violation highlighting is instant and any sense of lag is totally removed.

I really hope you enjoy this release – after almost 2 years we’re finally at the place I envisaged when I first set out to create this product.

The new release of StyleCop for ReSharper is available on CodePlex.

This is an interim update as the RTM version of StyleCop 4.4.0.12 and ReSharper 5.1 will be available soon. Once they are I’ll release a further update.

Work Smarter, Not Harder.

I thought the best way to represent my interest / exposure to document databases was via a graph. So here it is (click to enbiggen):

Sean Foushee improved upon Aaron’s original pretty checkboxes adding support for radio buttons – and presented his results in a very good step by step tutorial -

To save you some time I’ve pulled out the instructions into a simple demo which you can download here (you will need to rename it to a zip file)

All credits to Aaron Weyenberg & Sean Foushee

Berners-Lee has for the last few years been working on his vision of the future direction of the web – the Semantic Web – or more rather Semantic Data – so of course the first thing he asked the PM to do was liberate all the silos of data the Government has to allow the tax payers to have access and start to innovate. One of the biggest surprises was that the PM announced that he would be opening up the Ordnance Survey’s data sets (which generates several million pounds in revenue for the Government).

Ordnance Survey have the richest GIS data of the UK – the detail fair exceeds anything like Google or Bing Maps. While I’m sure the announcement sent shock waves through companies like Experian and many other who have built their businesses around supplying UK GIS data especially the converted Postcode / address information – the mother-load – has yet to be released – although in the first wave the Code-Point Open dataset has been published. Code-Point Open contains over 1.6 million GB postcodes (Isle of Man, The Channel Islands and Northern Ireland are not yet included) and their Longitude & Latitude. The data is available to download in CSV format.

This is a wonderful move and will benefit any UK developer who is interested in building in spatial services to their applications – but there is a massive difference between raw data and usable data. The Code-Point Open dataset comes in 120 separate CSV files, the data hasn’t been cleansed so some postcodes appear with ” and others don’t (which would affect searching) and there are many fields which wouldn’t be strictly relevant for developers doing spatial work.

So I set about the task of importing the data SSIS to deal with the bulk load – using some wonderful code supplied by SSIS MVP (and ex-colleague) Jamie Thompson. Once the data was in SQL Server it’s was much easier to convert it into various other formats:

• SQL Server 2005* – with separate longitude / latitude columns.
• SQL Server 2008* – utilising the Geography type for longitude / latitude.
• MongoDB – in a format that works with MongoDB’s geospatial functions.

These datasets, the SSIS package and the console app for importing into MongoDB are available on my GitHub Repository.

Work smart, not hard.

* These are large SQL Files – I would advice using the SQL Command Line to import the data – as the files are large enough to crash Management Studio, should you try to use it to open the files.

Prerequisites:

Installation:

We are working on upgrading the WiX installer to give users a more seamless installation and upgrade experience, but at the current time we are providing a .CMD file to install the plugin into the correct location. Please pick the appropriate .CMD file for your version of Visual Studio and also remember that YOU MUST UNBLOCK the StyleCop.For.ReSharper.dll (right click > Properties > Unblock) before you run the .CMD file. Windows automatically blocks any executable or dll that is downloaded from the internet and the plugin will not register with ReSharper until it is unblocked.

Also please ensure that you import the StyleCop for ReSharper Code Style Settings file (“Import-StyleCop-for-ReSharper-Code-Style-Settings.xml”). Failure to import this file into ReSharper (ReSharper > Options > Languages > Common > Code Style Settings > Import) will result in Quick Fixes and Code Clean Up Modules not working as expected.

There are essentially no new features in this release – updating the codebase to use the ReSharper 5.0 API was a pretty monumental undertaking. The size and scope of the API changes for ReSharper 5.0 were huge – hats off to the ReSharper team for delivering another great iteration of a great product.

I have taken advantage of some of the new features of the ReSharper 5.0 API – my favourite being the ability to have custom icons for custom Quick Fixes – I took the opportunity to make use of the excellent Silk Icons add a bit of UI polish to StyleCop for ReSharper, for the options dialog:

and Quick Fix Bulb Items:

I think they make StyleCop for ReSharper integrate rather seamlessly into ReSharper.

Work Smart, Not Hard.

@HowardvRooijen

During DDD8 I attended Neil Robbins “Hello Document Database” talk, where he demo’d how to get started with CouchDB. Although I have heard a lot about the NoSQL movement – this was the first time I had seen a document database in action. In the last 12 months I’ve been spending a lot of time with search technologies such as Lucene and Solr (which have recently merged into one project), so the idea of storing data in a repository other than a relational database has been in the forefront of my mind for a while.

2009 – the year ORM’s went mainstream

For .NET Developers 2009 was the year that many made the leap from working with relational data via SQL & Stored Procedures to using an Object Relational Mapper (ORM) Framework such as NHibernate or Entity Framework. This is a great leap to make as it can massively reduce the development, deployment and maintenance burden – because using an ORM means you have essentially excised a whole layer of your application architecture. This generally throws DBA’s into cataleptic fits because by using an ORM you are eroding away their speciality and they will generally throw performance, security & reliability arguments into the mix, but realistically for 99.99% of standard CRUD operations an ORM is a much better solution to hand cranked SQL.

The problem with ORM’s is the fundamental impedance mismatch between Object Orientation and Relational Databases – not only do you have to design each model adhering to the optimal rules of both domains but then you have deal the fundamental differences between deep object graphs and relational data; the best manifestation of this is the pain of mapping files – the bane of any NHibernate user’s life. This is one area where the “convention over configuration” mantra comes into its own and products like Fluent NHibernate makes this process easier – although not entirely pain free.

2010 – the year of NoSQL and Document Databases?

This is where the NoSQL movement and in particular document databases hit the sweet spot. If you’ve been following the latest trend of conversations about scaling web applications for a global audience – you generally wont hear relational databases being mentioned because relational databases don’t scale well (there’s a reason why SQL Server Azure allows a max db size of 10 Gb). When companies like Twitter & Digg talk about their future plans you hear far more exotic names that “Microsoft SQL Server 2008 R2” – you hear product names such as Cassandra, Voldemort, MemcacheDB, Redis, HyperTable and the aforementioned CouchDB.

One of the major pain points of using a standard relational database is not just the schema design, but all the problems associated with updating and versioning of the schema. Making schema changes on systems that have brittle architectures is the bane of the lives of the developers working on those systems. Because these document databases are schemaless (or rather they provide dynamic-schemas) - you do not face the same problems. You are free to modify the documents you store, they can be irregular and have different properties – the problem might be moved up a level where you convert the documents into your domain objects. If you are using dynamic languages you may have to ensure that the properties you access actually exist and add extra error handling to deal with this edge case. In any case this is much less of a burden than standard schema versioning hell.

One niche that document databases fit into exceedingly well is the CQRS pattern that is growing in popularity. If to provide scale and speed for your web architecture you decided to create View Model repositories so that you can quickly and easily populate your UI with data using an efficient Select * rather than performing traditionally expensive and slow relational queries with multiple joins, then document databases offer a very compelling solution. They are high performance, efficiently store data, they have built in support for replication and auto fail over, they are open source and run on Windows & *nix platform, thus can be cheap to scale and also support auto-sharding if you require cloud-level scalability and finally they have dynamic schemas so your development & maintenance effort is minimal.

The more I learn about Document Databases and in particular CouchDB and MongoDB – they more I understand that these are products of the web and by the web. Not only do they solve traditionally hard web problems such as scale but they also use web technologies at their core; MongoDB uses JavaScript in its Shell as well as internally for some Map Reduce functions and CouchDB uses a RESTful JSON API which allows you to query it directly from HTTP – which means you can query the database directly from your browser, which is very powerful if you want to perform fast AJAX read operations. With MongoDB you can query and receive results in a JSON like format (BSON) that any web developer should be familiar with.

CouchDB vs. MongoDB

I started investigating CouchDB – mainly because of Neil’s demo – but after doing a little research on the ecosystem – I stumbled across this blog post about CouchDB vs. MongoDB – and it was at this point I discovered that CouchDB can’t do dynamic queries – which is a pretty essential feature for me as I like to be able to easily explore the data. At this point I swapped focus and started to dig a little deeper into MongoDB and have to say that I really like what I’ve found.

The main problem that I discovered trying to get started with both CouchDB and MongoDB is that document databases really work well when they are accessed via  a dynamic language. The real problem is that using them via a static language, means you have to deal with data in the form of a dictionary. This just isn’t acceptable. Scott Watermasysk has a nice post on a potential work around combining the MongoDB-CSharp Driver and C#4′s dynamic keyword to replicate the behaviour in most dynamic languages.

NoRM – Making MongoDB mainstream for .NET Developers

At this point I discovered a new attempt to create a C# Driver for MongoDB that would allow seamless POCO conversion and LINQ support. The project is called NoRM and it’s the best hope for making MongoDB mainstream for the .NET Community. They guys working on the project are Andrew Theken, Rob Conery (the guy behind SubSonic and TekPub), Karl Seguin, James Avery, Jason Alexander, Olivier Oswald, Nate Kohari, Steve Mason and a few others. NoRM is incredibly simple to get started with and I’ve put together a small demo that you can play with.

Getting Started with MongoDB and NoRM

On a slight tangent – one of the great initiatives that the UK government has kicked off in the last few months is http://data.gov.uk, led by Sir Tim Berners-Lee, it is a program of work to open all the data of the UK Government, data that we, as tax payers have paid for and should be allowed access to. For developers – this is a treasure trove of novel “real” data with which we can play with. For this demo I picked an XML feed that contains information sports clubs in the district of Lichfield.

Firstly I created a self-contained solution that has everything you need:

GETTING-STARTED-WITH-MONGODB
+---Data
|   \---MongoDb <= Contains the MongoDB data files
+---ReferencedAssemblies <== contains .NET Libraries we reference
+---Solutions
|   \---GettingStartedWithMongoDB
|       +---Configuration
|       +---Contracts
|       +---Domain
|       +---Framework
|       +---Infrastructure
|       \---Properties
\---Tools
    \---MongoDB <== Contains MongoDB executables (server & client)

You can download this package from my CodePlex Project.

Step 1: Get MongoDB up and running

Extract the package to C:\Projects\OpenSource\ then navigate to C:\Projects\OpenSource\getting-started-with-mongodb\Tools\MongoDB in this folder you should see 2x .exe files, 2x .cmd files and 3 readme files.

The two .exe files are MongoDB’s client and server applications. Double click start-mongod-(the-server).cmd – this is a custom command file which starts the MongoDB Server and pipes in a configuration setting to specifically set the path where MongoDB stores it’s data files to C:\Projects\OpenSource\getting-started-with-mongodb\Data\MongoDb

Once you run the cmd file you should see the following Command Prompt window:

You may be presented with a Windows Firewall dialog – dismiss this.

If you want to access the web admin interface (it’s a bit sparse!) you can via http://localhost:28017 (as mentioned in the last line of text in the command prompt).

If you navigate to C:\Projects\OpenSource\getting-started-with-mongodb\Data\MongoDb you will notice this folder is empty – the database is running but we haven’t imported any data yet.

Step 2: Run the Solution

Open the Solution file located at: C:\Projects\OpenSource\getting-started-with-mongodb\Solutions\GettingStartedWithMongoDB.sln – this is a VS 2010 RC solution file. Once you’ve opened the solution file – there are only 4 files that you really need to look at:

1. /Program.cs – this is where the main logic sits
2. /Infrastructure/ClubRepository.cs – this pulls down the XML Feed and uses LINQ to XML to convert it into a POCO
3. /Infrastructure/Session.cs - this wraps NoRM and the calls to MongoDB
4. /Contracts/IUniqueIdentifier.cs – this is an interface that defines an Id Property required for MongoDB Updates.

I borrowed the Sessions.cs file from the NoRM unit test project which uses it as the default method for performing integration tests – I made one small adjustment as a workaround for a bug I discovered, which you can see on line 49:

While working on the demo I discovered that calling the UpdateOne method using the (item, item) signature didn’t actually update the item in MongoDB and that instead you need to provide a MatchDocument (essentially an criteria document that allows you to specify which document in the document database will be updated). To make this work I had to modify the generics constraints to require that the Type implements IUniqueIdenfitier which means we have access to the Id property in order to construct the MatchDocument. IUniqueIdenfitier.cs is as follows:

ObjectId is a NoRM type that represents a MongoDB ObjectId, which is is a 12-byte value consisting of a 4-byte timestamp, a 3-byte machine id, a 2-byte process id, and a 3-byte counter.

Finally it all comes together in Program.cs, where we retrieve the feed, convert it into POCO, create a new session with a MongoDB document database called “ClubDocDB”, insert all the clubs into a new collection called “Club” and perform a LINQ query against the MongoDB database collection for a club that has a Contact with a specific name, we then update that club’s telephone number, update and re-query, again using LINQ, to ensure that the update was successful. Session implements IDisposable which auto closes the connection.

Hit F5 to run the project for yourself.

Once you run the the solution you will see the following window appear showing the data we have parsed & inserted into the document database, queried, updated and re-queried:

If you swap to the MongoDB Server Window you can see there has been activity:

You can see the following has happened:

• The server has accepted an incoming connection
• It has build a new index for the Database called “ClubDocDB” and the collection called “Club”

If you navigate to the data directory – C:\Projects\OpenSource\getting-started-with-mongodb\Data\MongoDb - you will see that this has now been populated by two data files:

• ClubDocDB.0 (64mb)
• ClubDocDB.ns (16mb)

These files are preallocated – just in case you thought we had processed and indexed  80 mb of data. See the MongoDB FAQ for more info on data files.

Step 3: Using the MongoDB Shell

I’ve shown how you can access MongoDB via NoRM – it’s stupidly easy – if you are running MongoDB on your local machine then configuration is minimal – all you need to do is specify the document database name you want to connect to. Your collection name is based on the name of the POCO entity you specify. There is another way to connect to MongoDB – via the MongoDB Interactive Shell.

Navigate to C:\Projects\OpenSource\getting-started-with-mongodb\Tools\MongoDB and double click on start-mongo-(the-client).cmd – this will launch the MongoDB Shell:

To view the databases you can use the following command:

show dbs

To access the database you have just created enter the following command:

use ClubDocDB

This switches to our database. Next you can view the collections under this database using the command:

show collections

You should see the following results:

Club
system.indexes

Next we can actually query the collection in the same we queried it using NoRM and LINQ but using the native query syntax:

db.Club.find({ "Contact.Name" : 'Daryl Loynes' })

The results of this query are returned to the shell as BSON (binary-encoded serialization of JSON-like documents):

This is quite impressive as the field we queried is actually a sub-property of the document we indexed. This is really part of the magic of MongoDB in that the document as a whole is stored in the data file as BSON and the query language understands that we are querying complex objects rather than rows and columns. The query language supports this “dot notation” which allows us to reach inside documents and query it; this is extremely powerful.

We can perform a more fine-grained query and specify that we only want to return the Tel field:

db.Club.find({ "Contact.Name" : 'Daryl Loynes' }, { "Contact.Tel" : 1 })

This will return the following BSON result:

{ "Contact" : { "Tel" : "555-123-456" }, "_id" : ObjectId("4cdefa02b6f5465017110000") }

You can see that the result also provides us with an _id field which we could use if we wanted to change the Tel field value and apply this back as an update. For more information on querying have a read of the documentation.

For more information on the MongoDB Shell – have a read of the documentation.

If this post has got you interested in MongoDB, why not spend 10 minutes using the  interactive demo / mini tutorial of MongoDB at: http://try.mongodb.org/

Finally, for a broader overview of MongoDB and it’s features, I strongly recommend reading this presentation:

Work smart, not hard.
@HowardvRooijen