It has been an interesting few weeks in the world of web standards for email.

The boys from Campaign Monitor executed a successful awareness campaign in the form of fixoutlook.org which rapidly racked up over 24,000 Tweets and overtook the Iran Election in Twitter’s trending topics. Unfortunately for all of us, it has been a case of message received – but not understood.

The Core Problem

Back in 2007, Microsoft swapped the Outlook rendering engine from Internet Explorer to Word. This in itself is not a problem at all; and actually delivered some really good improvements. There was now one-to-one fidelity between the authoring and viewing experiences because they were one and the same.

I like having Word as my authoring tool. I like features such as SmartArt and the context aware picture tools.

In making this switch though, we inherited the woeful CSS support that Word has. Microsoft’s developer documentation lists Word 2007 as supporting “a subset of the standard HTML 4.01 specification, [...] the Internet Explorer 6.0 HTML specification [and] a subset of the standard Cascading Stylesheet Specification, Level 1.” That’s even less support than Internet Explorer 5 had.

Why does this matter?

This isn’t just some web standards movement for the fun of it – there is real business impact here. No, it’s not something that end users will bang their head against. It’s something that affects all of us web designers.

Two of the key areas that are lacking in the rendering engine are support for the float and background-image. The former throws us back to the dark ages of table based layouts and all their inherent accessibility and layout issues. The latter means that there are some designs you just can’t do at all. Try placing today’s date on top of a graphic header in an email and let me know how you go.

In a comment that I consider a bit unfair, Microsoft’s official response referred to Campaign Monitor as makers of “email marketing campaign” software (complete with those quotes). Another thread I stumbled across described the fixoutlook.org campaign as being about the ability to deliver “bloated HTML with pixel trackers, domain redirectors and Google Ads”.

This is not a movement to aid in the delivering of spam. There are legitimate reasons for delivering automated and/or bulk emails to users. Campaign Monitor goes above and beyond the legal requirements to make sure their system is not misused.

This movement is about being better online citizens:

• Bloated HTML? Float-based layouts are much leaner and faster to render than table-based layouts.
• Pixel trackers? We can do that in Word already – no change here.
• Domain redirectors? I don’t even know what they are in this context and Bing doesn’t seem to either.
• Google Ads? We’re not talking about running scripts at all.

Why does it really matter?

Personally, I think one of the most amusing demonstrations of why this really matters comes from one of Microsoft’s own newsletters:

Notice that message on the top? “Read this issue online if you can’t see the images or are using Outlook 2007.” The authors of this newsletter probably deemed that Outlook 2007’s rendering engine required too much extra work for them to support it that the business case just didn’t exist.

We’re going through this same experience at the moment for one of the largest online presences in Australia. Having got our templates working in all of the major email clients except Outlook 2007 and Gmail, it was time to see what we could do about these last two stubborn children. In the end, it took twice the amount of time to make it Outlook 2007 compatible than it did to develop it in the first place. (And no, Gmail is never a pretty story either but that’s not an excuse Microsoft should be using.)

It’s not all about mass marketing either.

Here’s how one of my opt-in Twitter notifications renders side-by-side in Word and IE:

(click for full size)

Why now?

There have been some comments floating around asking why we’re only just starting to care now. I think this is a valid question, with two answers.

First and foremost, email has always been a right pain and thus the Email Standards Project was born in 2007. This project has gone on to make head way with some of the biggest names in the email game. Unfortunately though, there has been lack lustre response from Microsoft to date (including even to this targeted campaign).

Secondly, while this problem has been present since Outlook 2007, the big concern is that there doesn’t appear to have been any recourse made in Outlook 2010. To be fair, no official builds have been released yet and thus the fixoutlook.org campaign is being driven on evidence gained from a pre-beta build. With all that in mind though, you’d think that Microsoft could have mentioned something in their reply if they were working in this area. They didn’t. Also, now is our last chance to try and make an impact on Outlook 2010 before it gets locked down into the full testing regime.

Standard? What standard?

Microsoft’s official response correctly identifies that “there is no widely-recognized consensus in the industry about what subset of HTML is appropriate for use in e-mail for interoperability.” They are also correct in identifying that “the Email Standards Project does not represent a sanctioned standard or an industry consensus in this area.”

As I highlighted at the start of this post, Microsoft have explicitly stated that the HTML and CSS support in Word 2007 is but a subset of existing standards. It is also interesting to note that they refer to the Internet Explorer 6.0 HTML Specification, another document which is not a sanctioned standard or an industry consensus in this area (or any, really).

It should be recognised that Email Standards Project is not about developing a new standard, or even a subset of an existing one. It does not portray itself to be a standards organisation at all.

This is demonstrated on their homepage by the clear mission statement:

Our goal is to help designers understand why web standards are so important for email, while working with email client developers to ensure that emails render consistently. This is a community effort to improve the email experience for both designers and readers alike.

In doing so, they have developed an acid test that they can use to measure the relative performance of each of the clients. This test is a subset of the existing standards, and a subset that they have arbitrarily agreed upon, however it is simply a tool for providing relative comparisons in the same way that we use the ACID1, ACID2 and ACID3 tests for web browsers. In fact, the IE 8 team considered passing ACID2 to be a milestone for their product’s development.

Meet in the middle?

The rendering comparison provided on fixoutlook.org does include one little morsel of hope. At the top of the Outlook 2010 rendering, you’ll notice an information bar that says “If there are problems with how this message is displayed, click here to view it in a web browser”. We can fairly safely assume that this will either flick the rendering engine to IE for that message only, or save it out to a temporary location and fire up the user’s default browser. The latter will have some challenges around embedded MIME data, however I imagine this is something they would have already solved in the pre-Word days of Outlook’s rendering.

Let’s get back to the original issue for a second: Word is being used to ensure a congruent authoring and rendering experiences, and a side effect of this is that emails authored with specific HTML and CSS do not render well.

Sound familiar? Web browsers solved this problem years ago with the introduction of multiple rendering modes driven by doctype switching. This has been adopted by every major browser manufacturer, including Microsoft, as a way of ensuring wide compatibility with varying levels of standards support.

The idea is by no means unique, but what’s stopping us from having Word as the rendering engine for emails received from another Outlook instance and IE as the rendering engine for emails the rest of the time?

1. There is evidently code there already to detect when an IE-based render would produce better quality results.
2. MIME already includes enough information for one to determine what application authored a message.
3. I don’t think anybody is currently too concerned about the amount of rendering that is preserved when one attempts to forward an EDM. This is troublesome ground across every email client out there.

We don’t need to radically change Word to support a whole bunch of new renderings. We don’t need to tear Word out of Outlook (despite what some of the campaign supporters have been saying).

All we’re asking for is a reliable and consistent way for the web developers of the world to deliver styled emails to Oulook, one of the best messaging platforms out there.

Updates

6th July, 1501: John Liu accurately brought up the anti-trust restrictions around the packaging of Internet Explorer. These restrictions apply to the shipping of Internet Explorer as a product, and do not relate to the underlying rendering engine (mshtml.dll). In fact, The Help & Support interface in Windows 7 relies on this rendering engine itself:

WinDbg is a tool that is immensely useful, but painfully hard to get any value from if you don’t know how to use it. On a few occasions I’ve found myself in a situation where I knew that WinDbg could give me the answer, but I didn’t have enough knowledge of how to use it.

This week we performed a production deployment on one of the projects I am involved in. Pretty soon after, we started to notice some issues cropping up that were affecting the performance and stability of the whole site.

In this scenario, WinDbg was a life saver for us. I can’t think of another way that we would have been able to identify and resolve this problem.

The first sign that something was wrong was this event getting fired across the whole front-end tier:

03 Jul 2009 08:32:43 AM
Computer: WEBTIER14
Monitor Title: "Event Log Monitor" (Type=Event Log Monitor)
Description:
* Event Time: 03 Jul 2009 08:32:42 AM
* Source: .NET Runtime 2.0 Error Reporting
* Event Log: Application
* Type: Error Event
* Event ID: 1000
* Faulting application w3wp.exe, version 6.0.3790.3959, stamp 45d6968e, faulting module kernel32.dll, version 5.2.3790.4062, stamp 4626467c, debug? 0, fault address 0x00022366.

We also noted that CPU usage was spiking erratically and was well above the average we would expect for the number of firewall connections we had open.

This event log entry was telling us that the w3wp worker process was faulting in an unrecoverable way, causing the application pool to be torn down in IIS. The nature of the fault was also causing the Windows Error Reporting dw20.exe process to trigger and try capturing a mini dump. These rapid recycling combined with the CPU and IO intensive task of capturing a mini dump was killing our servers and we needed to act fast.

The first step was to work out what was causing the process to be torn down in the first place, as the impact of Windows Error Reporting’s automated analysis was only a secondary issue. To capture this information, I used the adplus script in the Debugging Tools for Windows package to capture a dump.

In its default configuration, adplus will capture a crash dump on any first chance exception. Being an ASP.NET application there are a number of exceptions that we expect to be thrown as 404s are reported, redirects are performed, etc. As such, running adplus in its default configuration was causing it to grab the crash dump too frequently and not giving us the fault that was causing the complete teardown.

To capture the process teardown fault, I ran adplus with these parameters:

adplus.vbs -pn w3wp.exe -c Unknown.cfg

The first parameter just says that we want to capture dump data for the w3wp.exe process. The second parameter is the interesting one – I am passing in a configuration file that includes some exception filters. I grabbed this config from Tess’ blog.

The content of the configuration file is fairly simple XML that describes the unknown fault scenario:

<adplus>
    <settings>
        <runmode> CRASH </runmode>
    </settings>
    <exceptions>
        <config>
            <code>AllExceptions</code>
            <actions1>Log</actions1>
            <actions2>MiniDump;Log;EventLog</actions2>
        </config>
        <newexception>
            <code> 0xe053534f </code>
            <name> Unknown_Exception </name>
        </newexception>
        <config>
            <code> 0xe053534f </code>
            <actions1>FullDump;Log;EventLog</actions1>
            <actions2>FullDump;Log;EventLog</actions2>
        </config>
    </exceptions>
</adplus>

With the debugger now attached, it was a simple matter of waiting for the problem to occur. In our scenario it was happening every 10 to 15 minutes per server, so it took a while for me to capture the dump. Finally, the console showed “Dump file successfully written”.

To analyse the dump file, I opened windbg.exe, chose Open Crash Dump and selected the .dmp file that had been written out. You’ll usually find it in a CrashDump folder next to where you ran adplus from.

As we are working with a .NET application, but WinDbg is a native debugger, my next step was to load a set of debugger helpers called SOS. This is basically a plugin for WinDbg that lets us access information about the CLR state.

To load it, run:

.loadby sos mscorwks

The period at the start says it’s a WinDbg core command. The next parameter is the name of the library we want to load into the debugger. The final parameter is basically a relative file base – we’re telling WinDbg to load SOS from the same location that mscorwks had been loaded from. mscorwks is the .NET runtime, so this approach helps ensure we are loading the correct version and architecture of SOS and it saves us from having to write out the full path.

Next, I wanted to see what point of our application we were in when the process faulted. With SOS loaded, the command is quite simple:

!clrstack

Instantly, it was obvious that we had a stack overflow problem. This was made clear by literally hundreds of repeated frames on our call stack. It also explained why our ASP.NET Health Monitoring wasn’t reporting an exception, as a stack overflow causes the whole process to be torn down before any exception handling code can be run.

Here’s a short snippet (with the project details masked out):

...
CompanyName.Project.Web.Logic.Component.Method1(System.String, System.Int32)
CompanyName.Project.Web.Logic.Component.Method2(System.String)
CompanyName.Project.Web.Logic.Component.Method1(System.String, System.Int32)
CompanyName.Project.Web.Logic.Component.Method2(System.String)
CompanyName.Project.Web.Logic.Component.Method1(System.String, System.Int32)
CompanyName.Project.Web.Logic.Component.Method2(System.String)
CompanyName.Project.Web.Logic.Component.Method1(System.String, System.Int32)
CompanyName.Project.Web.Logic.Component.Method2(System.String)
CompanyName.Project.Web.Logic.Component.Method1(System.String, System.Int32)
CompanyName.Project.Web.Logic.Component.Method2(System.String)
CompanyName.Project.Web.Logic.Component.Method1(System.String, System.Int32)
CompanyName.Project.Web.Logic.Component.Method2(System.String)
CompanyName.Project.Web.Logic.Component.Method1(System.String, System.Int32)
CompanyName.Project.Web.Logic.Component.Method2(System.String)
...

The component in question is used across almost all of our pages and the site wasn’t completely down, so I could now derive that it was being caused by a very specific piece of data. This leads us to our next challenge, which is to find out what which record was triggering the problem.

To do so, I ran:

!clrstack -p

The argument of -p asks SOS to list out all of the parameters for each stack frame.

Finding out the URL of the current request is a non-trivial task, and not all parameters are available due to runtime optimisations. Rather than persisting in a search for the URL, it was much easier to grab another stack frame which included the information I needed:

1947ed28 1abb1c7e CompanyName.Project.Web.Logic.Presenters.RetailItemPresenter.View_ItemLoading(System.Object, CompanyName.Project.Web.Logic.Views.ItemLoadingEventArgs)
PARAMETERS:
this = 0x04144bf8
sender =
e = 0x04151b38

This is the event handler on one of our presenters being called as a result of the ItemLoading event being fired on the corresponding view. This approach is specific to our MVP architecture, but it was a known and simple entry point into most of the stack. Try and look for an equivalent call in your own stack.

I could now see that it was a particular retail item triggering the problem, and I knew that the information I needed was in the event arguments. You can see the parameter on the stack frame shown as e = 0x04151b38. That number is the memory location of the object we need.

To get the object, I then ran:

!do 0x04151b38

The !do command is a shortcut for !DumpObject, and gave us this output:

Name: CompanyName.Project.Web.Logic.Views.ItemLoadingEventArgs
MethodTable: 19ef2170
EEClass: 19d49b70
Size: 12(0xc) bytes
(c:\WINDOWS\Microsoft.NET\Framework\v2.0.50727\Temporary ASP.NET Files\root\aef04219\e69835b3\assembly\dl3\29e5062003a4007_a4fac901\CompanyName.Project.Web.Logic.DLL)
Fields:
MT Field Offset Type VT Attr Value Name
7932a400 4000183 4c System.EventArgs 0 shared static Empty
>> Domain:Value 001cc630:NotInit 00204700:03043914 <<
793308ec 4000819 4 System.String 0 instance 0412522c <RetailItemId>k__BackingField

We can now see the RetailItemId property and its corresponding backing field. The value column on that line is 0412522c, which is another memory reference.

Once again, we dump the memory reference:

!do 0412522c

This returns us the corresponding System.String object:

Name: System.String
MethodTable: 793308ec
EEClass: 790ed64c
Size: 28(0x1c) bytes
(C:\WINDOWS\assembly\GAC_32\mscorlib\2.0.0.0__b77a5c561934e089\mscorlib.dll)
String: 6T046
Fields:
MT Field Offset Type VT Attr Value Name
79332b38 4000096 4 System.Int32 1 instance 6 m_arrayLength
79332b38 4000097 8 System.Int32 1 instance 5 m_stringLength
793315cc 4000098 c System.Char 1 instance 36 m_firstChar
793308ec 4000099 10 System.String 0 shared static Empty
>> Domain:Value 001cc630:02fe01d0 00204700:02fe01d0 <<
7933151c 400009a 14 System.Char[] 0 shared static WhitespaceChars
>> Domain:Value 001cc630:02fe0728 00204700:06fe0e14 <<

Hidden away on the sixth line there is the raw string value, 6T046, which was the product id.

At this point, we had:

1. Captured a complete memory dump of the w3wp.exe process when the fault occurred
2. Identified a stack overflow to be the cause of the fault
3. Identified the area of our code that was involved in the stack overflow
4. Identified the particular piece of data that was exposing the bug

We could now:

1. Perform a quick fix to the data to circumvent the bug and get the website performant and stable again
2. Reproduce the bug locally using the same data that caused the issue
3. Have the lazy Friday we’d been hoping for

The video recording from Damian and my session at REMIX Sydney last week is now online. Thanks to some funky aspect ratio issues we both look really buff too.

In this session, two ASP.NET MVPs will share their experiences from building Australia’s largest e-commerce site with ASP.NET Web Forms. They’ll show you how to be a good web citizen by covering standards compliance (properly!), cross browser and accessibility considerations, non-JavaScript support, as well as coding techniques like the Model-View-Presenter pattern to improve maintainability and testing. Along the way, there’ll be plenty of discussion of the differences between Web Forms and ASP.NET MVC in each area, as well as an early peek at some of the new features coming in ASP.NET 4.0 and VS 2010.

Building great standards based websites for the big wide world with ASP.NET 4.0 – watch online

Building great standards based websites for the big wide world with ASP.NET 4.0 – download WMV

The resources from our talk are now all online (including the ones that hadn’t been released yet on the day):

• Damian’s ASP.NET Web Forms Model-View-Presenter (MVP) Framework (Open source under MS-PL)
• Damian’s Visual Studio XHTML 1.1 Templates (Open source under MIT)
• My XHTML Markup Sanitizer (Open source under MS-PL)

We also talked about the CSS Friendly Control Adapters which are again open source, under MS-PL.

While you’re checking out the videos, take a look at Jordan’s rockin’ Silverlight 3 Super Session. Not all of the videos are up yet (like, most of them aren’t), and I imagine that from the feedback flowing in there might be an updated version of the videos soon, so keep watching the videos page for some other really good content.

Updated @ 1601, 18th June: The video page interface has been updated to support deep linking so I’ve update this post to include links to the in-page players.

Updated @ 1053, 20th June: Added links to resources now that they’re all been published.

Last week at Remix I demonstrated a markup sanitizer that I’d been working on and announced that it’d soon be available as open source. After a few days and a bit of intellectual property management, I’ve finally managed to get it up on CodePlex under the MS-PL license.

http://markupsanitizer.codeplex.com

The XHTML Markup Sanitizer takes untrusted (X)HTML and massages it into real, trusted XHTML. While plenty of effort goes into preserving the original intent, markup validity and safety is the first priority. It’s particularly useful with content management systems where users are in control of markup, but you want to target XHTML1.1.

The sanitizer does not process entire pages. It is designed to massage snippets of text that users might enter in things like blog comments or product descriptions.

Check it out and try it in your project today! Now you have no excuse for not targeting full standards compliance.

http://opensearchvalidator.com

I’ve been doing a lot of work with both OpenSearch and IE8 Visual Search this week. These are the feeds to let you integrate your website into the browser’s search dropdown. Unfortunately the specs are spread across three different organisations (A9, Mozilla and Microsoft) and all rather poorly documented.

We’ve been implementing the feeds on a major e-commerce website, and thus we wanted to make sure we got them right. (We’ll be showing the site in the REMIX keynote next week.)

The solution? I built a validator. As we found new issues, I added them to the validator.

Today I’m releasing that validator for you to test your own sites with.

Some cool ones to check out are:

• hanselman.com, who has his SearchFrom element in the wrong namespace at the time of writing
• wikipedia.org, who are serving their visual search suggestions with the wrong media type at the time of writing
• au.yahoo.com, who are serving their OpenSearch description with the wrong media type at the time of writing
• bing.com, who don’t even publish a feed at the time of writing

Update 7th June 2009: The source code is now published on CodePlex.

This is the transcript of a presentation I’ll be giving at a joint schools careers night in Sydney tomorrow. The hosting school are my old high school, SHORE.

I’m here this evening to discuss small business.

It’s my personal opinion that a business degree will prepare you for running your own business little more than your HSC already will.

Your prospective universities may disagree.

Running a business is an expression of entrepreneurialism, and is not something that can be taught or learnt in a class room. They may be able to teach you financial ratios, an obscure set of legal structures and some basic marketing techniques, however each of these are somewhat superfluous to the core of running a business.

Many of the business people I value the most have either never attempted university, or more often, never completed it. The success of these individuals and their lack of (completed) formal tertiary educations should not however be interpreted as a free pass for anybody to skip uni, jump in head first and expect an equivalent outcome. Nor should it devalue the relevance of finance, law, marketing and the myriad of other business-related professions that I have skipped over. Instead, they each highlight what is at the very core of a successful business person; the individual themselves. If business success was a simple as following as defined set of parameters that you were taught in a classroom, the free market economy would simply fail to exist in the first place.

Partnerships

Running a business is a tough road. It requires one to act in many roles and demonstrate a diverse range of skills over an extended period, all whilst maintaining balance and focus. Larry Page and Sergey Brin suspended their PhDs to found Google. The pair’s success serves to highlight what I see as a very important aspect of the individual; the partner. All too often, starting a business is seen as something done by an individual.

Running a business demands a set of behaviours that one person will often struggle to deliver consistently. Particularly during the start-up phase, it can be a struggle to maintain focus in the right areas and avoid the wrong areas, particularly when you’re so passionate about getting it perfect instead of getting it done. Having a business partner can deliver the balance required, as well as a sounding board for ideas and the always needed set of helping hands. I know that I wouldn’t have achieved but half of what I have today without my business partner Tom.

In our business, Tom also provides a complementary set of skills. He’ll leave a lunch with three business leads and I’ll be asking him what the host’s name was again. Our partnership allows me to focus on building out product, and Tom to focus on the capitalization, two processes that require distinct skill sets.

There’s another description I quite like, which is that if you can’t convince one person to join you, you probably want to revisit your idea. I can’t remember exactly where I heard that, but I suspect it may have actually been one of my teachers here.

Someone who I see as being in dire need of a business partner is Mark Zuckerberg, the founder of Facebook. In 2006 Microsoft tried to buy Facebook and arranged an 8am breakfast catch-up. His office responded that “this simply would not be possible, as Mr Zuckerberg would still be in bed at that time.” Around this same time he was still attending trade fairs handing out a business card that read ‘I’m a CEO … bitch’.

Whilst we can all appreciate the humour, Facebook has failed to achieve anywhere near the profits required to match the US$2.2 billion that Microsoft laid on the table. I have some suspicions that Mark would be better served building the website we know and love and leaving someone else to make the deals.

Risk Capacity

Another business you might be familiar with from your computer lab periods is CollegeHumor.com. Launched by two US college mates spreading fliers around their campus, traffic grew organically – and quickly. At the three-month mark, the site was attracting 600,000 visitors per week, largely thanks to a video of a man sitting on a tree stump being hit on the back of a head with a shovel.

What started out as a joking search for advertisers to pay for their beer ended up as an online media business attracting advertisers like Coca Cola and Dreamworks. Had the business failed, Josh and Ricky would have lost little more than their initial $200 investment and their $30 per month internet bill. No doubt, they would have held on to their now extensive collection of funny videos.

Josh Abramson, the site’s co-creator once commented that “the greatest thing about starting a business in college is that there is very little risk.”

He’s right; and you’re in that sweet spot right now. You don’t go broke when you’re 18; you just go back to your parent’s place for dinner. You might miss a night out on the town; but you won’t miss a mortgage payment.

You’re all sitting at the peak of your risk capacity, and shouldn’t be afraid to take advantage of that. I’ve certainly had my fair share of failed ideas along the way. Most just lost time, some lost money and I certainly had varying amounts of cash in my wallet along the way. All of them were good fun though, and all of them prepared me better for the subsequent, and successful, attempts.

Skills Crossover

All of the businesses I have mentioned so far have been web based, but this does not make them purely technology businesses.

Lots of people author good content on the web, but that doesn’t make them good at promoting it. Google does that for them.

Plenty of people can make a funny video, YouTube demonstrates that. The simple act of publishing the video doesn’t make it popular with college students though. College Humor does that.

In their simplest sense, each of these businesses operates on the divide between two different skill sets. Generally, I find this to be the space where the most opportunity exists.

Brendan Powning is a 42 year old rigger on construction sites, with a side hobby of electronics. With 20 years of experience on construction sites, he’s seen countless tools go missing and no great solutions. Using his basic electronics skills he developed an inexpensive motion sensor that could be buried alongside the entrances to construction sites and activated from a mobile phone. Now, every time somebody drives on to one of his constructions sites out of hours, he gets a text message and can investigate.

Having being featured on the ABC’s New Inventors, he’s now developing a successful business around this device. Whilst not an overly complex or unique idea, it took the rare crossover of construction site experience and basic electronics skills for the idea to eventuate and be practical.

Think about the activities, sports and hobbies you participate in today, and where the frustrations exist in each of them. As its simplest level, business is about providing solutions and finding the frustration is generally a good place to start.

Enablement

This can all sound a bit ominous though; finding an idea, finding a business partner, and then (hopefully) finding your first customer. Picking the trifecta is hard, and it should sound at least a little bit ominous because if you’re successful in doing so you’ve done bloody well.

You don’t always have to do that though. There are plenty of useful ideas out there already; more often than not from people who will fail to execute them. Similarly, there are plenty of people ready to achieve, but still looking for that golden idea.

Think about which group you are in now; then find the other.

One of the businesses I run is a based in male fashion PR and export. I don’t know any designers, I don’t know any fashion editors, and I don’t know any fashion trends until they turn up in Chatswood Westfield. I do know how to build a website, my business partner knows how to market it, and our strategic partners know how to make it look good. Our involvement is silent, based around enabling someone else’s idea using our skills.

Closing

Stepping back for a moment, you may have noticed that I haven’t actually used the phrase “small business” since the very beginning of this talk. The techniques, behaviours and skills required in running a small business are equally applicable to operating larger businesses. The experience gained from making those front-line business decisions along the way has given me a respect for the wider organisational context. What may seem right and justified to you at one time is but a single point of view in the tug-of-war that is running an enterprise. Attempting to understand this wider context without having experienced it is an exercise in futility. The consulting role that I am currently engaged in may be one of technical strategy, but I won that role more so from my capacity to engage with the business as opposed to technical flair.

Closing on that statement, I’d like you to leave this evening thinking of business as both an end-game, as well as a flexible opportunity to develop a highly valuable set of skills.

If you’d like to review this talk at another time, the transcript is available from tatham.oddie.com.au, and I’ll ask Mr Scouller to circulate that address.

Thank you.

Working with the awesome team on my current project, I’ve come to the realisation that I never really did understand automated testing all that well. Sure, I’d throw around words like “unit test”, then write a method with a [TestMethod] attribute on it and voila, I was done; right? Hell no I wasn’t!

Recently, I challenged myself to write an asynchronous TCP listener, complete with proper tests. This felt like a suitable challenge because it combined the inherent complexities of networking with the fun of multi-threading. This article is about what I learnt. I trust you’ll learn something too.

What type of test is that?

The first key thing to understand is exactly what type of test you are writing. Having fallen in to the craze of unit testing as NUnit hit the ground, I naturally thought of everything as a unit test and dismissed the inherent differences of integration tests.

• A unit test should cover the single, smallest chunk of your logic possible. It must never touch on external systems like databases or web services. It should test one scenario, and test it well. Trying to cover too many scenarios in a single test introduces fragility into the test suite, such that one breaking change to your logic could cascade through and cause tens or even hundreds of tests to fail in one go.
• An integration test tests the boundaries and interactions between your logic and its external systems. It depend on external services and will be responsible for establishing the required test data, running the test, then cleaning up the target environment. This citizenship on the test’s behalf allows it to be rerun reliably as many times as you want, a key component of a test being considered valuable.

I always dismissed the differences as being subtle elements of language and left it for the TDD hippies to care about. Unfortunately, they were right – it does matter. Now, let’s spend the rest of the article building a TCP listener that is testable without having to use an integration test. Yes, you heard me right.

The Problem Space

As a quick introduction to networking in .NET, this is how you’d accept a connection on port 25 and write a message back:

var listener = new TcpListener(IPAddress.Any, 25);

using (var client = listener.AcceptTcpClient())
using (var stream = client.GetStream())
using (var streamWriter = new StreamWriter(stream))
{
   streamWriter.Write("Hello there!");
}

The first line attaches to the port, then AcceptTcpClient() blocks the code until we have a client to talk to.

In our challenge, we want to be able to talk to two clients at once so we need to take it up a notch and accept the connection asynchronously:

static void Main(string[] args)
{
  var listener = new TcpListener(IPAddress.Any, 25);
  listener.BeginAcceptTcpClient(new AsyncCallback(AcceptClient), listener); 

  Console.ReadLine();
} 

static void AcceptClient(IAsyncResult asyncResult)
{
  var listener = (TcpListener)asyncResult.AsyncState; 

  using (var client = listener.EndAcceptTcpClient(asyncResult))
  using (var stream = client.GetStream())
  using (var streamWriter = new StreamWriter(stream))
  {
    streamWriter.Write(”Hello there!”);
  }
}

If you’ve looked at asynchronous delegates in .NET before, this should all be familiar to you. We’re using a combination of calls to BeginAcceptTcpClient and EndAcceptTcpClient to capture the client asynchronously. The AcceptClient method is passed to the BeginAcceptTcpClient method as our callback delegate, along with an instance of the listener so that we can use it later. When a connection becomes available, the AcceptClient method will be called. It will extract the listener from the async state, then call EndAcceptTcpClient to get the actual client instance.

Already, we’re starting to introduce some relatively complex logic into the process by which we accept new connections. This complexity is exactly why I want to test the logic – so that I can be sure it still works as I continue to add complexity to it over the life of the application.

Split ‘Em Down The Middle

To start cleaning this up, I really need to get my connection logic out of my core application. Keeping the logic separate from the hosting application will allow us to rehost it in other places, like our test harness.

Some basic separation can be introduced using a simple wrapper class:

class Program
{
  static void Main(string[] args)
  {
    var listener = new TcpListener(IPAddress.Any, 25); 

    var smtpServer = new SmtpServer(listener);
    smtpServer.Start(); 

    Console.ReadLine();
  }
}

class SmtpServer
{
  readonly TcpListener listener; 

  public SmtpServer(TcpListener listener)
  {
    this.listener = listener;
  } 

  public void Start()
  {
    listener.BeginAcceptTcpClient(new AsyncCallback(AcceptClient), listener);
  } 

  static void AcceptClient(IAsyncResult asyncResult)
  {
    var listener = (TcpListener)asyncResult.AsyncState; 

    using (var client = listener.EndAcceptTcpClient(asyncResult))
    using (var stream = client.GetStream())
    using (var streamWriter = new StreamWriter(stream))
    {
      streamWriter.Write(”Hello there!”);
    }
  }
}

Now that we’ve separated the logic, it’s time to start writing a test!

Faking It ‘Till You Make It

The scenario we need to test is that our logic accepts a connection, and does so asynchronously. For this to happen, we need to make a client connection available that our logic can connect to.

Initially this sounds a bit complex. Maybe we could start an instance of the listener on a known port, then have our test connect to that port? The problem with this approach is that we’ve ended up at an integration test and the test is already feeling rather shaky. What happens if that port is in use? How do we know that we’re actually connecting to our app? How do we know that it accepted the connection asynchronously? We don’t.

By faking the scenario we can pretend to have a client available and then watch how our logic reacts. This is called ‘mocking’ and is typically achieved using a ‘mocking framework’. For this article, I’ll be using the wonderful Rhino Mocks framework.

This is how we could mock a data provider that normally calls out to SQL:

var testProducts = new List<product>
{
  new Product { Title = “Test Product 123″ },
  new Product { Title = “Test Product 456″ },
  new Product { Title = “Test Product 789″ }
}; 

var mockDataProvider = MockRepository.GenerateMock<idataprovider>();
mockDataProvider.Expect(a => a.LoadAllProducts()).Return(testProducts); 

var products = mockDataProvider.LoadAllProducts();
Assert.AreEqual(3, products.Count());

mockDataProvider.VerifyAllExpectations();

This code doesn’t give any actual test value, but it does demonstrate how a mock works. Using the interface of IDataProvider, we ask the mock repository to produce a concrete class on the fly. Defining an expectation tells mock repository how it should react when we call LoadAllProducts. Finally, on the last line of the code we verify that all of our expectations held true.

In this case, we are dynamically creating a class that implements IDataProvider and returns a list of three products when LoadAllProducts is called. On the last line of the code we are verifying that LoadAllProducts has been called as we expected it to be.

Artificial Evolution

Now, this approach is all well and good when you have an interface to work with, but how do we apply that to System.Net.Sockets.TcpListener? We need to modify the structure of the instance such that it implements a known interface; this is exactly what the adapter pattern is for.

First up, we need to define our own interface. Because we need to mock both the listener and the client, we’ll actually define two:

public interface ITcpListener
{
  IAsyncResult BeginAcceptTcpClient(AsyncCallback callback, object state);
  ITcpClient EndAcceptTcpClient(IAsyncResult asyncResult);
} 

public interface ITcpClient
{
  NetworkStream GetStream();
  IPEndPoint RemoteIPEndPoint { get; }
}

To apply these interfaces to the existing .NET Framework implementations, we write some simple adapter classes like so:

public class TcpListenerAdapter : ITcpListener
{
  private TcpListener Target { get; set; } 

  public TcpListenerAdapter(TcpListener target)
  {
    Target = target;
  } 

  public IAsyncResult BeginAcceptTcpClient(AsyncCallback callback, object state)
  {
    return Target.BeginAcceptTcpClient(callback, state);
  } 

  public ITcpClient EndAcceptTcpClient(IAsyncResult asyncResult)
  {
    return new TcpClientAdapter(Target.EndAcceptTcpClient(asyncResult));
  }
}

public class TcpClientAdapter : ITcpClient
{
  private TcpClient Target { get; set; } 

  public TcpClientAdapter(TcpClient target)
  {
    Target = target;
  } 

  public NetworkStream GetStream()
  {
    return Target.GetStream();
  } 

  public IPEndPoint RemoteIPEndPoint
  {
    get { return Target.Client.RemoteEndPoint as IPEndPoint; }
  }
}

These classes are solely responsible for implementing our custom interface and passing the actual work down to an original target instance which we pass in through the constructor. You might notice that line 17 of the code uses an adapter itself.

With some simple tweaks to our SmtpServer class, and how we call it, our application will continue to run as before. This is how I’m now calling the SmtpServer:

static void Main(string[] args)
{
  var listener = new TcpListener(IPAddress.Any, 25);
  var listenerAdapter = new TcpListenerAdapter(listener);

  var smtpServer = new SmtpServer(listenerAdapter);
  smtpServer.Start();

  Console.ReadLine();
}

The key point to note is that when once we have created the real listener, we are now wrapping it in an adapter before passing it down to the SmtpServer constructor. This satisfies the SmtpServer which would now be expecting an ITcpListener instead of a concrete TcpListener as it did before.

Talking The Talk

At this point in the process we have:

1. Separated the connection acceptance logic into its own class, outside of the hosting application
2. Defined an interface for how a TCP listener and client should look, without requiring concrete implementations of either
3. Learnt how to generate mock instance from an interface

The only part left is the actual test:

[TestMethod]
public void ShouldAcceptConnectionAsynchronously()
{
  var client = MockRepository.GenerateMock<itcpclient>();
  var listener = MockRepository.GenerateMock<itcplistener>();
  var asyncResult = MockRepository.GenerateMock<iasyncresult>();

  listener.Expect(a => a.BeginAcceptTcpClient(null, null)).IgnoreArguments().Return(asyncResult);
  listener.Expect(a => a.EndAcceptTcpClient(asyncResult)).Return(client); 

  var smtpServer = new SmtpServer(listener);
  smtpServer.Start();

  var arguments = listener.GetArgumentsForCallsMadeOn(a => a.BeginAcceptTcpClient(null, null));
  var callback = arguments[0][0] as AsyncCallback;
  var asyncState = arguments[0][1];
  asyncResult.Expect(a => a.AsyncState).Return(asyncState);

  callback(asyncResult);

  client.VerifyAllExpectations();
  listener.VerifyAllExpectations();
  asyncResult.VerifyAllExpectations();
}

Ok, lets break that one down a step at a time, yeah?

The first three lines are just about generated some mocked instances for each of the objects we’re going to need along the way:

var client = MockRepository.GenerateMock<itcpclient>();
var listener = MockRepository.GenerateMock<itcplistener>();
var asyncResult = MockRepository.GenerateMock<iasyncresult>();

Next up, we define how we expect the listener to work. When the BeginAcceptTcpClient method is called, we want to return the mocked async result. Similarly, when EndAcceptTcpClient is called, we want to return the mocked client instance.

listener.Expect(a => a.BeginAcceptTcpClient(null, null)).IgnoreArguments().Return(asyncResult);
listener.Expect(a => a.EndAcceptTcpClient(asyncResult)).Return(client);

Now that we’ve done our setup work, we run our usual logic just like we do in the hosting application:

var smtpServer = new SmtpServer(listener);
smtpServer.Start();

At this point, our logic will have spun up and run called the BeginAcceptTcpClient method. Because it is asynchronous, it will be patiently waiting until a client becomes available before it does any more work. To kick it along we need to fire the async callback delegate that is associated with the async action. Being internal to the implementation, we can’t (and shouldn’t!) just grab a reference to it ourselves but we can asking the mocking framework:

var methodCalls = listener.GetArgumentsForCallsMadeOn(a => a.BeginAcceptTcpClient(null, null));
var firstMethodCallArguments = methodCalls.Single();
var callback = firstMethodCallArguments[0] as AsyncCallback;
var asyncState = firstMethodCallArguments[1];
asyncResult.Expect(a =>; a.AsyncState).Return(asyncState);

The RhinoMocks framework has kept a recording of all the arguments that have been passed in along the way, and we’re just querying this list to find the first (and only) method call. While we have the chance, we also push our async state from the second argument into the async result instance.

Armed with a reference to the callback, we can fire away and simulate a client becoming available:

callback(asyncResult);

Finally, we ask RhinoMocks to verify that everything happened under the covers just like we expected. For example, if we had defined any expectations that never ended up getting used, RhinoMocks would throw an exception for us during the verification.

client.VerifyAllExpectations();
listener.VerifyAllExpectations();
asyncResult.VerifyAllExpectations();

Are We There Yet?

We are!

Taking a quick score check, we have:

1. Separated the connection acceptance logic into its own class, outside of the hosting application
2. Defined an interface for how a TCP listener and client should look, without requiring concrete implementations of either
3. Used mocking to write a unit test to validate that our logic correctly accepts a new client asynchronously

Having done so, you should now:

1. Understand the difference between a unit test and an integration test
2. Understand the importance of separation of concerns and interfaces when it comes to writing testable (and maintainable!) code
3. Understand how the adapter pattern works, and why it is useful
4. Understand the role of a mocking framework when writing tests

Was this article useful? Did you learn something? Tell me about it! ]]> http://blog.tatham.oddie.com.au/2009/04/09/testing-the-world-and-writing-better-quality-code-along-the-way/feed/ 1 tatham http://blog.tatham.oddie.com.au/2008/12/20/accessing-aspnet-page-controls-during-preinit/ http://blog.tatham.oddie.com.au/2008/12/20/accessing-aspnet-page-controls-during-preinit/#comments Fri, 19 Dec 2008 20:30:00 +0000 Tatham Oddie http://tatham.wordpress.com/2008/12/20/accessing-aspnet-page-controls-during-preinit/ ]]>

If you’ve read my previous post explaining a common pitfall with view state, I’d hope you’re preparing all your controls in the Init event of the page/control lifecycle.

Even if I’m not reusing them through my application much, I like to factor elements like a drop down list of countries into their own control. This centralizes their logic and allows us to write clear, succinct markup like this:

<tat:CountriesDropDownList ID="AddressCountry" runat="server" />

The code for a control like this is quite simple:

[ToolboxData("<{0}:CountriesDropDownList runat=\"server\" />")]
public class CountriesDropDownList : DropDownList
{
    protected override void OnInit(EventArgs e)
    {
        DataSource = Countries;
        DataBind();

        base.OnInit(e);
    }
}

The Problem

Once you start using this encapsulation technique, it won’t be long until you want to pass in a parameter that affects the data you load. Before we do, we need to be aware that the Init event is fired in reverse order. That is, the child controls have their Init event fired before that event is fired at the parent. As such, the Page.Init event is too late for us to set any properties on the controls.

The natural solution is to try and use the Page.PreInit event, however when you do you’ll often find that your control references are all null. This happens when your page is implemented using a master page, and it relates to how master pages are implemented. The <asp:ContentPlaceHolder /> controls in a master page use the ITemplate interface to build their contents. This content (child controls) is not usually prepared until the Init event is called, which means the control references are not available. For us, this represents a problem.

The Solution

The fix is remarkably simple; all we need to do is touch the Master property on our Page and it will cause the controls to become available. If we are using nested master pages, we need to touch each master page in the chain.

I often create a file called PageExtensions.cs in my web project and add this code:

public static class PageExtensions
{
    /// <summary>
    /// Can be called during the Page.PreInit stage to make child controls available.
    /// Needed when a master page is applied.
    /// </summary>
    /// <remarks>
    /// This is needed to fire the getter on the top level Master property, which in turn
    /// causes the ITemplates to be instantiated for the content placeholders, which
    /// in turn makes our controls accessible so that we can make the calls below.
    /// </remarks>
    public static void PrepareChildControlsDuringPreInit(this Page page)
    {
        // Walk up the master page chain and tickle the getter on each one
        MasterPage master = page.Master;
        while (master != null) master = master.Master;
    }
}

This adds an extension method to the Page class, which then allows us to write code like the following:

protected override void OnPreInit(EventArgs e)
{
    this.PrepareChildControlsDuringPreInit();

    MyCustomDropDown.MyProperty = "my value";

    base.OnPreInit(e);
}

Without the call to the extension method, we would have received a NullReferenceException when trying to set the property value on the MyCustomDropDown control.

You now have one less excuse for preparing your controls during the Load event.

(If you don’t get the title reference, Wikipedia can explain. A more direct title could be: Understanding and Respecting the ASP.NET Page Lifecycle.)

Page lifecycle in ASP.NET is a finicky and rarely understood beast. Unfortunately, it’s something that we all need to get a handle on.

A common mishap that I see is code like this:

protected void Page_Load(object sender, EventArgs e)
{
    if (!Page.IsPostBack)
    {
        AddressCountryDropDown.DataSource = CountriesList;
        AddressCountryDropDown.DataBind();
    }
}

The problem here is that we’re clogging our page’s view state. Think of view state as one of a page’s core arteries, then think of data like cholesterol. A little bit is all right, but too much is crippling.

To understand the problem, lets investigate the lifecycle that’s in play here:

1. The Page.Init event is being fired, however we are not subscribed to that.
2. Immediately after the Init event has fired, view state starts tracking. This means that any changes me make from now on will be saved down to the browser and re-uploaded on the next post back.
3. The Page.Load event is being fired in which we are setting the contents of the drop down list. Because we are doing this after the view state has started tracking, every single entry in the drop down is being written to both the HTML and the view state.

There’s yet another problem here as well. By the time the Page.Load event is fired, all of the post back data has been loaded and processed.

To investigate the second problem, let’s investigate the lifecycle that’s in play during a post back of this same page:

1. The user triggers the post back from their browser and all of the post back data and view state is uploaded to the server.
2. The Page.Init event is fired, however we are not subscribed to that.
3. Immediately after the Init event has fired, view state starts tracking. This means that any changes me make from now on will be saved down to the browser and re-uploaded on the next post back.
4. The view state data is loaded for all controls. For our drop down list example, this means the Items collection is refilled using the view state that was uploaded from the browser.
5. Post back data is processed. In our example, this means the selected item is set on the drop down list.
6. The Page.Load event is fired however nothing happens because the developer is checking the Page.IsPostBack property. Usually, they say this is a “performance improvement” however it is also required in this scenario otherwise we would lose the selected item when we rebound the list.
7. The contents of the drop down list are once again written to both the HTML and the view state.

How do we do this better? Removing the IsPostBack check and placing the binding code into the Init event is all we need to do:

protected override void OnInit(EventArgs e)
{
    AddressCountryDropDown.DataSource = CountriesList;
    AddressCountryDropDown.DataBind();

    base.OnInit(e);
}

What does this achieve?

• We are filling the contents of the drop down before the Init event is fired; therefore a redundant copy of its contents is not written to the view state.
• We are filling the contents of the drop down before the postback data is processed, so our item selection is successfully loaded without it being overridden later.
• We have significantly reduced the size of the page’s view state.

This simple change is something that all ASP.NET developers need to be aware of. Unfortunately so many developers jumped in and wrote their first ASP.NET page using the Page_Load event (including myself). I think this is largely because it’s the one and only event handler presented to us when we create a new ASPX page in Visual Studio. While this makes the platform appear to work straight away, it produces appalling results.

SQL 2005 and 2008 both have what’s referred to as an encryption hierarchy. The details of this are beyond the scope of this post, but in essence: we encrypt our data using a key. We need to protect our key somehow, and we don’t want to litter our stored procs with key passwords, so we use a certificate. We then protect the certificate with a database master key. This is in turn protected by the service master key which is finally protected by DPAPI, an operating system provided store.

The keys and certificates are stored within the database itself, but when we move the database to another server they can’t be accessed. This is because the new server doesn’t know how to decrypt the database master key and in turn can’t decrypt the keys and certificates we need to use.

Usually…

The database master key is always protected by a password. You would have had to provide this when you first created the key:

CREATE MASTER KEY ENCRYPTION BY PASSWORD = 'mypassword'

When you create the key like this, SQL generates the key value and encrypts it with the supplied password before storing it. It also makes a second copy which is encrypted using the service master key, and this is the copy that is normally used. When you move your database to another server, the service master key protected copy can’t be loaded but the password protected copy can be.

 OPEN MASTER KEY DECRYPTION BY PASSWORD = 'mypassword'

With the key now decrypted and loaded into memory, we can ask the new server to make a copy that is protected using the service master key.

 ALTER MASTER KEY ADD ENCRYPTION BY SERVICE MASTER KEY

Finally, we close the key to take it out of memory.

 CLOSE MASTER KEY

Voila, you can now access your keys and certificates on the new server with automatic key management (that is, with SQL automatically opening and closing the database master key for you as required).

Read-only databases

The approach described above is dependent upon the database being in a writable state as it makes modifications of the database master key itself. What happens when we want to work with a read-only version of a database such as a snapshot or a mirror?

With automatic key management, SQL Server will first attempt to decrypt the database master key using the service master key. In a read-only database, we are unable to create a copy of the key that is protected in this way.

After attempting that, SQL Server will look in the credential store (sys.credentials) for any credentials related to the master key. It will attempt each credential it finds.

Adding our credential to the store is easy:

USE [master]
GO

EXEC sp_control_dbmasterkey_password
    @db_name   = N'mydatabase',
    @password  = N'mypassword',
    @action    = N'add';

You can see evidence of the new credential in both sys.master_key_passwords and sys.credentials:

SELECT  d.name as database_name,
        c.*,
        mkp.family_guid

FROM    master.sys.credentials c

        INNER JOIN master.sys.master_key_passwords mkp
            ON c.credential_id = mkp.credential_id

        INNER JOIN master.sys.database_recovery_status drs
            ON mkp.family_guid = drs.family_guid

        INNER JOIN master.sys.databases d
            ON drs.database_id = d.database_id

Voila, just like above, you can now access your keys and certificates on the new server with automatic key management (that is, with SQL automatically opening and closing the database master key for you as required).

You might also notice that the sys.master_key_passwords view ties a master key password to a family id as opposed to a database id. A family id is assigned when a database is first created and it stays the same even if the database is detached, moved, reattached, mirrored, etc. As a result of this behavior, you could have multiple databases on the one server that share the same family id. In contrast, a database id is created every time a database is attached and is therefore unique for every database instance on the server. In the context of our master keys, the outcome is that adding a credential against one database will actually add it for all of the databases which have come from the same original instance. Even if you detach and reattach your databases, or drop and restart replication, the credential will still be kept in the store and work with the new database instance. Attaching any new instances which share the same family id will also automatically inherit the credential. (Unless of course you change the database master key password in which case the credential will still be attempted but will fail and cause the next one to be attempted instead.)