HOWTO: Source Control for VMWare Hosted Visual Studio on OSX

I've been wrestling with how to get some notion of backed-up source control for my Visual Studio projects that don't warrant going into the Esendex Team Foundation Server.

Didn't want them to be open source so code.google.com, codeplex.com, etc were all out and I didn't want to pay any money for a secure service.

Here's what I came up with.

Firstly I discovered that OSX ships with and SVN server, very handy. I followed this link to set it up: http://www.martellventures.com/blog/files/544f828ced9567b46fc92006af524d16-24.php.

With SVN running locally on the Mac, I could then rely Time Machine to make sure my source code was backed up.

Then downloaded the Ankh SVN plugin for Visual Studio so I could use and SVN repository.

So far so good, but then how to access the SVN repository. My virtual machine connects to the network independently so I had no consistent way of knowing the IP or hostname that I would need to browse to.

The answer: Bonjour for Windows. Installed that on the Windows Virtual Machine and bingo I can use the advertised machine name of my mac with the .local suffix to access the SVN, no matter what network I'm on.

Measure, Measure, Measure

The focus of Web 2.0 this year was the power of less. While there was definitely a feeling that nothing really new was happening in the industry, a couple of talks I attended focusing on optimisation of your business were probably the highlights of the event for me.

Eric Ries (blog: Startup Lessons Learned) was almost evangelical in his talk: The Lean Startup.

He espoused the virtues of measurement, feedback loops and iterative, agile development. All music to my ears and things we're doing and continue to improve on at Esendex.

What really got me thinking though was the notion of A/B testing of application features and measure how that translates into improving your businesses KPIs (key performance indicators).

To date I had considered A/B testing to be the domain of web sites, try different graphics, messages, calls to action, processes, etc and measure the goal completion percentages.

We're deep into building the new version of our new application at Esendex and we're making important decisions about the functionality and features we're going to make available.

The problem is those decisions are pretty much based on opinion rather than any objective measure. While we think they're a good idea, it remains to be seen whether our customers find them useful.

There is a certain amount of inspiration required and going with our gut instinct. Innovation generally involves a step change after all. As Henry Ford famously said:

If I had asked people what they wanted, they would have said faster horses.

But, if we take the leap and put the feature out there, wouldn't it be good to know if it gave the desired results. We need to be able to measure a) whether people use it and b) whether not it had the desired effect.

Outcomes in the web analytics world are generally fairly well defined. A site visitor bought something, registered on the site or, in our case, signed up for a trial.

The desired outcome of introducing a feature could be more grey. Outcomes are very likely not to contribute directly to one of our KPIs. The path to KPI improvement will probably be circuitous and require a degree of assumption but at each step we should be testing the hypothesis.

Often we'll be introducing a feature because we believe that it will improve on of our KPIs but that could just be by offering something other services don't, encouraging people to sign up with us rather than someone else.

In this case we will need to measure an indirect outcome until such time as we enough of a population to then measure more directly against our KPIs.

Very much more art than science.

I'm very much working this through at the moment. We're adding feature measurement into the beta product we're launching in May and I'm looking forward to using this process to improve the product in the direction our customers want.

I'll report back.

All Hail the Uber-Geek

One of the highlights of Web 2.0 was seeing Steve Souders presentation: Even Faster Web Sites.

His book High Performance Web Sites, is a well thumbed tome at Esendex and YSlow is required add-on to Firefox on our developer's machines.

Steve is an uber-geek. He has a passion for web-site performance like few others. He tirelessly investigates, tests, hypothesises, tests those hypotheses, and, importantly reports back to the rest of us.

His passion, and most importantly, the fact that he shares his results with all of us is something we should all be thankful for.

As developers we get adulation for building faster, more responsive and thus more usable web sites.

As web site users, we get a rich, responsive experience that does what we want quickly.

So all hail uber-geek and thank you to you all.

It's all about the resources

Our new web application, codename Doyle, is being built on the ASP.NET MVC framework. The primary driver for this was enabing Test Driven Development for web apps. Increasingly frustrated with the brittleness of the traditional approach this represents a fantastic architecture for building robust web apps.

The big challenge however has been conceptual. Patterns the team are comfortable with, have turned out to be unsuited to this approach. Discussing one such question at the whiteboard yesterday, it occurred to me that the issue we were having was we thinking in terms of method based architectures rather than resource based.

To date, our approach has been to have thin Data Objects that are passed to facades and sub systems that perform operations on them. We're a messaging shop so I guess that's where it came from. A message gets passed around, operations are performed on it, decisions are made about it.

The term Controller implied a co-ordination role with knowledge of controlling one or more sub-systems. However MVC tells us that is bad, Controllers should be thin and Models should be fat.

A good example is our new send message page.

The first pass involved the controller iterating through the recipients, constructing message objects and passing them to the Esendex ReST API SMS Message Dispatcher via our C# SDK.

Very quickly the controller got fat, validation, error condition handling, suddenly it was squatting behind the view like a Sumo Wrestler tucking into his 7th chicken of the day.

The light-bulb moment was the realisation that rather than thinking about sending a message, we were actually sending a batch of messages. Enter the MessageBatch resource.

All the controller has to do is load up the MessageBatch resource with the parameters (recipients, body, etc) and call Send. This returns either true or false, successful or failed.

The controller then just has to make one decision go to one page if successful, or another if not. Safe in the knowledge that the MessageBatch object will be duly populated with an error/validation info required for the view to render and give the user the option of what to do next.

Yes this has just moved the logic to another part of the system but being in the Model layer gives the opportunity to refactor, inherit, and most importantly, mask the complexities of sending a message batch from the front end. We're comfortable unit testing and doing it this way forces as much logic as possible into units.

So, if you're making the transition to MVC then consider looking at some of the thinking behind ReST web services (I read: RESTful Web Services as a good primer). I've come to the conclusion that ReST and MVC are intrinsically linked.

Photo credit: alarch

Converting NUnit Project to MS Test Project

Discovered this today while converting some unit tests. Saved me loads of time: http://social.msdn.microsoft.com/forums/en-US/vststest/thread/2192fac0-0de7-45c6-872a-2be49e95ce2c/

National Holiday Calendars

Found this great resource for my absence application.

http://www.mozilla.org/projects/calendar/holidays.html

Outputting a calendar without a gem

Had some fun yesterday with an application I've been working on to track employee absences. One of the requirements we identified was to publish a calendar that could be viewed in Outlook, iCal, etc, showing who was off and when.

As I was using Ruby and Rails, my first thought was to use a gem. I found this one iCalendar — Internet calendaring, Ruby style and was soon publishing the calendar. Marvelous, well not quite.

After giving up trying to setup an internal Linux server to host this application I've been trying Mor.ph. As this was a test project I didn't want to take resources away from the our Ops team, big mistake.

Morph seems really easy to use, I love the batched release process with capistrano, however there was a problem. They don't have the iCalendar gem.

Again this is as much a test project as anything so it seemed over the top to try and get them to host it. It occurred to me that I could use an ActionView template to achieve the same thing.

Here it is:

BEGIN:VCALENDAR
VERSION:2.0
CALSCALE:GREGORIAN
<% @absences.each do |absence| %>BEGIN:VEVENT
DTSTART:<%= absence.starts_at.to_datetime.strftime '%Y%m%dT%H%M%S' %>
DTEND:<%= absence.ends_at.to_datetime.strftime '%Y%m%dT%H%M%S' %>
SUMMARY:<%= absence.employee.name + ' - ' + absence.absence_type %>
END:VEVENT
<% end %>END:VCALENDAR

The code to render the view is dead simple:

@absences = Absence.find(:all)
headers["Content-Type"] = "text/calendar"
render

And being Ruby on Rails, I have a nice functional test that confirms the output.

Only tinge to the cloud is that I'm having a bit of a problem getting the timezone to appear in a format that Outlook recognises, any thoughts welcome.

Can Software as a Service go mobile?

The opportunity for Mobile SaaS is huge, as computing moves from the desktop to mobile devices, it is a natural evolution that software delivered to the desktop follows that migration.

One of the key tenets of SaaS is that the software can be delivered anywhere, as needed, but that anywhere comes with a caveat that is very significant in the mobile world.

The assumption made is that the desktop is always connected to the Internet, usually via a broadband connection. This assumption cannot be made about mobile devices.

To be truly successful, mobile solutions have to adapt to changes in connectivity. The key to their success is handling this gracefully.

As an example, consider mobile email, the most mature mobile SaaS offering. My BlackBerry copes brilliantly with being offline, allowing me to send, reply, sort and organise my emails wherever I am, coverage or not, seamlessly synching when connectivity is restored.

The iPhone Exchange integration however is far less robust, complaining when coverage is lost and regularly restoring emails I’ve deleted and sending emails multiple times.

So the implementation is one thing, but the architecture these applications rely on, conflicts with traditional SaaS implementations where the application is delivered page by page via a web browser. If there is no connectivity then client software is required on the device which then has to be kept up to date.

This is where the iPhone wins hands down. The iTunes Application Store seamlessly ensures software is up to date whenever the device is synched with the owners PC. Salesforce.com, the self-proclaimed poster-boys of the SaaS revolution have an iPhone app sitting alongside their BlackBerry version that enables offline access to your corporate sales data.

So implementation is key and working offline is critical but this software based solution brings another problem, multiple platform support.

As soon you start shipping software you have to cope with the plethora of devices and platforms that are out there. The beauty of a pure web-based model is that this is irrelevant, however offline working precludes this. So application developers have to focus on a few key handsets that they believe will be most used by the target market for their application, which probably means Nokia S60.

Google have produced an alternative with a technology called Google Gears. This allows a web browser to store information on the PC for later synchronisation, even better news is that they have a version for mobile devices. The bad news is that this only works on Windows Mobile.

However, this may not be all bad. Microsoft have done a very good job of making Windows Mobile the platform of choice for organisations wanting to extend their business apps into the mobile space. They provide a rich and mature set of development tools based around their .NET framework which enable developers to use the same languages and tools that the use to develop desktop and web applications to develop mobile applications. The same cannot be said for the iPhone and the S60 platforms.

Windows Mobile itself seems to be improving with every release and the manufacturers are producing more and more desirable handsets for it.

Another interesting development to match is Microsoft’s Live Mesh, a technology that seamlessly synchronises your data across all your devices. This can be pictures, music, video etc, but there is no reason not to include business data in this mix. Imagine working on a business workflow app on your desktop, stepping away from your desk and being able to continue on your mobile device at exactly the point you left off.

It’s for these reasons that I believe that Windows Mobile represents the best bet of the platforms to support Mobile SaaS, especially for business apps.

Recruitment Companies Feeling the Pinch

This popped into my inbox today

With the credit crunch looming above us a number of my clients are becoming increasingly reluctant to part with money for contractors in case they pick the wrong person, despite having urgent projects that need to go ahead.

With this in mind for the month of August I am offering 2 days free trial on any contractor placed at Esendex. Basically this service enables you, the client, to see a contractors work at first hand and also see how they fit in with your team. If you feel they are good for the project we can just roll the contract on (minimum 3 months), if not then I will take them off site and you wont have to pay a penny!

Contractors on free trial, whatever next?

Judging is complete

Finished the judging for the Esendex Developer Challenge today. A tricky exercise, the entries were diverse, in the end however we were all in agreement.

The winners are now safely tucked away into in a metaphorical gold envelope. As soon as I've spoken to them, we'll announce it to the world.

Security in anonymity

Amazon have had some recent outages (Bots to blame for Amazon.com outages? | The Register) which raise a spectre of doubt for me over the viability of cloud computing services.

These services allow smaller organisations to benefit from large investments in computing power made by a small number of organisations, Amazon, Google, etc and buy IT infrastructure as a service.

The concern for me is that they provide a very, very big and obvious target for people of nefarious intent to aim at. Global brands like Amazon represent a prize for hackers and it would seem if one goes down, everyone goes down. Their being S3 services were affected as well, taking down all customers services that rely on them.

Doing it for yourself, on your own servers gives you the protection of the crowd in much the same way as being in a shoal of fish provides protection to the individual members from predators. Of course, you have to do a certain amount of self protection but being far less likely to be a target makes that a lot cheaper.

The counter argument is that these large cloud services organisations have the resources to hire the best defensive minds in the business to protect their investment, a level you couldn't possible afford or justify on your own. The problem is that these people are defending against the best attacking minds in the business because they're protecting the biggest prize, it's a classic arms race.

Another argument is that these services allow you to scale your service rapidly in the face of unprecedented demand. You never know, your service might become de rigueur and in the Internet world that means millions of hits.'You only get one chance'.

Entrepreneurs lap this one up, being the eternal optimists that we have to be to survive the start-up trials, of course my service is 'the one' I'm just waiting for the market to realise and then I'll be ready. Looking at it a little more pragmatically, this is a million to one shot.

What's more likely, should your business succeed, is that growth may well accelerate but not more than is copable with. The trick is to architect your service so you can rapidly scale it, then it just becomes an issue of cash hardware. If you've got your business model straight then this shouldn't be a problem.

So, home grown for me. It's not hard or expensive these days with so many open source services and stiff competition in the ISP market; and I haven't even got onto the perils of locking your pride and joy to a proprietary app hosting architecture.

iPhone goes for the enterprise, BlackBerry fight back

It's an iPhone with a QWERTY keyboard!

BlackBerry 9000 in the wild

OK so no touch screen interface but this promises to be a serious bit of kit that will cement BlackBerries in the enterprise. Must check out the BlackBerry development kit so how that stacks up against the iPhone SDK that Jonathan is wrestling with at the moment.

Code Comments are they really pungent?

XP tells us that comments are a bad coding smell indicating that some functionality should be refactored out into a method. I've never been totally comfortable with this and Matt Stephens as posted a very good article as to why.

Comment judiciously, refactor if needed, avoid the 'f' word

Are Symbian and J2ME dead?

The launch of the iPhone and Android SDKs represent a huge leap forward in the opportunities for mobile device application development. The question for me is how do the current incumbents, Symbian and J2ME, react.

I've talked before about my continued disappointment with mobile applications (Mobile Phone Applications, when will they ever take off?). These experiences have been very much routed in the pre iPhone/Android era.

In the Symbian/J2ME world, applications are poorly integrated with the phone, hidden by layers of menus and generally underwhelming. The world according to iPhone/Android has the opportunity for the application to be front and centre, and in Android's case, closely integrated with the phone functions.

The second half of 2008 promises to be a fantastically exciting time for mobile device users. The iPhone has redefined the mobile device and the version 2.0 firmware with its enterprise features will deliver a device that kicks some serious Windows Mobile butt. Can't believe they're on version 6 and it's still an overweight, clunky to use, buggy monstrosity.

Now apps for the iPhone are not going to be without issues. The big one for me is the inability/refusal to allow background processes to run. It stops all sorts of useful features like background data updates and push facilities that would really make it a BlackBerry killer. That said, it's an amazing user interface experience, and as it continues to demonstrate, even without 3G data transfer speeds, iPhones are blazing a trail in mobile web usage. Though I do suspect most of this is on home WiFi networks.

The killer feature for me is the remote update capabilities that are built into the OS. The link to iTunes is critical to the devices operation and this allows Apple to push new updates, including strategy changes, at will. I can't remember the last time I updated the firmware on a Symbian phone.

Android is taking a very different line, an almost completely open platform. Full access to all the device's functions. Now this is really a desktop-like development experience. No-holds barred, access to everything, only constrained by your imagination type environment. Only one thing is missing, handsets. It seems Samsung and HTC are in a race to deliver the first but apparently this won't be until the end of this year! So develop what you like, it 'aint going to be on a device until 2009.

There seems to be a huge opportunity for Nokia to leverage their number 1, by a long way, position in the world handset market and bring something truly powerful and pervasive to the market. Unfortunately it seemed their response was to make it even harder to get Symbian apps on phones. Why Symbian Signed must die gives a good account.

Symbian can push out news stories about how many handsets ship with Symbian, 77.3m Symbian smartphones shipped in 2007 but for me that's 77m handsets that aren't going to run Symbian apps, it's just too difficult for people, especially enterprises.

Apple are treading a very delicate path with the iPhone, keep it exclusive but get it adopted by enterprises. Historically their products haven't been adopted by this segment. If their aspirations are to get the iPhone into the hands of key personnel and in turn those personnel buy a Mac for home instead of a Windows PC then that's probably a huge result.

I'm pretty neutral on Android. Yes it's flexible, yes it's open, but it's a while before before people can actually use it and who knows what will happen then. I do wonder whether there is a bigger play here involving GrandCentral a mobile device, closely integrated with that service could be very disruptive indeed.

Symbian, IMHO, have lost the plot already. They have huge penetration but low usage and no established update process. It's going to take something dramatic from them to enable them to maintain their dominance of the high-end mobile device market.

Putting SLAP to use - Dynamic Config

Now I've got the specifications of the SLAP protocol out of the way, I'd like to describe how we're using it and the benefits we are seeing. The first being the dynamic configuration of service end points.

In a service like ours we have a series of nodes that process messages and apply rules to manipulate and route them. If we want to change the location for a service, we have to first notify all the upstream services of the change of location.

In a planned scenario this can be laborious. All services that may make use of the service have to be reconfigured, often requiring a restart. It is possible to have these services monitor a config file for changes to avoid a restart but this doesn't remove the need to manually go through each service config and make the necessary changes. This becomes increasingly complex in a distributed system spanning many machines and networks.

The unplanned scenario is even less desirable. There are many reasons why a service can stop and this will generally be at the most inopportune times. Having to manually reconfigure services in response to a failure is not acceptable in a high availability scenario like ours.

The other problem with the unplanned scenario is that the client service has to deal with the unavailability at the point it's attempting to consume the service. Say for example this is a network service, it could be waiting for a network timeout before it recognises that the service is not accepting requests. It then has to fire-fight, cleaning up because it thinks the service is unavailable.

Far better for it to be told categorically that the service is unavailable, it can then queue requests or whatever has been coded as appropriate, while it waits to be informed that the service is operational again.

It is possible to use separate load-balancers to handle this kind of outage, but they cost money, need configuring, draw power and the solution the bring is by no means dynamic. I'll actually discuss load-balancing using SLAP in a subsequent post.

In the SLAP world, services are not configured to use fix endpoints like sip:ems3.prod1.esendex.com:8067 but rather are bound to well known service names eg: slap:smsrouter. The actual endpoints to the service are advertised in ANNOUNCE in order that clients can maintain a record of the current state of the services they need to consume.

If a client service wants to make use of an smsrouter service, it checks the state in it's local service state table before sending the request to the correct service URI. This information is kept up to date by the service. Most will announce their state on a periodic basis but I'd also consider it good practice for the service to send an ANNOUNCE when it's state changes, perhaps when it's under load or is shutting dowm, to ensure all clients are kept up to date.

The development team at Esendex have also found it very useful when building and debugging services as Jonathan describes in SLAP, my service’s up!. This was an unforeseen benefit but another one that has cut out a lot of the hassle with debugging services. The guys can step through the code, find the issue, write the unit test and rebuild very quickly which get's us to market far quicker.

More on load balancing soon.

A very RESTful experience

In May this year, a group of us (parents, friends and teachers) from my sons' school are cycling from Nottingham to Paris in order to raise money for the school. My role in this exercise, in addition to cycling, is to build and manage a website (www.r4rh.org) to promote the event.

Part of that web site includes retrieving a set of latest photos from Flickr tagged r4rh or RideForRoundHill. I've built the site using ASP.NET so naturally downloaded the Flickr .Net API Kit. My needs were very simple, just searching for a list of photos, and it worked brillantly until I published the site.

For some reason, I don't know why, the reference to the FlickrNet.dll didn't see, to carry across. Initial investigations didn't uncover a solution, one of the problems I find with doing this kind of thing these days is that my integration problem solving skills have got very rusty, so I browsed deeper into the Flickr APIs.

I've been reading RESTful Web Services by Leonard Richardson and Sam Ruby recently and the ethos of REST style web services has been intriging me greatly so I decided to have a go at consuming the Flickr REST API.

Boy it was easy.

• Do GET from URL with API key and search parameters
• Receive XML document
• Parse XML document to get list of photos
• Build list of images to render on web page.

Now if you end up reading Leonard and Sam's book you'll discover that in their view, Flickr's REST API is not truly REST but actually a REST/RPC hybrid. He does go to great length to make the case for pure REST services but in my view, from a usability point of view, this ticked all of my boxes.

• Simple to make the API call: just construct a URI
• Simple to interpret the results: wide support among development platforms for XML parsing
• Loosely coupled: Flickr can add new data to the output without breaking my code

It's the above criteria that have been used to design the new Esendex API that will be coming to the Internet very soon. We are going REST.

The first release will be available in March to support an exciting new service we are bringing to market. This will be a sub 1.0 version, beta subset of the full API which we will augment to support the full feature set over the coming months.

We will of course formally announce the release on the Esendex Developer web site, and we will be maintaining support for our existing API.

Committing to this approach has also opened up some fascinating possibilities for our applications as well how we can build composite applications that combine all sorts of other services with our own. It's going to be a very RESTful year.

SLAP - LOCATE Message

The LOCATE message is used by service clients to discover the existence and state of services on the network. Sent as a UDP broadcast it requests information about a specific service.

Upon receipt of a LOCATE, the relevant service is expected to respond immediately by broadcasting an ANNOUNCE message to enable the client to understand the current state of the service.

LOCATE [service name]

Where

service name

A name for the service, well known within the system domain

A typical LOCATE message would look like this.

LOCATE SMSCProxy 

Use of the LOCATE message is optional by the client if the intervals between ANNOUNCE broadcasts by the service are sufficiently small. In many situations though, this delay is unacceptable and the client will initiate the ANNOUNCE broadcast by sending a LOCATE message.

Having both LOCATE and ANNOUNCE messages also has the side benefit of allowing the protocol to have a comedy name.

Web 2.0 Expo, swapping Berlin for the Bay Area

I've just been booking travel and accomodation for Web 2.0 Expo in San Francisco. I went to the European version last year in Berlin and it was a bit of an eye opener for me. The venue was pretty poor, both location and internal, but the subjects covered struck many chords with me and work I'm engaged in.

I've spent the last few years focussing on the mobile sector, eschewing Internet and developer conferences in favour of mobile specific affairs like Mobile World Congress and Global Messaging Congress. While these conferences have been fascinating and tedious to various degrees they have always been successful in giving me time to reflect and cogitate on our services, how they fit into the mobile ecosystem and what they should become in future.

Web 2.0 Expo in Berlin made me realise that there is a lot of amazing work being carried by the darlings of the Web 2.0 world that have many parallels with the challenges we face at Esendex. By immersing myself back in to the web world as well, there was a lot that I could learn.

As we continue to grow at the phenomenal rate we have been and evolve our services we come across all sorts of scaling and performance challenges. These definitely rank in the 'nice to have' category of problem but they're no less challenging. Hearing about lessons learned by the likes of Flickr et al. can only help us decide how we continue to stay a few steps ahead of demand in the future.

These new services have also upped the ante when it comes to what an Internet service should be able to provide. Their use of AJAX and REST architectures have provided a rich user experience far beyond where, to be brutally honest, we are right at the moment.

That is changing, we have some exciting new services coming to our customers over the first half of this year that bring us bang up to date. We've spent so much time focusing on making sure our messages get through in a timely fashion we've probably taken our eye off the ball on the front end usability. So there has been a big focus in the development team on making our services richer and easier to use. Obviously I'll fill you in as soon as I can.

I suspect that the Berlin conference did suffer from being smaller than it's American parent, so I'm looking forward to seeing how it works on a bigger scale. There are also worse places to go than San Francisco, not least to get my fix of unbridled, Bay-Side, can-be-done-ness.

SLAP - ANNOUNCE Message

The ANNOUNCE message is the core message in SLAP. Sent as a UDP broadcast, it announces information about a service to the network. It's purpose is to inform interested clients about the current state of the service.

ANNOUNCE [service name] [state (Green|Amber|Red|Blue)] [service uri]
<[optional attributes (name=value)]>

Where

service name	A name for the service, well known within the system domain
state	The current state of the service as represented by one of 4 values Green, Amber, Red or Blue, see below
service uri	Unique network address for the service. Can be any format that the client would recognise.
optional attributes	An optional series of attribute value pairs seperated with a new line.

Service States

Service states are used in SLAP to give a simple indication as to the service state.

State	Description
Green	Service is fully operational and available
Amber	Service is operational but under load
Red	Service is not operational
Blue	Service state is unknown

If more information is required, then the service would include this with the optional attirbutes of the message. The content of these atttributes is not defined and is open to individual system implementations.

For example, information on current service performance (transations per second, queue size, memory utilisation, etc) could be included to allow the client to make utilisation decisions.

A typical ANNOUNCE message would look like this.

ANNOUNCE SMSCProxy Green node1.company.com:1234
Route=Operator1
Binds=3

Generally these messages would be broadcast at regular intervals to keep clients updated with service information. There are times when a client needs to actively interrogate the state of a service, for instance on startup, for which the LOCATE message is defined. More of that in the next post on SLAP.

SLAP, an introduction

Part of the reason I've been so quite on the blogging front of late is that I've got carried away with a bit of development spiking. Nothing that goes straight into production I hasten to add. I'm too far away from our development processes to be trusted with access to the live source tree, and anyway Nicholas (Development Manager) probably wouldn't let me.

I am in the enviable position of being able to create proptotypes, test things out and hand them over to the professionals to make them a reality. It's one of these projects that became SLAP (Service Location and Announcement Protocol).

One of the key technical challenges facing an organisation like ours is scaling our applications out to meet demand while at the same time keeping everything highly available.

Our architecure comprises a series of software agents that pass messages between each other routing and manipulating traffic based on a series of rules. For example

• A outbound bulk SMS will head straight to an SMS Router for it to decide based on the state of a set of Mobile Carrier connections which is the best current route.
• A premimum SMS message will be passed to a Subscription checking agent to confirm that the subscriber is valid, and then possibly onto a network lookup agent to resolve the destination network before heading onto the SMS Router for onward submission via the correct network for billing purposes.

As I'm sure you can imagine running multiple instances of these agents across multiple servers and have them know the state of the agents around them so messages get passed around reliably is no mean feat. Should an Mobile Operator connection become unavailable or we lose a server, the agents should be able to adapt to that.

One option is to use hardware load balancers, as we do in front of our web servers, but this quickly becomes very expensive and very costly to maintain as you end up with multiple layers of load balancing. Also it's very hard to dynamically configure the network of agents to cope with peak periods of traffic.

After reading Scalable Internet Architectures I was inspired to come up with a simpler, and cheaper, solution. SLAP was born which is an alternative, software approach that is deliverying huge benefits to us already.

Nodes broadcast one of 2 message types: ANNOUNCE and LOCATE over UDP which are subscribed to by other nodes on the network. These nodes can be application monitoring agents or clients wishing to consume services offered by the node.

It's simplicity hides a power that has allowed us to architect our system to self configure and repair itself in the event of failures. In subsequent posts I'll describe the specifics of the protocol and our describe our experiences using it.