These server issues I've had the past few days have gotten me thinking about resurrecting and completely rewriting eSonar, the server monitoring system that I was working on that ultimately was the catalyst to quitting my full time job almost 10 years ago to launch into the risky world of "being your own boss."
Check it out the old eSonar site on the Web Archive (I can't believe they actually archived the flash animation - that's the best part). Those of you who are familiar with the early versions of DKPSystem.com and my other work will definitely recognize the layout.
At the time, Perl seemed to me the appropriate technology choice due to CPAN and it's huge number of modules for talking many different protocols. Also, I had heard of Perl, which helped. So I learned Perl to build this system and bought a handful of used Pentium 3 Servers on eBay which I was going to scatter around the country in different data centers. I paid a then-16-year-old neighbor a couple hundred bucks to make me a demonstration flash animation for the homepage. I learned online payment processing, a handful of internet protocols, and how to send SMS messages.
But despite the fact that it was basically functional (what would in startup parlance be considered an MVP - Minimum Viable Product), I never quite finished the product. Being a bootstrapped 20-something I couldn't afford the $300/mo or whatever it would have cost to do this at the time, and I didn't realize that one should have the mentality of "just effing release it already" and so I resorted to contract programming, and eSonar ran on a single server for a few years, monitoring only my own systems.
Eventually, World of Warcraft happened and I got sucked in to it's tractor beam, but from that unhealthy obsession, came DKPSystem.com for WoW and other MMO guilds, which launched in May of 2006 on the same machine I had already in the data center that was currently the home of eSonar.
While that old server is long retired from years of upgrading and replacing hardware (going from one crappy server to up to a pile of quad core servers then to a hybrid of a few physical servers + a handful of VPSes), last night's maintenance until the wee hours of the morning has gotten me thinking that eSonar could have some value even still in a world where there are many server monitoring systems.
My current vision would be that of a hybrid between that of a home-grown solution (where homegrown solutions typically provide basic tools like filesystem monitoring, response time checking, database replication monitoring, all that) and a hosted solution (which typically just checks pings, but in this case would verify false positives, and have differing levels of alerts). The hosted part would be in the offering of the distributed architecture by using ("leveraging"? lol business-speak) the easy availablilty of cheap VPSes around the world, and notifying the customers that not only is there something worth notifying, but that it is actually verified, and not just something like an coding or routing error on one of my nodes.
I'd deploy an open source monitoring client for performing those tests and then sending the results to my master server (or another server, if they so choose). I'd also consider offering a simple open source master server for clients to run on their own machines. And if they wanted to take advantage of my system without the burden of paying their sysadmins to deploy the system globally, they could pay me a monthly fee.
This mentality definitely fits into my world view, where I like writing and managing open source software, but I also like running systems that, you know, pay me.
Further, since I'm already considering writing something like this myself as I'm starting to get sick of the false alarms from my monitoring services. Apparently 302 redirects apparently can trigger alerts with my current provider - I mean really? - as well as a slow response, which doesn't mean "down", just "slow": I'd like to see how long the request took to timeout. I don't see it being that difficult turning this into a general purpose product (especially if approached as such from the beginning rather than trying to re-architect an already-built homegrown solution). Either way, this would definitely help me out with my existing systems: DKPSystem, BracketPal, as well as my clients' sites.
And of course, since Erlang is basically built for this kind of distributed, concurrent system, I'd build it in 100% Erlang using Nitrogen for the web site, and straight Erlang for the backend notification servers.
Or maybe these are just the ramblings a sleep deprived programmer/father who was up until 4:30am working on his servers only to get up at 8:00am to take care of his kid who can't be in daycare because he had a fever yesterday. Maybe this is just another case of NIH Syndrome, and I should just use something else.