Reporting Passive Failures in Enterprise Systems

I promised that I would eventually get around to writing down my thoughts on reporting passive failures, but the more I think about it, the less there is to say and to think about.

When I say passive failure, I'm usually thinking about something that can only be detected over time. Because of this I assume the presence of some sort of persistence mechanism, like a database. I'm also assuming that the failure is related to the information making its way into that database.

..so boring.

1. If at all possible turn your passive failure into an active one. Suppose we have a batch processing system. The client pushes a batch of whatever to a server. If there's no whatever there's no push. Rather than trying to determine how long to go without a push to report a failure, make a way for the client to inform the server that there is nothing to push.

2. report a probability that the system has failed. The longer the system goes without doing something, the higher the probability that a failure has occurred. This will likely lead to having to check that your system is working, when it was working the whole time, but that's better than the embarrassing realisation that your system hasn't been working for the past month.

There has to be better ways to do this than what I've listed, but I got what I really wanted out of this, I figured out something I need to do at work tomorrow.

Jump shot in Wii Play billiards

"Okay now you have to blog about that," were the words of a wide-eyed Gina the first time I successfully sunk a ball using a jump shot.
It all started on a cool Spring afternoon. It was Good Friday, and there was nothing to do in a small town like Truro. We had some income tax coming back, so money wasn't as tight as it usually was.
Gina was at the computer when she turns to me with a look in her eye, the kind of look a person gets when they have an idea so fitting for the ambient mood that other people feel as though their collective minds have been channeled. This was exactly how I felt.
"You know how it's our anniversary tomorrow?" she started.
"Yah?" I said, in anticipation of finding out what I didn't know what was on my mind.
"We should get ourselves Wii Play."
"I'm getting my shoes on."

I get home with the new game, pop it in, and loose myself in the simple monotony and bright colours of the games. It would have lost it's appeal after mere minutes had I not had someone to share in my digital mirage.
Billiards became an instant favourite. With Gina and I being in our twenties, and it being billiards, it was a perfect match.
The physics were fairly realistic, and we began to expect more from them, including backspin. It did not disappoint, until it came to the jump shot. You can only play 9-ball for so long before the target ball has a solid wall of untouchable balls between it and the cue ball, and when it happens, you know exactly what needs to be done.
We tried pulling up at the end of the stroke, but the game was pretty tight lipped about how far off we were with our technique. We searched far and wide through the first 5 results on google, but with no luck. Our billiards playing comrades abroad were seeking the same knowledge as us, but none were forthcoming with the proper jump shot technique. The occasional super-champ casually mentioned their jump shot ability, usually describing how difficult they are to control, while entirely missing the aim of the thread: discovering the proper technique.

Then in a serendipitous moment of exploration, I found it. It was the exploration that happens between the time the novelty wears off, and the time you develop a new habit that it happened. I realised that in my fixation on making the next shot each and every time, I had never pushed the up arrow. To my surprise, the view changed. My view changed. I was compelled to explore further. I pushed up until I was looking down on the cue ball from above. I knew instantly what was happening.

I discovered this new perspective on the game not a second too soon. There was a wall of balls between me and the target ball, and not an artist's hope in calculus that I could bank the ball correctly. Like second nature I pulled back my wiimote and struck the cue ball hard on the side closest to the bottom of the screen. At the time I would have described it as the side closest to me, I was so involved in the game. It cleared the wall of balls the previously would have been a show stopper, and struck the target ball instantly upon the cue ball's return to the table, deadening the bounce, and keeping it on the table.

No sooner did my jaw begin to drop, than the cue ball went careening off across the table, striking a second ball, which had spent the better portion of that match peering down into the corner pocket. The second ball, I believe it was the 8, tipped lightly into the corner pocket while the cue ball came to rest a short distance away. It was done. Not only had I discovered how to perform a jump shot, I had made the exact shot I was hoping to make, plus another ball as an added bonus.
"Okay now you have to blog about that," were the words of a wide-eyed Gina the first time I successfully sunk a ball using a jump shot.

Detecting Passive Failure

I was intending to write a little bit on detecting passive failures in enterprise systems, but frankly I don't really have anything interesting to say. This is mostly because I don't have my notebook with me right now but don't worry, I'll write about the technical details in my next post. For now what's on my mind is a less technical kind of the same thing.

The way I defined passive failure for my own purposes is something that can fail without a specific point of failure. I promised some people I would write about that, possibly in the form of a children's book, but while thinking about making that post, I let too much time lapse between posts, which itself is a failure of the type I'm trying to detect, only in my own life.

Perhaps it was the length and anxiety caused by my last entry that made me neglect my writing practice, or perhaps it's just that I'm still not inspired to write, or maybe it's something else. The difficult part is that I'm not exactly sure at which point I've failed. This is one class of problem that I'm trying to figure out and solve at work.

Before I write about detecting passive failures in enterprise systems I'd like to know what some other people think about detecting and fixing passive failures in real life. What techniques do you use to keep yourself doing the things you should? Is it that doing what you should is less of an inconvenience than not doing it? Maybe you don't do things that aren't natural for you?