Nineteen years later is a bit late for these. But seeing as how the original has been getting so much traction I thought I’d write this up and post it.
Unlike the other pieces in this series (the one before and the two or three after this one, forthcoming) I’m writing this almost 20 years later. My attitudes on just about everything in my life have changed remarkably so the tone is going to be an awful lot different than in the other pieces, which I’ve attempted to preserve in my recent edits. So give me some rope.
A couple weeks after 9/11 the company I’d been working with (Is it impolitic to say it was TradeWeb 19 years later?) rented out a hotel conference room in Jersey City and told us to come in for a meeting. There was no real telling what the hell this was about in detail.
I slung my bag over my shoulder and headed in.
It was…what you’d expect, a hotel conference room with a strangely patterned carpet, nondescript beige walls and those metal chairs with vinyl cushions that seems to exist only in hotels.
We were there, milling around for a while, sitting down in rough social clusters, “sup” nodding with people all over. It was a good scene. There were some tears and some absentees and everything in between.
At some length I learned that everyone, all 105 or so of us that were there that day, had made it out that morning, which was rather something. The more I trace back over the events of that morning the less likely that was seeming to my mind. I’d only made it out of the mall and gotten about a block and a half before WTC-2 came down. It hadn’t seemed likely that we’d all have gotten out.
But…there was another factoid making the rounds.
We were fundamentally a software company, connecting trading firms for bond execution. Our business was our software.
Every Friday we’d make tape backups and they would be hand carried to a safe deposit box (no, not in the building) by our CTO, Bob.
This had been going on for the better part of a year, a couple dozen programmers plugging away day in and day out, Bob taking the backups offsite to be safe.
Well 9/11 came and went and the disaster recovery space we were paying for spun up. That involves a bunch of leased hardware and office space. The couple vital code monkeys and infrastructure team got to work setting up a mirror of our old infrastructure, which I’d mentioned were on the 51st floor of WTC-1 with our offices, a strategy that seemed sensible enough at the time.
They put everything together and it came time to restore our source code, the building blocks for everything, from tape.
Yeah, about that.
One thing about backups is that there’s two parts to making sure you’re secure. The second part is to make sure the backups you’re taking actually work. “What’s to test? We copied everything off.”
Murphy has a sense of humor and he was really stretching it out that day.
The backups of the source code were bad. It was all gone.
The CEO was telling us this (which was at this point was redundant to the grapevine) at the front of the conference room and he was just…hangdog. “We have some 9 month old backups from the London office, so…I…don’t know what we’re going to do. This might be the end.”
If I was then who I am now I would have cackled like an absolute maniac.
I used to get to the office pretty early, 7:30ish. I lived close enough to the office that I’d sometimes walk across the Brooklyn Bridge to get there. It totaled 3 miles, one to the bridge, the bridge is about one, and it was one to the building. It was especially lovely when it was snowing and I have to confess that at least once I’d be walking across the bridge during the snow, get to the middle and just marvel at the cables going up equally in each direction and how I couldn’t see anything but snow and the bridge. I’d stop and admire it, looking over each side unable to see the river below or hear nearly anything. Then I’d start walking only to find myself back on the Brooklyn side, having gotten myself turned around one too many times.
But it’s rare that you can have a silent moment in NYC and 7:00 in the morning on the Brooklyn Bridge in the snow is about as quiet as it can get. After all, what kind of fool walks through the snow when he doesn’t have to, and that for the sheer spectacle of it all?
During one of my aborted attempts to get back in to the college mindset I was taking Calc 2 at Westchester Community College. I found it utterly intractable. (Interestingly 3 and 4 I found a breeze.) I was a night class and I hated the professor. In retrospect we probably hated each other. The night of the final exam there was a full lunar eclipse. I parked my car and walked towards the building when I noticed it JUST beginning. I sat on the low “smoking wall” just outside the door and watched it, low in the sky with that red cast the moon has when the air is humid (or you’re looking over Jersey) as people walked in.
“Hey, c’mon man. Final’s starting.”
“Yeah yeah, gimme a minute.”
Eventually nobody was going in anymore and the moon was occulting behind the earth.
The professor came out.
“Mr. Wilson, We’re starting.”
“Yeah I’m gonna sit here and watch this.”
“If you miss the final you’re going to fail.” I shrugged.
“Yeah, but in 10 years I’m going to remember this moment.”
He turned bright fucking red and went back in.
And I did. It was a little scary to sit there. But I sat there and watched the entirety of the lunar eclipse which lasted longer than the Calc final. I know this because an hour later people started coming out.
“Dude, you did that fast.” I just chuckled.
Well dude, it’s about 30 years later and still one of my fondest memories. So you can still go fuck yourself because I’m still smiling.
THAT’S the type of person who walks across the Brooklyn Bridge in the snow as a goof.
I also used to like to noodle around with the code I was working on at the office when at home. So, before everybody got in I’d copy the whole source tree to my 20g external drive. Strictly speaking this was a big fucking no no amounting to what would today be called “exfiltration of the entirety of the firm’s intellectual property for unsanctioned purposes.”
So while Jim was up there talking about how he’d built the company up from an idea and didn’t know what was going to happen next…
I was smiling. Just grinning ear to ear. I whispered to someone sitting next to me.
He was done with his speech and the stable of crisis counselors he’d hired for the occasion started attaching themselves to groups and going through it all with us. Frankly I don’t recall what they talked about. But none of it really made me feel any better.
Well, 20 minutes goes by and Jim comes over with tears in his eyes, grinning.
“Am I to understand that you had the presence of mind to take a copy of our code that morning?”
I glanced nervously at my bag, where sat the drive containing everything, thinking I should not give up the whole thing in that moment.
“Yep. I copied out the source to an external drive that morning at about 7:30. So…unless everybody was VERY productive between 7:30 and 8:56, we didn’t lose anything.”
He just laughed.
Eventually we started work again and I copied the source back to a server and they checked it all in to source control and we just started working from where we were on September 10.
Test your backups kids. Backups don’t exist if you haven’t done a test restore.
My bonus that year was about half my yearly salary.
I remember talking with this one counselor for hours. In fact I was the last one out of the place. I was really wishing I’d had the sand to get her number.
Several people quit, too afraid to have anything to do with lower Manhattan, even by Jersey proxy. Some people even moved cross country, taking the cue to just bail out of The Life, which seemed positively silly.
Okay. So it took me 15 years.