Long ago when I was, I think, a sophomore in college and worked for the university IT group, I was trying to add an external drive to an early NeXT machine [1]. I wanted to try out their fancy GUI development stuff, you see. I was at best a modestly competent Unix admin, and this was circa NextStep 1.0, so the OS was... rough. It was in the dark days of SCSI terminators, so just telling if the drive was properly connected and, if so, how to address it was challenging.
After a couple hours of swearing, instead of working from a root shell in my own account, I just logged into the GUI as root. And there was a pretty interface showing the disks. I could just click on one and format it. Hooray!
Well either the GUI was buggy or I clicked on the wrong disk, because as the format was going, I realized the external drive wasn't doing anything. I was formatting the internal boot hard drive. And since nobody but me gave a crap about this weird free box somebody had given them, they had repurposed it. As a file server. For the home directories of a bunch of my colleagues. Who were now collecting around me wondering what was going on. Oops.
No problem, says I. I'll just restore from backups. But this thing used a weird magneto-optical drive [2]. The only boot media we had was on an MO disk. The backups were on another. And there was only one of these drives, probably only one in the whole state. The drives were, of course, incredibly slow, especially if you needed to swap disks. Which, I eventually discovered, I would have to do about a million times to have a hope of recovery.
Long story short, I spent 28 hours in a row in that chair. It was my immersion baptism [2] in the ways of being a sysadmin. The things I learned:
Fear the root shell. It should be treated with as much caution as a live snake.
Have backups. People will do dumb things; be ready.
A backup plan where you have never tried restoring anything may lead to more excitement than you want.
Be suspicious of GUI admin tools. Avoid new GUI admin tools if at all possible. Let somebody else be the one to discover the dangerous flaws.
If you were smart enough to break something, you're smart enough to fix it. Don't give up.
When some young idiot fucks up, check to make sure that they are sufficiently freaked out. If they are, no need to yell at them. Instead support them in solving the problem.
Seriously, my colleagues were awesome about this. I went on to become an actual paid sysadmin, and spent many years enjoying the work. The experience taught me fear, and a level of care that sticks with me today. I'm sure at the time I was wishing somebody would wave a magic wand and make it the problems go away, but working through it gave me a level of comfort in apparent disasters that has been helpful many times since.
"When some young idiot fucks up, check to make sure that they are sufficiently freaked out. If they are, no need to yell at them. Instead support them in solving the problem."
My first job was as a Marine avionics technician and I had the misfortune to work for bosses who reassured themselves by blaming you at full volume and making you feel like shit. Usually with the 'you could have killed somebody you useless idiot' which is actually not that often true. Not a great environment.
Last week I'm just starting my second year at this company as a real proper electrical engineer and I neglected to run all the steps of the pre-tests I said I ran and I therefore missed a (in retrospect, really cool) bug that showed up in front of the certifying examiner on a Friday afternoon. Said examiner costs a lot of money per hour, it's a government agency, etc. etc. I spent a weekend and Monday finding the bug and squashing it. It took til Friday before the new code was released and all the paperwork up to the certifying agency was fixed.
The whole time my boss was there and clearly frustrated but also very clearly keeping quiet. I definitely went and thanked him on Friday afternoon when it was all over for being so chill about it. He said it was no problem as he could see I had sufficiently embarrassed myself and was working hard to fix it. It's so true. I was keeping my head down, being calm but intent on the problem, and I don't know how I could've solved anything if he'd lost his shit.
Fear and care are definitely with me now. As is proper testing and better logging of my own day-to-day dev process. (Note to self: power cycle in the test when it tells you to power cycle. You're also testing the non-obvious functionality.)
root shell, plus rm with any sort of wildcard matching, plus a bit too much of a delay before you get your shell prompt back results in a very specific kind of panicked anxiety that almost anyone who has been programming or sysadmining for a while can easily relate to.
After a couple hours of swearing, instead of working from a root shell in my own account, I just logged into the GUI as root. And there was a pretty interface showing the disks. I could just click on one and format it. Hooray!
Well either the GUI was buggy or I clicked on the wrong disk, because as the format was going, I realized the external drive wasn't doing anything. I was formatting the internal boot hard drive. And since nobody but me gave a crap about this weird free box somebody had given them, they had repurposed it. As a file server. For the home directories of a bunch of my colleagues. Who were now collecting around me wondering what was going on. Oops.
No problem, says I. I'll just restore from backups. But this thing used a weird magneto-optical drive [2]. The only boot media we had was on an MO disk. The backups were on another. And there was only one of these drives, probably only one in the whole state. The drives were, of course, incredibly slow, especially if you needed to swap disks. Which, I eventually discovered, I would have to do about a million times to have a hope of recovery.
Long story short, I spent 28 hours in a row in that chair. It was my immersion baptism [2] in the ways of being a sysadmin. The things I learned:
Fear the root shell. It should be treated with as much caution as a live snake.
Have backups. People will do dumb things; be ready.
A backup plan where you have never tried restoring anything may lead to more excitement than you want.
Be suspicious of GUI admin tools. Avoid new GUI admin tools if at all possible. Let somebody else be the one to discover the dangerous flaws.
If you were smart enough to break something, you're smart enough to fix it. Don't give up.
When some young idiot fucks up, check to make sure that they are sufficiently freaked out. If they are, no need to yell at them. Instead support them in solving the problem.
Seriously, my colleagues were awesome about this. I went on to become an actual paid sysadmin, and spent many years enjoying the work. The experience taught me fear, and a level of care that sticks with me today. I'm sure at the time I was wishing somebody would wave a magic wand and make it the problems go away, but working through it gave me a level of comfort in apparent disasters that has been helpful many times since.
[1] http://en.wikipedia.org/wiki/NeXTcube [2] http://en.wikipedia.org/wiki/Magneto-optical_drive [3] http://en.wikipedia.org/wiki/Immersion_baptism