Leading a group working in an underground bunker on a live military radar site in the Australian outback, where it rains every few years. We had to open a rooftop cable duct and when the job ran overtime we closed it up with some rags that were to hand. That night it rained.
The next morning, the bunker was full to ground level and the automatic power cutoff had failed, as the float switch was directly under the cable duct and the water pressure of the deluge and kept the float depressed. By the time the water stopped flowing the float was under a foot of mud. The powered circuits were undergoing electrolysis and eating themselves away, made worse the the site managers refusing to drain the bunker or turn off the power until a week long arse-covering evaluation had been completed.
A few hundred million dollars of front line radar was out of action for several months.
Being a naive newly graduated engineer, I wrote a completely honest report and analysis. My boss said it was one of the best reports he had read and there was no impact on my career (if anything it got me noticed by the upper echelons of the organisation).
Lessons:
1. If you tell the truth you will be respected, even if it is incriminating.
2. If there is a way for something to go wrong it can do so (slight variation of Murphy's Law). Even if it's judged to be uneconomic to take preventative action, be aware of the possibilities, so you can make a conscious decision about the risk.
The next morning, the bunker was full to ground level and the automatic power cutoff had failed, as the float switch was directly under the cable duct and the water pressure of the deluge and kept the float depressed. By the time the water stopped flowing the float was under a foot of mud. The powered circuits were undergoing electrolysis and eating themselves away, made worse the the site managers refusing to drain the bunker or turn off the power until a week long arse-covering evaluation had been completed.
A few hundred million dollars of front line radar was out of action for several months.
Being a naive newly graduated engineer, I wrote a completely honest report and analysis. My boss said it was one of the best reports he had read and there was no impact on my career (if anything it got me noticed by the upper echelons of the organisation).
Lessons:
1. If you tell the truth you will be respected, even if it is incriminating.
2. If there is a way for something to go wrong it can do so (slight variation of Murphy's Law). Even if it's judged to be uneconomic to take preventative action, be aware of the possibilities, so you can make a conscious decision about the risk.