You don't get anything out of burning out. You just burn out. And that time is not coming back. This post seems like an apologist talking.
Production support alone is not that much of a problem. What the author skipped (conveniently? or forgot to mention?) is - it's really the "on call" phenomenon that's the problem.
The "typical" on-call - where when you are on-call you are magically on-call 24x7. Yes, during your sleeping hours as well; as if that's less important and the company can avoid spending money to hire dedicated support for those hours and instead make you suffer (yes, it's just that - there's no other name for it like "satisfaction", "learning", "growing" or any of those buzzwords).
You want engineers to do production support? Well, let them do it during normal office hours and only few times a month. Or heck, let them do it for weeks but let them punch in and punch out normal office hours. Let them choose to do only one half of the day and have someone else willing to do the another half.
There's no excuse for burning out engineers (esp. unsuspecting youngsters) by pushing them into ungodly hours of work ruining their health among other things while trying to constantly tell them - "do you even realise what a service to humanity you are doing!".
The point about on-call is really critical and really on-point.
If a company thinks an application is important enough to run 24x7 then it should staff for 24x7 support. Stealing wages from workers by expecting them to be available 24x7 (on-call) is an absolute abuse.
It also leads to burn out, poor performance during the day (how is a dev's development ability when they were up at 2:30am on an incident call the night before?), and clouded thinking causing mistakes or impacting recovery time during incidents.
> If a company thinks an application is important enough to run 24x7 then it should staff for 24x7 support
And where it really matters, they do. My team and I build and manage a large Emergency Services telecommunication network. We have Tier 1/2 operators on shift work 24/7. Tier 3 staff (programmers, system integrators and administrators) are their escalation point for critical issues outside of business hours.
> Stealing wages from workers by expecting them to be available 24x7 (on-call)
The Tier 3's that are on-call in our environment are on a rotating roster are compensated nicely for being prepared to answer the phone outside business hours. Frequently they don't get called during their week at all and it's free money.
> how is a dev's development ability when they were up at 2:30am on an incident call the night before?
Easy, as well as the financial compensation, we give them time in lieu. Two hours callout in the middle of the night, two (paid) hours given back on their next working day, or whenever they prefer, subject to availability of other staff.
There are simple solutions to these problems, and where they matter, they are applied. Granted things are very black and white for us as lives are potentially at stake, but any company that wants to have 24/7 engineers available needs to pay for that kind of support.
"Frequently they don't get called during their week at all and it's free money."
That's, I think, is a wrong perspective, when people are on call, they have to be somewhere near their computer/internet connection and be ready to work (so it is not just you can go to a party and if call happens do some quick fix in a toilet).
On-calls cannot do what they want with their time, so they don't get money for free.
Yeah this is true although our compensation for their time is pretty generous. As a team we are pretty flexible too and if somebody has an event they want to go to they can always find someone else on the team to cover or swap with them for the night.
> Two hours callout in the middle of the night, two (paid) hours given back on their next working day
This still sounds cheap to me. I have never worked on-call (and I never plan to), but the exhaustion cost of working two hours in the middle of the night is not equal to two hours of uninterrupted sleep. I would expect to get at least a half day off (paid) for any amount of middle-of-the-night work.
The thousand bucks extra they make a week whether they get called or not makes up for it. It's not like I clock watch my team anyway, if they took an extra hour here or there I don't bat an eyelid, they're all hard workers and they get the job done without complaining.
> Easy, as well as the financial compensation, we give them time in lieu. Two hours callout in the middle of the night, two (paid) hours given back on their next working day, or whenever they prefer, subject to availability of other staff.
Every company should do this but none I've worked at do. To be honest, I just take the makeup time myself.
Most companies don't watch the clock for their software engineers. If you get called in the middle of night and take the makeup time yourself, does the company give you time in lieu or not? (By outcome, I would say that they do.)
That's fair. It would be nice if it was an explicit policy though. Otherwise I feel like the company is just exploiting engineers who may not know any better.
>Easy, as well as the financial compensation, we give them time in lieu. Two hours callout in the middle of the night, two (paid) hours given back on their next working day, or whenever they prefer, subject to availability of other staff.
I didn't note this in my post above, but I always gave time-in-lieu for any late night activity. However, the thing that REALLY worked best was allowing the Engineers to prioritize reliability. I had to fight to make it happen, but going from nightly to every couple months volumes was worth it.
The companies do this for the money. And the people what work in those companies have no real sense of the risks, or they just care more for the numbers and want to roll the dice.
I would not trust someone who I just woke up at 2am to do something. He/she is mid-sleep. They will be prone to errors, they will be super tired, and I just ruined their next 1.5 days that it will take them to recover from that.
This is not a job where you live boxes where intellect is not needed as much, (strength and stamina will also be affected by a mid-night alarm). You want your folks to be 100% on par, otherwise they may make things worse.
Having worked in the tech sector for a while now, having burned out once, and having been on-call 24x7 far more than is healthy, I would say exploitation is the name of the game to a lot of managers.
You are less of a person and more of a means to an end. A tool to achieve something, and some tools are disposable. It can be of career advantage to a manager to burn out engineers. Maybe instead of spreading 24x7 on-call across 3 teams in three timezones, you put it on 1 team in 1 timezone. By doing so a manager can achieve a lot with less resources, and hopefully secure their own elevation up the corporate ladder before the cost of their strategy becomes evident.
The cost of burn out I think remains hidden, in technology there's a constant flux of staff anyway, teams being being created and dissolved, in all the noise a few people being exhausted and bailing from the company is hardly noticed. Perhaps they said something before they left, but it's best for everyone in middle management if the burnt out individual is labeled the problem, they were a bad culture fit you see, a grumbler who didn't have what it took.
Burning out from on-call comes from not being able to fix the underlying causes.
I'm happy to hold the pager if I've also got the right to block/rollback deploys until the system is stable - my current job has had two out-of-hours pages in the past year, and we're in the alexa top 10k so it's not like there's no traffic.
Bingo! There's nothing more motivating to "prioritize" fixing a problem than being woken up at 2 a.m. being affected by it. I've worked in places where one team writes the code and another team supports it and it's always the same: the code is absolute garbage. It ends up being too complex and having way too many moving parts, and is impossible to diagnose. I've worked in places where the team who writes it, supports it - 24x7. Their code is always super simple, easy to diagnose, and easy to maintain.
I've been at this for 35 years at many companies and working with many teams and it's always the same: if you want good software then make the team creating it also support it. In every case I've experienced it leads to software requiring little to no support, easy to maintain, and easy to extend. Why? Because nobody wants to get up in the middle of the night or work weekends and moreover, they'd rather be adding features than limping along with existing features.
I'm in ops and this is a huge, huge headache for those of us on our end at companies where we're not given Google SRE powers to control release trajectory based on failure budgets of some sort.
I've worked at too many places that had no SWEs on-call for the on-call alerts that I get, which the vast majority of the time involves throwing a bandaid (as in redirecting traffic, etc) in front of an internal bug that I hope eventually gets fixed once the RFO/etc has been submitted before it hits my NEXT on-call rotation or my poor coworkers.
Without SWEs on my rotation they don't understand the immediacy. They aren't the ones getting their Christmas week interrupted every 4 hours while ops keeps the house running. In Ops having your entire day ruined by various on-call alerts usually feels like you're working without any breaks and nobody even cares.
Anyone want a bad golang developer, wannabe ex-ops person who knows a lot about platform reliability and o11y and wants to focus on the golang end finally? I'll make your teams automation and o11y purr no matter where it is (bm, cloud, global pops, serverless..)..
I think there's only one way to solve this — which I've been unsuccessfully advocating for at my current company — and that is paid voluntary on call schedules.
It creates an actual market for on call work where engineers can simply say no to the extra cash if they don't like work taking up their nights and weekends. If the company is having trouble with no engineers wanting to be on call the pay is simply too low and needs to be increased. It's a job like any other and should be compensated as such.
In the end I honestly believe it will be beneficial for the company not having engineers burn out so quickly. Compensation also clearly sets the expectations — if you're being paid to do it you'll take it more seriously.
But does it give engineers a good incentive to improve product quality and reduce the number of production incidents?
Where I work we are not on-call. Nevertheless, I try to help the ops team when they encounter issues. This does make you improve logging and error handling since you know it takes a lot more time when it's difficult to filter logs for the interesting events.
Engineers not exposed to production issues and customers will never understand why you need these extra measures.
Our on-call system gives everyone an allowance per service plus additional time off if you're on-call on weekends and public holidays. You get both regardless of whether you get paged. Getting paid to do nothing is a great incentive for pushing quality, especially when you're on-call for more than one service in a given week.
> "do you even realise what a service to humanity you are doing!".
I don’t know who needs to hear this besides me twenty years ago, but if you want to do charity then go home at 6pm and volunteer at a real charity. Don’t do it for a wannabe robber baron who will not share with you. Don’t do it for someone where even an emotional payoff is years away or may never come.
Find something else you care about and help some people just because. Not because you’re getting under-paid and over-guilted to do it.
Well even "real charity", including very famous ones, are quite shady too: volunteers do work for free while donations goes towards C-exec 5-6 digits salaries.
Various places I've worked put certain people on-call. They were salaried but somehow paid extra for the time they were on-call. It seemed to be voluntary and the people who did it liked the extra cash.
On the other hand I had a friend at a very large and well known company. He got a job offer and was hired into one department, but he wanted to take a little time off between jobs before he started. They somehow convinced him to start saying there was a holiday coming up and he could take the time off then.
and as soon as he came in he started getting calls 1am 2am 3am etc...
so he left.
And they cajoled him back saying things were different and he finally bought it and went back.
same thing happened again, and he quit for a final time.
One part of the problem was that he was a US citizen working with a bunch of H1B visa folks and the company could get away with that sort of stuff. H1B folks will say yes sir, no sir because their dreams of living in the US are tied to keeping their job at all costs. and then the bad work culture festers.
I'm not gonna advocate for having engineers directly field customer calls at 3am (because that's a recipe for unhappy customers) but as someone who fields customer support calls/tickets as a large part of my job I feel very comfortable saying that a "sizeable enough to be a big problem" will not build systems that are supportable without engineering escalations unless they get support trying to escalate an issue or conference them in at 3am from time to time.
>The "typical" on-call - where when you are on-call you are magically on-call 24x7.
I ran the Engineering org for a startup and we had a small, 3 person Ops team that handled initial triage of events. About 75% of these issues were Engineering-related. My solution was to a) create an on-call rotation for Engineering and b) allow the Engineers to prioritize reliability work.
It sounds like a no-brainer, but I had to fight with the rest of the exec team to allow b) to happen, since it came at the expense of the product roadmap. I eventually won the fight and our nightly on-call volume went from 1-2 incidents per day to 1-2 every few months.
Vindication came about a year later when we were acquired by a large company. As part of the due diligence process (including 18 hours with me going over technical details in front of 30 senior folks from the acquiring company) we got major kudos for having a level of reliability that far exceeded what they typically saw for a company our size.
It’s usually a case if taking responsibility for things you have no power over.
When they ask it’s usually after a giant hole has been dug, and patterns have been set. If you knew from the outset that you would be the support team then you’d have prioritized some other tickets. You’d have increased the estimates on others. You would have refused to work on these three, you would have argued vigorously about these four decisions, and you would insisted your boss fire “That Guy” months ago because his code is garbage and his only real skill is articulate deflection.
This group of folks wants several somethings for nothing. One of them is labor, another is somewhere to assign blame. They are grooming you for failure and we all deserve better.
> There's no excuse for burning out engineers (esp. unsuspecting youngsters) by pushing them into ungodly hours of work ruining their health among other things while trying to constantly tell them - "do you even realise what a service to humanity you are doing!".
In the US, this is happening across the board and not just in tech. The expectation to always be available [often without monetary compensation] is sadly the new normal. Without strong labor laws in place, this implicit form of exploitation will never cease.
Police officers have pretty much the strongest unions out there and the junior folks on the force (i.e. exactly the kind of people who get put on call in tech) generally wind up stacking absurd combinations of shifts in order to be paid competitively. Rail workers, another strongly unionized profession, have it no better.
I get that you like unions but just because you have a hammer doesn't make every problem a nail.
I guess it depends on everyone's personal experience with unions.
From where I come from, unions are mainly a way for lazy employees to get immunity while doing nothing all day long.
I would still maintain that if you live in a country where it's legal to be called at any time, any day, then you have a third world class labor law - go downvote US :>
Historically, I don't think we've made much progress in terms of pro-worker labor laws without said workers first having to get together to create bargaining power. By default the power is in the hands of the ruling class, and they have no interest in handing out paid holidays, reasonable working hours, or sick leave.
You're talking as if they care about the exploitation. It's done purposely. They simply don't care when it means greater profits. They also believe there's an endless line of devs they can burn out and throw away.
If the engineers aren’t oncall, who is? Is it okay to exploit non-engineers? If anything, it is less exploitative to have those who are empowered to improve their situation oncall.
Our support folks handle the on-call support. They do one week each on rotation, they get some extra compensation and the following friday off.
If it's a serious issue they can't handle they might wake up one of us programmers, but usually they can find some temporary fix or workaround until the next morning.
Production support alone is not that much of a problem. What the author skipped (conveniently? or forgot to mention?) is - it's really the "on call" phenomenon that's the problem.
The "typical" on-call - where when you are on-call you are magically on-call 24x7. Yes, during your sleeping hours as well; as if that's less important and the company can avoid spending money to hire dedicated support for those hours and instead make you suffer (yes, it's just that - there's no other name for it like "satisfaction", "learning", "growing" or any of those buzzwords).
You want engineers to do production support? Well, let them do it during normal office hours and only few times a month. Or heck, let them do it for weeks but let them punch in and punch out normal office hours. Let them choose to do only one half of the day and have someone else willing to do the another half.
There's no excuse for burning out engineers (esp. unsuspecting youngsters) by pushing them into ungodly hours of work ruining their health among other things while trying to constantly tell them - "do you even realise what a service to humanity you are doing!".
It's just exploitation.