I once interviewed at a company that had you write a traveling salesman program where you were planning the shortest path of flights between cities. It was a take home test where you were given at least a week to complete it. I completed the test in record setting time (3 days earlier than most), mine was the only one that was fully documented with comments (no one else bothered to do this), they said I had the most visually stunning user interface (most were ugly or unusable; I even animated the plane flying between cities), the program came to the outcome in the shortest runnable time, out of all the submitters they said my program was the most optimized solution (not the best ever made, but the best submitted to this company so far), and mine was the only one that actually worked correctly and ran without flaws. They then proceeded to have me come in for the 6 hour interview in which they asked questions like, "Write a working hash table from scratch on the whiteboard." I got nervous and passed the question, they ended the interview early and rejected my application.
The problem with take-home tests is you may have had help, or someone else did the test for you. If you didn't do well in the in-person test, they may have reasonably assumed that was the case. Especially if the home test was unusually good.
I was aware candidates would do things like this when I applied for job long ago. I also was aware that employers were suspicious of these sorts of things. So I brought along listings of code I'd written, pulled them out, and walked the interviewer through how they worked to show I knew my stuff. Note that they didn't ask for this, I just did it proactively. I got an offer, too, and was told some time later by the interviewer that that was why.
My company gives a take-home test. When the candidate comes in for the interview I go over their code with them and ask questions about it: Why did you take this approach? How could you test this method?
I don't care how much help they got if they can discuss their solution coherently. This also shows the candidate that we took their solution seriously.
And how many candidates spend hours of their valuable time in those take-home assignments, only to get beaten by a purchased solution?
It's wonderful you don't care about the fact that these solutions are purchased. I'm sure your attitude is a great comfort for all the honest professionals who worked hard to find the time to do a 5 hour take-home, only to see their job taken away by dishonest candidates who paid for the solution.
Taken to its logical conclusions, if your attitude is representative, then everyone should just buy take-home solutions and just spend a bit of time understanding them before the onsite. Now we're back to square one, except at least we're not actively screening for dishonesty anymore.
That's not the problem that take home assignment is facing right now. As you said, it's the experience issue, people feel shit after doing it, rejected without any reason.
If copying/buying solution is actually a problem, that means take home project is a thing that's happening, and copying is the new problem to solve along the way.
> If copying/buying solution is actually a problem, that means take home project is a thing that's happening, and copying is the new problem to solve along the way.
In case you didn't know this: take-home is very much happening. I was job-searching not too long ago, and roughly 10% of the places I talked to insisted on take-home assignments. I declined all these requests because I'm a senior engineer with many options, but clearly many candidates are taking them.
There really is a problem with people buying solutions for take-home assignments, I saw that when I was hiring. It just isn't acknowledged, so effectively it has become a competitive advantage for dishonest candidates who are willing to cheat.
I was worried about that for senior candidates, and if I had a good recommendation or had some sort of screen call with them I’d usually skip the takehome.
Very occasionally the candidate came in and just was wildly off, but usually it was fine.
Later on we changed to having all candidates submit the takehome under the logic that if they’re a, “positive” candidate that they’ll be inclined to see this as an opportunity to strut their stuff. I was a little unsure about this approach but we went with it.
I’m curious how senior folks feel about these different attitudes.
> I’m curious how senior folks feel about these different attitudes.
It's quite simple: I'm a senior engineer and I will never do a takehome assignment.
Every time I'm on the job market, there's a certain percentage of positions that require a take home. I just skip them.
I don't explain my position, since I learned long ago that it just leads to arguments and confrontations with the very same recruiter who just a hours ago cold-called you begging you to interview.
I just terminate the process as quietly and unobtrusively as I can.
I wish I could comfort you by saying I ever regret this decision, but the truth is that the best jobs out there - the Google, Netflix, Airbnb jobs - do not require a take-home assignments. I guess they know better than to restrict their candidate pool in this competitive market.
It tends to be the below-average and (often) crappiest jobs that require take-homes, to the point it became a red flag to me. Now I often wonder "what kind of a desperate crappy senior engineer had to jump through all your hoops in this market, especially when you're just one anonymous company out of tens of thousands to be hiring engineers, with nothing special to recommend you this early in the process?".
In short, to insist on take-home tasks in this market is to give up on your strongest candidates. Neither me nor many senior engineers I know will at all consider spending 5+ hours on takehomes when we can easily get dozens (!) of onsites with short, fun remote screens. I will also be very skeptical of any engineer who in this market is willing to do these hours of takehomes after a long, hard day of coding at work.
One last point: it seems quite unfair rude to me for a potential employer to ask me to unilaterally dedicate 5+ hours of my rare valuable time, when we barely begun the process. It's especially out of place and comical when the same recruiter who literally begged you to interview in the first cold-call, is suddenly demanding you do 5+ hours of unpaid work. At least in remote screens, I know the employer is investing some of their own resources into this process as well.
Thanks for your candor, I generally had this sense. If we had something like, “we usually ask for a take-home but a pointer to a code sample you’d be willing to discuss will work.” Would that be acceptable?
My logic on the take home was that interviews are hard without something to talk about and contrived problems are not super reflective of real work. Therefore, I’d send a dumb version of a simple task we have, then we’d walk through trade offs and such.
If I have some code you wrote and can walk me through that’s a very useful datapoint not just for competence (if you’ve been around a while you probably are good) but for personality and how we’d work together.
Does that seem more reasonable as a trade off for your time?
I think in general all of these things just start to come off as dishonest, when senior engineers think about what makes them feel "senior," or what they respect and value in "senior" colleagues.
At some point I think senior engineers realize that coding is the absolute least of your worries as someone expected to do things like interface with the business side, plan medium/long-term roadmaps, improve dev tools or general architecture to make juniors more productive, or mentor/train/manage juniors.
When the entire process seems to be heavily weighted towards coding, it gives off the vibe that this is a company that recognizes overworking and churning out code as traits of a senior engineer, instead of shrewd decision-making, efficient delegation, smart scoping of projects, strong leadership skills, etc etc etc.
I don't want to spend my free time building a portfolio of random bullshit code any more than I want to spend time on your take home test. Because I can't for the life of me see why having an impressive (or average) github profile should be the leading indicator for senior engineering skills.
What is wrong with simple live-coding sessions on very small pieces of code while discussing objectives, edge cases, extensions, etc? Coupled with a high-level systems design session with whiteboard diagrams but no code, and a behavioral interview assessing leadership skills, extracurriculars, personality type, etc. Sane companies seem perfectly fine with sticking to this formula.
Where is this idea that we need more and more, deeper and deeper assessment of candidates coming from? What problem(s) is this solving to look at large amounts of pre-written code from candidates?
That's all very fair. The Q I usually ask of all candidates onsite is a design, "give-me-pseudocode/prototypes/class outlines" question which I find infinitely more useful in probing how someone thinks than whiteboard algorithm/coding qs. And to be clear, I wouldn't want to see BS code, but if someone had contributed to an open source project or had some sort of side project that they would have done anyway, that would have been useful.
I think our focus on code was based on the fact that everyone needed to write code. We were a small company so there was less of the architecture work. We may well have benefited from doing more of it, but we still needed people to ship lots of code.
You do ask a very important question for which I don't really have an answer: why do we care so much about these take-homes or coding interviews for people who have been around for a while?
For me I'd say because I want to have a point of comparison with all other candidates, but that's a pretty weak answer. I'll think more about it.
> In short, to insist on take-home tasks in this market is to give up on your strongest candidates.
This assertion makes absolutely no sense at all. Not being able to pass a simple fizzbuzz test doesn't mean the candidate is among the strongest there is. It just means that the candidate was either incapable of taking such a simple test or unwilling to provide any assurance that his alleged seniority made him a paper tiger.
I've personally interviewed a candidate who had a double major in math and CS, had a PhD, declared having over 10years of experience as a senior software dev, and on the first interview he showed he didn't even knew how to include a third-party library on what he claimed to be his programming language of choice he used most of his career.
That's the sort of stuff that take at home tests help weed out. If he took the test he wouldn't have wasted hours of his and our time.
Honestly it's annoying and does not tell that much. We recruted once a guy that passed algorithm tests on site. We had to fire him 9 month later. He wasn't abble to handle code he had not written and was disappearing every time there was a production issue. Tottally unreliable.
If I'm doing homework I expect an office visit. And pizza, or lunch, or something. I expect to be a serious candidate.
I code and troubleshoot for $$$, and unless there is a direct route to interview + hire (or at least lunch) I'm not interested. I understand there are sorting methods needed to do hiring, and I'll play that game if I have to, but if there is no direct, up front expectation that I'm in the running then it's just kind of patronizing without payoff.
I think the previous post wasn't literally saying they would be okay with someone purchasing a solution. Just that it's okay if they had help with the problem, even it's quite a bit, as long as they actually understand the code.
I can't imagine a scenario where a person can't solve the problem, so hires someone to do it, then explains it as well as the person that actually wrote the code. If you can explain it that well, with reasoning for tradeoffs and design decisions, you might as well have written the code yourself and saved some money.
> And how many candidates spend hours of their valuable time in those take-home assignments, only to get beaten by a purchased solution?
In the few hiring processes where I've participated as a hiring consultant, take at home tests were only used in the initial stages of a hiring process to filter out blatantly incompetent candidates, who in some cases represented about 90% of all applicants. We're talking about basic fizzbuzz stuff.
Even so, in subsequent stages the candidates who made the cut were again evaluated based on their technical skills, mostly to weed out candidates who might have cheated their way through the first stage.
This meant that it was impossible for dishonest candidates to eat the lunch of candidates who were cut due to their performance of the take at home test because anyone who failed that test was obviously not a competent developer. I mean, are you expecting to be hired as a professional software developer if you can't write a for loop?
Have you read any of the comments in this thread? We are talking about take-home tasks that take 5 to 10 hours or more, not fizzbuzz.
You are naive if you think you solved cheating by checking technical skills on the onsite. What if I'm a 7/10 candidate, and someone else is a 9/10, but I cheat and submit the solution of my friend who is a 10/10. I'll get the flight ticket to the onsite, the better qualified guy will not, and you will have to choose from a smaller pool of candidates, in which I might be the best one.
My companies do something a bit different: we ask people to pick any issue from any opensource project in our tech stack and try to land a patch closing it.
We think this is better then wasting the candidate's time on a toy project - at least the can write off their time on the opensource-karma account.
Also, we can check not only their code but also their communication style and other skills that matter in the real world (projects on our tech stack have a high threshold for documentation, unit tests and other stuff that matters).
We call it "social code challenge" and it is working wonderfully for our recruiting process.
I basically like the approach, but I'm a bit wary after playing devils advocate with it for a moment.
A candidate might reasonably assume that they can look better than the other candidates by spending more time on the problem and tackling a larger or more impressive ticket. If you imagine a world where all interviews were like this, candidates could end up working 60 hours a week on these projects in the normal course of job hunting (easier to imagine if you assume that the talent shortage isn't permanent). Probably, since they can see the size of the contributions made by previous candidates, the pressure increases over time as well.
edit: it would be amazing for FOSS in general though. Imagine how much faster GNOME would be if hiring committees required you to write bug fixes for popular projects.
> A candidate might reasonably assume that they can look better than the other candidates by spending more time on the problem
The same is true with a take-home, candidates see the (often underestimated) statement "this should not take more than X hours" and assume they will have a better shot if the invest more than the X hours.
I forgot to mention that we also accept past contributions - in the end the hypothetical candidate can spread this 60 hours investment over any other hiring processes using the same method.
I'm thinking about writing a "social code challenge manifesto".
Many employers coming looking for D programmers to hire have looked at accepted D contributions to the Dlang repository. It's a great way to build one's resume and find great people.
Exactly, many people complain about about not being hired because they lack experience and they can't build experience if they are not hired.
My advice is always "build experience contributing to FOSS projects in your spare time".
We are pushing the envelope and often find bugs or lacking features. I want to hire people that can fix/implement those and send the PRs upstream instead of nagging the project maintainers.
Seems like a good approach but beware "any issue" that's really a decoy filed by applicants aware of your hiring process. Unlikely yes, but worth considering, if only to appreciate the initiative behind such ruse.
Love this, especially with evaluating not only code but how well they collaborate. Ideally if the candidate had already submitted such a patch they could just explain it.
Exactly. And it is way easier to fake a resume than fake public reputation. For example, looking at a stackoverflow profile I can tell if someone knows how to ask and how to answer (a good question is 90% of the answer).
Why bother giving a take-home test if you don't trust the results?
You can have one-way trust in a take-home. If they don't meet your quality bar, you can discard the resume. And it takes near zero work on the company's part, that's why so many companies like it.
However, some people will just get a lot of help on the take-home. So it isn't fair to the other applicants if you treat a take-home as anything other than a filter.
> The problem with take-home tests is you may have had help
I think we over focus on these scary edge cases where some trickster is trying to con their way into the job. The worst case is you just fire them when they are unable to do the work, and that's on them.
Besides, what environment discourages help? I ask friends for help with my job all the time.
The cost of over optimizing for avoiding those false positives (ie: bad candidate marked good) is a huge drop in true positives (ie: good candidate marked good). It's silly.
> I think we over focus on these scary edge cases where some trickster is trying to con their way into the job. The worst case is you just fire them when they are unable to do the work, and that's on them.
The problem is once you offer them a job you stop searching for a candidate. Then you get a bad hire and even if it's quick you won't really know for a while, so then you're really behind.
It's bad enough when you hire someone who really wasn't qualified but didn't do anything bad. Now you have to get rid of someone who probably left a previous position and, like above, you have to go get someone to do the job you were hiring for in the first place. Worst of all possible worlds.
Perhaps the calculus is different in a big company, but for a startup your approach sounds pretty bad.
For what it's worth, I'm not convinced that my approach will actually let more "bad" people through. My personal opinion is that you will improve both your rate of good candidate hiring and lower your rate of good candidate rejection.
I believe that the existing system intends to optimize heavily for "reject bad candidates" and actually fails at that too.
Give candidates a take home, then review it with them.
I have a lot of reasons for believing this approach would be far superior to the "sit in a room and get interrogated for 7 hours" approach.
>I think we over focus on these scary edge cases where some trickster is trying to con their way into the job. The worst case is you just fire them when they are unable to do the work, and that's on them.
The whole reason for the long hiring process is that firing is hard (legally, financially, emotionally). If you're willing to dismiss this difficulty, you're assuming away the core problem.
In the US, where most startups are based, firing is actually quite easy. For a brand new employee, it's especially easy.
All that BS about it being hard financially or legally? Not true. At all. It's just an excuse startups use because they don't understand basic HR functions. And that's why every company of a reasonable size (i.e., more than a few dozen people) has an HR department, even if it's just one person. Because they know how to do this stuff.
My comments are based on what nearly every major company in the US actually does, not what people in Silicon Valley think other companies do. Since all those other companies have HR departments and most of the startups in SV don't, I'm pretty comfortable with my stance.
After all, managing hiring and firing are literally the primary reasons HR departments exist in every company making more than a few million $/year. Especially the profitable ones.
Every company in the US has an aversion to firing that leads them to use expensive defensive measures (in the case of the larger ones: severance pay, PIPs, standardized performance reviews, diversity training) for when they feel they have to go though with it.
Whether or not you’re talking about Silicon Valley, you’ve endorsed a much stronger claim, that such measures are pointless because firing is just so easy.
If you’re not ready to defend that claim, then maybe you should walk back on your original. Or at least show some some sign that you recognize things aren’t as easy or simple as your original blithe dismissal implies.
Remember, you didn’t even think the emotional side merited an acknowledgement.
> Every company in the US has an aversion to firing that leads them to use expensive defensive measures (in the case of the larger ones: severance pay, PIPs, standardized performance reviews, diversity training) for when they feel they have to go though with it.
Not really. It's just harder to hire again usually than figure out how to coerce the problem you have already to be less of a problem. That's the nature of the market right now.
If developers were easy to find and hire then they would be getting fired left and right like happens with just about any other commodity job.
This is simply not true. Employers have been hiring like crazy all across the country for the past few years.
And severance pay is not a thing for most employees, just the ones, like programmers, that have fairly cushy white collar jobs.
You need to stop seeing everything through a rose-tinted SV lens and look at it the way the rest of the country does. I'm ready to defend my claims. Are you?
But we aren't talking about non programming jobs. We are talking about programming jobs.
What everyone else does is completely irrelevant.
In the programming world, specifically, which is what we are talking about, firing is extremely costly.
This is proved by what companies in the programming world are doing right now.
If you disagree, then you are disagreeing with basically every major tech company in the world. And personal, my bet on who is right would be those tech companies, not you.
Oh if we're going to limit this to programming jobs then I know for a fact that the majority of programming jobs do not pay for relocation or a signing bonus. Especially not in the startup world.
And when they do, it's standard practice to require that relocation and signing bonuses be repaid if the employee resigns or is terminated within a certain time frame after joining the company.
I have worked at 5 different companies over the years, and Every single one of them was absolutely terrified of firing engineers.
Not only have I worked at a bunch of different companies, but I have also straight up failed at some of the earlier jobs, and have been in situations where I absolutely deserved to be fired, yet it took literally months for anyone to even approach the topic of my performance.
I am talking, situations of me not doing any work, for months, because I didn't care and was looking for new jobs during that time period.
My experience has shown to me that engineers are pretty damn untouchable. And yes, these have been (mid sized) startups.
The one time I had a reloc bonus, of a job that I left after 5 months, they did not ask for it back, and simply never talked to me again.
It is not about signing bonuses though. It is mostly about things like bad Glassdoor reviews. Many engineers would never ever work at a company where the engineers were constantly terrified of, and complaing about being fired quickly.
Also, disgruntled ex employees are something that you don't ever want to deal with.
I'm not saying to not interview people I'm saying that the existing process is petrified of a bad apple making their way through it, at almost all costs.
How many people can truly, reliably, "fake" their way through a take home? How many people could then do an in person code review and fake their way through that as well?
I doubt it's that many, but you'll get way better returns on good candidates, in my opinion.
Firing somebody can set you back $100,000 (or more if they're in a protected class). Worse, a bad hire can wreck your schedule.
I've encountered programmers who know their stuff, can talk about it intelligently and knowledgeably, but cannot write code worth spit. It's not as rare as you might think. It's sort of like someone with an art degree who has no talent for actually painting.
It's just silly that startups will go months without filling a position so they can avoid spending a week on a person that they might need to fire. This is an example of a premature optimization. A problem faced by less than 1% of companies nationwide is not something you need to worry about. It would be like leasing the top floor of an office building in Silicon Valley for top dollar because you heard that NYC occasionally floods in the winter.
If someone is in a protected class and they get fired within the first week, does anyone really think they'll win a lawsuit alleging discrimination? Does anyone actually think they'd get beyond a motion to dismiss (usually the first motion filed by a defendant)? If the employer was discriminating against protected classes they wouldn't have hired the employee in the first place. These types of claims are quickly dismissed and usually incur minimal legal fees.
So, to be clear, firing is not the goal, it's just the worst case. 100k sounds high but honestly, not something I'm familiar with. I wonder how much money is lost on recruiting being so ineffective? Given that the current process is about ~6-7 hours of eng time per on-site candidate, I'm not convinced that the cost of firing is going to outway the costs of recruiting being considerably optimized.
I'm not sure I see how someone is going to:
a) Write a take home project
b) Explain their design decisions, respond to code review, etc, in person
and somehow not be able to "write code worth spit" or otherwise game the system consistently.
edit: Would love to hear (in the US) where that 100k comes from. At-will state means I can fire you for whatever reason, can't imagine it really comes close to 100k tbh.
Don''t know why you're being downvoted, but 100k is definitely not the cost of firing a just-hired employee in the US.
If you're paying a new-hire-fast-fire more than the salary due to them for the period worked (and any other amounts listed in their offer letter, if one existed), then you need to talk to a labor lawyer because you're definitely doing something wrong...unless you're paying them to shut up about the experience.
6. it can take a while to determine they're incompetent if they're good at the facade
7. if they're in a protected class, you have to go through a period (often 6 months or more) of documenting their incompetence and leaving a trail of trying to help them improve
8. firing someone always comes with it the risk of a lawsuit, and meritless lawsuits are commonplace, but they still cost a LOT of money to defend.
9. disruption to your team, scheduling slippage, having to allocate someone else to fix the incompetent work, etc.
If you actually think those are the costs of firing a new hire, you need to talk to a labor lawyer immediately, because 1, 2, and 3 are hiring costs which don't apply to most jobs--startups included. Only in the rarefied world of FAANG and VC-funded money-losing unicorns do new hires get relocation expenses and a starting bonus. Even associates starting new jobs at BigLaw firms don't get paid relocation expenses or starting bonuses. And if you're doing the hiring in-house why are you paying a headhunter fee?
5. COBRA is paid by the employee, not the employer.
6. Somehow every other industry except programming manages to do this just fine.
7. This is completely and utterly false. If a new hire doesn't work out you can just let them go. Somehow, companies do this all the time without issues, including companies with much cushier jobs than startups.
8. Everything comes with the risk of a lawsuit. You're prematurely optimizing for the 0.0001% case, for an imagined scenario that isn't that expensive to defend.
9. Most of these apply to leaving the position open for months without filling. So let me add one to your list that applies if you don't fill the job: employee burnout leaving to existing employees leaving before the open position is filled, leaving you with more positions to fill than you started with.
Paid relocation expenses is fairly common outside of SV for highly compensated employees, however, they are structured so that you must pay it back if you don't stay employed for a certain time (usually a year). I've never heard of a relocation bonus that wasn't structured this way.
Retention bonuses and starting bonuses are structured the same, you sign an agreement to get that bonus that says you pay it back if either you or the company decides to end employment within a certain timeframe.
I agree with you for the most part, however, you keep claiming that relocation are not usually paid. I know that relocation packages are very commonly offered to programmers, even in entry-level positions, in the finance and oil industries.
Unless this is a contracted for benefit, this isn't a separate cost from lawsuit risk, but a mitigation of lawsuit risk: severance pay is paid to people on condition of waiving claims.
> if they're in a protected class
If you start a sentence this way, it's a clear sign you don't know what you are talking about. Except for a few special cases (“age over 40” and “veteran status”), every employment protected class is a dimension on which discrimination against any value is equally restricted, and everyone has some value on each axis (race, sex, etc.)
So, everyone is “in a protected class”. Well, several of them, actually.
> you have to go through a period (often 6 months or more) of documenting their incompetence and leaving a trail of trying to help them improve
No, you don’t, and you can’t really count this “requirement” and lawsuit risk as separate costs, since this is simply a policy companies adopt to mitigate lawsuit risk.
> you can’t really count this “requirement” and lawsuit risk as separate costs, since this is simply a policy companies adopt to mitigate lawsuit risk.
Think of it this way. If I sell my house, I pay a 6% commission. If I reject the offer, I still owe the commission. I pay the commission. Then if I sell it again, I pay another 6% commission.
If you sell your house, you pay a 6% commission on closing. If you reject the offer, there is no closing, and you don't owe the commission. Because the house didn't sell. If you sell the house, you can't sell it again unless you're committing fraud...But assuming you meant you find another buyer, and successfully close with the second buyer, you would then owe a single 6% commission to your real estate broker, not two 6% commissions.
This is also not how recruitment bonuses work, and I say that as someone that works closely with HR on the tax/legal side of things.
If you pay a recruiter a recruitment bonus, their recruited employee has to remain with the company for X amount of time (aka "trial period"). Usually at least 3 months, but it varies. Once that happens, the bonus "vests" and is due to the recruiter. (Many recruiters will try to get paid before the trial period ends, but you don't actually have to pay them yet.) If you terminate the employee before the trial period ends, the recruiter is not owed a recruitment bonus. These are all standard provisions for recruitment contracts involving 3rd party recruiters. (Note: this is for the contract with the recruiter, not the recruitment employee. The employee's employment contract, if any, may not include any language about a trial period.)
Of course, it's very possible for a startup without an HR department to leave out the trial period language from their recruiter contracts. Because they didn't know any better because they didn't think that HR personnel provided useful functions. Now you know why HR departments exist.
I understand, it isn't intuitive. I hope my analogy of someone well trained in art, yet still can't paint worth a damn, makes sense.
Or pilots who ace all the pre-flight training, yet simply are too uncoordinated to fly well. (That's me. I know all about flying, yet should never be in the pilot's seat.)
Some people are just unable to synthesis good code out of all their knowledge.
If they understand how it works well enough to go through a code review and modify it based on that code review, what's the problem? That just... sounds like what your work is going to be.
Good question. Like I said elsewhere in this thread, there are many people who are knowledgeable and intelligent about coding, but cannot code. They are different skills. I've met many people like this.
I could study Romeo+Juliet and know everything about it that lit professors know. But I could never write it.
For another example, from my time at Boeing, the engineers were divided into two independent groups. One group designed, the other group analyzed and checked the designs. They were different skills, and the engineers would gravitate to whichever path best suited them.
FAANG companies have this problem: when you hire thousands of people a year, a higher false-positive rate is a big deal.
When you are startupbro.com, you don’t have this problem, but you’re desperately cargo-culting your way to success, and doing anything in a way that goes against the grain causes the other bros to accuse you of “bad signaling”.
See also: why tech companies cram together in San Francisco.
>when you hire thousands of people a year, a higher false-positive rate is a big deal.
My friend's employer is currently hiring thousands of people a year with a single interview lasting 1-2 hours without problems. Some highly technical and some manual laborers.
Then do this by putting new employees on a probationary period for the first six months. Many of their employees are union, and it still works out, the unions agree to this arrangement.
Couldn't agree more. Almost every startup in LA I interviewed with seemed to suffer from some sort of imposter syndrome. Even a hint of a question about their technology choices or their business model and they look at you like you just ran over their dog. HOW DARE YOU QUESTION THE GREAT AND WISE WIZARD?!??
It's really not that hard at all to fire someone, at least in the U.S. I think most companies do it because they don't know any better and they heard that's what Google does.
>The problem with take-home tests is you may have had help, or someone else did the test for you
Ask about the code. Ask about the thought process. Ask how to debug it. Ask how to change it or improve it. Put one of the company's team members with them and pair code for half an hour. Find a bug or two and fix it.
There's no lack of questions to confirm if someone did the work themselves. This is easy to figure out once you have working code in front of everyone.
I've had candidates who can explain it all, and aced the take home. Failed all interviews. I spent 20 minutes chatting amicably with them letting them enthusiastically ask questions and then asked them to iterate an array and PR nt the elements. Print function was given. Array was given. They could not write the code in 40 minutes. I did this make good interview after the candidate had tanked their other interviews on tech and soft skills. Even if your Terrible on a white board you should be able to write a for loop.
Had other candidates totally ace the take home and flat out state first thing that they can't code and ask to switch to a non coding interview or to go home.
Some people spend a lot of time practicing to cheat. Given how much this industry pays it shouldn't be surprising.
Both of these happened 3 weeks ago when I was interviewing 32 candidates in a event. But several times in the last few years as well.
No I don't have statistics on this or pure confirmation... Kinda hard to get that info from suspected cheaters.
There are problems with every type of interviews a d they're hueristics for skill not a judge of skill.
I would love to meet someone who was at ease, could explain code in detail (including talking about looping over arrays) and then NOT be able to produce that code.
The second one flat out cheated, but I bet they wouldn't have been able to explain the take home so you might have caught them.
Kinda weak speculation on my part but I guess I could be wrong. I've literally never met anyone like that.
What is the take home like? Maybe the code/answers have leaked somewhere?
The second one could infact explain it all. And could solve design problems. This happens a lot actually, rusty former coders or people who learned in college and never actually did it so it stuck.
The first one could also explain the code fine. Interviews 2 and 3 they were comfortable and able to design decent systems but fell over on code. This why I was given the task of "get any code whatsoever after you put them at ease"
You can cheat by paying someone for the solution, and it also comes with a clear verbal explanation that you can read.
What GP describes is more common than you'd think. Not surprising when the prize for cheating is a 6 figure job, and there's no penalty at all for getting caught.
I've seen cases where people had a friend, someone they were dating or a cs major roommate who it was very likely did the code for them (they mentioned these people encouraging them in interviews)
In the gp scenarios they said they were able to Google almost all of the take home and stich the parts together. The overall solution/problem isn't Googleable, we checked.
I'd think that writing an extremely widely used package manager is good evidence of your competency as a software engineer. Whereas in your analogy the qualification is totally irrelevant to the position.
0. The actual job of a software developer is usually to develop software with some real-world use and benefit, not to write cs101 code on a whiteboard while someone watches you and grades you for speed and accuracy.
1. Software is what Google does, and Google does tons of software from tools to infrastructure to services to games. If you created a key software tool that tons of people use at Google to do their jobs, it's pretty relevant.
2. Software isn't like doing routine maintenance on an existing A/C system - it's usually somewhere between inventing A/C and creating a new design for data center A/C systems then building, deploying, and operating it.
Without making any judgement about this particular situation, this logic isn't sound:
1. I develop tool X, which is a simple popular solution.
2. Some company Y is hiring engineers to develop solutions that are far more complex and demanding than what my tool does, for instance because they have to support billions of concurrent users.
3. I deserve the job because most of Y's engineers use the simple tool X that I developed, even though the job requires far more advanced skills than I have demonstrated writing X, and I actually don't have these skills and will totally fail at the job if hired.
I think Noah's point was that homebrew wasn't a simple tool. And, not only was it evidence of the developer's capabilities but it was also used extensively by all of the employer's own engineers. A tool they all relied heavily on to get their own jobs done. Making it clear he could create software they found important for their own production. And, made it possible for them to deliver service to billions of users.
He got nervous on the whiteboard when asked a relatively simple question. They proceeded to shut him out of the position, completely disregarding his ample qualifications signified by their extensive use of his product. Which is a kind of tragedy.
Comparing an A/C unit to rocket science isn't an equivalent comparison. I can't think of a good analogy to be honest, but that isn't even close.
No other field I can think of really has the problem ours does. You can either make A/C units and rockets, or you can't.
But, in software engineering, it is highly possible to be able to say..... develop high quality GPS mapping software from scratch used by the Navy. Then turn around and have a company making word processing software reject you because you couldn't write a Mandelbrot fractal generator from memory on a whiteboard with a marker.
The fractal generator has zero to do with their word processing product and will never be a useful test of your abilities. Clearly the guy who made GPS mapping software can figure out how to write a word processor. Not to mention, they already have the word processor written so he'd only be assisting them in maintaining their existing code. In other words, he's already demonstrated his ability to write and maintain code of a high caliber. None of it makes any sense.
I really don't think any other field has this issue. It is weird as hell, and unusually pervasive in our culture.
Edit: It won't let me hit reply on your comment below. I think we've hit the maximum length for this comment thread. But, I'll respond by saying I don't know what position he was applying for. I'm not sure he ever explained beyond that tweet. The first I heard about it was Noah mentioning it. But, if it made their lives easier so they can deliver products on time to their own customers it makes sense they have a position somewhere in their company that would be a fit for him. At the very least it proves he is a good developer. Massive systems like the ones Google creates are made by many good engineers working together to make it possible.
Our field isn't quite so unique, and the rocket vs AC example is actually perfect.
Both are engineering projects. But one is simple and a solved problem, the other is far more complex.
The fact that many people at Google use Homebrew doesn't automatically qualify a Homebrew developer to work at Google. Maybe Google would develop Homebrew by itself if there was no tool like that available. Perhaps it would do a better job than said developer. Either way, it doesn't automatically imply that you can do whatever it is Google wants you to do just because you developed a tool that some of their engineers use.
Also, let's get real, to say that Google engineers use a package management system to "get their work done" and then describe said PMS as integrally related and essential to their work... more than a little bit of a stretch.
I'm just making a point about the abstract argument. I have no idea about whether Homebrew is technically impressive or not, but it sounds like people think that it is?
"I developed a simple popular tool that many of your employees are using, therefore you must offer me a job that requires far more complex skills that I cannot demonstrate."
Simple solutions for simple problems. More complex solutions for more complex problems. It's quite possible that the problem Homebrew solves is much simpler than the problems Google is hiring engineers to solve.
I always felt like this meant that google dodged a bullet. Not being entitled is perhaps the most important thing I want when hiring. I really don't see why developing homebrew should guarantee a job at google.
It doesn't guarantee a job at google, but I don't think that was the point. If he feels that the reasons for his disqualification are absurd, how should he voice that frustration while not being perceived as entitled? Are hiring practices beyond critique?
I once interviewed at a company that had me write a couple styles of matrix multiplication in C as a take home test, as well as give a one-page description of a project I had done. The project I described heavily involved graph searches.
Then I had a phone interview where they started off by commenting that I was the only one to include tests and address undefined behaviour. Then I had to write a piece of code to traverse a tree and add numbers, I can't remember what it was exactly, but I got nervous and messed up somehow.
Then I waited for 2 weeks just to hear back that they wanted someone more experienced with graph theory.
Expecting perfection on a white-board test is setting everyone up for failure. At best that's a large napkin exercise. Though having a failure, or a point where for whatever reason production is a bottleneck (if they happen to do it without obvious faults) is an opportunity to identify that either thing is the case in that section and move forward with how they'd react; what should be changed, etc.
We're all still human; expect failure to happen. See what happens next.
100% agree. I interviewed at Microsoft nearly a decade ago now. I was told I'd be interviewing for the company, and then if I was hired, based on preference and ability and need I'd be placed with a team
When I got there I was told I'd be interviewing for the SQL Server team, interesting work, but not something I'm interested in. I had never taken operating systems in university because it was always available at a time that another class I HAD to take occurred.
So by some random chance, I'd never heard of a semaphore. Perhaps inexcusable at this point in my career, but I was ignorant of the topic at that time.
My first interview question was to write a way to let many readers on a resource, and only 1 writer, with no readers reading at the time of the writer writing. I decided to use lists of something or another, ProcessId, or something similar.
Every time I would go to write the code, i'd get a few characters in, and the interviewer would interrupt me and ask me questions to help me along. Since I had no idea what a semaphore was, much less that I was supposed to use it, we spent like 30 minutes of this back and forth, me trying to write, him interrupting me. By the end of the session I had a single line initializing a list written.
Some of my other interviews during that battery of interviews went better than that one for sure but, needless to say, I didn't get the job.
My favorite interview experience was when I passed a live coding exercise on Monday and had time to optimize, then completely failed the exact same problem on Friday with a different company because my nerves hit me.
I feel for you man, I drove 4 hours from Sacramento to Palo Alto for a great start up job. I nailed the phone interview and thought all was going to go well. The first onsite interview went pretty well. The second part went south and my nerves got me. I really wanted the job but I blew it. I got good feedback and I'm working on interview problems. It was my first interview in 20 years but again I really wanted the job.
If I were ever to give a coding interview, I wouldn't be asking the candidate to implement a certain data structure, but rather explain the pros and cons of using say, an array vs a linked list and when you would use one over another. Because at the end of the day, you will not be implementing them but using them.
At a previous education startup I did a lot of interviewing of candidates. The question I feel gave me the most insight into candidates was very, very broad. After explaining (generally) how the company’s product worked, I’d ask them something like, “Let’s have a conversation about how you would design a system to distribute e-textbooks to students.”
I would reassure them there is not a right answer, that the exercise was just asking them to engage with the question. It was so effective as an indicator of future performance, and, according to those who eventually were hired, felt fair to them.
Of course this implies all my biases to people who can have a 1:1 convo about a very vaguely specified problem and so on. No idea if it’d be positive for everyone, but I wish someone would interview me like that.
Exactly, after years and years of interviewing and building teams I learned that the best way to interview about data structures is to do Q&A related to them.
Even if you are going to work in a project that requires building such structures, I demand that you don't do it from memory, but make sure to get help from resources to ensure that your structures are sound.
If they were willing to give you a top compensation, then OK, writing hash table from the scratch is a pretty common question at Google/FB. If it was for some generic job with generic pay, they did you a great service by rejecting you.
> "Writing a hash table from scratch" is a programming 101 problem just slightly harder than FizzBuzz, but not by much.
No, it's bloody not.
What's your hash function? Why? What do you do on hash collisions. How many buckets and how big? Does your hash function map to identity or not? Do you exclude certain fields from hashing? How do you handle contention from multiple threads? I can go on and on.
The fact that you think writing a hash is easy betrays a stunning amount of ignorance.
To be fair, the interview question a few posts up mentioned a working hash table, saying nothing about efficiency or thread safety. (though such a question could be gamed with a linked list as a "well akshuolly" a single-bucket hash table...)
That's the difference between a hashtable you might want to use in practice and a toy example to have someone write some code while quizzing some CS theory.
> I got nervous and passed the question, they ended the interview early and rejected my application.
When interviewing, we test for presentation, self-control and anxiety as well. For me it's not a big deal, but for the guys that were interviewing you it seems it was.
Software development is a very stressful job, if you can't even handle the interview, how would you handle real life (c) at this corporation?
Software development does not have to be a stressful experience. I'm sorry that you believe it does as that implies that your experience has largely been stressful.
In my mind, that perspective is a huge red flag for me when I'm on the job hunt. If you are trying to stress me out in an interview rather than make me comfortable then I have to thank you for letting me know before I started working there that you would be exhausting both physically and emotionally.
The company chose to spend a lot of time evaluating all these take home projects, in the first step, and then reject people for silly reason afterwards? It's a lose-lose situation.
Sounds like their hiring procedure is deeply broken.
Let me give a charitable and slightly less negative interpretation:
It may have looked like the applicant had someone else do the take-home assignment for him.
That said, I imagine that they could have interviewed in a more constructive way that may have been less stressful (e.g., walk us through your thought processes as you completed this take home assignment — most frauds can’t do this convincingly).
The company definitely could have done better. If you have someone hitting a home run on a take home assignment, at that point a no-hire decision better have some very strong justification behind it.
I went back and looked through my emails with them. When I asked why they rejected not only did the cite the interview questions as a problem they didn't like that I was vague in response to "What is your GPA?" I had responded "Average"
I went back and looked at the emails. I can't edit my original comment so I'll jot the corrections here. They gave 3 days not a week. I completed the submission in 7 hours, they required you track how many hours you spent and when. I used a genetic algorithm to solve the problem. And, I took advantage of multiple cores if they were available to divide the work load.
Looking at the email there were signs even in their review of my submission that their outlook wasn't completely rosy. They mentioned things like how I failed to correct for differential conversion factors based on global location and stuff like that.
Skipping the question obviously didn't work out for me as I didn't get the position.
It was a weird experience to be put on the spot with such great skepticism with all my past accomplishments and knowing how much they respected my test submission. In a way it was humiliating. I didn't leave that interview feeling very good about myself. I still regret it to this day. In some ways I wish I could go back in time and pass on the interview, I think I would be more confident today.
I bombed an interview where I couldn't understand the prompt, literally. Like the interviewer spoke it to me, and I restated the problem, and he said "no it's…" and drew something on the white board, and I listened and restated it, and we went back and forth like that for about 15 minutes. To this day I don't understand what he was asking for, only that involved a rate of bank customers and a Greek lambda.
Getting good jobs means pushing your limits and it's therefore a numbers game. Feeling shame from rejection is natural, but also generally unhelpful these days, so it's worth unlearning.
I remember doing a take home and adding it to github as requested by the company. Then I realized every one else who applied did the same thing and left up their solution and found all the other candidate answers using a simple search. This was prior to github making private repos free.
I've worked on pretty cool projects since then at other companies. And, it isn't the only interview I've ever had like that, it is more common than I would like. It is sometimes very hard for me to prove myself. Even though I've had plenty of jobs in the past with proof I can code and with good references. I sometimes feel like I'm an imposter, even while kicking ass on a project.
There is a lot of discussion in the comments on this thread about take home tests not being useful. I believe we've have a pretty good hiring process that makes significant use of take home tests (16-18 hours of them in fact). At the same time, I assure you that we care deeply about making the best use of our and the applicants time. This is what we do to make that happen:
- We make a commitment to reply to every candidate who applies as instructed. Not every reply can be detailed, but candidates are never applying to a black hole.
- We have a very detailed job description that answers a lot of the questions many developers have about a company up front. There is always a chance that we are lying or don't live up to what we say, but the info is at least there so the applicants can make an informed decision about whether they are really interested in working for us. The details about our application process are also stated clearly up front so that candidates know what they are getting into.
- We ask pretty in-depth follow questions about technical experience to make sure it aligns with what we are looking for. This filters out a decent amount of candidates who would be unlikely to pass the rest of our process saving them and us time. Other than roughly validating that the candidates work experience falls within the range we are looking for, we find resumes to be mostly useless.
- The initial "take home" exercise we ask applicants to complete takes 60-90 minutes. It's not a coding exercise and they are given a document that helps them prep for the work they are going to be asked to do. We have found over time that this is more predictive of success in our organization that doing an initial phone screen. Candidates are given relatively detailed feedback on this exercise, usually within 3 business days of submitting their work.
- The next step is a Zoom interview. We have a few very basic coding tasks that we ask them to do as part of this interview. These tasks are representative of real world skills a developer would need and not in any way convoluted or academic. The environment could be challenging for some due to the fact that we are watching over their shoulder, but again the tasks are very basic and the goal is simply to see that the tasks are able to be finished in a reasonable time. This isn't to tell us if the candidate can do the job, it just helps us filter out candidates who are likely to fail badly on the next part of our interview process. It saves them time and us time and money. We have occasionally passed candidates on this step who seemed like their bad performance might have been from nervousness and, to date, those candidates have never performed well on the rest of our skills tests.
The candidate has an opportunity to ask any questions they would like during this interview.
We also talk about money at this stage. Our salary range and benefits are published on the job description so we ask the candidate what their expectations are from a compensation perspective. If they are reluctant to share details, that's fine, but we at least confirm that they understand that our published compensation is what we are actually planning on paying and ask that if that is not suitable to them, that they let us know that now before moving along in the process.
Only if everything is aligning at this stage, do we then ask them to commit to a significant more time working through our skills tests process. Our increasing "ask" of the candidate's time is deliberately progressive. We are trying to only ask more of them when we have had the chance to vet them for obvious mismatches and vice versa.
- We then move into a two part skills test process (16-18 hours). All candidates who move on to these tests are compensated at around $50 per hour. All tests are timed and represent real world tasks our developers do on a daily basis. The first three have to be scheduled and proctored but the fourth is really "take home" and can be done at the developers leisure.
If they pass all of those tests, the work on the fourth test goes through a code review process and is then reviewed with the candidate as part of their second video interview, in a very similar way to how we would do a code review for one of our projects.
- Finally, if all that goes well, we do an on-site interview which is mostly a work day (or two) with the project they are working on, again, very representative of the work our developers do on a daily basis. The costs of travel and lodging are compensated at this stage, but we are not paying them hourly. The work they do for us is not customer based work and does not generate revenue for us.
I can't say this is a perfect process and it is a lot to ask of an applicant. But, we also put a lot of time and money into this process for those who get to the later stages. So it's a "give, give" if you will.
I'm sure some will take issue with this process and some don't apply because they don't like the intensity or commitment involved. But I honestly can't fathom trying to decide on someone's technical ability by spending less time seeing how they handle actual programming tasks. So, we do ask a lot from applicants, but our skills tests work out ok specifically because:
- As much as possible, our process is designed to be "evidence based." We rely heavily on skills tests and evaluations that look at how the candidate does work that is very similar to the work we actually need them to do in the job.
- We work hard to provide the candidate with all the info they need from us to determine if we'd be a good fit for them before we ask for more than a few hours of their time
- All programming tasks are done on their own machine with the editor/tools of their choice (we do choose the language the skills tests are in which align with the technologies which are posted as required in the job descriptions)
- All programming tasks are highly representative of actual work a candidate would do if they were hired. The fact that they are timed is probably the one thing that is most "contrived" about the tests.
- The skills tests are paid at ~$50 per hour
- The grading of the candidates work is pretty straight forward. We have a rubric of sorts for what we are looking for and, to the best of our ability, that rubric aligns with the same things we'd be looking for during a code review of one of our employees that was working on a customer project. There are no trick questions and there are no academic or algorithmic oriented questions (because our day to day tasks don't usually involve those skills)
- While it's possible someone could get another developer to take the skills tests for them, the on-site work day(s) would likely reveal their deceit rather quickly. And, if not, we are a small organization and would figure it out pretty quickly if they made it all the way through and actually started working with us.
I'd love to have a simpler and less time consuming process for everyone's sake (it takes a lot of our time to administer this process too). But without the validation that comes with the candidate participating in these exercises for us, hiring becomes more of a guess than an evaluation. At least, I haven't figured out how to do it differently yet and the advice I read hear on HN and elsewhere about dev hiring hasn't convinced me of a better way to do it.
We do offer some candidates the option of skipping the on-site interview and instead work with us in a 30-90 day contract to hire situation. This is actually, probably, the best scenario for us but it obviously doesn't work for all candidates to leave a stable employment situation for such an offer.
The key thing is that you filter out anyone you aren't willing to risk paying $800 or more to do the testing before asking them to commit to the extra time.
This is pretty onerous, but it should be pretty damn good at rejecting bad candidates.
I do wonder how many good or great candidates you lose with this method, though.
They'e also likely filtering out anyone who currently has a job, or a family, or even is interviewing at other companies. The process described here and in the posting at https://www.level12.io/careers/full-stack-web-app-developer-... is on the order of 30 to 40 hours. That's a massive time commitment.
I've wondered that myself. I don't know of a good way to evaluate that. I've thought about looking at bounce rate on the job description page or even showing a popup on bounce detection asking for feedback on the job description and application process, but haven't executed on either yet because I'm not convinced the effort would be worth it. If someone is going to bounce, how likely are they to take the time to tell us why.
It's also more than just the $800 we pay out. I personally proctor and grade the tests. That takes me 4-6 hours. Given my billing rate and roll in the company, that's arguably a greater cost to us than what we pay out to the candidate.
Is this really normal for web developer? I don't work in web dev, but I would absolutely not put up with this for only a 100k salary and getting paid roughly one grand for an interview process that takes in total days.
Reason why is because I've had interviews at other companies which are far less interrogative from the company towards me. For example, for my current job (embedded systems) I had:
- 30 minute phone interview with a technically competent hr rep
- a quick very easy online quiz (multiple choice/few sentence answers)
-one more 30 minute phone interview with a developer (open ended technical discussion)
-onsite for roughly 2 to 3 hours with a dew developers, all very open ended technical discussions
It was honestly the best hiring process I encountered. I didn't feel like I was even tested during the interviews, it was genuinely enjoyable technical conversations.
As best I can tell, our process is not typical. But it's also not fundamentally a bad thing. It obviously doesn't interest you and I'm sure there are others who wouldn't apply for the same reason.
But, there are others out there who appreciate a more thorough approach and might view the process you described as inadequate.
Also, our salary range is higher for senior devs.
I think what matters is whether the process being used by the organization is resulting in the hiring of developers that are a good match. I'm glad we are doing things this way even though it makes hiring take longer and likely filters out candidates who could do the work. It's an imperfect balancing act, one I'm always looking to improve. The main point of the post I made above was to show how IMO take home exams can be executed well, not to argue that everyone needs such an elaborate application process.
As best I can tell, our process is not typical. But it's also not fundamentally a bad thing. It obviously doesn't interest you and I'm sure there are others who wouldn't apply for the same reason.
But, there are others out there who appreciate a more thorough approach and might view the process you described as inadequate.
Path of least resistance my friend, path if least resistance.
I wouldn't do that to them. They have a right to interview how they want. I just wasn't good enough. They were looking for someone who could think on their feet under pressure in the moment and I'm not that guy.
I need a quiet place to sit and think something through and plan it out. It is why I did well on the test portion but not on site surrounded by interrogators. In those kind of situations I kind of feel like I'm the in the movie Swordfish where they're like, "Hack that server you have sixty seconds! (gun to head)" The difference is, I would have been dead.
