> We will attempt to directly build safe and beneficial AGI, but will also consider our mission fulfilled if our work aids others to achieve this outcome.
> Our primary fiduciary duty is to humanity. We anticipate needing to marshal substantial resources to fulfill our mission, but will always diligently act to minimize conflicts of interest among our employees and stakeholders that could compromise broad benefit.
> We will actively cooperate with other research and policy institutions; we seek to create a global community working together to address AGI’s global challenges.
I think one good thing to come out of all this tech elite flip flopping is that I now see these tech leaders for exactly who they are. It makes me kind of sad, because as someone who came of age early in the Web era I really wanted to believe that there was a bigger moral good to all we were doing.
I now view any moralistic statement by any of these big tech companies as complete and total bullshit, which is probably for the best, because that is what it is. These companies now exist solely to amass power and wealth. They will still use moralistic language to try to motivate their employees, but I hope folks still see it for the complete nonsense that it is.
I find it quite problematic that researchers get to talk about their own research and present it as facts without anyone taking a critical look.
Over time I’ve become more skeptical about this kind of psychology research (as more studies fail to replicate) and, as is often the case, here the sample size is quite small (76 students, split across 3 groups), predicting something relatively noisy as GPA. It is unclear to me that one would be able to detect reasonable effects.
Furthermore, some claims that make it into the piece are at odds with the data:
> Strikingly, not one interviewer reported noticing that he or she was conducting a random interview. More striking still, the students who conducted random interviews rated the degree to which they “got to know” the interviewee slightly higher on average than those who conducted honest interviews.
While Table 3 in the paper shows that there is no statistical evidence for this claim as the effects are swamped by the variance.
My point is not that this article is wrong; verifying/debunking the claims would take much more time than my quick glance. But that ought to be the responsibility of the newspaper, and not individual readers.
Politicians don’t get to write about the successes of their own policies. While there is a difference between researchers and politicians, I think we ought to be a bit more critical.
Of course we should be more critical, the methodology here is far from perfect. But right now, people have much faith in interviewing, and this research suggests that this faith may not be justified. As you mentioned, this is not the only research that reaches this conclusion. There is also the work by Daniel Kahneman[1] which I find pretty rigorous and draw the same conclusions about interviewing.
So obviously, the title of this article should be "Maybe interviewing is not that useful" instead of "The utter uselessness of job interviews", but besides this I find your comment unjustified.
In fact, it's quite the opposite, I believe this type of work contributed to make us more critical by questioning some basic facts about interviewing, that i would have never questioned just a couple of years ago.
> Over time I’ve become more skeptical about this kind of psychology research (as more studies fail to replicate)
It's easy to present data that shows flaws with today's interviewing.
It's a lot harder to present a better way of predicting candidate performance in the workplace, along with substantial data that indicates it's better than today's methods. Corporations would love more effective ways to determine effectiveness/performance before hiring.
Interviewing is terrible, but that doesn't mean there is a better option.
Nonsense. First step is to acknowledge that interviewing doesn't work very well. You can't keep deluding yourself because you haven't found a better way. Accept reality.
This is one of the biggest problems I see with business guys today. They want absolute certainty in a world which can't offer it. Start accepting that there is a lot of stuff we don't know and can't know at the moment and we simply have to work towards getting better.
You can't get better if you don't acknowledge that there is a problem in the first place.
These silly, must have +5 years of experience, and check all these boxes with technologies to get the job clearly shows the industry is completely lost at the moment and isn't willing to acknowledge it.
Of course people know interviews aren't the greatest, but it's the best tool they have right now. What people are pushing back against is criticism without solutions.
There are a bunch of other testing methods that are illegal in the USA too, such as IQ tests.
When users of code I've written complain that something sucks to use, I don't demand they come up with a better solution. Probably I don't even want to know what they think the answer is—unless they design user interfaces for a living, odds are their answer will be terrible and they won't even be happy with it if I build it.
This is as it should be—I'm paid to solve problems.
Why is this different?
The more I read about interviewing, the more I realize too many people think they have this problem solved—their amateur psychology is impeccable and their technical screens test for exactly the right things, no more and no less. Did they do a bunch of controlled studies to convince themselves of this, or are they taking sounding good, or intuition about the statistical outcomes of different techniques, to be equivalent to truth?
Maybe the first step is to collectively realize we have close to no clue what we're doing, and are being asked to solve a hard problem: individually, to talk to someone for an hour and make a hiring recommendation. In aggregate, to make the decision based on a handful of these one-hour conversations.
Maybe the first step is to realize this is a problem worth trying to solve.
> What people are pushing back against is criticism without solutions.
Which is bullshit. It's perfectly reasonable to criticise something without proposing an alternative. It's especially ridiculous to reject criticism provided without alternatives, when it's literally your job to do the work being criticised.
"Hey the way you're doing this part of your job produces results no better than random selection."
I still don't understand why employers can't simply set up a two week "trial contract" where as promising candidates simply work for two weeks so everyone can actually see and judge, with real world empirical data, how well the person does in the environment at the actual job.
Yes yes..of course I know this could be gamed as well, but no matter...you can't really argue that this wouldn't be magnitudes better then the typical current/broken interview process.
This will get you the most desperate employees, not the ones you want. I've got a home loan to pay, I'm not switching jobs if you can only guarantee 2 weeks of employment. And if there are multiple offers around, I'll take the 3 month or full time position, even if I'm half way through the 2 week contract.
Also, not all jobs/codebases lend themselves to being productive in 2 weeks, I'd argue they should be, but they aren't.
The concept of a two week trial contract is interesting, but also as flawed as anything else.
I switched roles to a new team, and the first 2 weeks were a trainwreck. It was almost of no predictive quality on how I would do.
Now, there are many reasons why that was the case, and perhaps those underlying issues should be addressed, but from all the role changes I've had, the first two weeks shows how well the group you're going into can onboard, more than it shows how productive the individual will be in the long term.
Yes, because it would be magnitudes worse. So do you give the assignment to? You have several hundred applicants, do all of them get the 2 week assignment? Who watches over them and answers their questions? After spending that much money, is the answer you get any better than a set of interviews
On the other hand, assume I am in a job and want to change positions, for any number of reasons... How many two week jobs do I have to take? Or should I quit my job first?
In the 2004-2007 timeframe, the company I worked for hired software engineers via a staffing company for three-month contracts. We interviewed the candidates with the intention of making a full-time hire. As the contract term approached, the management team did a 360 review, the decided to offer a full-time position or just not-renew the contract. This had some downsides, but overall I found it to be better that alternate approaches I've tried before or since. It stopped being viable once software engineering became a sellers market.
This seems to work ok for companies working on green field stuff, though you'd still probably struggle to entice people to leave their jobs for you. For other companies though, where technical debt and poor management is everywhere you look it doesn't. It gives employees a chance to see what they're really dealing with and to look for work two months later.
Indeed, though any kind of arrangement is difficult. You might end up in a dead end job, not matching your skills or otherwise soul crushing regardless of the method.
Counterexample: I did not apply to, nor interview for, my current gig, and I'm not cheating by working for myself.
And I bet almost everyone reading this has worked with at least one person they think shouldn't have survived the interview, and that person was making a boatload because they convinced the boss they're brilliant. Meanwhile 90% of their day was spent talking about how great they are, and 10% creating new bugs, and no one dared say anything because the thought of them being more "productive" was horrifying.
'It' mostly successfully matches employees to employers, but the quality of those matches may vary wildly.
Interview processes also vary wildly—you can't really say that it works without defining 'it.' Are we talking multiple technical screens that require writing code or a single fluffy buzzword-laden conversation with a C-level? Both have failure modes, but those failure modes sure are different.
> Interview processes also vary wildly—you can't really say that it works without defining 'it.' Are we talking multiple technical screens that require writing code or a single fluffy buzzword-laden conversation with a C-level? Both have failure modes, but those failure modes sure are different.
End of the day, everything evens out. People add the structure they need when they hire people. If your engineering interview process for some detail oriented gig is buzzword trivia with the CIO, the company will probably tank anyway. Conversely, if you do some nerd-fest whiteboard interview for a CTO in a bigger organization, you're probably not getting the right outcome either.
Did you read the article? People add the structure they think they need to interviews, but are clearly able to troll themselves into worsening their judgments by requiring steps that not only don't help, but actively hurt.
And this is a near universal phenomenon. Almost everyone wants to "get to know the candidate."
That's a syllogism and I'm not sure it tells us anything. If you replaced interviewing with a footrace across hot coals the same assertion would be (just as vacuously) true.
Everyone hates the process, and companies have investing major dollars and hours trying to improve. End of the day, little has changed since 1917. You either acquire-hire, get a strong referral or interview a pool of unknown applicants.
The tests and quizzes are little different to how a city hired an accountant in 1917. The old boys network evolved. Then you're left with the rest.
I strongly disagree. There are all sorts of situations where having a bad option is much worse than having no option, hiring and medicine clearly among them.
> Corporations would love more effective ways to determine effectiveness/performance before hiring.
This is irrelevant to evaluating the current methods. Even if there is no replacement, if they are useless, we should know it, bar none.
That feels a bit like saying we should just stick to blood letting and leeches because we don't know any more effective way of treating disease. Just because we don't have a superior alternative doesn't actually mean the current method is effective.
Leeches are often used in literal modern hospitals in the developed world as an effective treatment for certain ills.
B)
If we had no alternative, we should stick to it, but look at other things in the meantime. You seem to be advocating doing nothing at all, which is trivially easy to show works for no-one
I doubt there's that much, in general, since a lot of jobs just need a floor. But when it's used as a proxy for IQ test, the better you do at algorithms the higher IQ you probably have, and that typically correlates with better job performance, or being able to transform one's job into one with higher impact. (Even more if they have high Conscientiousness too but I don't think algorithms would correlate much with that.) It's also a weak proxy for seriousness. I hate algorithm questions (though fortunately not algorithms), almost everyone I talk to hates them, but if you're not expecting them and don't at least know the basic ones, you're not serious about applying to random tech job. (Which is fine, I don't mind that some people will get in a huff and walk out when asked to write a graph search algorithm or something as if it's beneath them or useless since the day-job never does such things, they just weren't serious about applying to that company.) Some tech companies have gotten rid of them, which is great, but you can't count on that yet as a candidate.
The problem with testing algorithms is that it in no way tests intelligence. I would think that 9 out of 10 programmers that know an algorithm would not be able to derive the algorithm from first principles. So you are just testing esoteric knowledge - it's qualitatively no different that asking someone questions about a specific framework / API.
You could make the argument that algorithms tend to be studied more by smarter people, but if that's what you're going for you may as well ask them about their hobbies, and hire the person that is into playing chess, or doing astronomy (or whatever intellectual pursuit you care to name).
If on the other hand you are interested in a person's ability to code, ask them to do so. The last time I had to hire someone, I wrote a small application with one module that was deliberately written in an obfuscated style. I asked candidates to bring that module under control - rewrite it in a readable code style. To do this, successful candidates needed to identify what the current code was doing by examining the public interfaces in a debugger, documenting what the calls seemed to do, prepare unit tests, and then rewrite the module in a readable style. It took about a day for most candidates to do.
At the end of that, you get to see a candidate's ability to read code, use a debugger, write unit tests, write documentation, and write well structured code, which is a pretty good coverage of the typical tasks in a developer's day. I feel this gives a much more realistic assessment of a candidate's capabilities that asking questions about a more or less randomly chosen algorithm.
I don't think someone with an IQ of 80 could program Djikstra's algorithm given a mathematical description of it with diagrams. I'm even skeptical of programming binary search. They might understand an intuitive explanation involving a phone book but I don't think they could program it. And even if they could, I think someone with an IQ of 120 would do it much faster, though both solutions would likely have the integer overflow bug that was even in Java's implementation for a long time. So I think algorithms do test intelligence, just not as well as an actual IQ test. It can easily be gamed by sheer memorization, whereas good IQ tests can't. I agree that other things like ability to play chess would probably test just as well as algorithms. If the industry switched to testing candidates to see if they can solve chess problems, or play a computer self-limited to some specific ELO, you can bet that everyone who was serious about getting a job in the industry would start playing a lot of chess, and those with higher IQs will on average play better chess.
When I have to give an interview and have to include an algorithms section I make candidates type code. Whether that's on a phone screen with a shared online text editor or in person with their laptop / an interview laptop, I want them to type stuff, not just rely on whiteboard pseudo-code and diagramming. As a vim user I discount that their editing environment may not be what they're used to but even if I was forced to use Notepad I could still bang out a function to test the even/oddness of a number (my own fizzbuzz) pretty quickly. So I at least make sure to test coding, even if poorly.
I agree work-sample tests are the best, but as another commenter noted if they take a lot of time for the applicant you're going to get people who refuse that just as some refuse to play the algorithms game. Especially if people have a github repo, especially if some of the projects they've worked on have had more than themselves as commiters, especially if they're currently employed as a developer at some other company that does general software. Unless you're trying to build a top team, which most projects don't need, you're wasting a lot of time trying to rank beyond "would work out ok" and "would not work out at all". I have a section in my phone screen that tests for regex knowledge, I'm primarily just testing to see if they know the concept or if when faced with a problem that regexes can solve (which actually does happen from time to time) they reach for writing some custom parser or not. If they vaguely remember there's a way to specify a pattern and find matches, that's a Pass. If they know grep/their language of choice's regex syntax and can give a full solution, great, I'll rank them slightly higher than someone who just knows the concept, but all I really care about is the concept. If they don't know the concept, that's a strong sign (to me) they won't work out.
I tried to do a semi work sample test with an intern candidate a few months ago instead of a different test, based on experience with a prior intern who struggled on something I thought was basic and left me wondering why I didn't catch that in the phone screen. Basically I gave them some stripped down code from several files that looks a lot like what we have in production (JS, using Backbone) explained the overall mapping from bits of code to what could be shown on the screen, and essentially asked them to add a new component (already written) to one part of the screen by modifying/filling-in-the-functions in a few places. It required them to read and understand some alien code, see what they can ignore, understand what was asked, and then do it (initialize something, pass it around, up to I think 3 indirect function calls of nesting, call a couple things on it). The candidate got through it, I'm not sure the old intern would have...
In my experience, IQ tests are a good indication of your ability to take IQ tests and very little else. Of course, you can differentiate an absolute dunce from someone who's not, but nothing more subtle than that.
Personnel selection research provides much evidence that intelligence (g) is an important predictor of performance in training and on the job, especially in higher level work. This article provides evidence that g has pervasive utility in work settings because it is essentially the ability to deal with cognitive complexity, in particular, with complex information processing. The more complex a work task, the greater the advantages that higher g confers in performing it well. Everyday tasks, like job duties, also differ in their level of complexity. The importance of intelligence therefore differs systematically across different arenas of social life as well as economic endeavor. Data from the National Adult Literacy Survey are used to show how higher levels of cognitive ability systematically improve individuals’ odds of dealing successfully with the ordinary demands of modem life (such as banking, using maps and transportation schedules, reading and understanding forms, interpreting news articles). These and other data are summarized to illustrate how the advantages of higher g, even when they are small, cumulate to affect the overall life chances of individuals at different ranges of the IQ bell curve. The article concludes by suggesting ways to reduce the risks for low-IQ individuals of being left behind by an increasingly complex postindustrial economy.
Employees will test the limits with lawsuits. I know at least one large state lost a class action lawsuit due to racial bias on civil service exams.
The problem with comparing IQ results at hire to employment success is that employment outcome is difficult to define over time. You're also unlikely to get statistically relevant data without focusing on large organizations with standardized HR processes. Most of the research is based on supervisory evaluations, which are not the most reliable indicators of anything for a variety of reasons.
The other thing I find amusing is that business folk who talk about this miss the fact that there are large workforces in the US that either have or do use standardized testing like this to hire and promote. Those are government bureaucracies, which function relatively well, but are hardly a model that most folks advocating this would aspire towards.
My point is not that this work is flawed, or that there should not be an article reporting on this research or the topic of interviewing. Rather, I think it'd be better for a third party to write about the topic in a more objective manner, rather than a professor promoting his own research (and thus with skewed incentives).
In particular, I was disappointed to find a (short) paragraph in the article that I find bogus. That does not mean the article shouldn't have been posted in the first place, but just that this paragraph should have been edited or removed.
I think there is something wrong when I feel like I have to look up the actual research paper and check whether the claims made in an article are supported by data and methodology. I should not be a skeptic when reading New York times articles.
To be fair, it is posted in the opinion section, but should we really just take this article as an opinion? That doesn't feel right to me either.
Then onto your last question and Daniel Kahneman, we can talk about that for a long time, but let me keep it short. The best place I know (though technical) is the blog by Andrew Gelman ([1][2] turned up in a 5 second Google, but there is way more on his blog), and Daniel Kahneman himself has "admitted" flaws in his studies [3][4].
I worked in the IT arm of a household name US non-IT company (just to provide basic context). Eventually we determined that interviewing just wasn't that helpful. We started telling the recruiting company, "Send me the best you've got that's available by Tuesday." Now certainly the recruiter is going to do some selection biasing there, but we found we had just as much success as our previous interviewing process that had a phone interview, face to face interview, and a test.
The fact that psychology (and other social sciences and even medicine) have a replication crisis in spades is well known. Just Google it, if you haven't been following the general science news for the last few years.
Even the "hard" sciences have trouble, because journals prefer publishing new and positive results, rather than replications or negatives.
The claim you mention about getting to know the person better in random interviews as well as "People can’t help seeing signals, even in noise." is misleading and unsurprising.
People ask questions in interviews that they want to know the answer to and that could go either way. All questions have equivalent expected surprise, either both answers are unsurprising, or one answer is surprising, but you think you already know the answer is the other one.
If interviewers were asking questions like "is 2+2=4" they would have detected random interviewers way easier, but they wouldn't be trying very hard to get to know the person.
As for getting to know the person, the more surprising someone's answers are, if you believe they are telling the truth, the more distinguishes them from "average person who gave answers I expected", so you say you "got to know" them. This is unsurprising.
This isn't to defend unstructured interviews, other studies for a long time have shown them to be worse than structured interviews and test scores. If I had to guess the only reason the research in the article got published as novel was the random interview part.
Edit: Here's a table from a meta-analysis of lots of studies on correlation between different factors and job performance: http://imgur.com/a/YRFTh. Basically unstructured interviews aren't as good as structured ones, but they are better than nothing. Work samples are the best, structured interviews and IQ tests tie for second.
Note that this meta-analysis is combining a bunch of different fields to yield general observations, a specific field may have different results. But in expectation for a randomly selected field these are fairly solid results, and I don't expect fields vary from them too much.
They also run expensive ad campaigns promoting their own policies, at least here in Australia. They don't even have to be successful policies to be able to talk about them...
I think the parent means, we don't take as objective fact a politician's claims that their policies worked; we interpret it as self-aggrandizement and scrutinize the claims quite heavily. And that—given the push to publish and the fact that null-result studies aren't very publishable—we should likely do the same for research conclusions.
I understood, but I think recent evidence in the US suggests that a great many people absolutely take as objective fact what a politician claims. A similar number probably absolutely believe the opposite with little or no evidence. Net, I think citing politicians was perhaps not the best analogy.
Sorry I was not more clear. What I meant was that the NY Times employs journalists and fact checkers and editors to validate stories, say on politics to make sure, to the best of their abilities, that the articles they post are correct.
Why is that not the case here, where a professor is allowed to sell his own work? It is as if Obama is the NYT reporter for Obamacare.
That people believe politicians blindly is a topic for another day :)
Unless you're talking about a very high-level position that either requires substantial leadership traits or very specialized knowledge, interviewing has never been about finding the right person for the job, it's been about finding a right person for the job. The reality is that most jobs can be done successfully by a large number of people. The accuracy of the assessment of the candidate's abilities is, to my mind, a secondary concern in the interview process and yet, just as in this study, is the only part of the interview process studied and critiqued.
But what I find far more important in the interview process is involving current team members in the process of selecting new coworkers. It's one way of getting teams to be bought into a feeling of shared purpose and is the first step in establishing a working relationship. If you don't give at least some of the current team a role in the hiring process, teams will feel imposed upon by those hiring and won't be as understanding about flaws in those added to the teams.
Focusing on selecting the "right candidate" is really myopic in a situation where there are likely many right candidates. We should, instead, be focusing on not selecting a wrong candidate and fostering the right team dynamic. We already know that strong teams significantly outperform strong individual performers who don't cooperate. Yet hiring still seems focused on optimizing for strong individual contributions. And I've yet to see a study that looks at the flaws of the interview process in building ineffective teams.
I think you may be misconstruing the argument here. I agree that any number of people will fit -- but the problem is, what if interviews aren't even selecting "a" right person for the job?
> what if interviews aren't even selecting "a" right person for the job?
We know that interviews are selecting a right person some percentage of the time. That percentage will never be 0 or 100. We will always have to accept some bad hires.
I would rather that successful hire percentage be somewhat lower and keep the team engaged in defining culture and hiring standards than to give up team involvement for a higher right-person rate.
My main point is that the people studying this issue and arguing against the interview process tend to only look at less than half of the benefit of the interview process. If we're going to ditch the interview process, whatever replaces it needs to have the same property of involving the existing team or it is, to my mind, automatically worse than what we have now. Because while we're all aware of how flawed the interview process can be, it does work to some extent. When I was hiring, only 2 out of the 50 or so that I hired didn't work out. Would I like to have avoided hiring one or both of them? Sure. Am I willing to sacrifice the team involvement benefits to got to do so? Absolutely not.
I don't see the problem. That researchers haven't done a perfect study in every possible way is hardly the issue here, but rather the blind trust in our ability to interview and hire people. This research paper does not come out of the blue. This is something that has been observed by many organizations and people for quite some time, that interviews are broken.
In fact there is an awful lot of wrong stuff society keeps doing despite evidence over many year suggesting it is stupid. The stupidity of high CEO salaries e.g. has been proven quite well, yet business keeps a blind faith in it. Getting employees in complicated jobs to perform by rewarding them on extremely narrow metric has also proven itself counterproductive yet the MBA crowd refuse to believe it doesn't work.
Business practices often seem to be more like religion than founded on reality. It is a GOOD thing that some researchers are trying to do their bit to correct this flawed picture.
> I find it quite problematic that researchers get to talk about their own research and present it as facts without anyone taking a critical look.
Are you suggesting anything out of the ordinary is going on here? Doesn't one have to present their work as legitimate and wait for feedback? So long as they are open to that feedback, I don't really get this argument?
> Over time I’ve become more skeptical about this kind of psychology research (as more studies fail to replicate) and, as is often the case, here the sample size is quite small (76 students, split across 3 groups), predicting something relatively noisy as GPA. It is unclear to me that one would be able to detect reasonable effects.
So present that back them as a refutation? Are you waiting for someone else to do it? Why are you debating the reliability here?
Structured interviews are better, but not really by much.
The problem with all of these things is they work, but not very well, and people make too much of any one thing. An interview is a very small slice of behavior, even if it's structured very well.
One quick comment: in general it is a bad idea to compute the inverse of a matrix (to solve a linear system). It's much better to compute the QR factorization or SVD instead (or simply call least square solver).
GMRES is definitely my go-to these days. Though it is worth noting that direct methods do have a benefit of letting you quickly solve many successive linear problems involving the same matrix, but different right-hand sides. But iterative methods scale very well for large sparse problems that they are very often the only tool to consider.
Block Krylov methods are a thing , but I haven't experimented with them yet.
Has anyone implemented them in production? I read their paper a while ago, but thought that the dual methods they showed might have practical limitations (like insane cache misses, compared to more standard methods)
These problems you sketch can already be overcome by using the approximate eigenspectrum info of the matrix obtained as a by-product of Krylov methods. This has been invented many times, and is used in 'thick restart', 'augmented Krylov subspace methods', 'deflation'. In particular this was developed for scattering problems with many electromagnetic incident waves hitting an object, changing only the rhs.
Also, you can also use an approximate LU-decomp as a preconditioner for Krylov methods.
Similarly, Tykhonov regularization (solving for a range of slightly perturbed matrices "A + labda I" where labda is a parameter) are easily tackled using Krylov subspace methods by noting that the Krylov subspace is invariant under shifts like these. So only once an orthonormal basis must be found for the Krylov subspace, which can then be used for every lamda of interest.
Depends on the number of unknowns, sparsity of the matrix, whether you want 3 correct digits or 16, etc.
Direct methods as Gaussian elimination with pivoting are proven to be stable. Iterative methods are not but can be a lot cheaper in computational costs. Also they can stop when a certain relative residual is reached, unlike direct methods.
If iterative methods like Krylov subspace methods where stable, then they could actually be seen as direct methods themselves, as the Krylov subspace has at most a dimension of N, where N is the number of unknowns; so after at most N iterations the solution could be extracted exactly from the search space. In practice this is not the case due to rounding errors.
Not very important and for a learning project I find using cvxpy a better idea as it's more readable ( like you did ) but:
Solving the full quadratic optimization problem for SVMs in basically impossible to do. You are forming an n^2 matrix, so I'm going to let you imagine what happens when n = 100 000.
Using people use either approximation methods ( Incomplete Cholesky, Nystrom ) or do it exactly but iteratively ( SMO, Pegasos... )
I'm implementing them for class right now so it's still fresh in my head haha
This short course is taught every quarter at Stanford, usually by a PhD student in ICME (Institute for Computational and Mathematical Engineering). This is the course page for the class as was taught a couple of years ago.
Edit: google CME193 to find more course pages, if interested
For a voting system (ranking of some candidates based on preferences of voters), it would be nice if:
- A single voter cannot determine the ranking (as a dictator)
- For every possible set of voter preferences, there is an outcome (not random)
- If everyone likes candidate A over candidate B, then in the final ranking candidate A should be ranked higher than candidate B
- If one prefers A over B when comparing just A and B, then one should also prefer A over B when an additional option C is offered
Sounds like some reasonable properties for a voting system, right?
Well, the theorem states that if there are more than 2 candidates, then there is no voting system that has all 4 properties above.
> Well, the theorem states that if there are more than 2 candidates, then there is no voting system that has all 4 properties above in the general case.
Nobel Laureate Amartya Sen[0] has demonstrated that, while there is no system that satisfies all four characteristics in the general case, there are systems that either satisfy all four conditions either probabilistically or satisfy all four conditions subject to some very weak assumptions.
The example I've heard him use is of the 2000 election in Florida, with Bush, Gore, and Nader (let's ignore Buchanan for simplicity). While technically there are 3! = 6 possible ways to rank the candidates, in practice, the ranking (Nader, Bush, Gore) is much less likely than (Nader, Gore, Bush) or (Gore, Nader, Bush). If we introduce one minor assumption about the relative frequencies of the rankings, we can prove that instant-runoff voting[1] does always satisfy all four of Arrow's criteria[2].
To use an analogy from computer science, the halting problem is undecidable in the general case, but that doesn't prevent static analysis tools from spotting many infinite loops; it just means it can't spot all infinite loops with 100% accuracy.
[2] A different example: instead of making assumptions about the relative frequencies, we could make assumptions about the number of axes that candidates may have and the way they cluster around them. This realistically depicts both two-party and multiparty elections in most parts of the world, since political positions are not uniformly distributed along n dimensions.
You're right that it was a smaller group, of course, but it wasn't an empty one.
Jello Biafra's endorsement of Nader plus Gore's attack on Twisted Sister for the Parents Music Resource Council means N-B-G was actually the order of preference for anyone where "the right to rock out" was their single voting issue.
In college I was part of a club that had an elaborate election procedure for officers. I'm pretty sure it violates Arrow's Theorem, but it was also nonterminating!
Oh, god. I don't remember the details, but it involved crossing off the bottom third of candidates every round until there was only one left. But somehow, this didn't always get rid of everyone, and you could get stuck in a state where there was no bottom third.
> the theorem states that if there are more than 2 candidates, then there is no voting system that has all 4 properties above.
There is no rank order voting system that has all those properties. (Rank order means, that the voter puts the candidates in the the order of preference: 1., 2., 3. etc.)
But there are other voting systems, e.g. a system where you give each candidate points between 1-100 or whatever, and you can give the same amount of points to several candidates if you want.
Regarding Arrow's theorem, most voting systems regarded as better than first-past-the-post tend to choose to violate the third criteria, commonly called "independence of irrelevant alternatives": adding another candidate shouldn't change the preference order of existing candidates.
>If one prefers A over B when comparing just A and B, then one should also prefer A over B when an additional option C is offered
That assumption seems fishy to me if in a real life application, in particular because we assume offering new choices can't fundamentally change the agent/voter's preferences isn't exactly true to life. I'll give a real life, but slightly historically oversimplified example:
Say I am a poor serf living in pre-revolution Russia about a century ago. I am usually offered the following two choices:
A. serve the bourgeois royal family and live off of their scraps
B. die like a dog
Then, one day, Lenin (or similar revolutionary) comes along and offers me:
C. fight for glorious USSR communism
In real life, I would think the offered choice of C would absolutely change my preferences for A and B.
Granted this doesn't invalidate the Arrow paradox in any way, I'm just saying the paradox isn't true to life because one of its core tenants doesn't quite hold.
> Granted this doesn't invalidate the Arrow paradox in any way, I'm just saying the paradox isn't true to life because one of its core tenants doesn't quite hold.
However, this plays rather fast and loose with the principles behind Arrow's theorem. First, you're violating the nondictatorial condition (you are the only person voting on your own choice), so it's kind of silly to apply the theorem here in the first place.
Secondly, you're saying that, as soon as Communism becomes an option (but only then), you'd rather die than serve the royal family.
I can see how this might be the case, but if so, you're fundamentally changing the meaning of choice B. It's no longer just the default state ("die of starvation"), but rather dying for a cause (Communism). It's not that the introduction of choice C suddenly means that you're now choosing B over A; it's that it also introduces choice D ("die as a martyr in the name of Communism", as contrasted with "die for no particular cause"). Your ranking is now C > D > A > B.
You have to understand that Arrow's theorem came about as the result of an effort to understand the way that firms make decisions, and the ways that voting structures of boards can influence corporations to make non-optimal decisions for the corporation. Your situation doesn't really apply to the definition of independence of irrelevant alternatives as implied by the context in which Arrow's theorem is applied (or the formal proof thereof).
>Secondly, you're saying that, as soon as Communism becomes an option (but only then), you'd rather die than serve the royal family.
Well, maybe. Supposing there's some information contained in option C, or the mere availability of option C, that reveals to you the futility of option A.
In other words, your preferences aren't necessarily transitive because the availability or unavailability of particular options are themselves a little information payload that could influence the decision.
Are the counterexamples offered by the theorem pathological, in the sense that they are unlikely to occur in practice but are theoretically possible? Or would they arise in practice frequently using standard rank voting systems?
As long as politics is not one-dimensional, there are completely reasonable cases where you get a rock-paper-scissors situation between 3 top candidates if voters select whomever is closest to them. That in turn violates the criteria that says the election result can't change if you add a non-winning candiate (rock beats scissors, but when paper enters the race now scissors is computed the winner).
This criteria is called "independence of irrelevant alternatives". A common criticism of Arrow's theorems usefulness is that it is a bit of a stretch to call paper an "irrelevant alternative" when rock-paper-scissors forms a cycle of preferences like that.
If politics is one-dimensional ("single-peaked preferences" in the article), then you never get this situation.
It's also important to note that a much more reasonable criteria exists called "local independence of irrelevant alternatives". This is the idea that total losers joining the race don't affect the result, however someone who is in the top rock-paper-scissors cycle (the "Smith Set") can still affect the result even if they don't win themselves. This is far more reasonable, as it's fairly arbitrary which of the candidates in that top cycle should win.
When there is no top cycle, the Smith Set is a single person (the "Condorcet Winner"). When there is a Smith Set, most reasonable voting systems will pick their winner somewhat arbitrarily as a member of the Smith Set, since everyone in the Smith Set beats everyone outside the Smith Set.
> Are the counterexamples offered by the theorem pathological, in the sense that they are unlikely to occur in practice but are theoretically possible? Or would they arise in practice frequently using standard rank voting systems?
Which particular problems occur, and the frequency with which they occur, depend on the particular voting system.
Plurality and majority/runoff (which are ranked preference voting systems with a vary narrow constraint on the preferences that are expressed on the input ballots, which is pretty much the same constraint as on the preferences reflected in the output of any single-winner voting system) hits problems fairly frequently in practice, but most of the common voting systems that most people think of as ranked-preference systems (IRV, etc.) still hit them in practice as well, though not generally as frequently and in ways which create as clear incentives to tactical voting.
The "worry" is not that this-or-that fantastical scenario might play out. It is that, as it turns out, the very mechanisms we use to measure group preferences simply cannot satisfy a list of very basic requirements. For example, the reason we do not use random drawings to determine who gets elected president is that we want the choice to "reflect" our preferences on the whole: but the result shows that this kind of "reflection" is probably not possible, and is always distorted in some fashion by the very procedures we adopt to make these decisions.
In "standard" FPTP-like systems we frequently see real-world cases that encourage tactical voting. E.g. look at the French Presidential election where Le Pen went through to the final round because there were four leftist candidates that split the leftist vote.
> We will attempt to directly build safe and beneficial AGI, but will also consider our mission fulfilled if our work aids others to achieve this outcome.
> Our primary fiduciary duty is to humanity. We anticipate needing to marshal substantial resources to fulfill our mission, but will always diligently act to minimize conflicts of interest among our employees and stakeholders that could compromise broad benefit.
> We will actively cooperate with other research and policy institutions; we seek to create a global community working together to address AGI’s global challenges.