Something else to consider is that "reasoning" (marketing term having nothing to do w/ actual reasoning) will only work for problems that can be broken up into chunks of quadratic complexity such that each chunk can then be re-encoded into the context to carry out more quadratic operations. I don't know if there is a theorem in CS proving obstructions for breaking up problems into such chunks but I think it is intuitively obvious that there will be problems which can not be solved w/ such restrictions.
There are only 4 optimizations in computer science: inlining, partial evaluation, dead code elimination, & caching. It looks like AI researchers just discovered inlining & they already knew about caching so eventually they'll get to partial evaluation & dead code elimination.
Don't know about the others, but FFT is the classic case of common subexpression evaluation (its mathematically equivalent), which I think by OPs definition would fall under caching.
That's a bit trite tbh. We all know of these techniques, but actually implementing them on GPUs in a low-overhead manner that maintains the model's fidelity is challenging. It's much more than just breaking out the old CS book and picking the next idea from there.
Your list is so short it doesn't even include the basics such as reordering operations.
It also feels incredibly snarky to say "they knew about caching" and that they will get to partial evaluation and dead code elimination, when those seem to be particularly useless (beyond what the CUDA compiler itself does) when it comes to writing GPU kernels or doing machine learning in general.
You can't do any partial evaluation of a neural network because the activation functions are interrupting the multiplication of tensors. If you remove the activation function, then you end up with two linear layers that are equivalent to one linear layer, defeating the point of the idea. You could have trained a network with a single layer instead and achieved the same accuracy with a corresponding shorter training/inference time.
Dead code elimination is even more useless since most kernels are special purpose to begin with and you can't remove tensors without altering the architecture. Instead of adding useless tensors only to remove them, you could have simply used a better architecture.
I think you can. If you have a neuron whose input weights are 100,-1,2, with threshold 0, you can know the output of the neuron if the first input is enabled, as the other 2 dont matter, so you can skip evaluating those.
I'm not enough of an expert to see if there's any actualy merit to this idea, and if you can skip evaluating huge parts of the network and keeping track of such evaluations, is actually worth it, but it intuitively makes sense to me that making an omelette has nothing to do with the Battle of Hastings, so when making a query about the former, the neurons encoding the latter might not affect the output.
Afaik, there's already research into finding which network weight encode which concepts.
MOE is a somewhat cruder version of this technique.
AI actually has some optimizations unique to the field. You can in fact optimize a model to make it work; not a lot of other disciplines put as much emphasis on this as AI
Yes, in "runtime optimization" the model is just a computation graph so we can use a lot of well known tricks from compilation like dead code elimination and co..
For inference assorted categories may include vectorization, register allocation, scheduling, lock elision, better algos, changing complexity, better data structures, profile guided specialization, layout/alignment changes, compression, quantization/mixed precision, fused kernels (goes beyond inlining), low rank adapters, sparsity, speculative decoding, parallel/multi token decoding, better sampling, prefill/decode separation, analog computation (why not) etc etc.
There is more to it, mentioned 4 categories are not the only ones, they are not even broad categories.
If somebody likes broad categories here is good one: "1s and 0s" and you can compute anything you want, there you go – single category for everything. Is it meaningful? Not really.
A uniquely important thing that a CEO brings to the table is accountability. You can't automate accounta- ...sorry, I can't continue this with a straight face :DDD
You can only replace someone who was useful. If one is useless, but is still there, it means they are not there for their contribution and you can't replace them by automating whatever it might have been.
The thing about C-suite executives is they usually have short tenures, however the management levels below them are often cozy in their bureaucracy, resist change, often trying to outlast the new management.
I actually argue that AI will therefore impact these levels of management the most.
Think about it, if you were employed as a transformational CEO would you risk trying to fight existing managers or just replace them with AI?
>I actually argue that AI will therefore impact these levels of management the most.
Not AI but bad economy and mass layoffs tend to wipe out management positions the most. As a decent IC, in case of layoffs in bad economy, you'll always find some place to work at if you're flexible with location and salary because everyone still needs people who know how to actually build shit, but nobody needs to add more managers in their ranks to consume payroll and add no value.
A lot of large companies lay off swags of technical staff regularly (or watch them leave), and rotate CEOs but their middle management have jobs for life - as the Peter Principe states, they are promoted to their highest respective incompetence and stay there because no CEO has time to replace them.
Disagree with the "jobs for life" part for management. Only managers who are there thanks to connection, nepotism or cronyism, are there for life as long as those shielding them also stay in place. THose who got in or got promoted to management meritocratically don't have that protection and are the first to be let go.
At all large MNCs I worked at, management got hired and fired mostly on their (or lack thereof) connections and less on what they actually did. Once they got let go, they had near impossible time finding another management position elsewhere without connections in other places.
Yes I was talking about middle managers mostly. Upper management, C-suite, execs are mostly protected from firing unless they F-up big time like sexual assault, hate speech, etc.
Ultimately, no. But when we get to this point - once we have AI deciding on its own what needs to be done in the world in general - then the bottom falls out, and we'll all be watching a new global economy, in which humans won't partake anymore. At best, we'll become pets to our new AI overlords; more likely, resources to exploit.
The make the decisions so I doubt they will soon themselves to be automated away. Their main risk will be that nobody can buy their products once everything is automated.
I wonder if capitalism and democracy will be just a short chapter in history that will be replaced by something else. Autocratic governments seem to be the most prevalent form of government in history.
They are constantly trying to reduce costs which means they're constantly trying to distill & quantize the models to reduce the energy cost per request. The models are constantly being "nerfed", the reduction in quality is a direct result of seeking profitability. If they can charge you $200 but use only half the energy then they pocket the difference as their profit. Otherwise they are paying more to run their workloads than you are paying them which means every request loses them money. Nerfing is inevitable, the only question is how much it reduces response quality & what their customers are willing to put up with.
Amodei is on the record about completely automating AI research in 6-12 months. He thinks it's an "exponential" loop & Anthropic is going to be the first to get there. That's not esoteric knowledge, that's the CEO saying so in public at the same time that their consumer facing tool is failing & their automated abuse detection is banning users for legitimate use cases.
I don't consider Anthropic to be one of the teams using AI particularly well. They're building the tools, they're not using the tools in the best, most skillful way possible.
This is very cool. I looked at the Claude.md he was generating and it is basically all of Claude's failure modes in one file. I can think of a few reasons why Anthropic would not want this information out in the open or for someone to systematically collate all the data into one file.
i read the related parts of the linked file in the repo, and it took me a while to find your comment here again to reply to. Are you saying that the failure modes of claude with "coding" webapps or whatever OP was doing? i originally thought it might have meant like... jailbreak. But having read it, i assume you meant the former, as we both read the same thing and it seemed like a series of admonitions to the LLM, written by the LLM (with some spice added by OP? like "YOU ARE WRONG") and i couldn't find anything that would warrant a ban, you know?
I'm not saying he did anything wrong. I'm saying I can see how Anthropic's automated systems might have flagged & banned the account b/c one of the heuristics they probably use is that there should be no short feedback loops where outputs of Claude are fed back into inputs. So basically Anthropic tracks all calls to their API & they have some heuristics for going through the history & then assigning scores based on what they think is "abusive" or "loopy".
Of course none of it is actually written anywhere so this guy just tripped the heuristics even though he wasn't doing anything "abusive" in any meaningful sense of the word.
reply