Altman miscalculated badly. OpenAI took what could have been amazing publicity, and in a rush to publish, gave reason for users to distrust their core product.
I have found satisfaction in thinking about activation multidimensionally (free energy surfaces). It lends the interpretation of reactants trying to escape gravity well-like local minima, crossing marginally more unstable transition states* to reach more stable global minima.
*A misconception is that transition states are local maxima. They are first order saddle points: maxima along one direction but minima along every other direction.
To drive a reaction forward, it doesn't always have to be lowering the transition state energy. Another technique is by destabilizing the resting state. In the analogy, the message would be: to not get too settled into one's comfort zone.
I'm glad to find someone who shares this perspective. Sigmoids show up everywhere (e.g. elo). One big area is AI ability. Coding agents have gone from completely unreliable to quite reliable at basic tasks.
Maybe this could be true about practiced abilities in general. For example, at some point circus entertainers must go from almost never succeeding to succeeding enough to put their lives at stake. It seems that at a certain point, with enough practice or intelligence, you reach a critical threshold where success rate switches from almost never to almost certain.
And I am also a big fan of potential energy surfaces - it always seemed like a huge upgrade going from crude 1D to multi-dimensional reaction coordinates. A big conceptual shift for me was to learn that transition states are not maxima, but first order saddle points.
Though I do wish I could have a better grasp on entropy's role. Also for example PES's connection to diffusion models and flow matching
> The VARIANT type shipped in DuckDB v1.5, and the way to think about it is JSON on steroids. Basically, imagine if JSON were fast. [...] DuckDB automatically detects the common structure hidden in your semi-structured data and “shreds” it, so it compresses well in storage
I am really looking forward to this hitting v2.0. I can't stand uncompressed JSON - so space-inefficient. But heterogenous JSON in parquet files is such a pain because of schema differences causing fields to be silently dropped. Having DuckDB solve this is exactly what I've been looking for.
I'm surprised that hypothesis 2 (that CSV serialization format mangles table columns) was falsified. Back in the gpt-3.5-turbo and gpt-4o era, I did needle-haystack tests and found that table format mattered a lot (csv, tsv, markdown). Most models "could not read vertically" for csv (they were horrible), but they could for markdown. I concluded that serialization format or tokenization played a major role.
Nowadays, LLM performance on csvs is much improved (I'm guessing after being explicitly trained on CSV question-answering.) But I still carry the impression that LLMs read columns only by "memorizing" column positions in a format-dependent manner. Maybe this impression is out of date.
Am I the only one who enjoys taking notes? But more in the sense of recording knowledge so that I have it in perpetuity.
For instance, I have almost completely forgotten how to solve ODEs, even though I had a good command of it when I learned it (by solving practice problems). In that sense, I wish that my prior self had taken good notes, so that I wouldn't have to dig up source material if I wanted to relearn it again.
Everyone likes nicely typeset LaTeX -- why not apply that craftsmanship to preserving academic notes?
It's not that I don't enjoy taking notes - I liked to take notes, distill knowledge, make "cheatsheets" outside of class.
To this day I still take countless notes when I learn new things, I just find it incredibly distracting to have to take notes during a lecture/class because it doesn't allow me to process the material properly.
>In that sense, I wish that my prior self had taken good notes, so that I wouldn't have to dig up source material if I wanted to relearn it again.
I don't really get it. If you've truly forgotten, you have to relearn it one way or another. What makes you think your past student self improvising a half-assed explanation in the moment could do better than a professional author who was already an expert when they wrote the book?
Because you never truly forget something. If you took effort to generate a rehash of your own mental brain model, first you will recognize it again and thus don't need to relearn everything, and second you already have it in your own brain model, instead of needing to invent it first from someone's else brain model.
There must be some merit to retyping LLM generated code, even verbatim. In school, I would rewrite or re-typeset notes as a study habit. In doing so, I'd review content, detect errors, synthesize concepts simply because rewriting notes forced me to pay attention at the per-word level.
While retyping LLM code is not something I personally do, I'd imagine it could bestow similar benefits.
Well elucidated. This problem has irked me for years in the form of multiple inheritance. When it's disallowed (like Java, unfortunately), trying to reduce a directed graph structure to a single dominant hierarchy is quite the bothersome choice.
"beyond comprehension" is a good way of putting it. I've been genuinely baffled by some of these AI designs - why any intelligent thing would write >10 lines of bloat for what should be a one-liner.
> "beyond comprehension" is a good way of putting it. I've been genuinely baffled by some of these AI designs - why any intelligent thing would write >10 lines of bloat for what should be a one-liner.
As Anthropic's drones say: treat Claude as your genius coworker. Don't think yourself, don't judge, the machine must know better than you. It is the genius, after all, not you.
It's like they're allergic to slowing down.