To be fair they could have done an experiment on a transaction that typically requires a person in the loop. Rather than choosing a vending machine which, already, does not require a person in the loop for the transaction.
You’re thinking of replacing a vending machine with a chatbot which indeed doesn’t make much sense. The experiment was replacing the management of the machine. It’s not crazy to think that, money being no object, it would be great to have a person who hangs around the machine, whose job it is to ask customers what kinds of things people might want to buy from the machine, and conduct trials, tinker with pricing, do promotional deals, etc. But that’s of course impractical to have a FTE per machine to do that. The idea of this experiment was to see if that could be done with Claude. And of course as others have pointed out, it’s a simple and cheap and low-stakes version of “a business” suitable for experimenting with.
If I recall, the idea was the AI taking the role of the vending machine manager, choosing and restocking products and such. Anything on top of that was, I assume, just added for fun.
Interesting choice of Pandas in this day and age. Maybe he’s after imparting general concepts that you could apply to any tabular data manipulator rather than selecting for the latest shiny tool.
You can assert whatever you want, but Polars is a great answer. The performance improvements are secondary to me compared to the dramatic improvement in interface.
Today all serious DS work will ultimately become data engineering work anyway. The time when DS can just fiddle around in notebooks all day has passed.
Pandas is widely adopted and deeply integrated into the Python ecosystem. Meanwhile, Polars remains a small niche, and it's one of those hype technologies that will likely be dead in 3 years once most of its users realise that it offers them no actual practical advantages over Pandas.
If you are dealing with huge data sets, you are probably using Spark or something like Dask already where jobs can run in the cloud. If you need speed and efficiency on your local machine, you use NumPy outright. And if you really, really need speed, you rewrite it in C/C++.
Polars is trying to solve an issue that just doesn't exist for the vast majority of users.
Arguably Spark solves a problem that does not exist anymore: single node performance with tools like DuckDB and Polars is so good that there’s no need for more complex orchestration anymore, and these tools are sufficiently user-friendly that there is little point to switching to Pandas for smaller datasets.
> Pandas is widely adopted and deeply integrated into the Python ecosystem.
This is pretty laughable. Yes there are very DS specific tools that make good use of Pandas, but `to_pandas` in Polars trivially solves this. The fact that Pandas always feels like injecting some weird DSL into existing Python code bases is one of the major reasons why I really don't like it.
> If you are dealing with huge data sets, you are probably using Spark or something like Dask already where jobs can run in the cloud. If you need speed and efficiency on your local machine, you use NumPy outright. And if you really, really need speed, you rewrite it in C/C++.
Have you used Polars at all? Or for that matter written significant Pandas outside of a notebook? The number one benefit of Polars, imho, is that Polars works using Expressions that allow you to trivially compose and reuse fundamental logic when working with data in a way the works well with other Python code. This solves the biggest problem with Pandas is that it does not abstract well.
Not to mention that Pandas is really poor dataframe experience outside of it's original use case which was financial time series. The entire multi-index experience is awful and I know that either you are calling 'reset_index' multiple times in your Pandas logic or you have bugs.
"Data Science" has never been related to academic research, it has always emerged in a business context. I wouldn't say that researchers at Deep Mind are "data scientists", they are academic researchers who focus on shipping papers. If you're in a pure research environment, nobody cares if you write everything in Matlab.
But the last startup I was at tried to take a similar approach to research was unable to ship a functioning product and will likely disappear in a year from now. FAIR has been largely disbanded in favor of the way more shipping-centric MSL, and the people I know at Deep Mind are increasingly finding themselves under pressure to actually produce things.
Since you've been hanging out in an ivory tower then you might be unaware that during the peek DS frenzy (2016-2019) there were companies where data scientists were allowed to live entirely in notebooks and it was someone else's problem to ship their notebooks. Today if you have that expectation you won't last long at most companies, if you can even find a job in the first place.
On top of that, I know quite a few people at the major LLM teams and, based on my conversations, all of them are doing pretty serious data engineering work to get things shipped even if they were hired for there modeling expertise. It's honestly hard to even run serious experiments at the scale of modern day LLMs without being pretty proficient at data engineering related tasks.
I have not work with Polars, but I would imagine any incompatibility with existing libraries (e.g. plotting libraries like plotnine, bokeh) would quickly put me off.
It is a curse I know. I would also choose a better interface. Performance is meh to me, I use SQL if i want to do something at scale that involves row/column data.
This is a non-issue with Polars dataframes to_pandas() method. You get all the performance of Polars for cleaning large datasets, and to_pandas() gives you backwards compatibility with other libraries. However, plotnine is completely compatible with Polars dataframe objects.
Pandas is generally awful unless you're just living in a notebook (and even then it's probably least favorite implementation of the 'data frame' concept).
Since Pandas lacks Polars' concept of an Expression, it's actually quite challenging to programmatically interact with non-trivial Pandas queries. In Polars the query logic can be entirely independent of the data frame while still referencing specific columns of the data frame. This makes Polars data frames work much more naturally with typical programming abstractions.
Pandas multi-index is a bad idea in nearly all contexts other than it's original use case: financial time series (and I'll admit, if you're working with purely financial time series, then Pandas feels much better). Sufficiently large Pandas code bases are littered with seemingly arbitrary uses of 'reset_index', there are many times where multi-index will create bugs, and, most important, I've never seen any non-financial scenario where anyone has ever used Multi-index to their advantage.
Finally Pandas is slow, which is honestly the least priority for me personally, but using Polars is so refreshing.
What other data frames have you used? Having used R's native dataframes extensively (the way they make use of indexing is so much nicer) in addition to Polars both are drastically preferable to Pandas. My experience is that most people use Pandas because it has been the only data frame implementation in Python. But personally I'd rather just not use data frames if I'm forced to used Pandas. Could you expand on what you like about Pandas over other data frames models you've worked with?
I initially considered using Pandas to work with community collections of Elite: Dangerous game data, specifically those published first by EDDB (RIP) and now by Spansh. However, I quickly hit the maximum process memory limits because my naïve attempts at manipulating even the smallest of those collections resulted in Pandas loading GB-scale JSON data files into RAM. I'm intrigued by Polars stated support for data streaming. More professionally, I support the work of bioinformaticians, statisticians, and data scientists, so I like to stay informed.
I like how in Pandas (and in R), I can quickly load data sets up in a manner that lets me do relational queries using familiar syntax. For my Elite: Dangerous project, because I couldn't get Pandas to work for me (which the reader should chalk up to my ignorance and not any deficiency of Pandas itself), I ended up using the SQLAlchemy ORM with Marshmallow to load the data into SQLite or PostgreSQL. Looking back at the work, I probably ought to have thrown it into a JSON-aware data warehouse somehow, which I think is how the guy behind Spansh does it, but I'm not a big data guy (yet) and have a lot to learn about what's possible.
R and Matlab workflows have been fairly stable for the past decade. Why is the Python ecosystem so... unstable? It puts me off investing any time in it.
The R ecosystem has had a similar evolution with the tidyverse, it was just a little further ago. As for Matlab, I initially learned statistical programming with it a long time ago, but I’m not sure I’ve ever seen it in the wild. I don’t know what’s going on there.
I’m actually quite partial to R myself, and I used to use it extensively back when quick analysis was more valuable to my career. Things have probably progressed, but I dropped it in favor of python because python can integrate into production systems whereas R was (and maybe still is) geared towards writing reports. One of the best things to happen recently in data science is the plotnine library, bringing the grammar of graphics to python imho.
The fact is that today, if you want career opportunities as a data scientist, you need to be fluent in python.
Mostly what's going on with Matlab in the wild is that it costs at least $10k a seat as soon as you are no longer at an academic institution.
Yes, there is Octave but often the toolboxes aren't available or compatible so you're rewriting everything anyway. And when you start rewriting things for Octave you learn/remember what trash Matlab actually is as a language or how big a pain doing anything that isn't what Mathworks expects actually is.
To be fair: Octave has extended Matlab's syntax with amazing improvements (many inspired by numpy and R). It really makes me angry that Mathworks hasn't stolen Octave's innovations and I hate every minute of not being able to broadcast and having to manually create temp variables because you can't chain indexing whenever I have to touch actual Matlab. So to be clear Octave is somewhat pleasant and for pure numerical syntax superior to numpy.
But the siren call of Python is significant. Python is not the perfect language (for anything really) but it is a better-than-good language for almost everything and it's old enough and used by so many people that someone has usually scratched what's itching already. Matlab's toolboxes can't compete with that.
I love R, but how can you make that claim when R uses three distinct object-oriented systems all at the same time? R might seem stable only because it carries along with it 50 years of history of programming languages (part of it's charm, where else can you see the generic function approach to OOP in a language that's still evolving?)
Finally, as someone who wrote a lot of R pre-tidyverse, I've seen the entire ecosystem radically change over my career.
The pandas workflows have also been stable for the last decade. That there is a new kid on the block (polars) does not make the existing stuff any less stable. And one can just continue writing pandas for the next decade too.
and shorting something priced in a currency is effectively going long on the currency as well. If the USD takes a dive due to, idk, increasing populism from both major parties, stocks will do quite well in nominal terms. Your shorts will burn and you'll end up far worse than just staying in cash.
For most people, the best way to short is to just hold cash equivalents like short-term treasuries.
eg: If you think NVDA is overvalued relative to the overall tech sector, you could short NVDA, go long QQQ.
And if you have a more opinionated trade in the same currency, eg: you think AAPL will be fine if the AI trade pops, you can do short NVDA, long AAPL.
Finally, an even more advanced version would be to go long on something else in the same sector, but which is less overvalued in your opinion. eg: Short ORCL, long NVDA.
Positions are not necessary to be single transaction. They can be multi-step trade.
For global currency risk (meaning on USD), You will have to hedge your shorts with a non currency long position which historically hold value during defaults/ runs etc. Assets like gold (ETFs/Gold bars) or real estate (REITs or physical land holdings) or rights to commodity revenue like oil, copper etc [1].
If the currency risk is not for USD, then mix of other currencies particularly USD would work well as as hedge.
Currency risk is independent of shorting, i.e. it is risk in Long positions as well, current may inflate faster than your position increases in value etc.
---
[1] Commodity come with additional shorter term market volatility and risks - due their own supply/demand volatility and depend performance of economy.
However after assets like Gold, they will have highest correlation of returns against inflation as long the economy doesn't completely crash, because the demand for them is foundational
I’d change to “can be the same as being wrong” and agree. All these people out there thinking their being oh so clever with bubble this short that etc. Everyone knows.
One of the idiosyncrasies of modern human society is that we’re pretty good at knowing how things we create or initiate can go wrong, particularly with the economy. We’re just not great at perfectly understanding the degree of risk or the probability or at what point/level it goes wrong. That’s why I’ve never really got all the chat of “economists have predicted xx of the last x recessions yadda yadda”. I’m fine with that, I’d be more concerned if they predicted 0 of the last x recessions.
Your quote is something that AI mania speculators often like to reassure themselves with, but consider the fact that it took 17 years for the NASDAQ to recover from the dotcom bubble when adjusting for inflation. What's being early by a year or two when the consequences take decades to heal over?
If you want to sell all your tech stocks because you think that it's irrational then take your neutral position. You won't profit from stonks going up but you won't get anything from them going down either. You've isolated yourself from them.
What the quote is advising against isn't neutral positions or pulling out early during an upturn. It's about trying to time downturns.
If you want to profit from a stock going down, you need to hold inverses like shorts or selling call options / buying puts. These inverses are always short term positions, there is no such thing as a cheap long term asset that profits when stocks go down.
Basically if you want to profit from a predicted downturns, becuase you think some asset is irrationally overvalued, then you don't just need to be right, you need to be right and time it. Because it doesn't take long before you go bankrupt holding these sorts of inverses. Aka market stays irrational longer than you can solvently hold these risky positions.
The 17 year recovery time literally has nothing to do with this btw. It's all about short term.
See the other posts in this thread discussing Nassim Taleb's strategy of small bets spread over time with highly asymmetric rewards. You can afford to lose it all on small bets nine times in a row, if on the tenth bet you achieve a 100x payout.
The Nasdaq 17 year figure is only true from the peakiest peak of ~4800 in March 2000. Two years earlier, in March 1998, it was at 1750. It had hit 1750 again by August 2023.
All this to say... shorting 1-2 years early doesn't work. You don't have the patience or capital to actually maintain a short position for two years while the market goes from 1750 to 4800. You can cheaply sit out in cash, if you want, but that's not a short position. And the S&P500 hasn't seen the kind of 300% run-up over an 18 month period that Nasdaq did in the dotcom boom.
My understanding is that an extremely OTM put on a clear, strongly held thesis would be Burry-like, and many people would be able to do so.
But Taleb's point is that (non-insiders) cannot accurately predict regarding individual securities (hence derivatives), but can identify over-/under-priced OTM options — and that, trading these systematically, one can suffer many repeated "small" losses that become outweighed by the Big One that eventually (yet unpredictably) hits, thus generating overall positive expected value. But, as I further understand Taleb, most people don't have the huge capital that enables such a strategy, and that doctors, lawyers, dentists, etc., are better off making money by plying their professional services and perhaps investing in index funds and the like.
it's not 100% real, I bought a call last week at a deep discount to the ASK.
Do you remember the stock market a few decades ago? Stocks once had large bid/ask spreads. Now the hft/dark pools eat it all, with 99% profit on trades. High speed, lasers and fiber optics front-running..
Buy the options with the intention of them either hugely appreciating in value or expiring worthless. Under Taleb's system, your bet sizes should be small enough that it doesn't matter if individual options tranches expire worthless. The bid/ask spread only matters when you try to cut losses on a large bet, which is outside the scope of this strategy.
look at strikes where higher OI is. everything will be arbd out by bots on anything that isn't thin though. i trade futures for this reason though, because they are actually centralized unlike us equities. selling calls for a credit or spreading into free long entry is a better strategy in every way though.
What is the human need to bash SQL at all costs? At its core, such a simple syntax, yet its so powerful at aggregating/manipulating tabular data and the like. Instead we’d rather declaratively say what we want in a more verbose/disjointed way… fascinating
I'm with you. The more SQL alternatives I've seen, the more I'm convinced that SQL doesn't have that much fat or disfunction in it's syntax. There is inherent complexity. Follow an SQL-killer's guide far enough and you will see them quietly reintroduce the very same abstractions SQL solved decades ago.
For me SQL feels like PHP. Sure it's usable, but it doesn't spark joy.
Syntactically PRQL is much simpler, cleaner and more flexible. You simply chain pipeline stages instead of having a single monolithic query statement.
Data model wise EdgeQL is close to what I want (links instead of joins, optionality instead of null, nesting support), but it's syntax is almost as bad as SQL.
I just wish they had mutation in there too. I don't like the idea of swapping between PRQL and SQL, let alone some complex update statements where i'd rather write the query in PRQL. .. Yea you could argue they shouldn't be that complex for updates though heh.
Yeah, we deliberately left out DML to focus on DQL exclusively. I also find that appealing from a philosophical angle since it allows PRQL to remain completely functional.
I haven't thought about DML too much but what I could envision is an approach like the Elm or React architecture where you specify a new state for a table as a PRQL query and then the compiler computes the diff and issues an efficient update.
For example
DELETE FROM table_name WHERE id = 1;
would be something like
table_name = from table_name | filter id != 1
SQL:
INSERT INTO table_name (id, name) VALUES (1, 'Jane');
PRQL:
table_name = from table_name | append [{id=1, name='Jane'}]
Update is the trickiest to not have clunky syntax. For example what should the following look like?
SQL:
UPDATE table_name SET name = 'John' WHERE id = 1;
I can think of `filter` followed by `append` or maybe a case statement but neither seems great.
I used to use SQL very extensively, but then I moved into different roles, and now I always have to look up its syntax, stuff like "how did HAVING work again?", "how do I search for a date two days before today". With Text-to-SQL I can express my query in natural language. Because I have a very solid understanding of relational data models I can craft my prompt very precisely, but without constantly having to go back to the documentation. So at least for casual users it's very helpful.
When you're having to work with someone else's godawful convoluted database schema you're sometimes really happy not to have to write the query yourself. Giving the LLM context with previously written queries can be quite effective.
The writing SQL experience is a product of both SQL's syntax, the structure of the database you're querying over and the complexity of your query.
When things get hairy, and you have a good number of representative queries already written that you can use as context, LLMs can be really nice tool.
Agreed, but maybe the step change there is refactoring the data model, not continuing to author “hairy” sql via LLM that’s all fine until it breaks… and you end having to mend the nastiness back into compliance the ol’ fashion way
You could definitely load your billions of records with millions of relationships into memory, denormalize, restructure and rewrite the data (flawlessly) a lot cheaper (computationally) than running a large LLM on all that hardware.
“We filtered first instead of reading an entire table from disk and performing a lookup”
Where both OLAP and OLTP dbms would benefit.
To your point, it’s clear certain workloads lend themselves to OLAP and columnar storage much better, but “an endless sequence of misbehavior” seems a bit harsh .
Recent example, have 1GB of data in total across tables.
Query needs 20 minutes. Obvious quadratic/cubic-or-even-worse behavior.
I disable nested loop join and it's 4 seconds. Still slow, but don't want to spend time figuring out why it's slower than reading 1GB of data and pipelining the computation so that it's just 1 second, or even faster given the beefy NVME where files are stored (ignoring that I actually have good indices and the surface area of the query is probably 10MB and not 1GB).
How can the strategy be slower than downloading 1GB of data and gluing it together in Python?
Something is just off with the level of abstraction, query planner relying on weird stats. The whole system, outside of its transactional guarantees, just sucks.
Another example where materializing CTE reduces exec time from 2 seconds to 50ms, because then you somehow hint to the query planner that result of that CTE is small.
So even PostgreSQL is filled with these endless riddles in misbehavior, even though PhDs boast about who knows what in the query optimizer and will make an effort to belittle my criticism by repeating the "1 row" vs "agg all rows" as if I'm in elementary school and don't know how to use both OLTP or OLAP systems.
Unlike column dbs where I know it's some nice fused group-by/map/reduce behavior where I avoid joins like plague and there's no query planner, stats maintenance, indices, or other mumbo-jumbo that does not do anything at all most of the time.
Most of my workloads are extremely tiny and I am familiar with how to structure schemas for OLTP and OLAP and I just dislike how most relational databases work.
I think part of the problem is that the people working on Postgres for the most part aren't PhDs, and Postgres isn't very state of the art.
Postgres implements the ancient Volcano model from the 1980s, but there's been a ton of query optimization research since then, especially from the database groups at TUM Munich, University of Washington, and Carnegie Mellon. Systems like HyPer and Umbra (both at TUM) are state of the art query planners that would eat Postgres' lunch. Lots of work on making planners smarter about rearranging joins to be more optimal, improving cache locality and buffer management, and so on.
Unfortunately, changing an old Volcano planner and applying newer techniques would probably be a huge endeavor.
I feel your pain. I've been through all stages of grief with `enable_nestloop`. I've arrived at acceptance. Sometimes you just need to redo your query approach. Usually by the time I get the planner to behave, I've ended up with something that's expressed more simply to boot.
I love a semantic layer as much as the next guy...
Pivoting a decent sized BI shop toward using one instead of splashing the same SQL all over the place is *tough*. It's one of those: "the analyst could have been building important report for director and you want them to create re-usable logic??? we'll do that later, get report done now. Just copy/paste that SQL over here"
This is how you end up with the the 1000 model, "the numbers don't match up", hot mess situations that gain momentum and are hard to slow down.
The flip side is, you often don’t know what needs to be reusable until you’ve had some iterations. Wrong abstractions can be way worse, and also gain their own momentum.
The problem is that often these quick or maybe not reusable are written in such a haste that there's no breadcrumbs left to do the right thing whenever you are done getting that urgent thing out (most likely never because "everything is urgent" :( )
Yeah, minimizing the gap between the semantic layer authoring and adhoc is what you need to do to close that - there has to be a progressive model both for consumption (take this semantic layer, slightly extend/tweak it in an adhoc fashion) and for organically promoting up the adhoc works to the layer.
Right now a lot of semantic tools introduce a big discontinuity in both workflows that keeps the two worlds separate.
That tracks. The semantic layer is like a capital investment that pays off over time. So it can be hard to justify the initial investment if people don't grok the payoff.
