I've been playing w/ this product self-hosted for a few weeks. It can join across multiple tables, windowing functions. I haven't tried self-joins yet, nor I have put much effort into tuning w/ Golden SQL or other documentation.
I would put this at the skill level of a junior data engineer. It's pretty impressive.
For large databases, LLMs do not perform well if you pass the entire schema (either run into context window issues or confuse the LLM with too much info). There is a schema linking step that identifies the relevant schema and only passes that. The schema linking is also done in the fine-tuning process.
The largest we have successfully deployed is on the OSQuery schema https://osquery.io/ which is 277 tables and lots of business context (malwares, vulnerabilities, Windows registry keys, etc).
Really cool to see people already testing the capabilities of OpenAI's latest release. Not sure how useful this will be in practice since it will likely lack business context/tribal knowledge, but still really interesting.
Yes, based on what I've experimented with OpenAI's assistants, it still requires engineering and developing tools to get the best performance on large databases.
Open source LLMs are catching up with OpenAI. They are also considerably cheaper in the long run. We took a look at how they performed in terms of accuracy vs. OpenAI in an NL-to-SQL workload.
tldr -> For zero-shot prompting, the closed source LLMs are still far ahead in terms of generating accurate SQL. However the gap is closing. With the right dataset for fine-tuning the open-source LLMs might be able to perform at the same level.
That's absolutely fascinating! The use of single-molecule fluorescence to verify rotational motion is indeed impressive. I'm curious, how do you think this groundbreaking technique could potentially impact future research in the field?
seems like there's a high potential for the AI to hallucinate and get confused with ingredient measurements... last I checked, LLMs struggle with numbers.
