When we got accepted into S23, our call started with Seibel saying something along the lines of "We like you, but we're pretty sure you're going to have to pivot." A non-trivial amount of my batch mates also joined with minimal product strategy, so I don't think you can read too much into the rejections.
I don't have anything to add re: YC over the years, just that my anecdotal experience differs.
A few weeks back we shared a demo [1] of this on HN and got to chat with a lot of interesting folks but we weren't able to support everyone just because we're a small team.
We're opening up a free developer tier of our CLI tool in hopes that it can be helpful to more people! We gated it behind email verification just to CYA because we're using our own LLM API calls and compute behind the scenes.
If you have any issues please comment here or shoot me a note at matt at talc ai.
EDIT: I'm also just noticing that in the blog text I wrote "generated for Boeing". I meant Boeing as a topic, not as a company-- sorry for the sloppy language. I'll push a fix for this later today.
One of my regrets about the demo is that we paid a lot of attention to showing off our ability to generate high quality Q/A pairs, but not nearly as much to showing what a thoughtful and thorough grading rubric can do.
It's totally possible to do a high quality grading given a rubric that sets expectations! Great implementations we've seen use categories like correct / correct but incomplete / correct but unhelpful / incorrect to better label the situation you describe. We've found that we can grade with much more nuance given a good rubric and categories, but unfortunately didn't focus on that side of things in the demo
I'm not familiar with wikicrow, will check it out!
Not a pricing expert by any means, but here were some of the considerations we thought of when we made that decision:
1. There's plenty of usage based infrastructure/dev tools (ie. AWS, databricks), so I don't think we incentivize minimal usage.
2. The value we're providing feels directly tied to how much testing we're running, so when we tried to construct tiers they didn't feel helpful.
3. In our experience, enterprises have been fine with usage based pricing. They're already paying for human QA / labelling on usage based terms (even if it's part of a larger fixed contract), so our pricing isn't a deviation for them.
Usage based pricing with some sort of free tier ideally is typically is _best_ for individual developers and smaller businesses who are price sensitive but want to get off the ground quickly. Larger businesses can benefit from a usage based model but typically enterprises will want a more guaranteed price and stability quarter to quarter for planning purposes. This doesn't mean necessarily that usage based pricing doesn't work for enterprise businesses however it does mean that they typically are keen on having alerting and limits setup so that they don't accidentally go massively over budget.
One common strategy to continue with usage based pricing for enterprises but have more stability is to have them buy the usage "upfront". As an example, to use this product you may charge them "tokens" which correlate to dollars, and they can buy 1000 tokens upfront which get burned down over their time using the platform.
Another reason why enterprise plans are not typically usage based, or at least why its less common, is that many enterprises want to self host the product in their own systems in order to meet their internal security and compliance standards for their data. If that ends up being the case, self hosted products typically are not usage based because the act of sending consistent usage data back out of the enterprise's VPC adds a lot of additional complexity for the product and both business parties. Typically an annual license based structure is the model most businesses go with for self hosted.
Just to disclose a bit, I help companies do billing and figure out their best pricing strategy. Also a YC company. Congrats on the launch, the demo is technically impressive, and the value prop to me is really clear.
If you want to chat at all about it, happy to just talk for a few minutes. My email is in my profile.
I agree that the current grading is a bit harsh -- the rubric we're using in this demo is fairly rudimentary. What we've seen be more helpful is a range of grades along the lines of correct / correct but unhelpful / correct but incomplete / incorrect. This somewhat depends on individual use cases though.
Let me know what questions generated you thought could be more complex! We're always working on improving our ability to explore the knowledge space for challenging questions.
1) There's an interesting subtlety in the phrase "the correct question-answer pairs". While we don't often find factually incorrect pairs because of how we're running the pipeline, the bigger question is whether or not the pairs we generated are "the" correct ones -- if they are relevant and helpful. This takes some manual tweaking at the moment.
Inconsistent outputs over different runs are definitely an issue, but most teams we've worked with barely even have the CI/CD practice to be able to measure that rigorously. As we mature we'll aim to tackle flakiness of tests (and models) over time, but a bigger challenge has been getting regular tests like these set up in the first place.
2) In this scenario, we go to the documents powering a RAG application to both generate and grade answers. For example, the knowledge base might know that (1) product A is being recalled, and (2) customer #4 is asking for a warranty claim on product A. Using those two bits of information, we might generate a scenario that tests whether or not customer #4 gets the claim fulfilled. In other words, specific user information is simulated/used during the test set creation.
I think this could be more useful to most people as a prompt/RAG testing service rather than an llm testing service. If I ran a test and found out the llm I was using is 60% accurate on some topic what would I do with this knowledge - build a more accurate llm? Switch to another? On the other hand if a service offered me suggestions to improve accuracy by providing a score for various prompt or RAG inputs, I think this would be very useful to many people. It could even uncover a general prompting strategy depending on the underlying Llm or inputs available which would be really useful
I don't have anything to add re: YC over the years, just that my anecdotal experience differs.