Sure, but when people say something is 5x cheaper or 500% cheaper, what is the math that they are implying? That the expensive thing is 5x / 500% as expensive as the cheap thing?
castform founder here. unfortunately, we are cloud-hosted at this point.
but some easy options on the open-source side include huggingface's trl & unsloth.
you can run our data-generation scripts here: https://github.com/castform-ai/benchmax and then hook it up trl/unsloth for training. should be able to do all of this on your own compute.
castform founder here. i'm personally a little against techniques like self-consistency/majority voting during rl training because they tend to result in the model's output distribution "sharpening" a lot. this means the model will lose it's exploration ability and probably won't be able to explore/discover new solution strategies, which can be harmful for both rl training + generalization to unseen cases
castform founder here. i'd say it's a threat but the frontier ai labs' argument would basically be that the market opportunity for intelligence is infinite so it doesn't matter. at the same time, i do believe there will continue to be a massive, growing market for big labs, esp for super-frontier use-cases. today that's longer-horizon coding tasks but in the future it can be things like scientific discovery, etc.
(founder of castform here) the model you post-train should ideally learn general patterns & search strategies over your dataset that should transfer to new docs you add to the search corpus (unless its super out of distribution)
i'm a little confused by this, should most piles of company documentation look pretty similar? what are we getting by tuning at the org level?
what if you have a bunch of teams or apps that have different documentation patterns?
how much does this degrade over time, it beats leading models with that static data set but clearly this edge will degrade with data drift, how quickly does that happen?
also, any "this is 100x cheaper" blogpost means nothing if not discussing TCO (I know your team didn't write this.) I don't care what inference costs are if I don't know training/overhead costs. what's the breakeven point. and again, how long is this RAG stack going to be worth keeping, you beat 5.6 Luna but at some point un-tuned models will beat you, so this is a temporary solution that needs to be re-upped at some point. benchmarks against data drift would help there
- if you have teams that have totally different documentation patterns, yes i think a model trained one teams' docs probably won't directly transfer to another. but let's say you have a model trained on SEC 10-K filings, it should work well as new 10-K filings are added (since terminology & structure of these docs are similar)
- on TCO, the training costs for the experiments you saw were <$200
- you bring up a good point on data drift, don't have concrete numbers/experiments here. something we should defn look into (though i imagine this also changes a lot based on use-case and company)
could you share more about what you mean by "general patterns & search strategies"? I can think of it being along the lines of searching over specific tables or databases for queries in certain context. It's an exciting line of work and I'm interested because I need something like this for the problem I'm solving atm. So, I'd like to understand how the training generalizes
castform founder here. the roi optimization makes sense. i think there are lots of usecases for which even a 2% gain in accuracy can be quite useful. off the top of my head
- high volume customer support. higher accuracy means fewer escalation, reducing labor costs
- fraud detection. catching even one extra fraud attempt could mean a lot in savings
- and ofc the classic ads use-case where at scale bps in improvement could mean millions in revenue :)
the bigger model would still cost more :)
at the same time, i see prompting as being orthogonal to post-training. i'd imagine post-training a smaller model with a better prompt would make it perform even better
