After looking at freely available SVGs of pelicans and bicycles, I have a hard time imagining what they could be using to game this.

Then maybe what you want is email?

Sprinkled with a little bit of https://delta.chat ?

And potentially a dash of iroh https://news.ycombinator.com/item?id=48542480 https://www.iroh.computer/solutions/delta-chat ?

Delta.chat no longer uses email though, right?

delta.chat is email

https://delta.chat/en/help#can-i-use-a-classic-email-address...

Seems like what Apple's going for with afm3. Their latest model that will be embedded in macOS 27 is a quantized dense 20B that only select between 1 to 4B at inference, based on the prompt, not token by token. If only they could make a 100B or 400B dense that selects ~5 to 15B...

I don't understand, if they are only using a subset of the tokens then it's a sparse model. What do you mean by dense?

Nothing to understand. Straight up hallucination. I could have sworn I read that they used a novel architecture where the model is dense but you could select specific layers or something at inference. reread the announcement: just said MoE. Corrected my brain's weights so thanks.

https://machinelearning.apple.com/research/introducing-third...

Thanks for responding.

Could it be some sort of permanently routed MoE where they detect and switch for the whole prompt instead of token by token?

I would wager not a lot. There are some real hard constraints in space, from power consumption, to weight to thermal output (lack of convection is a real PITA for thermal shedding), and the list goes on.

This was a preview release. They haven't finish training. The Pro contains more knowledge but it probably takes longer training than flash for the smarts to kick in.

But with Apple's AFM 3 architecture, we might end up with huge SOTA adjacent on devices with limited RAM.

They use a technique where you only load between 1B and 4B of a 20B dense model for an entire prompt run, not token by token like a MoE, and use mostly the low power ANE instead of GPU cores.

Now, imagine if/when they scale up to 100B or more? On a chip using 2W?

I think we're also ignoring a potential innovative move in how models work.

If someone could splinter or fragment the models into more specific tasks i.e "spellchecker AI" and get these working as well as Sonnet 4.6-4.8 on those tasks on a personal laptop. You then question the $100 a month fee.

Bear in mind these laptops are likely to be $5000 or so because of the memory, HDD and M7 chip they likely need.

It feels to me like the beginning of the inflection point but software updates not hardware updates will be the accelerant.

  "That’s where EMO comes in.

  We show that EMO – a 1B-active, 14B-total-parameter (8-expert active, 128-expert total) MoE trained on 1 trillion tokens – supports selective expert use: for a given task or domain, we can use only a small subset of experts (just 12.5% of total experts) while retaining near full-model performance."

https://allenai.org/blog/emo

Don't worry. Most people spend most of their compute time on a phone, where you're ability to filter ads is way more enshitified.

I wonder where's the line between using a font and copyright/trademark infringement.

If the font is proprietary then using that without permission could be copyright infringement.

If the logo can be confused as owned by another entity, then that could be argued as trademark infringement.

But these cases would need to be decided in court. And different jurisdictions might be more restrictive than others.

"That’s where EMO comes in.

We show that EMO – a 1B-active, 14B-total-parameter (8-expert active, 128-expert total) MoE trained on 1 trillion tokens – supports selective expert use: for a given task or domain, we can use only a small subset of experts (just 12.5% of total experts) while retaining near full-model performance."

https://allenai.org/blog/emo