I think you're massive underestimating intelligence - not ASI, but just, like, human-level intelligence. Most of the existence of human civilization has taken place in conditions wildly different from the ancestral environment, and we adapted anyway - and the reason why it was possible was because we have a tiny bit of general intelligence.
For your simulation argument, see e.g. https://alicorn.elcenia.com/stories/starwink.shtml - I think you're arguing that nothing like this could possibly happen, because humans are too dumb to ever figure out unfamiliar physics without prolonged experimentation, even given hundreds of years of theoretical research and several attempts in the real world? I think the closest real-world example of this is space exploration - it's extremely expensive and takes years to put stuff in space, and yet probes and rovers we launch end up working most of the time. On my model, it's because we have a decent grasp of the underlying physics in enough detail to predict real-world complexity, and we have toy testing areas we can test the tech on, and if you do this cautiously enough you can, in fact, launch a thing that ends up working.
I think there's a fundamental difference, between the sort of careful planning that lets you launch space probes, and blind optimization like evolution does. The former can generalize to unfamiliar domains even when the latter doesn't. That's what intelligence gives us.
> human civilization has taken place in conditions wildly different from the ancestral environment
"Wildly different?" That's a wild exaggeration, if you don't mind me saying. Subjectively, sure, there are lot of important differences. Objectively, they're basically the same. The intellectual environment machines bask in is many orders of magnitude more different from physical reality than living on Earth is from living on Jupiter. Even the topology of it is without common measure.
> and the reason why it was possible was because we have a tiny bit of general intelligence
Also, hands. Also, an upright posture. Do you think just any being with the level of intelligence we have would have followed the same path? Attributing our success solely to our intelligence is simplistic -- it is our intelligence plus a slew of innate capabilities, which we did not create, and without which our intelligence would be worthless.
The kinds of simulations you find in LessWrong's half-baked thought experiments are almost always ridiculous and this is no exception. "World simulations" in real life are massively limited stuff like Minecraft where you might be able to embed a 1 MHz computer if you manage to bump it to run 20K ticks per second. The resources required for the simulated beings to do all that work wouldn't be available and their generations will be slower than ours, not faster.
I didn't spell it out, but the simulation I had in mind was for a toy world, because only a toy world could possibly run fast enough.
> I think you're arguing that nothing like this could possibly happen, because humans are too dumb to ever figure out unfamiliar physics without prolonged experimentation
I'm talking about operational, practical, instinctual knowledge for real-time action. Knowing physics doesn't make you capable of catching a ball, and you don't need to know physics to catch a ball. In real life, we never deal with the fundamentals of physics, we deal with macroscopic objects and systems that are very far removed from these fundamentals. They are more varied, messier, faster, and overall harder to deal with.
> I think there's a fundamental difference, between the sort of careful planning that lets you launch space probes, and blind optimization like evolution does. The former can generalize to unfamiliar domains even when the latter doesn't. That's what intelligence gives us.
Blind optimization has one definite advantage over careful planning, however: it is not self-limiting. Any goal-directed optimization process is de facto limited to the subset of possibilities that it can prove furthers its goals. That's why most human technology is simple and unsophisticated compared to natural systems: it needs to be predictable, reliable, build-upon-able, and therefore easy to model.
And when it is more sophisticated, like current AI, it is so heavily inspired from natural optimization processes that I would argue it is actually a form of blind optimization. I mean, think about it: we understand AI so poorly that you and a lot of other people are afraid of losing control of it. If we were designing it intelligently, we wouldn't be worried. But we aren't: we are doing it blind.
For your simulation argument, see e.g. https://alicorn.elcenia.com/stories/starwink.shtml - I think you're arguing that nothing like this could possibly happen, because humans are too dumb to ever figure out unfamiliar physics without prolonged experimentation, even given hundreds of years of theoretical research and several attempts in the real world? I think the closest real-world example of this is space exploration - it's extremely expensive and takes years to put stuff in space, and yet probes and rovers we launch end up working most of the time. On my model, it's because we have a decent grasp of the underlying physics in enough detail to predict real-world complexity, and we have toy testing areas we can test the tech on, and if you do this cautiously enough you can, in fact, launch a thing that ends up working.
I think there's a fundamental difference, between the sort of careful planning that lets you launch space probes, and blind optimization like evolution does. The former can generalize to unfamiliar domains even when the latter doesn't. That's what intelligence gives us.