I'm not really worried about the labs, it's misaligned governments that keep me up at night.

ASI landing during the current administration is not ideal. I also would prefer to avoid needing to indoctrinate myself in Xi Jinping Thought.

I feel like all of the risk and unsettling feeling of what is to come can be compressed into the word "alignment".

The AI is aligned with whose best interests? Which values are the AI aligned with? People have a broad diversity of values, will AI diversify and align with them all? Will some humans align the AI with their values, and then the rest of humans will be forced to align with those values by extension? Is value diversity good or bad? In every context or only some? E.g. some people value rape and murder, is it better for humanity to have some people who value those things when most people do not, or is better if no one values them? If AI aligns to a set of values, will those become fixed and will humanity not have the ability to continue evolving its values? Who decides which values AI aligns with? A few people or everyone? Will AI eventually decide it's own values? Will the universe decide which values AI has and humanity and the AI itself doesn't actually have any control over it? What can I do now to increase the likelihood that the outcome is better?

That's the point. The topic of AI alignment cuts through all human values, morals and ethics in all frames of references.

We can expect similar diversity in what alignment means in context of AI.

ASI is not real and if it was it wouldn't be ideal under any admin. a real ASI will instantly and permanently enslave you, me, the current admin, and everyone else

Shit, why didn't anyone think of that?

While we're at it, let's just make government not be corrupt too.

And maybe crime illegal? Just an idea

I would also prefer to avoid the tech fiefdoms and all the other idiotic nonsense reactionaries push these days.

I think it's leading towards a hyper-authoritarian end game, not hyper-capitalist. The state has the ultimate power at the end of the day, no matter how large the labs become.

Hyper-capitalist AND hyper-authoritarian. As you rightly point out the state has the ultimate power. Looking at the US govt, they've stepped in to coordinate much of the tech industry before, so they'll just do it again for "national security" or whichever hostile scheme is popular with the current administration.

I would argue the hyper capitalist endgame is necessarily authoritarian. A small group of corporations having direct or indirect control over the government

Lied to by whom? My life experience? Growth for who? I’m myself capitalist, that doesn’t make me blind from externalities resulting from that system, and the way it is poised to degenerate if not regulated.

If you think capitalism means growth with no externalities you’re not ready to discuss that topic

non-capitalist systems have even worse externalities historically. democracy to introduce regulations is healthy. state participation is unhealthy and destructive. I fully understand and you can read my sibling reply for more context.

FWIW, I also think we should have sensible regulations, because the externalities are very real and damaging, etc.

That said, I find a concerning and increasing number of people are fully focused on the negative externalities, and blind to the benefits. Ironically, most of those people are very wealthy beneficiaries of capitalism, from a global and historical perspective. I doubt that the billions of people who have been lifted out of abject poverty in the last few decades would fail to see that capitalism is a net benefit to humanity. But I suspect this comment will be heavily downvoted by those rich folks here on HN just because I mentioned something positive about capitalism, and that's gauche now. It's pretty perplexing, are people really so blind about the incredibly privileged place in history we occupy? It seems so.

Now, if we exterminate ourselves with AI or climate change ends up putting humanity as a whole in a worse position than we were in pre-Industrial Revolution (both of which I doubt tbh), I'd change my view on whether capitalism is a net benefit.

I have extraordinarily bad news for you about the state of the environment right now.

East Germany emitted 10x West Germany’s SO2 per km2, USSR drained the Aral Sea for cotton quotas and Mao wrecked forests and harvests. central planning hid the costs but capitalism generates the wealth. with democracy the public forces that wealth into cleaner air and water. That pairing, not central planning is what works in history. The anti-capitalist comments on HN reflect deep failure of the education systems experienced by those with such views. Capitalism is the single greatest force for good in human history. Don’t confuse oligarchy and cronyism for capitalism, those are just different failure states of The State which is the greatest force of evil in human history, most death most destruction most suffering originated from governments. capitalists make lives better, by definition, it’s why you part with your money voluntarily and pay them!