> overall a benefit to humanity.

By that metric a complete ban on LLM’s might be on the table, which I don’t think is something you’re advocating for here.

I'm advocating for a litmus test that can't be easily abused, and I'm advocating against unfair equivalences. I also am not calling "overall a benefit to humanity" a metric, it's more of a conclusion we could arrive at by an application of relevant metrics.

> a conclusion we could arrive at by an application of relevant metrics.

That’s dangerous ground because of how you can arbitrarily change weights of different metrics. If the goal is the “overall benefit of humanity” then that’s what’s important not metrics.

How do you propose assessing "overall benefit of humanity" without metrics?

Qualitative assessment etc not just metrics.

How do you propose to asses “overall benefit to humanity” with metrics?

Number of successful outcomes reported (e.g. with disabled students, cancer patients). Labor costs. Latency of services. Scientific discoveries found. Etc.

I'm not trying to discount the qualitative approach, but I think it's not impossibly hard to find metrics we can associate with "overall benefit to humanity" from a quantitative viewpoint.

> I think it's not impossibly hard to find metrics we can associate with "overall benefit to humanity" from a quantitative viewpoint.

One would hope for the slightly higher ambition of metrics that are causally connected to "overall benefit of humanity", settling for merely correlated has that unpleasant risk of the association breaking as you try to optimise.

Qualitative metrics are also metrics. You have to have a spectrum in order to weigh things against each other qualitatively, even if the spectrum is just a 2-point binary.

Qualitative assessment and qualitative metrics are different things.

Qualitative assessment is the overall process of using qualitative metrics to evaluate something.

The important distinction is it covers a wider range of analysis than qualitative metrics alone.

This matters due to the difference between a hypothetical perfect analysis covering every possibility and one than can actually take place in the real world.

I feel like both viewpoints are valid ways of determining what you personally view as "good", and I don't want to take away from that diversity of opinion

There is no realistic possibility to effectively ban LLMs worldwide, which makes advocating for this useless.