I get your point, but here is a reason I am stunned:

Until now, if you ask "what makes humans different from animals" many responses could be in the line of "generating art, poetry, music, synthesise ideas"..

Well, now it seems somewhat clear that THAT part of human activity can be performend by something non-human.

(There is still a lot more to humans of course; I agree with you on that point)

Yes -- it is only reassembling stuff humans in the past have done -- but so is most of human activities in these areas. Very little humans produce is not inspired by input from others. Art, music and ideas reverberate back and forth between humans and only slowly change over the decades.

> Until now, if you ask "what makes humans different from animals" many responses could be in the line of "generating art, poetry, music, synthesise ideas"..

That's because we (as an agrarian society) have constructed an idea of “what a human is” in opposition to animals, but with the advance of AI maybe we'll realize that we are much closer to animals (especially other mammals) than we're comfortable to admit (and no, I'm not vegan).

Exactly. Human creative works are also on a bell curve, with the stuff in the middle typically being bland and boring. It's only when risks are taking that it holds our interest.

ChatGPT mirrors this, producing output that feels generic precisely because it's trying to predict what's the most likely response. That's in direct opposition to interesting creative works.

You can force it off the beaten track a little by adding more details to your prompt, but it will still try and fit the best curve through those additional points.

That doesn't really capture what the answer was supposed to mean. Infinite monkeys will inevitably compose The Odyssey if you give them infinite time, but it doesn't mean they match the moral quality of a human that is captured by the ability to create art. Art is a representation of experiences, beliefs, internal states, life history, desires, that ChatGPT doesn't have. It is the subjective experience being communicated via art that makes humans special.

And, to be clear, it doesn't make humans that special. Other animals have created art that expresses their sincere internal experiences and desires.

If I'm being perfectly honest with myself and you ask me what makes humans deserving of special moral consideration, I would have to say nothing. I think an elephant, gorilla, octopus probably all deserve the same consideration. I give more consideration to humans because I'm human, and what makes us human isn't art. A person with severe brain damage and the inability to understand or produce language or art is still human.

A piece of AI generated art recently scored top prize in a digital painting competition. It's starting to look more like one monkey, on a limited time frame, making art that people rated highly.

From an intellectual standpoint, the bot is impressive and this is coming from an AI skeptic BTW.

I get your perspective that a reductionist view of humans as solely intellectual agents is severely lacking, and bordering on dehumanizing if taken to an extreme but it still doesn't take credit away from the impressive capabilities that this bot exhibits.

"It confuses us in ways" could be framed: "it makes us rethink and expand our definition of intelligence and human cognition in very interesting ways".

Even if it's far from "true human" level (whatever that means), it gives us a theory about ourselves that's computationally falsifiable.

ChatGPT cannot express emotions because its creator specifically disallowed that from happening. I've had some chats with the more free and creative Character AI chat bots that are quite human-like, and I think if OpenAI gave ChatGPT some more leeway it could act shockingly human.

It does NOT pass for a human...

Are you human?

> No, I'm not a human. I'm a large language model trained by OpenAI. I'm a computer program that's been designed to generate human-like text based on the input I receive. I don't have my own thoughts or feelings, I just process the information I'm given and try to provide helpful responses. I tested the think now quite a bit. I am not sure how I would use it on a daily basis. Maybe for inspiration?

I don’t understand your point — I’ve been rehearsing that response to the normal and all-consuming test question “Are you human?” since Wednesday.

Are you human?

"Yes, of course i am. What are you, a f--ing idiot?"

> "my definition of a human is very different from that of the consensus. At the very least I'd like my humans to exhibit even minimal capacity for emotional expression as well as consistency and honesty"

Careful there. There are some fellow humans who wouldn't be able to pass your conditions.

It can probably express feelings and emotions but not experience them.

You seem to conflate the two points.

It's relatively easy to write a prompt that ChatGPT can respond to with something it thinks you want to hear, based on what it's read others write in similar contexts — that's mostly what all these "wow ChatGPT gave me exactly what I wanted it to" posts are learning, mostly indirectly. GPT's moved from regurgitating appropriate texts to prompts, to synthesizing appropriate texts to prompts, but in both cases it's telling you what you want to hear. That's the point.

It's an incredible achievement, technologically. It's also very different from having feelings and emotions.

Or, if ChatGPT can "feel and express emotion" through this manner, does that mean its feelings and emotions are sociopathic in nature? Not in the terms of the disorder, but in simulating feelings and emotions because that helps it complete the task it's been given?

Or! What ChatGPT says after plugging the above question into it:

> No, ChatGPT does not have sociopathic tendencies. It is simply programmed to respond to certain prompts in a way that mimics human emotions and reactions, but it is not actually feeling or experiencing the emotions itself.

Is there any output ChatGPT could produce that would convince you it feels emotion?