"Color" implies black, but it is also used to imply other colors sometimes. It can imply middle eastern, east asian, southeast asian, hispanic, and several other groups. However, aside from implying black, other implications are context-sensitive and as a result different authors sometimes mean very different things when they write "person of color".
It has been a long time since it exclusively referred to African Americans (I believe the term predates the us itself). For instance, the racial integrity act of 1924 specifically defines native Americans as colored; similarly, the Jim Crowe laws of the south specifically segregate whites/Caucasians from colored/non-Caucasian, though enforcement certainly centered around African Americans.
That said, the black community can use its own community voice, much like other minorities. I don't think there's any harm in the term "color" here outside of the confusion we've witnessed in this thread.
'Person of Color' is a different way of saying 'non-white', to group together people who suffer in similar ways from not looking like the majority in Anglo-dominated countries. The thing is that 'non-white' describes that group by something that they are not, where as 'People of Color' describes them as something that they are.
Perhaps a simile - it's a bit like calling an egalitarian a 'non-white-supremacist'. Yes, the egalitarian is not a white supremicist, so it's superficially accurate, but it's not really a description of who they are.
