If you change yellow to silver, it gives you "colour silver," but if you look at the assumptions, it assumes silver is an element. Clearly it's not really getting the meaning of the sentence, and the fact that it's picking "yellow" is mostly coincidental, I think, as is also illustrated by your examples.
Query: The 2nd colour in purple, belly, yellow, arm, white and blue
Answer: yellow
Query: The 3rd colour in purple, belly, yellow, arm, white and blue
Answer: yellow
Query: The 7th colour in purple, belly, yellow, arm, white and blue
Answer: yellow
Query: The bluest colour in purple, belly, yellow, arm, white and blue
Answer: yellow
:-)