Just look at the price of cocaine compared to other similar countries (UK) and it's obvious that while drug use exists in Australia, that specific drug is used less frequently.
Higher price is a function of demand and scarcity. Scarcity from more difficult smuggling routes and demand from popularity. Although you are correct insofar as a lot of people can’t afford to do it regularly.