Django's default subset is enough to do 99.9% of what you'd need to do (I know from experience) and there are various packages you can install to extend the ORM if necessary or if it just makes your life easier.
Of course, as the other person said, you can always always always descend into raw SQL if you must or if it just saves you time.
Like many 9s of development are, it's brownfield anyway, so you're saying the ORM should only be used if one's prepared to make large schema changes?