Care to elaborate? A-priori is generally used to mean that it does not rely on experience or observation. So while the words and symbols we use to explain 2+2=4 (i.e. Two, plus, four, equals) are leaned concepts, the truth and the ability to understand that truth is independent of that experience.
He's getting that argument from good ole Kant and his Critique of Pure Reason from 1781 (http://en.wikipedia.org/wiki/Critique_of_Pure_Reason). The basic argument, so far as I understand it, is that there must exist certain "structures in the brain" that make all our understanding possible. Since these are brain structures they must be a priori (everyone is born with them). On the side of aesthetic senses he boils these down to space and time. From our sense of space it is possible for us to reason about geometry and therefore much more abstract forms of mathematics without having to have recourse to the "outside" world. So we can envision and prove new geometrical proofs and our internal brain structures for understanding space ground these proofs in a sort of sense certainty. To go to your example of 2 + 2 we can say that nowhere in these symbols is contained the meaning 4 - so no amount of analysis (dividing these symbols up, reducing them to more primitive symbols, etc) will get us to the equality - we have to perform some operation internally and that operation has to be grounded on something a priori - these would be our sense of space where we can envision two things and two more things and perform the count and arrive at 4 - all done outside of experience.
Now there are plenty of philosophers who came later who have tried instead - following in the shoes of Hume - to perform a materialist grounding of these faculties not in the brain but derived from sense experience itself - sense experience somehow constitutes the structures of the brain and changing external sense experiences can reshape or reconstitute these brain structures. For example, Gilles Deleuze seems to me to do something like this.
If I understand you correctly, you are saying that math isn't invented, it is discovered. I am not sure this is true. I think they are models that are not true or untrue, just consistent and more or less useful than other particular models. It is trivially true that if I have two things in my left hand and two things in my right hand and I put them together, there are four things because this can be mapped onto identity of things. But really quickly it doesn't scale and you need non-real concepts to solve more complex problems. Roman numerals mapped to things (I,II,III) but Arabic were more symbolic and enabled more complex calculations (but still were base-10, which is a choice, not reality, some bases are better for certain calculations than others.) But again they still represented things in some sense. Zero was invented, it had to be invented because it's not representing a thing that can be counted, it's the lack of a thing so it's, I think, fundamentally a concept and not "real." It's still fairly intuitive (despite the fact that it took millions of years to invent) but then there are concepts like infinity. But then it turned out that that just described an infinite number of "things", there was also uncountably infinite. Then there are imaginary numbers. They can solve real problems, but I don't even know what they are. They seem to be an artifact of math that let you solve quadratic equations even though they map into something that can't even be said to be a "lack" of something, like zero is. It is a mapping into a completely virtual space. It was invented to solve problems, but it's not real in any material sense. I'm not a math person, so there are a lot more that I don't know anything about. Maybe I missed the point of your statement, but I wanted to explain what I think. That math isn't reality, it is a human invention that describes reality. You can have things in math that aren't reality but are useful models.
The a-priori/a-posteriori distinction is essentially: is its truth subject to observation of the world. Most mathematicians will argue that even complex ideas like imaginary numbers are the result of the system of mathematics, and can be arrived at and proved without observing the world. Physics is a-posteriori because the math involved is used to describe the world, and the correct formulas and the truth of the formulas cannot be arrive at without observation. Even if someones work is just performing certain calculations on others work, somewhere, working backwards, observation is necessary. If you start with imaginary numbers, and work backwards to simpler and simpler math, it is purely self referential. You don't need anything other than the base concepts to prove the truth, and the base concepts are true by definition, not by observation. You don't have to see a triangle to know it has three sides. It has three sides because that is the definition of a triangle.
