You are mixing up what something is with how it is conventionally formally modeled or defined. This is easy to do in modern mathematics, where students are trained from early on to develop every concept from basic axioms, imposing an ordered structure of definitions for convenience of exposition and proof without worrying as much about the inherent nature of the objects under study.

A concept of number is not required to draw or describe graphs. Indeed, if you wanted you could define numbers in terms of graphs. For example, you could start with the combinatorial game Hackenbush and build up numbers from there. https://en.wikipedia.org/wiki/Hackenbush