Switzerland is a country with a total population approximately the size of the state of New Jersey. This being too centralized for most things, they then further divided that population into 26 cantons ranging in size from approximately the size of New Hampshire at the high end to "that number of people would be classified as a town rather than a city" at the low end.

The median size looks to be around 200,000 people, so maybe start by dividing the US population into cantons of around that size and doing most of the rulemaking at that level.