For one there isnt a single housing market, there are a lot of segmented housing markets that only partially overlap. In high demand areas, land is already limited.

For another zoning restrictions are severe in the US, with most residential land zoned for single family housing with large lot sizes.

With high upfront costs, limited land, and political issues, developers would rather build high end housing, which has led to over supply in the high end market, and under supply in the low end market