> As far as I can see, commercial software isn't limited by "major unknowns"

I'm sorry, I needed to get in an afternoon run and actually deleted a paragraph that explained this a little more that I couldn't get just right before posting.

I completely agree with you. There aren't a lot of technical unknowns for most things people are trying to do.

Thing is, I wasn't actually talking about technical challenges with that point, rather, the requirements that are given at the beginning of a project. The requirements for the new Bay Bridge, for example, are pretty darn clear: make a new bridge from Oakland to Yerba Buena Island.

The requirements for making G+ attract Facebook users... errr... uhh...

Or how about the requirements for the next Call of Duty to be fun...

A great many of today's software projects -- especially those discussed here on HN -- aren't straight up engineering problems, they're more design and UX problems. Even for something like the software for the Joint Strike Fighter, you're more into UX than hard-hittin' avionics for a lot of it these days.

And I think that's what makes it hard to predict software development schedules. If we were all building ATM software again and again, we'd be knocking it out of the park. A competitor to Snapchat that is able to sway users over... that might be harder to define up front.

Ah man, this hit slightly hard as someone who just moved from Postgres to MongoDB to avoid database migrations only to realize that I've now lost the power of a many to many relationship.

How would one avoid database migrations by switching databases?

MongoDB guys call a "database migration" something like "adding a column to a table". They'll tell you it's an incredibly complex and time consuming thing to so, AND THAT"S WHY YOU NEED MONGO!

Of course out here in the real world, an "alter table" statement takes no time at all even on a large table, and can be done while the system is running.

Now I am not saying that MongoDB has no uses at all; but I am saying that the people around it don't really have much of a clue about databases in general and aren't in any sort of a position to be choosing technologies.

Could you please explain to me what you dislike about MongoDB?

I downvoted you before I realized that you coming at this from a position of ignorance rather than ego.

Mongo is a way of storing 'stuff'. You don't have to define the form that the stuff takes before you shove it in the database. This makes things easier by lowering the knowledge barrier to getting started, which is a double-edged sword.

What I've gathered is that the tradeoff of not having to deal with schema migrations now is something I'll have to pay for with data integrity down the road. Do you think that's a good summation?

Yes, that's an excellent way of putting it.

Right. Thanks for the help. I'm gonna move back to Postgres.

MongoDB is schemaless, so you don't have to do traditional migrations. Unless you're making a joke about me migrating to a different database to avoid migrations, in which case, good joke.

An undocumented and unvalidated schema, you mean. A database without a schema is a contradiction in terms.

Depends how you define database. Perhaps it's more precise to say that Mongo is a schemaless key-value store, but in truth it provides many functions that people think of as a "database", and the differences between it and traditional RDBMS seem inessential to what distinguishes a database from other things.

My apologies, the official documentation referred to MongoDB as "schemaless" so I figured it was a real thing.

You aren't wrong, MongoDB is schemaless in the sense that you don't need to define a schema to use Mongo. But to use any datastore, you'll be defining a schema somewhere - generally, in your app itself.

Migrations are about data, not schema. You don't need migrations with a "schemaless" database at the beginning, but as soon as you have data you care about, you're gonna need to ensure some sort of consistency.

Do you think I should move back to Postgres? If so, why? Genuinely looking for well backed technically minded opinions here.

I think you should learn the relational model (not SQL). Understand the problems it was trying to solve and how the relational model solves them. Then figure out how that applies to your app. Database in Depth was helpful for me http://www.amazon.com/Database-Depth-Relational-Theory-Pract... .

Yes, you should move back to Postgres. The reason is you can do many more operations on structured data than on structure-less data. You found that out already with the many-to-many thing. There are plenty of other intangibles that will bite you in the ass later.

Wouldn't you look for better migration tooling rather than jumping technology?

As far as I know, the best migration tool for SQLAlchemy is Alembic, and it can't magically handle migrations (which is what I wanted). I jumped technology because it seemed like MongoDB did (and still does) meet my use case better. I'm open to any opinions or advice though.

Granted, most engineers are not exactly employing quantum physics to parking lot design. Most people, in general, work on gluing together pre-built parts that they do not understand all of the fundamentals of.