Small, incremental upgrades are something I'm finding are essential to maintaining sanity as the only dev at my company doing what I do. Waiting to upgrade only increases the pain. Now instead of having one potential issue to troubleshoot every now and again, I have dozens when I finally do buck up and upgrade.

So I get in the habit of upgrading the dependencies of all the apps I work on, every time I work on them. Issues happen, but only to one dependency at a time. It's manageable.

What I would love is to eventually have a CI server do it for me. Every single day, it would run bundle update, run the tests, and deploy unless there's a problem. If there is a problem, it drops me an email with the trace and fixing it becomes part of my morning routine.

If subtler problems surface this way, then I've discovered an oversight in my test coverage, or an overly complicated architecture that I need to remove dependencies from.

I'll probably implement this sometime this year. I'm thinking I'll want to redo deployment instead of relying on Capistrano, then finally growing my own CI solution. I'm slowly moving away from big monolithic apps to smaller, homegrown solutions that do only what I want them to do. I've already reimplemented provisioning and configuration management. I believe in DevOps as code.

> What I would love is to eventually have a CI server do it for me. Every single day, it would run bundle update, run the tests, and deploy unless there's a problem. If there is a problem, it drops me an email with the trace and fixing it becomes part of my morning routine.

Agreed. A while back, there was a security fix in the default php5-fpm configuration (0666 -> 0660, I believe), and I skipped it when updating. My website went down until I learned to change the owner of the process. And this wasn't even a dist-upgrade.

Sometimes manual intervention is required.

In my case, I run updates almost every day. If I had to wade through 6 months of backlogged updates, I would have wasted a lot more time identifying where it broke. Smaller feedback loops are a win.

Gemnasium provides something similar to what you describe in the context of Ruby applications with Gemfiles. It can be configured to automatically generate pull requests updating outdated dependencies, which could trigger your test suite to automatically check for regressions.

I use Travis CI to maintain my Ruby gems, and one thing that I do is turn on a build that allows failure, where it uses the HEAD version of my upstream dependencies. That way I know within a day when upstream has done something broken, which is especially nice if it's by accident.

The problem is that OpenBSD has had some pretty significant user-facing changes in only the past year. The past few releases have seen apache replaced with nginx, and the next version is going to be getting rid of nginx for their own httpd. If you're just using the basic webserver, that's some pretty significant change right there. While they're advertising clean and simple, they can and do introduce potentials for heavy breakage in nearly every release.

Maybe that's an indication that the configuration of the web server should be partitioned into "basic configuration" (which they can all support) and a few other categories (which are supported by a subset), i.e. the configuration itself should be "standardized".

I realize that this is a nontrivial proposition, but if we're ever going to realize the dream of "interchangeable parts" we're going to have to learn to do this stuff.

The old code is one pkg_add away, though. Breaking binary compatibility is at least as likely to hurt you - and simply upgrading PostgreSQL without a dump-reload cycle can hurt plenty.

That still requires admin time to update the configuration and change management tools, QA time to run a regression test that will exercise all the potential failure paths, development time if you count on the ability to do configuration updates based on application state, etc. Updates that look small can still require considerable changes.

If you're used to it, you can streamline these processes and automate as much as possible. Even if it's a change you can't easily test or automate, by making just the one change at a time you know what to blame if it breaks. :)

Presumably that's just the default web server, and you can use the webserver of your choice as well?

> ... perhaps if were to actually get serious about doing small/incremental updates all the time, we'd get better at actually doing them without incident.

That would require more developers to actually give a fuck about the user experience not breaking from release to release.

I simply expect that doing an upgrade of the system (as opposed to grabbing the latest version of applications and libraries) is going to totally destroy my box and I'll spend the day recovering, because that's happened to me multiple times, from multiple OS vendors.

Hopefully (and I realize that this is somewhat wishful thinking), more users doing haphazard upgrades and complaining about them not working would be feedback into the whole cycle.

(I'm not very well placed in the software cycle to actually start this feedback loop, but one can wish, right?)

Given the general quality of response to user reports in free software ("The source is there, fix it yourself", "WONTFIX", "That's a feature", "you're doing it wrong."