I feel the same way; although I've spent some time with both Gnome and KDE, I never developed an attachment to either of them, mostly because I felt that there was a good likelihood that mastering the Gnome of today might be a pointless exercise because if you sat me down in front of the Gnome of one year from now, it'd be unrecognizable to me.
Around the advent of Gnome 3/KDE 4 I also realized that I didn't really use any of the Gnome or KDE apps. If I want to burn a cd, I know wodim well. If I want to listen to music, I probably use my phone. The browser is the only desktop app that I use regularly, and that's fairly agnostic.
I've spent years tweaking rxvt Xresources, for example, such that I can drop it on a brand new machine and everything behaves as I expect.
I'm well past the point in my career where I'd be looking for, say, a better terminal emulator, or shell, or chat app or any of that stuff; the stuff I've got is working as well as I could hope, and when I need something new I can compose it from the abilities that the shell and the window manager expose.
The last big change I made was after I tried a tiling WM (Xmonad) for a couple of days and realized right away I'd never go back; that's been 7 or 8 years ago. Occasionally stuff changes underneath me in ways that seem somewhat capricious (Pulse Audio, Systemd) but I usually give that stuff a chance and try to adapt to it unless its absolutely unbearable.
I enjoy checking out the new stuff on machines I use at home for fun, but at work I treat the workstation as an instrument; I know what I want and almost everything has a reason.
I've been going through a similar experience over the last few years, mostly driven by my hardware getting older and not handling the requirements of modern DE's (when I'd prefer the resource to be used for applications).
It's nice stripping back to the original structures, though it does feel like computer archaeology sometimes. There's a lot of services in the upper layers that have to be switched off to get back to plain .xinitrc and Xresources!
On the upside, I know my configuration and it's all in a repository now!
Except I never actually got on board with KDE or GNOME. Command line, .xinitrc, and .Xresources were my jam lo these many years.
When I first tried Linux in the 90s it was already less of a pain than Windows. Why would I want to reinstate all that Windows-like cruft to be at most 0% more productive than I was without it?
It's a good feeling.