I think a lot of the issues of "dependency hell" come from the non-obvious way to specify a specific version of a shared library to link against (in fact, I can't even find out how to do this in the documentation for gcc or ld now, but I know I've done it, and that it's possible).
Take libncurses for example; I'm using it as an example because there was a time when multiple versions were installed on many linux distros. Both ncurses4 and ncurses5 could be available, and in fact, this used to be case with a lot of linux distros, ncurses4 was often installed for binary compatibility with stuff that was previously compiled.
So you end up with the following files on your machine:
libncurses.so.5.7
libncurses.so.5 (symlink to .5.7, maintained by ldconfig)
libncurses.so (symlink to .5.7, mainted by ldconfig)
Ideally, you'd be able to install ncureses4 on the same machine, and you'd end up with these files also:
libncurses.so.4.x
libncurses.so.4 (symlink to .4.x, maintained by ldconfig)
the .so entry is a link, maintained by ldconfig, to the latest version. There is no conflict because shared linkage is against a specific A.B.C version, resolved at runtime (see below).
So if you compile/link with -lncurses, you get the latest one (which is a reasonable default), as that's what the .so points to. But if you have ncurses4 installed, and you know you want to link against that API, you need to link against the specific file (with a full path and version number in it), rather than use ld.so.cache and gain the advantages of runtime library location resolution.
So the work around for the inability to specify an exact version number to link against without having to give the full path to the library is to move some of the version number to before the .so and make it part of the library name. So you now you link with -llibraryX-2.4 which looks for liblibraryX-2.4.so. No one calls this, or thinks of it as, libraryX-2.4, they think of it as libraryX. Since there is no standard way to do name or indicate version numbers of with this work around, you sometimes end up with these different formats:
Each library maintainer making their own version number format. This could all be the exact same library version.
A quick demo to show that when using -llibrary (and -shared), the full path isn't stored in the binary, just the filename and version number, which is resolved at runtime by the dynamic linker in ld.so to an actual full path using the cache created by ldconfig (hardly conclusive, but you get the gist -- could also use binutils stuff here to see this):
In this case, we see that ld.so.cache maintains a library name to filename mapping, rpm mentions just the library name (with version number), and ldd resolves that to a full path, and that there is no mention of the full path in the rpm binary itself.
I'd really like to see a -llibrary=version option that helps with this, allowing a specific A, A.B, or A.B.C (or whatever string is after .so in the filename)... maybe I was thinking of some linker other than gnu ld that works like this.
If you know what version of a library you want to build against and you know the path to it you can link against it. The compilation and linking is no longer automatic because you have to specify your dependent library specifically, but you can put it anywhere you want and it won't conflict with a different version of the same name.
One way to simplify this is to customize your library builds and associate a unique pkgconfig .pc file with them. When you build your application, reference the .pc file for the library version you want. If the application does not support pkgconfig you could write a wrapper that parses it and provides the build/link options you desire. If you're building packages you're doing enough work by hand that this should not be unnecessarily complex.
If you know what version of a library you want to build against and you know the path to it you can link against it. The compilation and linking is no longer automatic because you have to specify your dependent library specifically, but you can put it anywhere you want and it won't conflict with a different version of the same name.
But that's my point: we should be able to get automatic linking. Different versions of the same library already don't conflict because the file names are different:
libncurses.so.4.0.1
libncurses.so.5.1.1
When you build your application, reference the .pc file for the library version you want. If the application does not support pkgconfig you could write a wrapper that parses it and provides the build/link options you desire.
True, I see what you're saying. It always seemed to me that custom pkconfigs for this case, that generate full paths to the libraries instead of -llibrary options, is a bandaid on not having automatic linking against a specific version.
i too hope one day all this will be automatic (though we still have to specify a version, and keep up to date on what version supports what, so its hardly automatic for the dev or packager). the real fix to me is to embed all relevant information in the binaries and let the dynamic linker figure out which one should be used at run time. compile time would merely embed what version was used to build the app. maybe this is already possible? i'm curious...
While that would be cool, it doesn't seem very pragmatic. I think I'd rather have control over exactly which version is chosen (where I can use implementation details in making a decision), rather than have it decide automatically based on claims expressed in the library metadata.
though we still have to specify a version, and keep up to date on what version supports what, so its hardly automatic for the dev or packager
I think the common case is for new development, you'd most likely develop against the most recent release version, but you don't need to keep up to date on which version supports what because you can continue to use the old versions with long-since compiled binaries as long as it can be installed along side the latest version. I think people and distros, in general, are too quick to remove older versions from being installed (or even available), which increases the on-going maintenance requirements of still popular binaries.
On the other hand, during transition periods, distros have been pretty good about this, like the libc5 vs libc6 transition, and by providing compat packages. But this has mainly been an issue with closed source abandonware (Skype for a long time was still using OSS and needed an ancient version of some audio libs).
Take libncurses for example; I'm using it as an example because there was a time when multiple versions were installed on many linux distros. Both ncurses4 and ncurses5 could be available, and in fact, this used to be case with a lot of linux distros, ncurses4 was often installed for binary compatibility with stuff that was previously compiled.
So you end up with the following files on your machine:
Ideally, you'd be able to install ncureses4 on the same machine, and you'd end up with these files also: the .so entry is a link, maintained by ldconfig, to the latest version. There is no conflict because shared linkage is against a specific A.B.C version, resolved at runtime (see below).So if you compile/link with -lncurses, you get the latest one (which is a reasonable default), as that's what the .so points to. But if you have ncurses4 installed, and you know you want to link against that API, you need to link against the specific file (with a full path and version number in it), rather than use ld.so.cache and gain the advantages of runtime library location resolution.
So the work around for the inability to specify an exact version number to link against without having to give the full path to the library is to move some of the version number to before the .so and make it part of the library name. So you now you link with -llibraryX-2.4 which looks for liblibraryX-2.4.so. No one calls this, or thinks of it as, libraryX-2.4, they think of it as libraryX. Since there is no standard way to do name or indicate version numbers of with this work around, you sometimes end up with these different formats:
Each library maintainer making their own version number format. This could all be the exact same library version.A quick demo to show that when using -llibrary (and -shared), the full path isn't stored in the binary, just the filename and version number, which is resolved at runtime by the dynamic linker in ld.so to an actual full path using the cache created by ldconfig (hardly conclusive, but you get the gist -- could also use binutils stuff here to see this):
In this case, we see that ld.so.cache maintains a library name to filename mapping, rpm mentions just the library name (with version number), and ldd resolves that to a full path, and that there is no mention of the full path in the rpm binary itself.I'd really like to see a -llibrary=version option that helps with this, allowing a specific A, A.B, or A.B.C (or whatever string is after .so in the filename)... maybe I was thinking of some linker other than gnu ld that works like this.