Don't often upgrade ? No.
Did you realize that package managers can be run from the command line as well ?
Servers might not need a GUI wraper for a package manager, but yes they need a package manager
And it would go without saying that if we're talking about servers we are not talking about web browsers, games, and desktop suites.
I said I run Slackware. So yes I'm aware that package managers run from the command line. That's the only way you can do it with Slackware. There isn't a graphical one. I do all configuration from the command line.
Yes, a package manager is helpful. No, a package manager is not essential.
Right, it goes without saying, so why start with a distro that is desktop oriented? But as you say, it really doesn't matter. Just don't use the stuff you don't need, but why bother with a distro that comes loaded with all that stuff?
A shell is a shell, all distros have your choice - as far as servers go you have basically one descision to make - how do you want to install/update/patch your software ? Apt ? RPM ? Ports ? Maybe you might have a preference for where your defaults and config files are kept - other than that there is no real significant difference in the software.
You don't have to use Apt, RPM or Ports. They are not even distributed with some distros, e.g., Slackware.
You can also download, manually resolve dependencies, compile, and install. I often do. Package managers are a convenience, not a necessity.
If you're a person who knows *nix, you can sit down and run or manage any of the distros and not need to know or care which particular one it is beyond do I need to use Apt or Ports, or whats the default location of the .config file I'm looking for.
Seriously there is basically no difference.
I agree if you are a person who knows *nix you can manage any distro. But that is not because they are all alike. Just to point out one difference, init styles differ. Most distros use System V style init, some like Slackware use BSD-style (although Slackware provides compatibility for Sys V via the /etc/rc.d/rc.sysvinit script). Some are unique, like Gentoo.
I never tried to make the point that there was any significant difference in how the various distros are managed, so there's no reason why you should argue that point. We don't disagree. What I said was why build a server on a desktop-oriented distro when you aren't going to use most of the desktop-oriented software to begin with?