You can go down to 50W idle, but it requires some very specific hardware choices where the ROI will never materialize, some of which aren’t available yet for Zen4.
I have…
* AMD Ryzen 7 PRO 5750GE
* 128GB ECC DDR4-3200
* Intel XL710-QDA2 (using QSFP+ to a quad-SFP+ passive DAC breakout)
* LSI 9500-16i
* Eight WD 16TB HDDs (shucked)
* Two 2TB SK Hynix P41 Platinum M.2 NVMe SSD
* Two Samsung 3.84TB PM9A3 U.2 NVMe SSD
* Two Samsung 960GB PM893 SATA SSD
So that’s the gist. Has a BMC, but dual 40GbE and can sustain about 55GbE over the network (in certain scenarios, 30-35GbE for almost all), running TrueNAS scale purely as a storage appliance for video editing, a Proxmox cluster (on 1L SFFs with 5750GEs and 10GbE idling at 10W each!) mostly running Apache Spark, a Pi4B 8Gb k3s cluster and lots more. Most of what talks to it is either 40GbE or 10GbE.
There is storage tiering set up so the disks are very rarely hit, so they’re asleep most of the time. It mostly is serving data to or from the U.2s, shuffling it around automatically later on. The SATA SSDs are just metadata. It actually boots off a SuperMicro SuperDOM.
——
The Zen 3 Ryzen PRO 5750GEs are unicorns, but super low power. Very tiny idle (they’re laptop cores), integrated GPU, ECC support, and the memory protection features of EPYC. 92% of the performance of a 5800X, but all 8C/16T flat out (at 3.95GHz because of an undervolt) caps at just under 39W package power.
The LSI 9500-16i gave me all the lanes I needed (8 PCIe, 16 SlimSAS) for the two enterprise U.2 and 8 HDDs, and was very low idle power by being a newer adapter.
The Intel dual QSFP+ NIC was deliberate as using passive DACs over copper saved 4-5W per port (8 at the aggregation switch) between the NIC and the switch. Yes, really. Plus lower latency (than even fiber) which matters at these transfer speeds.
The “pig” is honestly the ASRock X570D4U because the BMC is 3.2W on its own, and X570 is a bit power hungry itself. But all in all, the whole system idles at 50W, is usually 75-80W under most loads, but can theoretically peak probably around 180-190W if everything was going flat out. It uses EVERY single PCIe lane available from the chipset and CPU to its fullest! Very specific chassis fan choices and Noctua low profile cooler in a super short depth 2U chassis. I’ve never heard it make a peep, disks aside :)
I'm looking at an LSI 9300-16i which is 100€ (refurbished) including the cables. I just have to flash it myself. Even a 9305 is triple the cost for around half the power draw.
My build is storage, gaming and a bunch of VMs.
Used Epyc 7000 was the other option for a ton more PCIe. I have no need for more network speed.
Yep. 9300s are very cheap now. 9400s are less cheap. 9500s are not cheap. 9600s are new and pricey.
As I said, you can't recoup the ROI from the reduced power consumption, even if you're paying California or Germany power prices. Though you can definitely get the number lower!
I had this system (and the 18U rack) in very close proximity in an older, non-air conditioned home for a while. So less heat meant less heat and noise. I also deliberately chased, "how low can I go (within reason)" while still chasing the goal of local NVMe performance over the network. Which makes the desire to upgrade non-existent, even 5+ years from now.
Not cheap, but a very fun project where I learned a lot and the setup is absolutely silent!
Nope. The extra E was for "efficiency", because they were better binned than the normal Gs. Think of how much more efficient 5950Xs were than 5900Xs, despite more cores.
So the Ryzen PRO line is a "PRO" desktop CPU. So typical AM4 socket, typical PGA (not BGA), etc. However they were never sold directly to consumers, only OEMs. Typically they were put in USFF (1L) form factors, and some desktops. They were sold primarily to HP and Lenovo (note: Lenovo PSB fuse-locked them to the board -- HP didn't). For HP specifically, you're looking at the HP ProDesk and EliteDesk (dual M2.2280) 805 G8 Minis... which now have 10GbE upgrade cards (using the proprietary FlexIO V2 port) available straight from HP, plus AMD DASH for IPMI!
You could for a while get them a la carte from boutique places like QuietPC who did buy Zen 3 Ryzen PRO trays and half-trays, but they are long gone. They're also well out of production.
Now if you want one, they're mostly found from Taiwanese disassemblers and recyclers who part out off-lease 1L USFFs. The 5750GEs are the holy grail 8-cores, so they command a massive premium over the 6-core 5650GEs. I actually had a call with AMD sales and engineering on being able to source these directly about a year ago, and though they were willing, they couldn't help because they were no longer selling them into the channel themselves. Though the engineer sales folks were really thrilled to see someone who used every scrap of capability of these CPUs. They were impressed that I was using them to sustain 55GbE of actual data transfer (moving actual data, not just rando network traffic) in an extremely low power setup.
-- -----
Also, I actually just logged in to my metered PDU, and the system is idling right now at just 44.2W. So less than the 50W I said, but I wanted to be conservative in case I was wrong. :)
44.2W that has over 84TiB usable storage, with fully automagic ingest and cache that helps to serve 4.5GiB/sec to 6.5GiB/sec over the network ain't bad!
Agreed! Despite being PCIe 3.0, these were perfect home server CPUs because of the integrated GPU and ECC support. The idles were a bit higher than 12th gen Intels (especially the similarly tough to find "T" and especially "TE" processors) mostly because of X570s comparatively higher power draw, but if you ran DDR5 on the Intel platform it was kind of a wash, and under load the Zen 3 PRO GEs won by a real margin. Plus you really could use every scrap of bandwidth and compute these chips could muster. You use ALL the chip. :)
My HP ProDesk 405 G8 Minis with a 2.5GbE NIC (plus the built in 1GbE which supported AMD DASH IPMI) idled at around 8.5W, and with the 10GbE NICs that came out around June, are more around 9.5W -- with a 5750GE, 64GB of DDR4-3200 (non-ECC), WiFi 6E and BT 5.3, a 2TiB SK Hynix P31 Gold (lowest idle of any modern M.2 NVMe?), and modern ports including 10Gb USB-C. Without the WiFi/BT card it might actually get down to 9W.
The hilarious thing about those is they have an onboard SATA connector, but also another proprietary FlexIO connector that can take an NVIDIA GTX 1660 6GB! You want to talk a unicorn, try finding those GPUs in the wild! I've never seen one for sale separately! If you get the EliteDesk (over the ProDesk) you also get a 2nd M2.2280 socket for mirroring.
I have three of those beefy ProDesk 805 G8 Minis in a Proxmox 8 cluster, and it mostly runs Apache Spark jobs, sometimes with my PC participating (how I know the storage server can sustain 55GbE data transfer!), and it's hilarious that you have this computerized stack of napkins making no noise that's fully processing (reading, transforming, and then writing) 3.4GiB/sec of data -- closer to 6.3GiB/sec if my 5950X PC is also participating. I don't need the cloud, we have cloud at home!
-----
If you want a 5750GE, check eBay. That's where you'll find them, and rarely NewEgg. Just don't get Lenovo systems unless you want the whole thing, because the CPUs are PSB fuse-locked to the system they came in.
4750GEs are Zen 2s and cheaper (half the price), and pretty solid, but I think four fewer PCIe lanes. Nothing "wrong" with a 5750G per se, but they cap more around 67-68W instead of 39W.
Just if you see a 5750GE, grab it ASAP. People like me hunt those things like the unicorns they are. They go FAST! Some sellers will put up 20 at a time, and they'll all be gone within 48 hours.
-----
I really look forward to the Zen 4 versions of these chips, and the eventual possibility of putting 128GiB of memory into a 1L form factor, or 256GiB into a low power storage server. I won't need them (I'm good for a looooong time), but it's nice to know it'll be a thing.
Intel 15th gen may be great too, as it's such a massive architecture shift plus a new process node. Intel also tends to have really low board chipset power consumption, and really low idles.
Obscenely capable home servers that make no noise and idle in the 7-10W range are utterly fantastic.
I have…
* AMD Ryzen 7 PRO 5750GE
* 128GB ECC DDR4-3200
* Intel XL710-QDA2 (using QSFP+ to a quad-SFP+ passive DAC breakout)
* LSI 9500-16i
* Eight WD 16TB HDDs (shucked)
* Two 2TB SK Hynix P41 Platinum M.2 NVMe SSD
* Two Samsung 3.84TB PM9A3 U.2 NVMe SSD
* Two Samsung 960GB PM893 SATA SSD
So that’s the gist. Has a BMC, but dual 40GbE and can sustain about 55GbE over the network (in certain scenarios, 30-35GbE for almost all), running TrueNAS scale purely as a storage appliance for video editing, a Proxmox cluster (on 1L SFFs with 5750GEs and 10GbE idling at 10W each!) mostly running Apache Spark, a Pi4B 8Gb k3s cluster and lots more. Most of what talks to it is either 40GbE or 10GbE.
There is storage tiering set up so the disks are very rarely hit, so they’re asleep most of the time. It mostly is serving data to or from the U.2s, shuffling it around automatically later on. The SATA SSDs are just metadata. It actually boots off a SuperMicro SuperDOM.
——
The Zen 3 Ryzen PRO 5750GEs are unicorns, but super low power. Very tiny idle (they’re laptop cores), integrated GPU, ECC support, and the memory protection features of EPYC. 92% of the performance of a 5800X, but all 8C/16T flat out (at 3.95GHz because of an undervolt) caps at just under 39W package power.
The LSI 9500-16i gave me all the lanes I needed (8 PCIe, 16 SlimSAS) for the two enterprise U.2 and 8 HDDs, and was very low idle power by being a newer adapter.
The Intel dual QSFP+ NIC was deliberate as using passive DACs over copper saved 4-5W per port (8 at the aggregation switch) between the NIC and the switch. Yes, really. Plus lower latency (than even fiber) which matters at these transfer speeds.
The “pig” is honestly the ASRock X570D4U because the BMC is 3.2W on its own, and X570 is a bit power hungry itself. But all in all, the whole system idles at 50W, is usually 75-80W under most loads, but can theoretically peak probably around 180-190W if everything was going flat out. It uses EVERY single PCIe lane available from the chipset and CPU to its fullest! Very specific chassis fan choices and Noctua low profile cooler in a super short depth 2U chassis. I’ve never heard it make a peep, disks aside :)