September 2018 Archive

Guide: 10 Gigabit Networking

September 11th, 2018

For small office/home setups, 10-gigabit networking is probably a lot cheaper than you think.  For larger environments, it’s certainly going to cost more, but also your IT costs should be a smaller percent relative to human/salary budgets.  (If you’re in a “real” office without 10gbit, and you throw a lot of data around to network resources–get it fixed).

The Limits of Gigabit

Gigabit isn’t that fast!  You’ll see ~100MB/s on network transfers.  Ever needed to temporarily move a few terabytes from a computer so you could reformat it?  It’s painful.  Back in the days when most HDs could read/write at a little north of 100MB/s, it wasn’t as noticeable.  These days, SSDs, multiple-HD arrays, and especially NVMe drives blow way past that limit.

Minimal 10gbe Setup: Direct Host-to-Host

The cheapest possible 10gbe setup is to directly link two hosts.  Very commonly, this is a single server and some network attached storage device.  For that, you will need:

  • 10gb NIC (x2)
  • Direct Attach Cable (x1)

You can find $40-50 eBay listings with two Mellanox ConnectX-2 cards (MNPA19-XTR) and DAC cable for ~$40-50.  That’s everything you need.

For these sorts of direct connections, simply give each network card an IP address in a subnet different from your home network.  If your internal home network is using 192.168.0.X IPs, give the cards something like 10.11.11.11 and 10.11.11.12 IPs.  (The entire 10.X.Y.Z class A subnet is private).  Windows/macOS/Linux/whatever will figure out routing for you.

2-port cards are just a few dollars more, if you want your storage/server to connect to up to two different hosts.

10 Gigabit Switches

The complete approach is to connect your 10-gigabit hosts directly into your network via a 10-gigabit switch.  Unfortunately, this is where things get a little more expensive.  I use UniFi kit, and have a USG-16-XG at home, and have been really happy with it.  But that’s also $600.  If you’re a small office with 5-10 people tossing around videos, photogrammetry data, or gigantic data sets, it’s absolutely worth it.

The setup here is pretty straightforward–just connect your workstation to your network per normal.  My 10gbit hosts at home just have a single cable.

Other switch options:

  • The Quanta LB6M24 24-port SFP+ switch is often recommended, easily available enterprise gear.  It’s cheaper, at ~$280, but also LOUD, and idles north of 100W.  If you have a garage networking rack at home, or a network closet far from your desks at an office, consider it.
  • The MicroTik CRS317-1G-16S+RM is low power, quiet, and has 16 SFP+ cages for ~$380.  Their switch lineup has a few options with 2-4 SFP+ uplinks, too, which might be good options for small setups.  I don’t have experience with this switch, but I have used MicroTik gear in the past and been happy with it.

Other SFP+ Cards (macOS)

The best card I’ve found for macOS is the Solarflare SFN5122F. You should be able to find them on eBay for $40.  These cards perform really well–here’s macOS-to-macOS through a US-16-XG switch on an iperf3 test:

Intel networking cards are generally considered the gold standard for home use.  The Intel X520-DA1 and Intel X520-DA2 are the 1- and 2-port SFP+ cards, but will cost you $80/$120 or so, respectively.

Fiber Options

SFP+ connectors are new to most home users.  The world of fiber can get very confusing–there are different transceivers for different lengths of cable, cables with different wavelengths, etc.  The basics are:

  • “Direct Attach Cables” are copper cables with built-in transceivers.  SFP+ versions can carry full 10gb just fine.  Upside is price, downside is a pretty short length limit.
  • Fiber optic cables are some sort of magical hollow glass.  A 20m cable costs $35, and you’ll need a $20 transceiver at each end.  Upside is no practical length limit, downside is cost and the inability to terminate your own ends.

Fiber is also electrically isolated and totally nonconductive.  I have a fiber strand between my cable modem and my router–if an Arizona monsoon storm lands a nasty lightning strike nearby, in theory I have some extra protection from outside current.

Network Attached Storage

10gbe networking is nice when you need to copy large files around, obviously, but it also opens up a lot of options in terms of your normal file usage patterns.  My main desktops no longer have any HD-based storage inside of them; what appears as local drives are backed by storage on my network.  My “Photos” drive is using ~6TB–I see read speeds of 500MB/s, and there’s another ~16TB available for it.

With a large-enough HD array, you can easily saturate a 10gbe link at full 1000Mb/s read/write.  This is very easy to achieve with even a small SSD-only storage array.  There are two general approaches here:

Filesystem Owned by Server

Your Synology/FreeNAS/Unraid/whatever box owns the filesystem.  You modify things over a file share–probably SMB, especially since macOS is deprecating AFP, but maybe NFS.

Filesystem Owned by Workstation

Your workstation owns the filesystem locally.  This is probably iSCSI, where the server exposes a bunch of block storage, and your workstation/client treats it like any other block storage device–it will format however it pleases and thinks of it as a local drive.  This is especially nice because things like watching files, system-level notifications of changes, etc, all work.

On my macOS desktops, I just have a .DMG disk image on a network share for my bulk storage drives.  Apple doesn’t ship iSCSI support with macOS, because they have awful enterprise-related blind spots, but there are a few 3rd-party options.  I decided to use DMG since it’s built-in, and those mounts still appear as external drives (especially important for me with BackBlaze and my Photos drive).

What About Laptops?

Well!  If you absolutely must get a laptop on your 10gbe network, the easiest approach is an external Thunderbolt enclosure.  Thunderbolt basically connects directly to your PCIe lanes, so you can drop any old PCIe card in there (not just GPUs).  This $200 enclosure works great for non-GPU cards.

Thunderbolt 3 enclosures are backwards compatible on macOS, from something like 10.12.6 onwards.  This compatibility is an Apple-specific thing, so I’m not sure where things lie with Thunderbolt 2 laptops and Windows.  If you have a Thunderbolt 2 MBP, and are using a Thunderbolt 3 enclosure, you’ll also need this adapter and a Thunderbolt 2 cable.  Ouch.  You could also just get a Thunderbolt 2 enclosure, but might as well get something that will be useful for awhile yet.

Why Not Cat5e/RJ45?

I haven’t recommended Cat5e/Cat6 stuff here for a few reasons.  Primarily, it’s far more expensive and hard to come by.  You’ll probably have to buy new equipment, which will cost a lot more, since SFP+ has been the preferred enterprise format for years.  It’s also slightly higher power consumption across the board (NICs/switches both).

If you’re already in an office space with a LOT of cat5e/cat6 cable already run through your walls, it might be worth using.  I just don’t have experience with it and can’t really recommend anything.

Why Not Gigabit Link Aggregation?

There are a multitude of ways to aggregate >1 network ports.  Many consumer NAS devices have 4 gigabit ports, for instance.  The short answer is that single network streams are still limited to gigabit speeds.  This is a great option for something like a NAS that wants to show up with the network with IP over 4 ethernet ports.  Four clients could each be pulling 100MB/s from the server, but an individual client will still be limited to 100MB/s (even if they are aggregating multiple ports on their end too).

Questions?

I am by no means a networking expert, but feel free to ask any questions!  Either over email (mwegner at that Google email service), or here in the comments.  I know–leaving comments on a blog is so mid-200s–but at least they’ll stick around for when someone finds this article a year from now.

8 Cores and 64GB of RAM for $500: The HP Z420

September 6th, 2018

Dear Everyone:  Please stop buying expensive computers.  You probably don’t need them.  My current desktops are both HP Z420s, which are absolutely fantastic given their price.  So fantastic that I’ve decided to write this guide (please don’t wipe out eBay stock entirely).

Some quick caveats:

  • This is a hardware guide.  It might go stale, and maybe soon?  It’s September 2018 right now, and I’ll try to update this prefix with any major information that changes.
  • Because I’m talking about the used enterprise market, it’s very US-centric.  My understanding is that HP is more common in Europe than Dell, so maybe this applies there too?  Sorry, Australia, South America, and literally everywhere else.  I hear your deals are rough 🙁

HP Z420 Prices

This is currently the best desktop hardware sweet spot available.  It’s great.  Here are some current configurations on eBay:

  • E5-1620, 16GB memory:  $270 shipped
  • E5-1650v2, 32GB memory:  $405 shipped
  • E5-2650, 64GB memory: $457 shipped

Very roughly, you’re looking at $300/400/500 shipped for 16/32/64GB models.

Desktop Specs

To be clear:  This is for a complete desktop.  Case, power supply (600W), CPU, memory, and depending on the particular listing, maybe a DVD R/W drive, small HD, sometimes a small SSD, sometimes a video card.  For the purposes of this guide, I’m assuming you’re a technical person/developer/etc, and will be installing your own operating system, storage, and GPU.

 

Wait, What are the Cons Here?

Mostly, you’re buying previous-generation hardware.  The major cons are:

  • Older hardware in general.  However, enterprise/workstation stuff is built-to-last.  Xeons are durable, this is ECC memory, etc.
  • Speaking of memory, Z420 is using DDR3.  Looking at benchmarks from this article about Sandy Lake, which supported both DDR3 and DDR4–they’re pretty linear, in that DDR4 has higher clock, but about the same performance at the same clock (actually a bit worse):
  • You can find HP Z420 listings with 14900R memory (1866Mhz), which I think is the fastest it will go.
  • DDR3, and older generation CPUs in general, will use a little more power.  My desktops idle at 100W and sit at 200W under max CPU-only load.

CPU Selection

The Z420 takes a few different Xeon processors.  Be aware that there are two revisions of the Z420 motherboard, which show as different boot block dates.  2011 boot blocks can’t use V2 cpus, while 2013 boot block can.  If a seller isn’t selling a Z420 included with a V2 cpu, there’s a chance you won’t be able to upgrade.

There are multiple versions of the HP documentation for the Z420, as new CPUs were released, but this should be the latest (click for full-size):

 

There’s a fairly common tradeoff between fast single-threaded performance and more cores.  Just plug any CPU you find into PassMark for an idea.  I’d recommend the E5-1650v2, which is cheap and fast, with six cores for when you do max things out.  It has a PassMark score of 12,697/1,994.  Compare that to the i5-8600K’s score of 12,812/2,521, which is currently $250 for just the CPU.  I’d recommend an E5-1650v2 or E5-2690v2, depending on what you can find on a listing.

Other CPU notes:

  • The Z420’s that ship with an E5-2687W use a watercooling CPU setup.
  • You can replace the CPU cooler.  The Noctua NH-U9DXi4 fits in the case (see fan connector notes in “transplanting entirely” below).
  • The E5-2xxx processors all support dual socket boards.  Irrelevant for a Z420, unless you plan on eventually moving to a Z620/Z820 and taking your CPU/memory with you.

Upgrading From Stock

A totally stock Z420 might work for some tasks (especially if you get one with an SSD)!  I’d recommend the following, though.  FYI, the best place to find deals on components is /r/buildapcsales:

NVMe Upgrade

You’ll need a $15-20 NVMe to PCIe m.2 adapter to drop into an open slot.  NVMe devices just sit on the PCIe bus normally, i.e. on a newer motherboard with a built-in m.2 NVMe slot.  These adapters are basically just physical adapters–the boards are mostly just a bunch of traces.  You’ll be able to get full speed out of an x4 slot, so don’t sweat the slot placement much.

Currently, you can get a 512GB 970 Pro for ~$240, or a 1TB ADATA or something else for $200.  NVMe speeds are totally worth it for heavier work.

GPU Upgrade

The Z420 has a 600W power supply with a one internal rail.  However, it only has a single GPU power connector.  In theory, you could run a spitter to a high-power card, and it’d be fine, but I wouldn’t recommend it.  You’re better off stepping down to a single-power-connector card (i.e. RX 570 or GeForce 1070 currently).  You will probably need a 6-to-8 pin PCIe power adapter.

Power Supply Upgrade

Maybe you absolutely need a beefy GPU, or you want to drop in a fanless power supply in the pursuit of quieter computing (the ultimate here is to just put your PC on the other side of the wall–my main desktop sits in the adjacent room’s closet).

None of the Z420’s power supply mounting holes line up with a standard power supply.  If you’re running your Z420 on its side, you can get away with some foam tape, but in a vertical orientation you’ll probably need to bond in your own L brackets to hold it or something.

The Z420’s motherboard power connector is non-standard.  You’ll need a ~$10 wiring harness to connect it to a standard ATX power supply.

Transplanting Entirely

Maybe you have a nice silent case already, or you found a cheap barebones Z420 motherboard and want to give it a home.  Or you’re kind of bored on a Sunday and have the equipment lying around already.  Whatever the reason, this is where owning a Z420 gets a little hairy.  Some notes on the process:

  • The CPU fan header is PWM, but 5-pin.  The last pin does nothing on a Z420, but the BIOS may freak out and ramp up all fans without it grounded.  Use a jumper wire to ground it (“last” pin being the one that isn’t connected with a 4-pin fan plugged in, and ground being the black wire on the 4-pin fan connector).
  • 4-pin PWM fan headers on the motherboard don’t have a groove for the middle plastic ridge.  No idea why.  You’ll have to Dremel those ridges off any fans you want to connect.  They’ll still connect fine to normal stuff later–there are two grooves on the connectors to line things up.
  • Front USB 2.0/USB 3.0 headers use a “cable sense” pin.  The BIOS will error on bootup and require F1 to continue if either are missing.  The USB 2.0 header needs to be grounded–it’s the pin next to the empty key, and ground to the pin adjacent to it.  The 3.0 header is a huge pain because its pins are tiny and won’t fit into a jumper.  Here’s the info anyway.
  • There’s a temperature sensor in the front panel connectors.  If missing, your fans will run at full speed.

The practical solution to the front connectors is just to rip out the entire plastic front connector array from the Z420 case (they come off pretty easily), and reconnect to your new case.  Really, the default Z420 case is pretty great–heavy, so no fan resonance, and fits large GPUs just fine.  Don’t bother with a case transplant unless you have very good reasons.

Hackintosh

I’m tired of typing words, but a final note–you can absolutely hackintosh these machines.  That is, run macOS natively on non-Apple hardware.  The only thing that doesn’t work is sleep support, which seems to be an Intel C602 chipset issue.  My hackintosh setup is:

  • RX 570/580 for out-of-box driver support. NVIDIA drivers work via the web drivers, but they’re getting worse, and Apple and NVIDIA aren’t likely to make up their little spat anytime soon.
  • FL1100 chipset USB 3.0 card, to avoid the hassle of injecting onboard ports (i.e. this or this).
  • I use a Solarflare SFN5122F card for 10gbit networking.  They have macOS drivers, and it’s ~$40 on eBay.

Maybe one day Apple will have solid, extensible developer-focused desktop hardware.  Let’s be real, though.  At the time of this writing, the Mac Mini hasn’t been updated in 1,421 days, and the last Mac Pro update was 1,722 days ago.  Apple is a $1 trillion company out crashing self-driving cars.  They don’t give a shit about desktop machines for developers.  If they did, I’d be typing this on one.

I expect the era of the hackintosh will fade once all supported Apple machines use their T2 security chip or descendents.  Until then, enjoy the Z420.  If you’re interested in macOS on a Z420, just email me and I’m happy to share my Clover setup.  It’s pretty minimal, although there is some dark magic in generating the power states to fully modulate your CPU frequency up/down depending on load (get a E5-1650v2 if you want to use my config verbatim).

Other HP Options

I have an HP Z820 in the house, which was previously acting as a VM host.  It’s great–16 cores, 128GB memory, and was originally purchased for $900.  Prices have since gone up, thanks to memory getting nuts again.  The Z620 and Z820 are quite similar–both support dual sockets, both can go much higher for total memory.  Be aware that both are substantially more custom, though, and you’ll likely be unable to do any modification on a Z820 without major hassle.  (Seriously, look at this thing–it’s gorgeous).

The Z440/Z640/Z840 is the next generation, but prices are a little crazy due to DDR4.

The Z4/Z6/Z8 are current, and can go absolutely nuts with the latest Xeons, but holy cow, expect iMac Pro style Xeon prices here.

Questions?

I like computers enough to talk random shop about them and write blog posts like this.  Feel free to drop me a line–I’m “mwegner” at the gee-mails.