Review of ASRock Rack EPC621D6U-2T16R for huge NAS

Today we are reviewing the asrockrack motherboard, which is designed for those who build a server for Big Data or a data storage system for hundreds of Terabytes. The EPC621D6U-2T16R model resembles the previously reviewed EPC612D4U-2T8R for Socket 2011-3, but with the only difference that this time the manufacturer did not limit itself to an outdated platform. Just imagine, the EPC621D6U-2T16R motherboard allows you to connect 30 (thirty) SATA devices, plus an NVME disk and a USB flash drive.

That is, you can connect all the 3.5-inch front 4U compartments of the case with just one Board, so there will still be free channels for SSDS placed inside or behind. And not to be short, you have a Slimlink port for connecting NVME U2 drives, you have a download from SATA DOM, and two 10-Gigabit RJ45 ports for connecting to your infrastructure, all in MicroATX format. Of course, with this announcement, this Board is asking for the build, but let’s first find out why it is exactly like this.

CPU - LGA3647 Xeon Scalable

In general, if the motherboard has many channels for connecting SSD/HDD, it is most likely created for storing information, not for processing it.

Today, both Intel and AMD have special low-cost processors for storage applications, such as the Xeon-D or EPYC 3000. If you do not get involved in the eternal dispute “Intel vs AMD”, then choosing a discrete processor Xeon Scalable gives you the following advantages:

  • first, you have a 6-channel memory controller versus a 4-channel Xeon-D controller
  • second, you have the Intel c621 South bridge that has 16 SATA channels with software RAID support
  • third, you can use the cheapest Xeon Bronze 3104 processor with 6 cores, support for the cheapest ddr4-2133 ECC Reg memory and a frequency of 1.7 GHz without Turbo Boost. Its power is quite enough for a file server, and it is cheaper than Xeon-D.

In general, the idea of building a storage unit on a 6-core Xeon Bronze is also interesting because this processor has a very low power consumption, around 50 W, so in a normal server, it may be enough for a passive radiator, such as Supermicro SNK-P0068PS, which will allow you to build a very quiet data storage system that can be kept in an open cabinet in Open Space. When choosing a cooler, keep in mind that ASRock Rack EPC621D6U-2T16R has a Narrow ILM socket, and if you still do not know how this socket differs from Square ILM, I recommend reading our review of coolers for Xeon Scalable. Intel’s Junior 6-core processors have 48 PCI Express channels, the same number as the top-end Xeon Gold and Platinum, so you don’t lose anything in terms of connections.

Of course, you can install here at least the most powerful 28-core CPU, 6-phase VRM modules are ready for high load, but this decision is hardly justified from the point of view of storage. The manufacturer has not released a separate list of compatibility with processors, and this usually means that the entire Skylake series up to Xeon Platinum 8280 is supported.

By the way, I just want to note that the Board has a simpler memory power filtering scheme than say a two-processor monstrous EP2C621D12-WS for workstations. It is understandable: the MicroATX format is so small, and you have to save space on the motherboard. The processor uses a 6-phase power supply and a 2-phase power supply for memory modules, and the manufacturer claims support for DDR4 ECC RDIMM with frequencies from 2133 to 2933 MHz. The latter works with processors based on the Cascade Lake architecture and the latest BIOS versions.

Network: Intel X550-BT2

In the field of high-speed optical interfaces, every year there is some kind of revolution, and the 10GBASE-T standard has stalled at the level of previous years (read the comparison of 10G SFP+ and 10GBase-T), and from my point of view, the best network controller here is Intel X550-T2. The EPC621D6U series motherboard can be equipped with either a 2-port 1-Gigabit Intel i350 controller (in models without the “2T” index), or a 2-port 10-Gigabit Intel X550-BT2.

On some motherboards, you can find the X550-AT2 controller, but its difference "BT2 “is only in support of PCI Express 3.0, while” BT2 " supports PCI Express 2.1 with a channel width of up to 8x. In terms of bandwidth, this difference does not matter, but I want to warn you against firmware network controllers firmware version 2.10 from the official site of Intel. In our testlab, three different versions of Intel X550 (one discrete and two embedded on motherboards) started losing the second port after flashing to firmware version 2.1, and we had to roll back to version 1.08.

Theoretically, the controller supports intermediate speeds of the NBase-T standard, equal to 2.5 Gbit/s on a cable of the 5th category and 5 Gbit / s on the 6th, but only under Linux, and moreover-these speeds are not declared by the manufacturer of the motherboard. In any case, I don’t think it’s the intermediate speeds that matter for 30-disk storage. The built-in network card supports booting from PXE, as well as connecting to the IPMI controller in out-of-band mode, and if you choose the 1-GB version of the motherboard, it also supports booting over iSCSI.

Data storage subsystem

The distribution of data storage channels is as follows: 12 SATA-600 ports are output from the Intel C621 southbridge via 4-channel SFF8647 connectors in the lower-left corner of the Board. Two more SATA-600s are represented by regular ports, and the red one has power for SATA DOM drives, and the white one is parallelized with the M. 2 slot, and will turn off as soon as you install an SSD M. 2 drive with a SATA interface, so use the latter only for NVME, especially since this slot is connected via the PCI-E bus directly to the processor.

Special mention should be made of the Slimline U. 2 x8 port for connecting a bucket with PCI Express drives directly to the processor. Today, the cost of PCI-E SSDS is equal to the price of SATA drives, and the only reason for installing the latter is the lack of enough lines to connect PCI-E devices. At the same time, PCI Express drives, according to Micron, in some cases can reduce the access delay by 100 times!

And, of course, the gem of this motherboard is the lsi3616 HBA controller, which supports both SAS-12 and SATA-600, and PCI Express drives (including NVME). This chip promises performance at the level of 1 million IOPS, the ability to work at speeds of SAS-12Gb/s when using SAS-600 backplanes and automatic distribution of PCI Express lines in configurations of 4x4, 16x1 or 1x16.

The output from the lsi3616 controller is via 4 sff8647 ports, that is, you can access: 16 SATA-600 channels, 12 SAS-12 or 4 NVMe PCIe Gen3. 0. the controller mode (PCI-E or SAS/SATA) is switched by two jumpers near the sff8647 ports, that is, you can use two ports for PCI-E devices and a pair for 8 SAS/SATA devices.

Naturally, the LSI3616 uses all 16 lines of the PCI Express 3.1 bus, so it is connected in parallel with the PCI-E4 slot via a divider, and if you install an expansion card in this slot, the controller will not work. That is, figuratively speaking, you have one slot for expansion boards, which you can use without restrictions, while the other is sacrificed to the SAS-12 controller, and is completely useless on this Board.

Another unpleasant point associated with placing the LSI3616 on the Board is that the chip has an oversized radiator that overlaps the topmost PCI Express slot in the case. Here is a question of saving: a normal microATX case has 4 slots for PCI cards, one of which is logically used for the FC controller,and the other for the SSD sled. So, the bottom slot you lose from the PCI Express switch, the top - because of the cooler HBA controller, you have one PCI compartment above the M. 2 port and one full PCI Express 16x.

Fortunately, if you do not adhere to high moral principles and on a short leg with the supplier of motherboards, the radiator of the lsi3616 chip can be slightly filed: it is clearly taken with an excess for a 10-Watt controller (this is how much the maximum consumes HBA LSI3616 without RAID), and without a guarantee, you can install 4 SSD drives in empty compartments for PCI cards. The allowable chip temperature is 85 degrees Celsius, but in normal mode, the controller does not heat above 60 degrees even with a “modified” radiator.

It is clear that this motherboard is not bought in order to install some kind of expansion card in the lower slot of the PCI-E4, while losing the built-in LSI 3616 controller, but if you put the SSD sled there, keep in mind that they can block access to the pad for the COM port, again if you need it for something.

In principle, ASRock Rack has already performed a miracle by putting a huge lga3647 socket, network controller and SAS HBA on one MicroATX Board, while leaving Slimlink X8 so that you can build the fastest storage on PCI Express disks. But for some reason, the developers completely forgot that there is a lot of empty space on the “back” of the motherboard, using which they were once able to perform the feat and install Xeon Scalable on the Mini-ITX.

BIOS, remote management and monitoring

BIOS settings are fairly standard: modes of operation of PCI slots, SATA controllers, as well as basic properties of power consumption, including the restriction on the processor’s TDP. I would like to note that ASRock Rack EPC621D6U-2T16R allows you to control the speeds of only PWM fans, and only speed monitoring is available for 3-pin fans.

Like all modern motherboards, the ASRock Rack has a separate 1-Gigabit network port for remote control based on the ASpeed AST2500 chip. This chip gives you full HTML5 control with an interface optimized for smartphones, as well as an IPMI interface for connecting centralized monitoring systems. Here’s what exporting IPMI data in text format looks like:

root@ubuntu:/tmp# ipmitool -H 192.168.1.206 -U admin -P admin sensor
3VSB             | 3.400      | Volts      | ok    | 2.880     | 3.060     | na        | na        | 3.740     | 3.900     
5VSB             | 5.040      | Volts      | ok    | 4.260     | 4.500     | na        | na        | 5.490     | 5.760     
CPU1_VCORE       | 1.790      | Volts      | ok    | 1.250     | 1.320     | na        | na        | 1.980     | 2.070     
VCCM ABC         | 1.230      | Volts      | ok    | 1.020     | 1.080     | na        | na        | 1.320     | 1.380     
VCCM DEF         | 1.210      | Volts      | ok    | 1.020     | 1.080     | na        | na        | 1.320     | 1.380     
CORE_PCH         | 1.000      | Volts      | ok    | 0.720     | 0.760     | na        | na        | 1.100     | 1.150     
1.05_PCH         | 1.050      | Volts      | ok    | 0.890     | 0.950     | na        | na        | 1.160     | 1.210     
1.80_PCH         | 1.790      | Volts      | ok    | 1.530     | 1.620     | na        | na        | 1.980     | 2.070     
BAT              | 3.140      | Volts      | ok    | 2.000     | 2.700     | na        | na        | 3.400     | 3.560     
3V               | 3.320      | Volts      | ok    | 2.880     | 3.060     | na        | na        | 3.740     | 3.900     
5V               | 4.980      | Volts      | ok    | 4.260     | 4.500     | na        | na        | 5.490     | 5.760     
12V              | 12.100     | Volts      | ok    | 10.200    | 10.800    | na        | na        | 13.200    | 13.800    
PSU1 VIN         | na         | Volts      | na    | na        | na        | na        | na        | na        | na        
PSU2 VIN         | na         | Volts      | na    | na        | na        | na        | na        | na        | na        
PSU1 IOUT        | na         | Amps       | na    | na        | na        | na        | na        | na        | na        
PSU2 IOUT        | na         | Amps       | na    | na        | na        | na        | na        | na        | na        
PSU1 PIN         | na         | Watts      | na    | na        | na        | na        | na        | na        | na        
PSU2 PIN         | na         | Watts      | na    | na        | na        | na        | na        | na        | na        
PSU1 POUT        | na         | Watts      | na    | na        | na        | na        | na        | na        | na        
PSU2 POUT        | na         | Watts      | na    | na        | na        | na        | na        | na        | na        
MB Temp          | 32.000     | degrees C  | ok    | na        | na        | na        | 55.000    | na        | na        
Card side Temp   | 54.000     | degrees C  | ok    | na        | na        | na        | 68.000    | na        | na        
TR1 Temp         | na         | degrees C  | na    | na        | na        | na        | 65.000    | na        | na        
CPU1 Temp        | 41.000     | degrees C  | ok    | na        | na        | na        | 87.000    | 88.000    | na        
PCH Temp         | 43.000     | degrees C  | ok    | na        | na        | na        | 85.000    | 86.000    | na        
DDR4_A Temp      | 49.000     | degrees C  | ok    | na        | na        | na        | 84.000    | 85.000    | na        
DDR4_B Temp      | 54.000     | degrees C  | ok    | na        | na        | na        | 84.000    | 85.000    | na        
DDR4_C Temp      | 50.000     | degrees C  | ok    | na        | na        | na        | 84.000    | 85.000    | na        
DDR4_D Temp      | 40.000     | degrees C  | ok    | na        | na        | na        | 84.000    | 85.000    | na        
DDR4_E Temp      | na         | degrees C  | na    | na        | na        | na        | 84.000    | 85.000    | na        
DDR4_F Temp      | na         | degrees C  | na    | na        | na        | na        | 84.000    | 85.000    | na        
Lan Temp         | 54.000     | degrees C  | ok    | na        | na        | na        | 103.000   | 104.000   | na        
SAS Temp         | 60.000     | degrees C  | ok    | na        | na        | na        | 65.000    | na        | na        
CPU1_FAN1_1      | na         | RPM        | na    | na        | na        | 100.000   | na        | na        | na        
FRNT_FAN1_1      | 1900.000   | RPM        | ok    | na        | na        | 100.000   | na        | na        | na        
FRNT_FAN2_1      | na         | RPM        | na    | na        | na        | 100.000   | na        | na        | na        
FRNT_FAN3_1      | 2400.000   | RPM        | ok    | na        | na        | 100.000   | na        | na        | na        
FRNT_FAN4_1      | 1900.000   | RPM        | ok    | na        | na        | 100.000   | na        | na        | na        
REAR_FAN1_1      | 700.000    | RPM        | ok    | na        | na        | 100.000   | na        | na        | na        
CPU1_FAN1_2      | na         | RPM        | na    | na        | na        | 100.000   | na        | na        | na        
FRNT_FAN1_2      | na         | RPM        | na    | na        | na        | 100.000   | na        | na        | na        
FRNT_FAN2_2      | na         | RPM        | na    | na        | na        | 100.000   | na        | na        | na        
FRNT_FAN3_2      | na         | RPM        | na    | na        | na        | 100.000   | na        | na        | na        
FRNT_FAN4_2      | na         | RPM        | na    | na        | na        | 100.000   | na        | na        | na        
REAR_FAN1_2      | na         | RPM        | na    | na        | na        | 100.000   | na        | na        | na        
ChassisIntr      | 0x0        | discrete   | 0x0080| na        | na        | na        | na        | na        | na        
CPU1_PROCHOT     | 0x0        | discrete   | 0x0080| na        | na        | na        | na        | na        | na        
CPU1_THERMTRIP   | 0x0        | discrete   | 0x0080| na        | na        | na        | na        | na        | na        
PSU1 Status      | 0x0        | discrete   | 0x0080| na        | na        | na        | na        | na        | na        
PSU2 Status      | 0x0        | discrete   | 0x0080| na        | na        | na        | na        | na        | na        
PSU1 AC lost     | na         | discrete   | na    | na        | na        | na        | na        | na        | na        
PSU2 AC lost     | na         | discrete   | na    | na        | na        | na        | na        | na        | na        
CPU_CATERR       | 0x0        | discrete   | 0x0080| na        | na        | na        | na        | na        | na      

In addition to the processor, memory modules and chipset, the motherboard uses IPMI monitoring to track the temperature of the SAS controller and the network card, for which the upper temperature threshold is 103 degrees Celsius. A graphical dashboard can be easily assembled in Grafana:

In general, the Board manages five fans, monitors 14 temperatures, the state of two power supplies and the main server voltage. In the field of management, everything is very decent, so you don’t have to worry about monitoring storage.

Compatibility with ZFS/Btrfs

Since the lga3647 socket has already become quite an ordinary phenomenon by 2020, we do not have any problems with modern operating systems: Windows Server 2019 DE, Centos 8.1, FreeNAS 11.3 and ESXi 6.7 are perfectly supported by the considered platform, the HBA controller is defined and allows you to read the S. M. A. R. T. of connected drives. Once again, I want to note that you still have full compatibility with ZFS, BTRFS, and other tools that require Raw access to the drive.

Please note that the SAS controller requires support for the MPT2Fusion driver family, so the operating system must not be older than 2018.

Recommendations for ordering

Choosing ASRock Rack EPC621D6U-2T16R as a storage platform, you can be sure that you will not be offered an analog at tenders: this is a unique and unique Board created for a single highly specialized task. Please note that when using SAS12 drives, there can be no more than 12 of them connected, and if you use VMware ESXi 6. x+ as the main operating system, then you can throw both the LSI 3616 controller itself and two SATA controllers integrated into the South bridge into the guest OS.

Most rack-mounted server cases today support ATX and EATX motherboards, so I wouldn’t say that the MicroATX format is directly a necessity here. Quite another thing is the Tower format cases, which can be filled to the top with baskets for SSD/HDD and get a powerful storage system with 20 gigabits “out”. In this case, the scenario of using Xeon bronze series processors and the ASRockRack EPC621D6U-2T16R motherboard in a compartment with cheap DDR4 Registered PC2133 memory looks harmonious and is a great purchase!