sviko.com

ASRock Rack EPC612D4U-2T8R review: motherboard for cloud storage

For storage of “cold” data, where the volume in terabytes on one unit of a server rack is important, the compact 1U cases containing 12 hard disks of a format of 3.5 inches are issued. Integrators and assemblers can use for these needs as ready-made server sets, and to collect storage independently in special cases, such as Chenbro RM14500.

In such cases, the motherboard is given a minimum of space, and it is required not so much – to have a maximum of SATA/SAS ports and be as compact as possible to give as much space as possible for hard drives. For such purposes, ASRock Rack, the server unit we are familiar with all of the company ASRock, the motherboard produces EPC612D4U-2T8R for processors Intel Xeon LGA2011-v3.

This Board is interesting because the manufacturer has integrated into a compact MicroATX form factor 8-port controller SAS-12Gbps (chip LSI3008), two 10-Gigabit network ports BASE-T on the controller Intel X540, two slots PCI Express 3.0 16x and one PCI Express 3.0 8x. In total, one Micro-ATX Board can connect as many as 16 hard drives and 14 network ports, which you will agree, is impressive!

This Board is interesting because the manufacturer has integrated into a compact MicroATX form factor 8-port controller SAS-12Gbps (chip LSI3008), two 10-Gigabit network ports BASE-T on the controller Intel X540, two slots PCI Express 3.0 16x and one PCI Express 3.0 8x. In total, one Micro-ATX Board can connect as many as 16 hard drives and 14 network ports, which you will agree, is impressive!

To provide high-speed communication with other nodes, the Board is equipped with a dual-port Intel X540-T2 network controller with RJ45 interfaces for copper cable environment. Today, almost all ASRock Rack motherboards with a 10-Gigabit network are released under the BASE-T standard, as the General trend in storage systems is to use a conventional twisted pair for network connectivity, and the optical environment remains for faster interfaces. The Intel X540 chip is not the newest in the 10-Gigabit controller family, it does not have the power-saving features of Intel Ethernet Power Management, but like all Intel 10-Gigabit chips, supports iSCSI, FCoE and NFS hardware offloading, and has the SR-IOV feature that allows you to share Board resources between virtual machines. At a maximum 10-gigait speed, the length of the 6th category cable can be up to 55 meters, and when using the 6A category cable – up to 100 meters. Network ports can be grouped according to the IEEE-802.3 ad standard to improve throughput and fault tolerance.

The next important element of the Asrock Rack EPC612D4U-2T8R is a 12Gb/s SAS controller Broadcom (LSI) 3008. It is the simplest chip in the SAS-12 family of Broadcom chips, having a PowerPC 476 processor with a frequency of 1.2 GHz. It supports RAID 0,1,1 E and 10, has PCI Express 3.0 8x bus, supports T-10 data protection type and DataBolt technology, which allows operating at SAS-12Gb/s speeds when using SAS-6 Gb/s disks and backplanes. The total performance of the controller is declared at the level of 1 million IOPS , so the speed margin here is huge.

By the way, continuing the topic of storage, 8 more SATA-600 ports work on two SATA-controllers built into the Intel C612 chip, and the devices can be combined into RAID 0, 1, 5 and 10 arrays.

Because of this dense layout, there is no space left on the motherboard for an Ultra M2 drive, although this is where it would be very useful for caching I / o operations, or for the operating system, if you plan to install Windows. For Linux, VMware and FreeBSD, there is an internal USB 3.0 Type A port where you can install a low USB flash drive with bootloader and OS kernel.

The Board has a large number of connectors and a whole block of jumpers. Here are just a few:

Pins for connecting the NMI button (a softer alternative to reset)

  • A module for connecting the TPM
  • Thermocouple connection sensor (TR1)
  • Output to front VGA port
  • Output to two USB 3.0 ports
  • Output to two USB 2.0 ports

Rebooting through the NMI is a softer alternative to reset, as it helps to avoid data loss when writing heavily to the disk.

One network port LAN1 allocated for the needs of remote management of the BMC. Remote monitoring is assigned to the Aspeed AST2400 chip (network controller – Realtek RTL8211E), the most common server solution for the current day. In fact, although the Board is declared as a 3-port, there are two active network ports here, but their total speed of 20 Gbit/s will be more than enough for the next 5-6 years.

It should be noted that although the motherboard has 3 PCI Express 3.0 slots, they share the data bus with the following restrictions:

  • Upper PCI Express 3.0 16x (PCIE5 slot) when you turn on the SAS controller LSI3008, goes from 16x mode to 8x mode.
  • The bottom PCI Express 3.0 16x (slot PCIE7) shares 8 lanes with the middle slot PCI Express x8 (PCIE6), so when you connect there expansion Board switches to 8x.

The form factor of the average pcie6 slot allows you to install long expansion cards designed for 16x, but when you install the motherboard for its intended purpose, in the cloud storage, most likely all three slots will remain empty, since there is simply nothing to put in them: everything you need is already integrated on the motherboard.with such a tight layout, there is no space left on the motherboard for an Ultra M2 drive, although this is where it would be very useful for caching I / o operations, or for the operating system, if you plan to install Windows. For Linux, VMware and FreeBSD, there is an internal USB 3.0 Type A port where you can install a low USB flash drive with bootloader and OS kernel.

The processor socket has a Narrow ILM format, which is important to consider when choosing a cooler. For outdoor servers you can find on sale cooling system Noctua NH D9DX i4 height 3U. With low-profile coolers, in General, there are no problems – for 2U cases you can use Supermicro SNK-P0048AP4, and for 1U – Dynatron T318 or Dynatron R31.

The motherboard has 4 DDR4 RDIMM/LRDIMM slots with frequencies 1600/1866/2133 MHz and 2400 MHz for Xeon E5-1600/2600 V4 processors. The maximum supported capacity of one RDIMM module is 32 GB and that of the LRDIMM module is 64 GB. If we assume that on 1 TB of disk space it is desirable to have 1 GB of RAM for cache and deduplication tables, then this will be enough for 1U case, clogged with 12 hard drives of 21 TB each.

We would like to note the support of 6 fans, each of which is connected to a 4-pin PWM socket with led indication. If the cooler stops, a red light will light up near the PWM connector. The automatic adjustment of the CPU cooler usually keeps the fan speed at a minimum.

On the back panel only a minimum set of outputs: two USB 2.0, two USB 3.0, three RJ45, VGA, RS232 ports and a UID indicator.

Remote Control

BMC supports the virtual desktop function with the ability to connect ISO files as a CD-ROM, flash drive or disk drive, the so-called Virtual Media function. Any of the system sensors, such as CPU temperature, can be opened as a widget and constantly monitor its changes on the chart. At the same time, surfing between the Windows of the web-interface does not affect the open widgets. The KVM software console allows you to record video, take screenshots, and dump the blue screen of death settings for easy diagnosis. However, if you plan to use the GPU to display the image on the screen, the virtual KVM console will be unavailable, and in such cases it is better to purchase a separate IP-KVM, such as ATEN KN2124VA.

We tested the speed of Virtual Media and got about 3.7 MB/s , so the Gigabit connection here is clearly unnecessary, and connecting remote monitoring to a 100-megabit network switch, you will not lose anything, on the contrary – it is very convenient when installing updates supplied as ISO images for hypervisors on FreeBSD.

After testing the BMC for two weeks, let’s say – some serious complaints about the operation of the device does not arise: the mouse cursor is synchronized almost perfectly, keystrokes are transmitted without delay, but sometimes with the intensive work of the main network adapter Intel X540, the BMC interface becomes unavailable, and it was not possible to understand the pattern of such behavior. I would also like to note that the power management is carried out in the window of the java-widget responsible for KVM, so to enable and disable the server will have to allow pop-UPS in the browser and JAVA.

Very pleased with the hardware monitoring section – the Board displays the readings of all temperature sensors, including the system Board, built-in network card, South bridge, memory modules and processor, as well as readings of all voltage sensors and all speeds of coolers that are in the system.

Compatibility with operating systems

We have tested how ASRock Rack EPC612D4U-2T8R is compatible with the latest operating systems. In all cases, the installation was carried out via a remote connection to the BMC using Virtual Media.
Install_1

  • Windows 10 x64 build 1807 – network driver installation required after installation
  • VMWare ESXi 6.7 – everything works out of the box, including hardware monitoring
  • Windows server 2016 build 1705 – everything works out of the box
  • Debian 9 Stretch initial release – everything works out of the box

In VMware vSphere 6.7, PCI Passthrough hardware mode was available by default for the Intel X540-T2 network adapter, as well as for USB controllers and LSI 3008. Also, when physically forwarding hard drives with SAS and SATA interfaces to the guest operating system, the latter received full hardware access to the device with the ability to read S. M. A. R. T. data.

UEFI (BIOS)

In ASRock Rack EPC612D4U-2T8R uses AMI-BIOS from traditional text-based interface. It is in the General BIOS that the built-in SAS controller LSI 3008 is configured, which by the way, by default works in the “IT Mode” mode, which is important for modern file systems such as Btrfs and Zfs.

In common, most of the settings refer to different functions of the processor, serial ports and control system. Supports the function of waking the motherboard from sleep mode by PCI Express signal, as well as over the network. But the ability to enable the signal from the USB is not provided.

The motherboard has a built-in watchdog that operates in two modes-simple cold reset and NMI call interruption. The latter is recommended only when using intelligent file systems with extensive caching in RAM. The operation of the watchdog requires support from the operating system, and from the tested OS, the easiest way to achieve this is only in Windows Server 2016.

Testing

For testing, we used a computer with the following configuration:

  • Intel Xeon E5-2603 v4 (6C 15M L3, 1.7 GHz)
  • 16 Gb DDR4 ECC RDIMM
  • Noctua NH-D9DX i4 3U
  • 8 x SSD Samsung MZ-7KM480E, 480 Gb, SATA-600

Let’s test the iSCSI speed and compare the response time on the built-in and discrete network controller.

Lan_speed1-1 Lan_speed2

When testing, we noticed that the only strongly warming element on the Board was PCH, whose temperature without the installed fan exceeded 60 degrees Celsius, so if you want to use the motherboard ASRock Rack EPC612D4U-2T8R for a floor server in a non-industrial housing or for NAS-a (ideal, by the way, option) – be sure to provide good air flow to the radiators.

Price

The average cost of ASRock Rack EPC612D4U-2T8R is $570 . For comparison, the retail cost of the LSI 9300-8i controller on the LSI 3008 chip is 210$, and for the Intel X540-T2 network card they ask 230$. Thus, when assembling cloud storage savings on the face.

Conclusions

ASRock Rack company is a specialist in motherboards, and the best part is that the model range of the company has non-standard highly integrated solutions in a compact form factor. Such boards are universal, which can be used for computer virtualization nodes, for SDS-solutions, and for high-density storage systems, including flash-arrays in cases of size 1U and above.

The universal microATX form factor allows the Board to be installed not only in high-density full-depth enclosures, but also in half-depth shortened models designed for installation in wall-mounted telecommunication cabinets. This flexibility allows the server manufacturer to offer the customer a unique solution that will have no competitors in retail and will be able to pass through tenders without the risk of supplying analogues from cunning competitors. It is this feature that makes an ASRock Rack EPC612D4U-2T8R great buy.