Intel is actively promoting its LGA-3647 platform not only for the server market, but also for top-end computers designed for enthusiasts, as well as for workstations optimized for complex parallel calculations. Figuratively speaking, today this platform allows you to assemble a working machine with 56 physical cores (112 CPU threads) in one desktop case. Usually, such power is required for any scientific calculations with high accuracy performed on the processor. However, both Intel and motherboard manufacturers understand that the customer is likely to use GPUs or discrete accelerators for machine learning functions, so Xeon Scalable processors in this case are considered as the only solution from Intel that has a controller on 48 lines PCI Exprees 3.0. Want to connect more than three PCI-Express 16X GPUs? Choose a dual-processor motherboard, and put at least the simplest 6-core Xeon Bronze 3104, you will have 96 lines PCI-E.
SHOW FULL ARTICLE
Motherboard EP2C621D12 WS from ASRock Rack this is exactly the same universal case when you choose a platform for the machine under the LGA-3647, want to have a large number of PCI-Express slots and the basic capabilities for today’s storage system. The main advantage of the motherboard is as much as 7 slots PCI Express 16x format. As usual, this requires clarification: the PCIE slots 3 and 4 share a PCIE x16 bus and work or in the format of x16 + 0 or in the format of x8 + x8. PCIE 2 operates in x8 mode, that is, in fact, remains 5 slots for speed x16 slots, three of which you will be able to take 2-slot GPU cards without additional extension cords.
But in addition to the traditional PCI Express slots for expansion cards, the EP2C621D12 WS motherboard has two M. 2 slots with support for speeds up to PCIE (x4). The first of them is connected to CPU1, and the second – to the southern bridge Intel C621 PCH. Both slots support SATA3 drives ranging in size from Key 2230 to Key 2280. The use of PCI Express M. 2 format drives does not affect the speed of expansion card slots.
Of course, the main storage system in this case is transferred to SATA-array. To load the hypervisor on the Board there are two ports for SATA DOM drives (red) with additional power, but for the main array there are 10 SATA 600 ports, 8 of which are connected to the first SATA controller, and two more – to the second. In total, you have access to 14 SATA ports, which will be enough without exaggeration for any storage needs. The motherboard supports RAID levels 0, 1, 10, and 5 through Intel drivers.
But since the motherboard ASRock Rack EP2C621D12 WS is not created for servers, but for workstations, there is no SAS controller or PCI Express ports for connecting 2.5-inch SSD. How important is it? From our point of view, two M2 slots are enough to keep the operating system with GUI and cache data from SATA drives. In extreme cases, it is always possible to install an accelerator Board for 4 M. 2 drives.
But with peripheral ports, the situation is, to put it mildly, strange. In General, the Board is designed for workstations, but this is indicated only by the built-in audio path, and even then not very high quality: the Realtek ALC892 codec is rooted as much in 2011, on the Board the analog path is located in the lower corner, from which without additional shielding the signal simply goes through the entire very long Board to 3.5 mm sockets, which in their form, come from the same place, from the early 2000s.
The choice of network connections is even stranger: 4 1 GB / s interfaces, for which a rare Marvell 88e1543 controller is selected, a separate BMC port for remote control, and only 2 USB 3.0 ports. At least this network controller is supported by ESXi 6.x+, and the Board has connectors to connect the front two USB 3.0 ports and two USB 2.0: it is not enough, my friends.
In General, I believe that on the motherboard EEB format enough space to install and USB 3.1 controller, and I do not understand the meaning of installing 4 network ports in the device, which well obviously does not want to be a network gateway. Need a fast network – put 10 / 25G, BMC control for the workstation is not relevant at all. In this regard, Asus C621E Sage is more clear and balanced.
Perhaps ASRock Rack just unfairly put this fee in the section “workstations”. If you look at all the same from the point of view of the server, everything is clear, except for the Marvell network chip.
The motherboard supports processors with TDP up to 205 W, that is, all existing today Xeon-s series Skylake, including the top Xeon Platinum 8180. To each processor power is supplied through the 8-pin ATX12V connector, ATX 24-pin connector here is one, so that one power supply for the machine will be enough.
For RAM modules, a 2-phase power supply circuit with e88e solid-state polymer capacitors with a temperature rating of 105 degrees Celsius is used. Each CPU has a 6+1-phase power system with Renesas ISL69127 controller.
Each CPU has a 6+1-phase power system with Renesas ISL69127 controller. On the reverse side of the Board there are polymer solid-state capacitors, as on most modern professional motherboards.
For fans here, frankly, paradise: 7 dual 6-pin connectors, each of which has its own led indicator of failure. The Board has automatic speed control of CPU, front and body fans.
IPMI / BMC
BMC are due to modern chip Aspeed AST2500 the most contemporary solution with a normal HTML5 interface, compatible including mobile devices. You do not need to install any client software on the operator’s computer, do not bother with compatibility with browsers – everything works under PC, and under MacOS, and under Linux and Android.
From the KVM interface you can connect virtual media from the local computer, but this is the last century. It is much more convenient to mount the folder on the NAS with the installers of operations and choose which image and how to connect. The machine scans the CIFS / NFS folder in search .ISO files and offers you a connection not only in the form of a floppy disk or CD-ROM, but also in the form of HDD, so you can be downloaded over the network. However, the installation showed that Virtual Media works at a speed of about 1.5 MB / s , which is frustrating: the breakthrough did not happen, but in vain, because it is not just a method of installing the OS remotely, but also a practical replacement of PXE boot.
Of course, the Java-application is also left, but now it is only necessary to run the on-screen keyboard, if for example you find it difficult to press F11-F12 under MacOS.
A separate item is dedicated power management, separately-sensors and logs. It is a pity that there are no graphs in the interface, I really liked to monitor the temperature change, but this is partially offset by a very clear interface for viewing logs.
The motherboard has a standard AMI (Aptio) BIOS with a standard set of settings for server boards. I want to note that the motherboard has a function WatchDog, which for some reason in the workstation-model do not put (has two options: NMI and Reset).
Basically, the BIOS has everything you need for a server motherboard, including boot logs and errors.
- Processor: 2x Intel Xeon Gold 6146
- Memory: 32GB (2 x 16GB) DDR PC2400
- Vmware ESXi 6.7
- Debian 9
Let’s test the speed of Virtual Media:
- AMI Virtual CD (the image is stored on NAS) – 1512 KB/s
- AMI Virtual CD (the image is connected with the computer) – 1680 KB/s
At times, the speed jumps to 4 MB/s, but it is more like the cache. So for BMC monitoring needs, a normal 100-megabit switch will be more than enough.
ASRock Rack makes excellent server motherboards, and if you consider the EP2C621D12 WS as an option for a machine for scientific calculations and machine learning, then this is the best option. There are 7 PCI Express slots for expansion cards, and a large number of SATA-ports, well, in General, everything that should be in a modern highly integrated motherboard.
What I liked:
- Design of the Board under the rack housing with air flow perpendicular to the ports of the motherboard.
- Two M. 2 slots that do not share performance with other PCI Express ports
- Two SATA DOM ports
- Very good IPMI/BMC interface
- Traditional BIOS
- 6-pin cooling connectors for high-power fans
What’s not to like:
- Only 2 USB ports on the Board panel
- No 10g Ethernet ports
- The audio interface is suitable only for RDP operation
In General, today there are 3-4 similar motherboards available on the market separately from server platforms. If you want to purchase the motherboard in question through a tender, specify 7 PCI Express slots of 16x physical format, 12 SATA ports and 4 network ports. These specifications do not include any Asus or Supermicro or Tyan Board, so you will get exactly what you need.