sviko.com

QNAP TDS-16489 U - testing Tiering, SSD caching and virtualization in enterprise storage

Modern storage systems are primarily software functions aimed at storing, protecting and distributing information. I would like to say that in the second place - it is “iron”, but no - for small companies it has become important, what services can be integrated with the storage system, whether it can reserve or completely replace any “cloud” and how many other paid services replaces one NAS today and will replace tomorrow. The developers of network storage went further and offered to transfer the entire virtual infrastructure to one device: both data and applications themselves. And when the manufacturer says: “Inside our NAS, you can run virtual machines and containers, work with CAD applications using a video card, on the same device - store hundreds of terabytes, which are available without delay to your programs and which are constantly protected by snapshots, replications and backups to the cloud”, here we begin to think, and how many devices in the rack will replace one such NAS, and here we remember about “hardware”.

QNAP needs no introduction. Starting with the desktop model is available in SOHO, and today she releases a powerful virtualization host series TDS-16489U built on two Intel Xeon E5-26xx V3 (8 cores, 16 threads) with 256 GB of memory, with PCI Express slots for expansion cards, and most importantly - with the software, and data storage function and the functions of the hypervisor consolidated in a single web-based interface and are controlled from a common window.

The well-known dilemma of the last five years: “what drives to put - fast SSD or large HDD” company QNAP decides simply: “put both, we have SSD-cache does not use hard drive bays: in our flagship model TDS-16489U, created to work with Big Data, you are waiting for 16 compartments for 3.5-inch HDD (SAS/SATA), plus 4 compartments for 15-mm SSD (SAS/SATA), 4 slots PCI-Express, and all this - in 3U case”. Different types of SSD-media are needed in order to use both technologies of SSD-acceleration, which today is in this world: Tiering and SSD cache.

Tiering - removal of hot data on SSD

The term “Tiering” means the distribution of data by media types, as by layers of the pie. It’s very simple - let’s say we have a large amount of information that we store on slow 7200 RPM hard drives. Periodically, we access these files and start working with them, reading and saving them to disk. This is a typical situation when preparing a report or analyzing a large archive. The storage controller understands that if this data suddenly aroused interest, and the application or users access them, it is necessary to move them to faster media and automatically transfers them to where the access speed and the cost of a terabyte is much higher.

Tiering_scheme

QNAP has three levels of storage in Qtier technology:

  • Hard disks 7200 RPM SATA/NL-SAS (Capacity layer)
  • 10K/15K RPM SAS hard drives
  • SSD (Performance layer)

Tiering creates another RAID array from the SSD, the volume of which is added to the existing disk pool, and this has a huge advantage over simple SSD-caching, where the amount of solid state drive is not taken into account in the total disk array. It would seem, it is possible to collect inexpensive high-speed SAS-hard drives and to hammer them an additional disk shelf to receive higher speed, but in the current realities disks on 10/15 thousand revolutions became so unprofitable that producers remove them from the conveyor: this sector is completely forced out by solid-state drives therefore in real life the most demanded files at activation of multilayered storage will always lie on solid-state drives. That is why it is very desirable to use RAID 6 for ultra-fast level.

Specifically in QNAP TDS-16489U under SSD allocated 4 additional compartment with hot swap, so that solid state drives are not put at the expense of the Capacity-layer, and in addition to it. Given that at the time of writing, there were 7.68-terabyte 2.5" SSDS (HGST Ultrastar SS200 and SS300) on sale, these four additional compartments will be enough for almost 30 GB of terribly fast data.

The Qtier process is fully automated - the self-learning algorithm takes into account the frequency of access to the data blocks and transfers the most popular to a higher layer. This process is performed either on a schedule (at night, when everyone is asleep), or at a minimum load on the storage - how to configure. You can force “right now” move, but you can’t affect what the system wants to put on the SSD and what on the HDD: you can only choose the directories and LUNs for which QTier is activated.

One of the most useful features of multi-layer storage is storing new files directly on the Performance layer. This is a mega-useful option, and here’s why: if you’re using large-volume LUNs stored on a HDD, then be prepared for the fact that the storage will take a very long time to analyze their use before transferring to an SSD. And so you can copy an existing LUN, or create a new one - and it will immediately be on a higher layer, and at high load, there will remain forever. And remember that if you upgrade an existing disk pool to the QTier level (this is what the manufacturer calls multi-layer storage activation), you will not be able to disable it.

To show you the beauty of this technology, we took 4 enterprise Hitachi SSDS (HGST) of the 400GB UltraStar SS200, and “upgraded the disk pool to QTier” by creating RAID 5 of these drives, adding another 1 terabyte of SAS SSD to the existing disk terabytes. The upgrade process took about 30 minutes, after which the patterns of real-world tasks were launched, taken by the specialists of PureStorage, to assess how our storage is changing.

Diagram-1

Yes, of course, we are all used to the fact that the SSD is ten times faster than the HDD, but note - we have not sacrificed a single hard drive, and all 16 compartments of the Capacity-layer is still at our disposal, and since QTier is part of the Qts operating system and does not require the purchase of licenses, we pay only for solid-state drives. Each of the used HGST Ultrastar SS200 in our test costs about $440 - is it a lot or a little from the point of view of QTier?

Diagram-2

And if we take into account that with the growth of SSD volume, their speed indicators grow (along with the price, however), then the use of terabyte SSDS can give even more impressive coefficients.

If the user will be available settings “how and what and where to transfer between layers”, the need for a traditional SSD-cache will not be at all.

SSD caching

SSD caching works at the block level, and immediately when you configure it, you will be prompted to choose for what access the fast media is used: for random or serial, as well as whether it will work only for reading, or for writing, too.

In General, the ideology of the SSD cache is that it acts as an intermediate link between the client and the SSD hard drives, and the data in it is not delayed for a long time, being replaced by LRU or FIFO algorithms. That’s why choosing SSD for read/write mode, QNAP operating system will warn you twice that you may lose information on the disk pool if the SSD is broken or disconnected. Before you can replace a caching SSD, you must first disable it in the operating system settings, wait for its contents to synchronize with the hard drives, and then remove it from the machine. Our tests have shown that in some situations, it can take up to 10 hours (!) to reset the contents of the SSD cache to the disk pool.).

For more efficient use of SSD provides a mechanism of “cut-off” long blocks of data at random access, designed to prevent the cache that can be quickly read or written to hard drives (for example, large long files that are read and written entirely, such as video recording).

SSD cache does not depend on the division of data into layers, and you can use either of these technologies separately or both at the same time, allocating for caching the fastest and most expensive NVMe media, and under Tiering - SAS/SATA SSD, which are cheaper.

In our testing we used 1.6 TB HGST SN150 Board with the declared capacity of 740 000 IOPS 4k at random reading. Immediately it is necessary to mention that this Board is not from the list of recommended QNAP, but we still decided to try it, as there was no other alternative at hand. Unfortunately, acceleration, random access with her work, though she decided. We tried LUNs of different sizes, switching the caching mode to random and sequential access, but there was no difference with the disk array in our tests. Therefore, in practice, we recommend choosing components from the list of compatible models on the manufacturer’s website.

Diagram-3

Regardless of the operating system settings, serial access acceleration worked fine. On the example of 4 LUNs with a capacity of 900 GB, it is seen that at the beginning of the test, the storage saturates the cache, reading data from hard disks, and then the flow goes completely from the cache until it is emptied. By the way, the graph is obtained for the mode “acceleration of random read/write access” with a cutoff of 4 MB.

This: accelerating the QNAP TS-16489 U disk array, it is better to use the Tiering technology, and to abandon the SSD cache at all.

Virtualization with passthrough of the video card and USB

QTS operating system from QNAP (version 4.3 at the time of this review) supports two virtualization environments: traditional and container. The first uses its own hypervisor Virtualization Station 3, which is based on KVM. For containers used plugin Container Station, which supports both format - Docker and LXC, the repository QNAP also available private server storage object for accessing third-party servers to containers on OpenStack and S3 protocols.

The built-in hypervisor allows you to allocate hardware resources with redundancy, that is, each guest you can present all the physical CPU cores and all the memory, but when you start the guest you may be denied if more virtual resources are used than is available. But especially interesting is the function of passing thru the video card to the virtual machine. Why might I need to do this in the NAS?

If you look at the new SOHO/SMB/Enterprise class QNAP servers, many of them support GPU installation, and have additional power for powerful graphics cards that are planned to be used for applications related to artificial intelligence and machine learning. On the entry-level models can be quite a deal with experimentation and development, and the top models - to use in combat. It is already quite obvious that GPU in NAS is a new trend that can only be welcomed, because in addition to AI and ML applications, now you can create a full-fledged virtual workstation with 3D graphics and sound on the file server, which we will do now.

Our test QNAP TDS-16489U was released 2 years ago, and does not have power cables for the graphics card, so in terms of the voracity of the GPU, we are limited to 75 Watts, which can give the slot PCI-E 16x. NVIDIA’s professional series of Quadro graphics cards has the P2000 model, which does not require additional power, has 4 ports for monitors and drivers optimized for professional applications, including work in virtual environments. The technical ability to install a video card in the NAS and display the image on the video wall of 4 monitors looks very tempting, especially if you realize that you have no restrictions on performance or file download speed: on the left - 32 Xeon cores (16 physical and 16 logical), on the right - 2-level storage of 180 TB with SSD cache, on top - 256 GB of RAM…

Specialists working with 3D graphics and VDI repeatedly dissuaded me from the idea of setting up a workplace for CAD using the RDP Protocol. All as one agreed that it will be buggy and slow down, regardless of the network, and if you use a video card for such purposes, then immediately display the image on the screen via Displayport locally or IP-extension. We didn’t have a high-quality Video-over-IP extender at hand, but we did have a flagship KVM switch with IP access, Aten KN2124VA, designed to monitor and maintain a large fleet of servers in the data center. This IP-KVM provides direct access to the console from a browser window or through its own software, and can be connected to video cards with VGA, DVI, HDMI and DisplayPort outputs, and the device itself can work in both Gigabit and 100-megabit networks, which uses special algorithms for compressing the media stream. Console modules are connected to IP KVM directly by CAT5 cable, do not require installation of software on the server side and are powered by USB. Assuming the video-over-IP compression algorithms would kill all the picture quality in 3D, I thought we wouldn’t need this device, but I was wrong…

Aten_gallery_0

Everything went well - in one click we create a virtual machine, select the pool on which its files are stored, connect ISO Windows 10 x64 from the image and install the operating system in 5 minutes, allocating 8 CPU cores and 16 GB of memory. We launch Windows 10 using the built-in VNC console, install a package of virtual tools, a demo version of 3D Studio Max 19, the latest version of SketchUp, check the availability of the machine through RDP: everything works except SketchUp, which swears at the lack of a 3D accelerator, and now we will give it the whole Quadro P2000.
In QNAP QTS, the GPU assignment of the Board is configured in the HD Station application. You can give the video card to the built-in programs of the operating system, for example, for transcoding video streams, and you can choose the Passthrough mode for the virtualization station. Select the second and in the settings of the guest OS add our Quadro P2000 tab “GPU” on the principle of one video card to one VM. Run - not loaded. The built-in VNC console of the hypervisor does not work with the hardware GPU, and remote access via RDP does not connect at all, so it’s time to connect via IP KVM Aten, and lo and behold - it worked! Windows 10 has booted up, drivers have been installed and the NVIDIA logo has habitually spun in the remote desktop window, which means 3D support. Perhaps the video card needed to see the monitor connection to activate at boot.

SketchUp editor started almost instantly, feeling the presence of hardware 3D video core, but 3D Studio Max, which worked perfectly with a virtual VGA driver, refused to start. After downloading a large SketchUp project and playing with objects, I came to the following conclusions: a virtual workstation for working with graphics on the NAS is not fantastic, the QNAP QTS 4.3 operating system supports the entire range of professional and gaming NVIDIA graphics cards, and 3D acceleration works both on direct connection and via remote access protocols. But the example with 3D Studio Max still says that not everything is as smooth as we are described by marketers, and even QNAP itself draws attention to the fact that the graphics cards in their NAS are installed primarily for computing on the GPU.

And here, by the way, full expanse. First, the container hypervisor has built-in templates for artificial intelligence frameworks. Second, there is a separate QuAI application to manage various machine learning models on the GPU (supported by Caffe, CNTK, MXNet, and Tensorflow) that runs on top of the container hypervisor and can easily monitor the load on the graphics card. However, on our TDS-16489U this app was unavailable.

Interestingly, when creating a new path, it automatically enabled snapshots, and some configuration changes, the hypervisor itself creates a snapshot. Snapshots can be configured on a schedule and viewed in a convenient directory.there is, incidentally, full expanse. First, the container hypervisor has built-in templates for artificial intelligence frameworks. Second, there is a separate QuAI application to manage various machine learning models on the GPU (supported by Caffe, CNTK, MXNet, and Tensorflow) that runs on top of the container hypervisor and can easily monitor the load on the graphics card. However, on our TDS-16489U this app was unavailable.

Virtual switches allow not only to configure the topology of the network, rigidly tying the network port to the guest machine, but also to combine ports before forwarding, as well as to include the Protocol of protection against the formation of loops, STP. Note that Qnap Virtual Switch does not have a dedicated network port for management, and if you are keen on distributing IP addresses of the internal subnet to virtualka, you can cut off access to the Web-interface of the entire NAS. The only thing lacking in such an abundance of virtualization is the ability to fully migrate virtual machines from one storage to another. This feature does not exist at all, and you can only migrate a guest OS by cloning a powered-off virtual machine and then importing a copy.

By the way, in terms of importing foreign VMS everything is more than pleasant: VMX, OVA, OVF and QNAP formats are supported. We tested the migration of Ubuntu 18.04, Debian 9 LTS and Windows Server 2016 from VSphere 6.0 to QNAP Virtual Station - everything went smoothly: the copy is deployed in a new directory, so you can import the same VM multiple times.

In General, all the software functions of QNAP servers in one article will not consider, so let’s continue to get acquainted with the NAS itself.

The design of the QNAP TS-16489U

TDS-16489U series consists of 4 models with indices SA1, SA2, SB1 and SB2, different processor models and RAM volumes. Our top TS 16489 USB3 has two Xeon E5-2630 (2.4 GHz, 8 cores, 16 threads and 256 MB of DDR4 memory EC). Structurally, the NAS is made according to the 1-controller scheme typical for servers on Intel Xeon and assembled in a 3U high case, which allowed to place 16 bays for 3.5-inch hard drives with SATA-600 / SAS-12G/s interfaces on the front panel.

The storage on a separate Board installed as much as 3 controller SAS 12 Gb/s production LSI, so no expanders are not used - all storage devices are directly connected to the controllers and give the maximum speed.

At the very beginning of the article I pointed out that for SSD-caching in TDS-16489U there are special 2.5-inch compartments, which are accessed from the back side of the server. Here it is possible to install drives up to 15 mm, but when installing such hot SSDs as Hitachi Ultrastar SS200, it is useful to turn on the NAS ventilation at maximum, because at room temperature in idle mode, these SSDs easily heat up above 55 degrees, and already at 66 degrees begin to trigger overheating warnings.

The QNAP TS-16489 U motherboard can be completely removed from the server without removing the storage from the server rack, but servicing the same fans requires removing the top cover of the case. The motherboard has two slots for M2 Gen2 x4 cards, which together with the battery module can be used to store the write cache in case of power failures. The TDS-16489U itself has such a large amount of RAM that with a sharp power outage, the entire cache may not have time to write to one M2 module, so for such cases they are installed in pairs and never changed. On sale the battery pack is not found, so these bays are empty.

The storage has a pair of Gigabit Ethernet RJ45 ports and 4 SFP+ 10Gbps Ethernet ports. The latter are implemented on the controller Intel XL710, which does not support iWARP/RDMA, so if you want to put a 40-Gigabit interface - you will have another reason to do this, especially since the system is natively supported by the 3rd generation of network cards Mellanox ConnectX3 at speeds up to 56 GB / s.

But we must bear in mind that the machine has only one PCI Express 16x slot, and the other three are PCI Express 8x. Interface cards 10/40G Ethernet usually use PCI-E 8x slot, NVMe cards - 4x or 8x, and only rare cards on 4 NVMe Ultra M2 slots have PCI-e 16x interface. So in simple terms, the possibilities of expansion are above the roof. The NAS has a built-in IPMI Manager, under which a separate network port is output. With a great Web-based interface and open access via SSH, it is difficult to simulate a situation in which you have to run the IPMI console. But if you have such situations, you may also need the QRM+ package, your own QNAP application for monitoring Supermicro servers in your network.

The motherboard has 16 DIMM DDR4 ECC slots, 8 of which are already filled with 32-Gigabyte modules, and the maximum amount of RAM can be 1 TB. Do not think that this is only required for Big Data, because the server constantly caches information from disk pools in RAM for faster access.

Ecology and environment

For power storage is responsible fault-tolerant power supply 650 W with two hot-swap modules. The typical power consumption of a device with hard drives should not exceed 360W, but when using expansion cards, the power can reach 400W.

Power_consumption

The NAS itself has a very deep body - as much as 530 mm, it is deeper than many industrial servers. The noise level is up to 68 dB, and it is better to install the server in a full data Center with a cold corridor and powerful ventilation. This machine is not intended for installation in the same space as the working personnel.

Replication and fault tolerance

For fault tolerance, the algorithm Doulbe-Take Availability is used, which performs real-time incremental copying of data from the main server to the backup. Not only files on shared folders are replicated, but also virtual machines with all settings, as well as snapshots of volumes and LUNs. One of the interesting features of this technology is to reserve a physical server in its own virtual environment, so that in case of failure of any old Dell or HP machine, while you are looking for spare parts, your server will continue to work in the QNAP hypervisor.

There are a lot of data backup tools here: various tools for working with backups (Acronis True Image, Archiware P5), clients for cloud storage (Azure Storage, Google Cloud Storage, Hicloud S3), synchronization between several NAS-s and other things like backup to an external JBOD, which can then be put in the trunk and taken to another city if you need to quickly transport dozens of terabytes of information. It does not make sense to list all the possibilities of backup - in the “backup” section in QNAP’s “marketplace” more than 30 plugins - reserve for your health.

Horizontal and vertical scaling

QNAP TS-16489 U allows you to connect up to 8 disk shelves with 12 or 16 compartments for 3.5-inch drives (total - 144 HDD with a total capacity of 1152 TB). The connection is made via interfaces SAS-6Gbps or SAS 12Gbps, what head unit have to have a dual-port interface card QNAP SAS-12G2E.

Also among the accessories there are 10-Gigabit 2-port network cards with SFP+ slots and RJ45 ports, 2-port 40-Gigabit network card with QSFP+ slots and memory modules up to 64 GB. Replacement of processors by the manufacturer is not declared.

Load balancing between head units and horizontal scaling are not declared by the manufacturer.

Warranty and support

The standard warranty for QNAP TDS-16489U is 3 years and is provided by authorized service centers throughout Russia. The manufacturer does not currently offer extended warranty packages with a fixed response time.

Licenses and additional packages

All the considered functionality of the storage was provided by the built-in functions of the operating system QTS 4.3 and plugins from the official QNAP repository and do not require the purchase and activation of any licenses. The features of Tiering and SSD Cache, for which some manufacturers ask for money, are available here for free.

Compatibility with third-party equipment

QNAP has two lists - white and black for hard drives and SSD drives. We recommend you to study these lists very carefully before buying the device and drives. 12-terabyte HGST HE12 hard drives with SAS interface had to participate in our testing. In the QNAP compatibility list, such hard drives with SATA interface are recognized as compatible, and 6 and 10-terabyte SAS from the same series - in the black list. Ours was neither there nor there, and they normally didn’t earn, causing delays in RAID 5 up to 20 seconds. I had to use 2-terabyte WD Red, which of course is a shame.

Just keep in mind that with the version of QTS 4.3, QNAP storage refused to support PCI Experss card NVMe third-party - they can be used for SSD cache, but to create new volumes will not work. This is some strange policy: what was compatible before, suddenly stops working.

Real-world patterns

Test package allows VDbench to run patterns, removed programs I/O treysing with real problems. Roughly speaking, the special software records how the application, whether it is a database or something else, works with the file system: the percentage of writing and reading with a different combination of random and sequential operations and different size of the block of writing and reading.

We used the patterns taken by Pure Storage specialists for four cases: VSA (Virtual Storage Infrastructure), VDI (Virtual Desktop Infrastructure), SQL and Oracle Database. The test was conducted at 16 threads for each virtual machine, which created a query depth of approximately 64.

Diagram-4

Surprisingly, the storage produces a large delay in the patterns, but the performance values are still very high.

Purchase price and class

QNAP TDS-16489U comes under the order with the price calculation under the final customer. According to our information, the price of a device without disks is in the range of 20 000 - 25 000 USD. Since we do not have the exact value of the cost, it is not possible to calculate the economic indicators.

Conclusions

QNAP TDS-16489U is no longer a class of storage systems, but a ready-made device such as “server+storage” in one device, which you can put into the big Data processing environment, and configure the operation of the main applications locally. This makes it possible not only to significantly save on the network infrastructure of the data Center, but also to facilitate the management and updating of software.

QTier technology, which is available to the customer for free - this is really the moment to focus on when buying, and if possible to choose an SSD with such a volume that they completely fit the most popular LUNs, because this algorithm works best when you create new logical cuts on the QTier-array and they are immediately placed on the Performance-layer. Testing of modern SSD from Hitachi showed that the TDS-16489U disk controllers do not reveal the full potential of modern solid state drives, so when configured in order, it is necessary to give preference to the volume of SSD, not their performance.

In the field of virtualization, the focus on machine learning applications and GPU support is commendable, as well as work on your own object storage server, which, in principle, today allows you to completely abandon the third-party cloud, access to which, as practice shows, can be blocked at any time.

And, of course, it is worth noting that the Tiering, and virtualization, and GPU forwarding to the guest operating system, and even SSD caching - this is part of the overall concept of QTS software, which means all this is available in simpler devices, including the SmB/SOHO class.