Triple M.2 RAID setup

I went a littler further down the VROC rabbit hole. I have a PCIe m.2 adapter: http://www.amazon.com/gp/product/B01N78XZCH/ref=oh_aui_detailpage_o00_s00%3Fie=UTF8&psc=1

Edited by eric_x - 16 Jul 2017 at 5:19am

If you use VROC, do you have the VROC key that costs ~$100? You must have it for third party NVMe support. Also isn't VROC only usable with a Skyake X processor? That is what the confusing description in the manual is trying to tell you.

Don't forget that all the restrictions, details, and requirements for VROC, etc, are all created by Intel only. Mother board manufactures only pass this stuff on, they don't create it.

Also, you cannot boot from non-Intel SSDs RAID array using VROC, that is at least what I have read many times. That seems ridiculous, but Intel considers X299 to be related to their professional server equipment, where you must pay for added features like this. PC builders won't accept this, and Intel has never done this in the past with their HEDT platforms. A big mistake by Intel this time.

Still also, using VROC means installing the RSTe enterprise RAID driver (Intel Rapid Storage Technology enterprise driver and utility ver:5.1.0.1099) as far as I know. See that Alternate ID option is Storage Configuration? That is used to switch between IRST and RSTe, or at least must be enabled to get the RSTe driver active. Can both IRST and RSTe be active at the same time? I don't know, I don't have an x299 board. If you don't have it installed, that may be why Windows does not see the 960 EVO in your adapter. But BEWARE, installing it might make your IRST RAID 0 array unusable.

I can tell you now you won't get a bootable, three 960 EVO array using the adapter and the IRST driver with the other two SSDs in the M.2 slots, sorry to say.

Maybe this article about VROC will answer some questions, and show you a new adapter card:

http://www.pcworld.com/article/3199104/storage/intels-core-i9-and-x299-enable-crazy-raid-configurations-for-a-price.html

Let me guess, you are using IRST version 15.7, from your board's download page? A reasonable thing to do, but...

Forum member clubfoot told me about the problem with Crystal DiskMark's write test when using IRST 15.7, and when I tried it, I also had the same issue. We both use IRST 15.5 now, which does not have that bug. I had issues in Windows using IRST 15.7, which disappeared after changing to IRST 15.5. It will take Intel a while, but I predict IRST 15.7 will disappear from Intel's download page.

The large file, sequential read speeds you can get with NMVe SSDs in RAID 0 are nice, but the concurrent loss of small file, 4K read speed, a given with RAID, is a tradeoff I don't like.

Nothing can be done about ATTO, it is stuck where it is until it gets new options and coding to deal with NVMe SSDs. Plus ATTO uses all compressible data, the easiest type to use for a benchmark. ATTO results are the best case situation results, that don't reflect real data.

Glad you figured the M.2 port thing, I can only guess about the X299 platform at this time. Plus some things I assumed would be inherited or used in the X299 chipset that exists in the Z170 and Z270 chipsets, did not happen. That is in particular, the three M.2 slots connected to the chipset, that then can use IRST RAID. That might be a case of deciding which resources to use for various things. On the Z170 and Z270, we loose two SATA ports for each M.2 slot in use, which does not happen with X299. Use all three M.2 slots on the 'Z' chipsets, and no Intel SATA ports. Or perhaps X299 does not have the same flexibility, or is considered worthless in a professional level chipset by Intel.

I hope that non-Intel NVMe SSDs in RSTe RAID are bootable via VROC, but I believe that was the big complaint about X299, that is was not possible. Possibly the Pro level VROC key will allow it.

Initially, the older X79 chipset only used RSTe RAID, and its benchmark results were not as good as IRST. Later Intel added IRST support to X79 and UEFI/BIOS updates allowed its use. It remains to be seen how the new RSTe works out. At least both types are supported from the start.

Speaking of my crazy Optane x3 SSD RAID 0 array (clubfoot ), which Intel apparently said could not be done, but works fine, surprised me with this:



Yes the write speeds are nothing amazing for an NVMe SSD, but this is with all of six single layer Optane chips, two per SSD, with each Optane SSD using a DMI3 (PCIe 3.0) x2 interface. The 4K random read speed is beyond anything else out there. I'm still not sure if in actual use it is better than my 960 EVO x2 RAID 0 array, since the slower write speed should offset any better read speed when doing tasks that both read and write.

clubfoot, did you ever get Sequential Q32T4 read speeds at that level with your triple Samsung NVMe RAID 0 array? It's been a while, I don't remember.

Sorry I seemed to confuse you about DMI3 and PCIe 3.0 in my last post. It's simple actually, and I made a mistake, or perhaps Intel sorted out their terminology in their latest document.

The Intel 100 and 200 series chipsets, like the Z170 and Z270, have what Intel calls their DMI3 interface. As shown in the picture you posted, that is the interface between the chipset and the processor. They are data channels or lanes that are really PCIe 3.0 lanes, but Intel sometimes calls them DMI3 (Direct Media Interface) when used for a specific purpose, as communication between the chipset and processor.

Most of the 100 and 200 series chipsets, like the Z170 and Z270, have PCIe 3.0 lanes, which in the past was only PCIe 2.0. I was calling the PCIe 3.0 lanes in the chipset DMI3 lanes, but in reality they are equivalent, and I should be calling the resources used by the M.2 slots on these boards from the chipset, PCIe 3.0 lanes. The M.2 slots on your and my board are connected to the chipset's PCIe 3.0 lanes.

The processor has its own PCIe 3.0 lanes, 16 for Desktop processors, and normally 28 to 44 for HEDT processors. The new Skylake X processors changed that to only 16 for that model only. The X299 chipset also has PCIe 3.0 lanes.

So the processor and chipset both have PCIe 3.0 lanes. But the Intel IRST software only works with the PCIe 3.0 lanes from the chipset. The IRST software cannot control the processor's PCIe 3.0 lanes. That's why we cannot use the processor's PCIe 3.0 lanes with the chipsets PCIe 3.0 lanes, to create a RAID array with the IRST software.

Before the 100 and 200 series chipsets existed, such as on the Intel X99 platform, M.2 slots were connected to the processor's PCIe 3.0 lanes. We could create a bootable, single NVMe SSD volume on that M.2 slot, but there was no RAID capability, because it did not use the chipset's PCIe lanes, which are only PCIe 2.0 on X99 anyway. We could also connect a single NVMe SSD to the processor's PCIe 3.0 lanes on Z97, Z87, and even Z77 boards, using an M.2 to PCIe slot adapter. All an NVMe SSD like the 960 series needs is four PCIe 3.0 lanes. But again no RAID support, and the UEFI/BIOS must include NVMe support to allow booting from the NVMe SSD. Most older chipset boards do not have NVMe booting support.

VROC is a new feature only on X299, that is adding a RAID capability to the processor's PCIe 3.0 lanes, using the RSTe RAID software. Note that is not the IRST software, and RSTe and IRST cannot be combined or work together.

In case you haven't seen this, here is the Intel 200 series chipset document, including X299. According to the table on page 21, the X299 chipset could have up to three "Total controllers for Intel RST for PCIe Storage Devices". That means three M.2 slots connected to the chipset PCIe 3.0 lanes for IRST RAID. The Z270 chipset has the same spec (page 20). (NVMe SSDs have their own NVMe controller chip built into the SSD. SATA drives use the SATA controller included in the board's chipset, or add on SATA controllers like ASMedia and Marvell.) Why some (all?) X299 boards are limited to two NVMe controllers, I don't know. That may simply be a choice in the use of available resources, allowing more SATA III ports on X299, that are not available on Z270.

https://www.intel.com/content/www/us/en/chipsets/200-series-chipset-pch-datasheet-vol-1.html?wapkw=200+series+chipset

On to my Optane RAID 0, which is three 32GB Optane SSDs, that each use a PCIe 3.0 x2 interface. Note that is only x2, instead of x4 used by a 960 EVO or Pro, and most other NVMe SSDs. If you click on a single SSD in the IRST utility, you can see this:



If you checked your 960 EVOs like that, you would see a PCIe link speed of 4000 MB/s, with a PCIe link width of x4. Again, that is for each single SSD.

That bandwidth is the theoretical maximum, that does not take into account overhead, latency, etc. So in theory I should only be getting a max theoretical 6000 MB/s bandwidth, right? Three, 2000 MB/s interfaces. So how am I getting 8GB/s+?

Note that the 8808 MB/s is the "Q32T4" test. That means Queue depth 32, with four threads. That means 32 IO requests in the memory IO queue, which was the maximum for SATA AHCI, but NVMe does many more than that. So we have 32 multiple IO requests running in four threads, which adds up. Also we have caching going on, where the data is written to DRAM memory, and fed to the benchmark program. Note that the simple Seq test, one IO request in one thread, is only 3435 MB/s. The same applies to the 4K Q32T4 and simple 4K tests. There is more to understanding these benchmarks than simply interpreting the numbers as the speed of a drive or RAID array.

Benchmark results with NVMe SSDs in RAID 0 have always been disappointing, given what we tend to expect the result to be. Users of single NVMe SSDs often complain about their benchmark results that don't match the specs. We don't know what the limiting factor is, the benchmark program, the IRST software that acts as the NVMe driver with these SSDs in RAID, or something else.

I was surprised by your comment that you thought the IRST 15.7 software was affecting your CPU over clock. Is that right?

I noticed today that ASRock has a new UEFI/BIOS version for your board. Or should I say the first update from the initial factory version. But please read all of this post before you update:

http://www.asrock.com/MB/Intel/Fatal1ty%20X299%20Professional%20Gaming%20i9/index.asp#BIOS

Of interest to you may be the update to the Intel RAID ROM, according the the description. What that will provide for your board remains to be seen, but I would suggest applying the new UEFI version.

I don't know about your experience using ASRock boards, but I strongly suggest using the Instant Flash update method, labeled "BIOS" on the download page. Instant Flash is in the Tools section of the UEFI, or at least should be.

This and all UEFI/BIOS updates will reset all of the UEFI options to their defaults (with some exceptions), and remove any saved UEFI profiles you may have.

Early on with the Z170 boards, users that had NVMe SSD IRST RAID arrays discovered that after a UEFI update, even if the PC just completed POST when NOT in RAID mode and without the PCIe Remapping options enabled, their RAID arrays would be corrupted and lost! That did not happen with SATA drive RAID arrays (no PCIe Remapping settings), so no one expected that behavior.

I don't know who made the mistake here, Intel or ASRock, but it was not a good thing for NVMe SSD RAID users. Of course that was the first time IRST RAID worked with NVMe SSDs, with the then new IRST 14 version. At first we would remove our NVMe SSDs after a UEFI update, reset to RAID and Remapping, and install the SSDs again.

ASRock now does NOT reset the "SATA" mode from RAID to AHCI or reset the PCIe Remapping options during a UEFI update in their newer UEFI versions, after learning about this issue. I imagine they remembered to do that with your board, but just an FYI.