Triple M.2 RAID setup

Hey there,

New user here and I'm trying to understand the RAID arrays using PCI-e lanes via the CPU and Chipset.

On my x99, I run two 960 pros at RAID 0 using windows software. Obviously I'm using the CPU PCI-e lanes and I can get seq reads of 7000 MB/s.

I have seen folks using the z170/z270 motherboards and creating RAID 0 arrays using the Chipset lanes, but they are only getting seq reads of 3500 MB/s.

Why is this?  Are the CPU lanes able to 'double up' while the Chipset lanes cannot?

I ask because some of the motherboards I'm looking at disable certain lanes when using a 28 lane CPU and thus I may only be able to use Chipset lanes to create a RAID 0 drive.  If this is the case, then I'm guessing it would be SLOWER than the RAID 0 drive I currently have because i'm using CPU PCI-e lanes instead of Chipset lanes. 

I agree, it would be a great thing if you could use a CPU lane/Chipset lane m.2 to create a RAID drive across both.  That would solve a lot of issues with a 28 lane CPU.  Hell for me it wouldn't have to even be bootable just a non-os drive!

Am I correct in this?  I would love to do a triple m.2 RAID but it sounds like it would be slower as a boot drive since you have to go through the chipset than a non-os drive and using the CPU lanes.

Finally concerning VROC, I don't think that even WITH a key you'll be able to use the 960 EVO.  From what I understand VROC is only for Intel drives UNLESS you are using a Xeon processor.  The key is only if you want to do RAID 1 and another key for RAID 5/10.  It really does suck that Intel won't allow non-intel drives to use VROC unless you have a XEON. :(


Edited by Brighttail - 15 Aug 2017 at 7:08am

Assuming the other two are on CPU lanes that should work. If you don't want to boot from the CPU raid array you can actually make that configuration today with RST from inside Windows if I'm not mistaken.

Edited by eric_x - 22 Jul 2017 at 8:15pm

marko.gotovac wrote: @Eric, new registered user here. I'm relatively new to PC building so would greatly appreciate some clarification as to some questions I have: 1a. From my research, speaking about the 10 core 7900x, this CPU provided 44 lanes. These lanes are all spread out across the 5 x16 slots, as seen in my Gigabyte x299 Gaming 7 mobo manual: https://scontent.fybz1-1.fna.fbcdn.net/v/t1.0-9/20155742_10155549067923669_8406735989774525926_n.jpg?oh=6fd6a1bbc0776c62b6c8e9c99139e9c7&oe=59F756C0 Is this correct?  (16 + 4 + 16 + 4 + 8) = 48? Isn't it supposed to equal 44? 1b. The chipset has 24 lanes. Correct? 1c. These chipset lanes are responsible for driving different components on the chipset such as the Ethernet connection, 8 SATA connections and the THREE M.2 drives? 1d. If so, how many lanes of the 24 are dedicated to the 3 m.2 drives? From my understanding, in order to be able to not bottleneck the potential bandwidth throughput of an M.2 drive, it requires at least 4 PCI-e lanes. Is that correct? I've attached a photo of a page from the mobo describing what happens when utilizing the 3 m.2 slots: https://scontent.fybz1-1.fna.fbcdn.net/v/t1.0-9/20245516_10155549070448669_8167031257432050270_n.jpg?oh=2829db001b070f9e09450d24a118d916&oe=59F315A6 1e. I've got my hands on two 960 EVOs and was hoping to get a RAID 0 going. Will I be bottlenecked by a lack of chipset PCI-e lanes? What is the maximum number of M.2 drives I can install in these three m.2 slots before I begin to experience a degradation in performance? And if there is some degradation, is my only alternative to buy a M.2 to PCI-e adapter card and plug it into one of the x4 slots being fed by the CPU lanes? I apologize if these questions have been answered again and again, but I've been going from forum to forum, and all of these threads involving x299 is making my brain hurt.  Any help would be appreciated! Marko from Canada

First understand that the way a mother board manufacture allocates the various resources available on a platform can differ between manufactures. So what ASRock does may not be the same as what Gigabyte does on their X299 boards.

As the second picture shows, there is a sharing of the chipset resources (PCIe lanes) between the M.2 slots and the Intel SATA III ports. That is normal for the last several Intel chipsets. You may use either a specific M.2 slot, or the SATA III ports that are shared with that M.2 slot, but not both at the same time.

You won't be bottle necked using two 960 EVOs in two M.2 slots, or even three. All will be PCIe 3.0 x4, as long as that many PCIe 3.0 lanes have been allocated to each M.2 slot. Given the sharing of resources situtation, you have either full bandwidth on the M.2 slots and lose the ability to use the shared SATA III ports, or vice versa.

But we also apparently have PCIe 3.0 lanes from the X299 chipset allocated to one of the PCIe slots on your Gigabyte board. That would account for the 48 vs 44 lane count. So you will need to further verify which PCIe slot uses the X299 chipsets's PCIe 3.0 lanes, and how that affects things like the M.2 slots, IF it does so. Normally the PCIe lanes used by a slot are only actually used when a card is inserted in the slot.

Interesting about VROC and the three M.2 slots working together. I assume anything with VROC would use the RSTe driver. The VROC acronym (Virtual RAID On CPU) contradicts itself if the two M.2 slots that must be connected to the X299 chipset, can be used with VROC. Using the M.2 slots does not use up any of the CPU's PCIe lanes, so if it is possible, it is more magic from Intel, as the PCIe Remapping in a sense is.

If you could not use IRST and RSTe at the same time, they you could not even have SATA drives in RAID 0, which would not be good. It will be interesting to see how IRST and RSTe RAID drivers coexist.

I forgot to explain something you asked about earlier, why your third 960 EVO does not appear in the IRST Windows program as a detected drive. That is normal for the IRST RAID driver and the IRST Windows program on the Z170 and Z270 platforms. Only NMVe SSDs that have been "remapped" with the PCIe Remapping options appear in the IRST Windows program. Since the M2_1 slot (right) does not have a Remapping option, an NVMe SSD connected to it will not be shown in the IRST Windows program and UEFI IRST utility.

We are so accustomed to SATA drives being "detected" in the UEFI/BIOS that we expect NVMe SSDs to behave the same way. NVMe is unrelated to SATA, and their controller is not in the board's chipset as the SATA controller is, it is in the NVMe SSD itself. On X99 boards, the only place NVMe SSDs are shown is in the System Browser tool, and in an added NVMe Configuration screen (if it is available), that simply shows the NVMe SSD listed. A more extreme case is my Ryzen X370 board, with no NVMe RAID support. It has an M.2 slot, and will list an NVMe SSD in the slot. But if I use an M.2 to PCIe slot adapter card for the NVMe SSD, as I have done as a test, it is shown nowhere in the UEFI except the Boot Order. That 960 EVO works fine as the OS drive in the adapter card, using that PC right now. The Windows 10 installation program recognized it fine.

I can give you some information about this spec:

Please note that if you install CPU with 44 lanes or 28 lanes, RSTe does not support PCH PCIe NVME and VROC (Intel® Virtual RAID on CPU) is supported.

Socket 2066/X HEDT processors are actually from two Intel CPU generations, Skylake and Kaby Lake.

The Kaby Lake 2066 X processors are the i5-7640X and i7-7740X, both only provide 16 PCIe 3.0 lanes.

The Skylake 2066 X processors are the i7-7800X, i7-7820X, and i9-7900X. Both i7s have 28 PCIe 3.0 lanes, and the i9 has 44 PCIe 3.0 lanes.

So that spec is saying, as far as I can tell, that the Skylake 2066 X processors do not support using the RSTe RAID (driver) with the PCH (X299 chipset) NMVe interface, the two (or three?) M.2 slots.

But the Skylake 2066 X processors support using the RSTe RAID (driver) with VROC.

We basically know that we need to use the Intel IRST RAID driver for the PCH NVMe interface (M.2) RAID, which this spec states in a round about way. As I thought, it seems the RSTe RAID driver is only used with VROC. VROC does use the PCIe slots for drives in RAID. I hope all the M.2 slots are usable with VROC, but I'm a bit skeptical about that.

What extra RAID support (RSTe, VROC) the two Kaby Lake 2066 X processors have, if any, I don't know. Probably none, since they are cheaper processors. Only IRST RAID with the X299 chipset's two M.2 slots can be used with the Kaby Lake processors, no VROC. Since VROC will use the processor's PCIe lanes, and the Kaby Lake processors only have 16 PCIe lanes, which in the past have always been allocated per slot as 16 in one slot, or 8 and 8 in two slots, how can you use a video card if all the PCIe lanes are in use in two slots? Actually, the 16 PCIe lane processor spec for your board is one slot at x8, and the other at x4, if that is correct.

Intel already has NVMe SSDs that use a PCIe slot interface, the consumer 750 model, and their many Data Center DC series professional SSDs.

As you can see, I am a PC storage drive enthusiast, so all this interests me. I'd enjoy having an X299 board mainly to play with the storage interfaces, with the appropriate CPU of course.

Thanks, I didn't see that. I think I will try the Windows 10 image backup in case anything happens, I already accidentally erased the array once in testing. I did notice a bug in the UEFI where if you were in the VROC control and pressed escape it would bring you to a dialog with no text so maybe they fixed that with this update.