0

Folks,

AWS exposes HDDs in their D3 instances (https://aws.amazon.com/ec2/instance-types/d3/) as nvme drives and these drives DON'T set the rotational flag as they're supposed to (https://lwn.net/Articles/408428/).

I have cut a ticket to AWS, but their response was that this was automatically set by the linux kernel and hence they can't do anything about it. Any ideas on how to fix this / how to programmatically identify drive types with 100% accuracy?

FWIW this breaks existing software like ceph that tries to detect drive types using this way. Right now the only option I have is manually fixing this. Any ideas are welcome!

Thanks. Subu

Subu Sankara Subramanian
  • 101
  • 2

1 Answers1

1

It should. And it sets it. But Linux sets this flag based on what hardware exposes. Linux sends an SCSI INQUIRY command during hardware initialization phase to each discovered device, to know what it is, and other identification commands, and sets those flags based on what devices reply. If it replies it's a rotational HDD, it sets rotational to 1.

AWS is a virtualization cloud, so you see not a physical hardware, but virtual. It's emulated I think with some version of QEMU. You can confirm this from within a VM, if you have seen any mention of virtio devices at various places (dmesg, lspci and so on). QEMU, in turn, really has an option to expose each virtual storage device to the VM either as "rotational" or as a "solid state".

So, Linux sets whatever AWS cloud made it to set. It is the cloud duty to run QEMU with proper options. So I'd consider the reply as a formal boilerplate or run-around reply, they just don't want to investigate the problem.

Nikita Kipriyanov
  • 8,033
  • 1
  • 21
  • 39
  • 1
    It is worth noting there's unlikely a single HDD behind what AWS provides, so it won't have the same performance profile as read hardware. So treating as non-rotational could be better. – rvs Feb 04 '22 at 11:32
  • This is a speculation. We just don't know and there is no way to know unless they tell us. – Nikita Kipriyanov Feb 04 '22 at 13:13
  • they don't say it directly but many things suggest that EBS works similar to things like Ceph RBD. Their FAQ directly say things like "EBS is a multi-tenant block storage service". Performance profile suggest that. And finally, as a person working on direct competitor to EBS I can say, there practically no way to build a product with similar price, durability and feature set without some sort of distributed network-attached storage underneath. – rvs Feb 04 '22 at 13:36
  • 1
    That said, I agree it's a speculation whether setting rotational to 1 is better or worse. This needs to be tested, and it may depend on specific kernel version, filesystem and workload in the guest. – rvs Feb 04 '22 at 13:37
  • So again, as I said, the answer "can't do anything with it" is a run-around. They can, but don't want to do anything. The best, as I think, is to make this configurable in the VM instance control panel. Also it is worth giving the recommendation to use `noop` I/O scheduler for disks (because the real scheduling is to be performed by the host). – Nikita Kipriyanov Feb 04 '22 at 14:01
  • So the interesting thing is - D3 instances provide 'local HDD'. These are not EBS instances. But I agree there is probably virtualization and drive sharing between various 'instances' they expose from the bare metal hardware. I will bring this back up again with AWS. – Subu Sankara Subramanian Feb 04 '22 at 16:34