2

We use a StarWind SAN which has the concept of thin-provisioned disks that grow as needed. You can allocate a 4TB drive but it starts off tiny and grows as blocks are written to the virtual disk (via iSCSI).

The virtual disk used for our main file system has grown to 1.5TB and has plenty of virtual space left (2.5TB) but disk space on the SAN is another matter - it's getting a bit tight. That's the downside of thin provisioning - you can overcommit disk space.

So we're busy archiving old folders off the main disk system to the archive area.

However, this will only make any difference if Windows 2008 re-uses the deleted blocks before it adds new blocks to the disk when new files are added.

Is this the case or are we wasting time archiving (aside from the benefit of keeping things tidy) and need to consider expanding the SAN disk soon?

Basil
  • 8,811
  • 3
  • 37
  • 73
Rob Nicholson
  • 1,678
  • 8
  • 27
  • 53

2 Answers2

4

What you're looking for is actually called "thin reclamation", a process where the server OS tells the underlying storage when it has unmapped a block, even if it's not been zeroed out. Windows 2008 can be configured to act this way with some vendors, however (at least the last time I read about someone testing starwind) not yours.

I believe the decision of where to write new data to disk is not configurable in Windows. I'm pretty sure that it's complex enough that you wouldn't want to configure it lightly, though. In any case, you have to assume that Windows will write where it wants regardless of what the back-end storage is up to.

If you do delete a large quantity of files, consider migrating them to another thin provisioned LUN. It's a terrible waste of time if you're going to be doing this frequently, but it'll buy you enough time to grow your storage hardware or decide on a more permanent solution.

Basil
  • 8,811
  • 3
  • 37
  • 73
  • I can see the advantages of "thin reclamation" in allowing the SAN to unmap the associated block in the virtual disk thus effectively shrinking the virtual disk in real time. But I would also have thought it was more efficient for Windows to re-use blocks ASAP as wouldn't it generally keep the data closer together and therefore reduce head movement? – Rob Nicholson May 02 '12 at 14:21
  • Windows's decision of where to put new blocks is based on an algorithm designed to optimize the layout, including minimizing the number of fragments it has to create. Where it chooses to place new data is not configurable that I've heard, and I don't believe that it prioritizes previously used blocks at all. In fact, since they're more likely to be part of smaller contiguous free space blocks, it might be more prone to using virgin sectors, which will tend to have a larger amount of contiguous free space available. – Basil May 02 '12 at 16:37
  • What it boils down to is that the algorithm windows uses to decide where to write was developed before thin provisioned SAN volumes. Now that we have this new problem, windows is focusing their efforts on integrating the free block reclaim technology that will allow any storage device to be made aware of when sectors are actually unallocated. – Basil May 02 '12 at 16:43
  • @RobNicholson I don't know much about Windows disk internals, but I *believe* Basil is correct: Windows uses "Keep files contiguous if possible, use earliest contiguous block" as its allocation strategy. The problem with "reuse previously used sectors first"/"Fill earliest sectors", at least with rotating media, is that re-using sectors this way might keep your free space (relatively) unfragmented, but it means your files themselves will be scattered around the drive (so a sequential read of say a Word document might mean moving the head all over the place - bad for speed) – voretaq7 May 02 '12 at 16:44
  • It's worth noting that thin-provisioned SANs (and SSDs) obviously blow some of those assumptions out of the water. Filesystems have not really caught up to storage technology yet... – voretaq7 May 02 '12 at 16:45
  • I know that disk architectures are an art form in themselves. I can see the obvious benefits of keeping files contiguous but I'm perplexed how Windows knows how many blocks a program is about to write at the start to keep them together. Must do some more research – Rob Nicholson May 03 '12 at 19:01
  • 1
    It doesn't. It simply does its best to put files into large blocks of empty space, as far as I know. Anyway, ask your vendor what kind of "stay-thin" technology they have. If they have zero block detection, you might be able to find a way to write zeroes to the unallocated space and re-thin your LUN. Otherwise, have the OS copy it to a new LUN and delete the old one. – Basil May 03 '12 at 20:26
  • Basil - yes, cloning the virtual disk to another one is the sure fire way of shrinking it but, as with all drive migrations, that's not easy if you are also trying to maintain up time of a large file share – Rob Nicholson May 12 '12 at 08:33
  • Not much I can tell you then. Thin reclamation is tricky. Maybe ask starwind whether they do thin reclamation on zeroed out data. If so, you might be able to format your unused space. – Basil May 14 '12 at 13:54
0

Check with the vendor but you are likely wasting your time archiving.

The allocated space to the LUN will not shrink by virtue of deleting files in Windows. Note that the allocated space to the LUN is different than the size of the LUN itself. If you thin provision a 100GB LUN, and write 10GB of data to it, the SAN will allocate 10GB worth of raw disk blocks on its underlying disks to the LUN. Then, when Windows wants to write to a new block, this grows the % of space on your thin LUN that becomes allocated/provisioned. Over time, as Windows requests to write to pristine (never-touched) blocks, those blocks will be allocated by the SAN from its global pool of unused blocks, and the allocated/provisioned size of the LUN will increase further.

Eventually with enough data churn, a thin provisioned LUN will become thick provisioned. It may take a long time, but it depends entirely on the OS's behavior.

Without special software (that basil has mentioned), the SAN has no way of knowing which blocks can be reclaimed, as the SAN can't "see" NTFS (or any other filesystem) by itself. Additionally, most of the time you need to have this software running in Windows before the volume becomes thickly provisioned, but again check with the vendor.

In general, thin provisioning buys you time (you don't have to allocate all your storage at the get-go) but eventually you will need to back your volumes 100% with storage.

Note, my understanding is that Linux does prefer to overwrite blocks instead of using pristine ones, but I don't have a reference to back that up.

Jeremy
  • 938
  • 2
  • 7
  • 18
  • I'm not talking about shrinking the LUN - that I know is not possible with StarWind. I'm asking about the Windows methodology/algorithm for re-using the blocks that have already been allocated (and therefore extended the virtual disks) and then freed up by deleting files. Take an example of a 1TB disk onto which 500MB was written. The virtual disk ends up @ ~500MB. Delete half the files (size wise) and it stays at 500MB but internally 250MB is free. Write another 250MB of files and what happens? – Rob Nicholson May 12 '12 at 08:31
  • From the comments above, I infer that Windows will re-use as many of the free blocks as it's algorithm allows but the virtual disk will continue to grow as some new blocks will be added. So with the above example, write another 250MB of files (lots of smaller files) and the VD will grow in size but not by the full 250MB as it will overwrite previously allocated blocks – Rob Nicholson May 12 '12 at 08:38
  • >but eventually you will need to back your volumes 100% with storage. That's not quite true. With thin disks, you may as well give the OS partition 200GB but you are unlikely to ever use that amount. With data drives, yes it's different – Rob Nicholson May 12 '12 at 08:44
  • I edited my answer for clarity. In summary - due to the I/O profile of a data drive it may take much less time for a data drive to become thick provisioned, but *any* thin provisioned drive on a Windows box will eventually become thick provisioned as long as there are new writes going to the drive. It may take a really long time, but it *will* happen. The way to get around this is to run an agent in the OS, as has already been discussed. – Jeremy May 13 '12 at 17:32