Yesterday, I tried to restart our DPM server and got the following error message:
“999DPM01V001 failed to start… The requested operation could not be completed due to a file system limitation (0x80070299)”.
Now, before I continue with the bad news, let me quickly describe our DPM environment:
We are running Data Protection Manager 2012 R2 within a Virtual Machine. This VM runs on an Hyper-V 2012R2 server. This physical server hosts all the VHDX (all 24 of them) for the DPM server, and of course dedup is enabled to save previous space on all the volumes that contains those VHDX.
After digging a little more in the problem, this message directly relate to the inability of NTFS to handle a lot of fragment within each file.
An NTFS volume has a fix allocated size were it stores the information about each fragment for each file, and unfortunately, once this allocation is full, the file becomes unusable until you defrag it.
What does that have to do with dedup? Well, the dedup process effectively creates fragmentation on your volume, so the same principle applies, once this fragmentation allocation for a file is full, you can’t start the VM that uses that VHDX.
The solution? This is were the bad news I spoke about early comes into play:
One of the solution is to increase the size of the “Bytes per filerecord segment” (that’s the space allocation) from the default 1024 to 4096 (you can find out what the size of this table is for you volume by running “fsutil fsinfo ntfsinfo <volume:>”), and the only way to do this is to, well, format the volume with the /L option (format <volume:> /L)!
Another solution is to move the affected file to another volume so it gets “undeduped” and the fragment disappear, but you’ll still very likely to incur the same problem further down the track with your other files anyway. So a more permanent solution is to go back to formatting.
I hope you you read this post before the problem hits you, so you can move those VHDX and reformat the volume while you still have enough space.
The partially good news is that Microsoft has fixed this problem with ReFS, but unfortunately, this is only available on Windows Server 2016.