Server Fault Stack Exchange
Server Fault Stack Exchange

Questions tagged [deduplication]

94 questions
24
votes
2 answers

How large is my ZFS dedupe table at the moment?

I have read a lot of information about planning RAM requirements forZFS deduplication. I've just upgraded my file server's RAM to support some very limited dedupe on ZFS zvols which I cannot use snapshots and clones on (as they're zvols formatted as…
Josh
  • 9,001
  • 27
  • 78
  • 124
13
votes
5 answers

Apparently Robocopy does not understand Windows Server 2016 deduplication. Is there a way to make it work without corrupting the volume?

Scenario: Doing a server migration from old Server 2008 R2 to new Server 2016, following this Server Fault guide: File server migration using Robocopy After Robocopy completes, enable deduplication on Server 2016 for the copied volume, and then use…
Dale Mahalko
  • 725
  • 1
  • 6
  • 16
13
votes
3 answers

Windows Server 2012 R2 Deduped 356GB to 1.32GB

I'm experimenting with deduplication on a Server 2012 R2 storage space. I let it run the first dedupe optimisation last night, and I was pleased to see that it claimed a 340GB reduction. However, I knew that this was too good to be true. On that…
Mark Henderson
  • 68,316
  • 31
  • 175
  • 255
12
votes
1 answer

Technical details for Server 2012 de-duplication feature

Now that Windows Server 2012 comes with de-duplication features for NTFS volumes I am having a hard time finding technical details about it. I can deduce from the TechNet documentation that the de-duplication action itself is an asynchronous process…
the-wabbit
  • 40,319
  • 13
  • 105
  • 169
11
votes
2 answers

ZFS - destroying deduplicated zvol or data set stalls the server. How to recover?

I'm using Nexentastor on a secondary storage server running on an HP ProLiant DL180 G6 with 12 Midline (7200 RPM) SAS drives. The system has an E5620 CPU and 8GB RAM. There is no ZIL or L2ARC device. Last week, I created a 750GB sparse zvol with…
ewwhite
  • 194,921
  • 91
  • 434
  • 799
11
votes
10 answers

Block-level deduplication on Linux

NetApp provides block-level deduplication (ASIS). Do you know any filesystem (even FUSE-based) on Linux (or OpenSolaris, *BSD) that provides the same functionnality ? (I'm not interested in false deduplication like hardlinks).
Benoît
  • 1,331
  • 3
  • 11
  • 23
9
votes
3 answers

Backup storage server with ZFS

I am IT everything man at a small company. I want to design a new infrastructure including a new server and a separate backup server with company wide backup policy. The most important thing in the company is the SQL Server and its databases. There…
Krystian Lieber
  • 190
  • 1
  • 4
8
votes
1 answer

NTFS Compression with Deduplication?

I have installed the Data Deduplication role on Windows Server 2016. Per Microsoft, it's using NTFS compression to help save space: Data deduplication stores more data in less physical space. It achieves greater storage efficiency than was possible…
Andrew Bucklin
  • 435
  • 1
  • 5
  • 12
8
votes
5 answers

What's the best way to copy deduplicated files onto a new Server 2012 drive?

We have a deduplicated volume on a Windows Server 2012 machine that is approaching it's limits. It is a 1.3TB drive with ~10TB of duplicated data. We want to copy all of this data onto a larger 4TB drive. What is the best way to perform this copy…
Screndib
  • 231
  • 1
  • 2
  • 8
8
votes
2 answers

What is "deduplication"?

I mean, I can look up the dictionary definition, but why is everyone suddenly talking about it in reference to virtual tape libraries? What's "new" here so that it's so much in the news lately?
lavinio
  • 238
  • 2
  • 7
7
votes
3 answers

Windows 2016 Storage Spaces Direct + Data Deduplication

Has anyone combined S2D (Storage Spaces Direct) with Data Deduplication? Is this even possible or recommended practice? Please elaborate on why this is a good idea or not. EDIT: Just stumbled across this article…
Paul Masek
  • 722
  • 2
  • 7
  • 16
6
votes
1 answer

Windows 2012 Robocopy with Deduplication active

Current Issue: File Cluster Disk of 14 TB Size needs an CHKDSK which can't be done because it lasts too long. So I'm going to replace this disk in the storage of the cluster. I run many times the copyjob with robocopy. No problem at all till I saw a…
donduc_jr
  • 73
  • 5
6
votes
2 answers

How to interpret the results of DDPEval.exe?

I have read the documentation for DDPEval.exe and various blogs and so on and cannot find any information on how to interpret the results. I ran the tool on a drive on one of our servers and found the results confusing, particularly with regards to…
Stephen Kennedy
  • 161
  • 1
  • 10
6
votes
4 answers

ZFS: Memory issues with dedup even though zdb -DD looks fine

I've been experimenting with ZFS on a machine with Ubuntu 12.10, 32GB RAM (non-ECC, production system will have ECC) and a 2x2TB Linux-managed RAID1 (will be moved to RAIDZ1 for production). I just created the tank on the 2TB soft-RAID1 device,…
user206444
6
votes
2 answers

Server 2012 deduplication: run on Hyper-V host or guest VM?

I have a Windows Server 2012 Hyper-V host with VMs on a dedicated RAID-5 volume. One VM is also running Server 2012 and is being used to serve files. The other VMs are smaller and mostly used for temporary testing scenarios. The main file server VM…
WaxTrax
  • 217
  • 1
  • 4
  • 9
1
2 3 4 5 6 7