ZFS disk usage explained in one handy guide
I use ZFS extensively. ZFS is my favorite file system. I write articles and give lectures about it. I work with it every day. In traditional file systems we use df(1) to determine free space on partitions. We can also use du(1) to count the size of the files in the directory. But it’s different on ZFS and this is the most confusing thing EVER. I always forget which tool reports what disk space usage! Every time somebody asks me, I need to google it. For this reason I decided to document it here – for myself – because if I can’t remember it at least I will not need to google it, as it will be on my blog, but maybe you will also benefit from this blog post if you have the same problem or you are starting your journey with ZFS.
zpool
Let’s create some test pool:
# mdconfig -s 1G
# mdconfig -s 1G
# mdconfig -s 1G
# zpool create ztest raidz1 /dev/md0 /dev/md1 /dev/md2
# zpool list
NAME SIZE ALLOC FREE CKPOINT EXPANDSZ FRAG CAP DEDUP HEALTH ALTROOT
ztest 2.75G 431K 2.75G - - 0% 0% 1.00x ONLINE -Does it mean that we can store 2.75GBs on the pool? Unfortunately not. zpool under column FREE reports to us the free bytes in the pool. This means that it doesn’t count the data redundancy in it. So, every time we write data on the disk a parity data will be written to the pool. In the case of RAIDZ1, the size of the one disk will be used for the parity data. The SIZE value reports the size of the whole pool (so all the disks in the pool).
zpool shows the total bytes of storage available in the pool. This doesn’t reflect the amount of data you can store on the pool. To figure out that you should refer to the AVAIL space from the zfs.
$ zfs list
NAME USED AVAIL REFER MOUNTPOINT
ztest 261K 1.71G 29.3K /ztestWhat is interesting is in the case of a mirror it will show the size of a single disk.
NAME SIZE ALLOC FREE CKPOINT EXPANDSZ FRAG CAP DEDUP HEALTH ALTROOT
ztest 960M 87.5K 960M - - 0% 0% 1.00x ONLINE -zfs
# zfs list
NAME USED AVAIL REFER MOUNTPOINT
zroot 146G 49.3G 14.4G legacy
zroot/home 2.0G 49.3G 12K /home/
zroot/home/def 1.0G 49.3G 1.0G /home/def
zroot/home/oshogbo 1.0G 49.3G 1.0G /home/oshogboThe zfs(1) command shows us the used and available space per dataset. The used space (USED column) is hierarchal. It means that the size of the zroot/home/oshogbo (1GB) is also added to the zroot/home (2GB). zroot/home contains 2GB because both zroot/home/oshogbo and zroot/home/def use 1GB and it probably doesn’t contain data by its own.
The available space (AVIL) means how much data we can actually write to the dataset. This value refers to the size of data stored after compressions, deduplication, and all the RAIDs stuff.
The available space in our example is exactly the same because all datasets have access to the whole pool. This value may be changed per dataset, for example using quotas and reservations.
The reference data (REFER) means how many data are REFERENCED to the particular dataset (not stored in the dataset). The zroot/home refer to 12KBts of space. In this space there is only some metadata, as it is not real data that is stored there. Those data basically say that such a file system exists. Let’s look at the example below:
NAME USED AVAIL REFER MOUNTPOINT
ztank/test 11.0G 614G 11.0G /test
ztank/mytests 0 614G 11.0G /mytestsThe ztank/test is using 11.0G and it has REFERance 11.0G. The ztank/mytest REFERENCE to 11.0G but is using 0 storage space. How is that possible? This is because the ztank/mytest is a clone of the ztank/test. It means that if we would like for example to send ztank/mytest to the file, the created file will have 11.0GB size, but physically on our disks ztank/mytest doesn’t use any blocks.
If we were to start writing to the dataset ztank/mytest the USED and REFER amount will be increased:
NAME USED AVAIL REFER MOUNTPOINT
ztank/mytests 1.19G 612G 12.2G /mytests
ztank/test 11.0G 612G 11.0G /testWhat if we were to remove the data from dataset ztank/mytest which refers to the ztank/test? The USED value wouldn’t change because the data wasn’t freed from ztank/mytest, but the reference count will drop.
NAME USED AVAIL REFER MOUNTPOINT
ztank/mytests 1.51G 612G 1.55G /mytests
ztank/test 11.0G 612G 11.0G /testAnd the last thing that would happen if we freed some space in ztank/test? ztank/test is we would have a snapshot because ztank/mytest was created from it.
NAME USED AVAIL REFER MOUNTPOINT
ztank/test 11.0G 612G 48.9M /testThe snapshot is using and REFERing to the 11.0GB of data. As mentioned before the USED is hierarchal and means that it counts all datasets and snapshots. This means that 11.0G used by the `ztank/test` is a value of the all underlying datasets and snapshots. If we were to rollback to the state of test snapshot:
NAME USED AVAIL REFER MOUNTPOINT
ztank/test@test 0 - 11.0G -It will turn out that the snapshot doesn’t use any space because all of our data is stored in the dataset:
NAME USED AVAIL REFER MOUNTPOINT
ztank/test 11.0G 612G 11.0G /testTo see more details about used space we can run the `zfs list -o space` command.
zfs list -o space
NAME AVAIL USED USEDSNAP USEDDS USEDREFRESERV USEDCHILD
ztank/mytests 612G 1.51G 0 1.51G 0 0
ztank/test 612G 11.0G 0 11.0G 0 0The USED and AVAIL columns we know already.
The USEDSNAP is a space used by the snapshots. If we removed a file like previously this value would go up to 10.9G.
NAME AVAIL USED USEDSNAP USEDDS USEDREFRESERV USEDCHILD
ztank/mytests 612G 1.51G 0 1.51G 0 0
ztank/test 612G 11.0G 10.9G 48.9M 0 0The USEDDS column show the size of files in the dataset – only files without snapshots, reservations etc.
The USEDREFRESERV value is reporting the space used by refreservation for this dataset.
The USEDCHILD value is reporting the space used by its children. If we would go back to this example within the hierarchy:
NAME AVAIL USED USEDSNAP USEDDS USEDREFRESERV USEDCHILD
zroot/home 49.3G 2.0G 0 0 0 2.0G
zroot/home/def 49.3G 1.0G 0 1.0G 0 0G
zroot/home/oshogbo 49.3G 1.0G 0 1.0G 0 0GWe see that zroot/home does not USEDDS any of the data and its child (USEDCHILD) is using 2.0GBs.
df
The df(1) output may be a little bit confusing.
Filesystem Size Used Avail Capacity Mounted on
zroot 64G 14G 49G 23% /
zroot/tmp 51G 1.9G 49G 4% /tmp
zroot/usr 87G 38G 49G 44% /usr
zroot/var 54G 5.0G 49G 9% /varNormally df(1) reports the size of all of the filesystem in the operating systems. The problem is that all the filesystems (datasets) are using the same pool of data and all of it is available to any of the filesytems. So, if we were to add it up as we use to it would turn out that our disk is much bigger.
This is also the reason why we can’t depend on the capacity value. You also may notice that the size of the file system shrinks as space is used up and grows when space is freed. This tool will give us incorrect answers. This may also confuse other tools and windows machines while we mount datasets via SAMBA.
The situation in which we can depend on the df is the used size. This value correspondents to the REFER value of the zfs list, and also to determining where the mount points of datasets are.
du and ls
Let’s examine this example:
# zfs list
NAME USED AVAIL REFER MOUNTPOINT
ztank/test 23K 625 23K /test
# du -h file3
512B file3
# ls -lah
-rw-r--r-- 1 root wheel 1.0G May 12 13:40 file3The zfs list says that we are using 23KB of data. du(1) is saying a few bytes and ls(1) is reporting a GB. The case is the written file is compressed or full of zeros which ZFS also compress.
The du(1) tool reports how many bytes are used to store the contents of the files after compression, dedupe and so on.
The ls -l shows the real size of the file. If you plan to copy a file to a different FS without compression you need to prepare to have enough disk size.
Summary
The understanding of how ZFS is uses space and how to determine which value means what is a crucial thing. I hope thanks to this article I will finally remember it!
