6171 dsl_prop_unregister() slows down dataset eviction.

Review Request #235 - Created Aug. 28, 2015 and submitted

Justin Gibbs

A change to a property on a dataset must be propagated to its descendants
in case that property is inherited. For datasets whose information is
not currently loaded into memory (e.g. a snapshot that isn't currently
mounted), there is nothing to do; the property change will take effect
the next time that dataset is loaded. To handle updates to datasets that
are in-core, ZFS registers a callback entry for each property of each
loaded dataset with the dsl directory that holds that dataset. There
is a dsl directory associated with each live dataset that references
both the live dataset and any snapshots of the live dataset. A property
change is effected by doing a traversal of the tree of dsl directories
for a pool, starting at the directory sourcing the change, and invoking
these callbacks.

The current implementation both registers and de-registers properties
individually for each loaded dataset. While registration for a property is
O(1) (insert into a list), de-registration is O(n) (search list and then
remove). The 'n' for de-registration, however, is not limited to the size
(number of snapshots + 1) of the dsl directory. The eviction portion
of the life cycle for the in core state of datasets is asynchronous,
which allows multiple copies of the dataset information to be in-core
at once. Only one of these copies is active at any time with the rest
going through tear down processing, but all copies contribute to the
cost of performing a dsl_prop_unregister().

One way to create multiple, in-flight copies of dataset information
is by performing "zfs list" operations from multiple threads
concurrently. In-core dataset information is loaded on demand and then
evicted when reference counts drops to zero. For datasets that are not
mounted, there is no persistent reference count to keep them resident.
So, a list operation will load them, compute the information required to
do the list operation, and then evict them. When performing this operation
from multiple threads it is possible that some of the in-core dataset
information will be reused, but also possible to lose the race and load
the dataset again, even while the same information is being torn down.

Compounding the performance issue further is a change made for illumos
issue 5056 which made dataset eviction single threaded. In environments
using automation to manage ZFS datasets, it is now possible to create
enough of a backlog of dataset evictions to consume excessive amounts
of kernel memory and to bog down the system.

The fix employed here is to make property de-registration O(1). With this
change in place, it is hoped that a single thread is more than sufficient
to handle eviction processing. If it isn't, the problem can be solved
by increasing the number of threads devoted to the eviction taskq.

Associate dsl property callback records with both the
dsl directory and the dsl dataset that is registering the
callback. Both connections are protected by the dsl directory's

When linking callbacks into a dsl directory, group them by
the property type. This helps reduce the space penalty for the
double association (the property name pointer is stored once
per dsl_dir instead of in each record) and reduces the number of
strcmp() calls required to do callback processing when updating
a single property. Property types are stored in a linked list
since currently ZFS registers a maximum of 21 property types
for each dataset.

Note that the property buckets/records associated with a dsl
directory are created on demand, but only freed when the dsl
directory is freed. Given the static nature of property types
and their small number, there is no benefit to freeing the few
bytes of memory used to represent the property record earlier.
When a property record becomes empty, the dsl directory is either
going to become unreferenced a little later in this thread of
execution, or there is a high chance that another dataset is
going to be loaded that would recreate the bucket anyway.

Replace multiple calls to dsl_prop_unregister() with a single
call to dsl_prop_unregister_all() (O(n^2) -> O(n)).  There are two
software layers in ZFS that use the property mechanism: the DMU,
and the ZPL. The DMU registers all of its callbacks with the
objset_t pointer of the dataset as the callback argument. The
ZPL uses the zfsvfs_t pointer. In both cases, these software
layers want to unregister all of the callbacks they registered
on a dataset, regardless of the property name. This new API uses
the callback argument pointer as a "software layer identifier"
for filtering. The list of callbacks is traversed once with
items selected for deletion by a pointer comparision.

Replace use of dsl_prop_unregister() with the new
dsl_prop_unregister_all() API.

Ztest on illumos.


  • 0
  • 11
  • 0
  • 11
Description From Last Updated
Dan McDonald
Justin Gibbs
Robert Mustacchi
Matthew Ahrens
Justin Gibbs
Matthew Ahrens
Prakash Surya
Justin Gibbs
Justin Gibbs
Prakash Surya
Justin Gibbs
Review request changed

Status: Closed (submitted)

Change Summary:

commit 03bad06fbb261fd4a7151a70dfeff2f5041cce1f
Author: Justin Gibbs gibbs@scsiguy.com
Date: 2015-09-17T12:05:51.000Z

6171 dsl_prop_unregister() slows down dataset eviction.
Reviewed by: Matthew Ahrens <mahrens@delphix.com>
Reviewed by: Prakash Surya <prakash.surya@delphix.com>
Approved by: Dan McDonald <danmcd@omniti.com>