How to Improve the Pruning Ability of Dynamic Metric Access Methods

How to Improve the Pruning Ability of
Dynamic Metric Access Methods
Caetano Traina Jr.
Agma Traina
Roberto Santos Filho
Dept. of Computer Science and Statistics
University of Sao Paulo at Sao Carlos - Brazil
[caetano| agma | figueira]@icmc.sc.usp.br
Abstract
Retrieval of complex data is accelerated through
the use of indexing structures. They organize the
data in order to minimize comparisons between
objects aiming to prune blocks of data during the
process. In metric spaces, comparison operations
can be specially expensive, so the pruning ability
of indexing methods turns out to be specially
significant. This paper shows how to measure the
pruning power of metric access methods (MAMs),
and defines a new measurement, called
"prunability,” which indicates how well a pruning
technique carries out the task of reducing distance
calculations at each level of the tree. We also
gives a new measurement for it, and presents a
new dynamic access method, aiming to retrieve
objects in a metric space minimizing the number
of distance calculations required to answer
similarity queries. We show that this structure is
up to 3 times faster and requires less than 25% of
the distance calculations to answer range and
nearest-neighbor queries as compared with the
Slim-tree, one of the most efficient metric access
method (MAM). This gain in performance is
achieved by taking advantage of a set of global
representatives.
Although using multiple
representatives, the whole structure remains
dynamic and balanced. Experimenting with
existing metric access methods, we have
discovered that the use of one node representative
works well at the upper levels of the existing
MAMs, but is almost useless at the lower levels.
Our proposed method assures higher rates of
pruning at all the levels of the tree.
1 - Introduction
The amount of information stored in databases has grown
at a very fast pace over the last years. Data today is far
more diversified and complex than it was in the past, when
all the information was represented by text and numbers.
Christos Faloutsos
Dept. of Computer Science
Carnegie Mellon University - USA
[email protected]
The massive use of multimedia information, such as
images, audio, video, time series and DNA sequences have
generated new challenges for researchers in the database
area. Thus, a cornerstone for the database area is the
development of fast and efficient access methods, which
can cope with large amount of complex data in these
domains.
In multimedia systems, it is important to search the
database for similar data. In image databases, for example,
the user looks for images that are similar to a given one
according to specific criteria. These data domains do not
pose an order relationship property; in other words,
relations such as “less than” and “greater than” are
meaningless, rendering the usual access methods useless.
However, if a similarity function is defined, similarity
queries make sense. Note that a metric distance function
points out the dissimilarity between objects. When a
distance function has non-negativity and symmetry
properties and satisfies triangular inequality, it is said to be
a metric function, the domain is said to be metric, and
metric access methods can be used. In metric domains, the
so-called similarity queries most frequently used are the
following:
• k-nearest neighbor or k-NN query: kNN=<sq, k>,
which asks for the k objects that are the closest to
a given query object center sq.
• range query: Rq=<sq, rq>, which searches for all the
objects whose distance to the query object center
sq is less or equal to the query radius rq.
Note that calculating distances between complex
objects are usually expensive operations.
Hence,
minimizing these calculations is one of the most important
aspects to achieve acceptable response time in a metric
access method (MAM). This paper presents a way to
reduce the number of distance calculations to answer
similarity queries and introduces a new and efficient MAM,
the DF-tree.
The main approach is the use of global representatives
for the whole tree. These global representatives will work
together with the representatives of the nodes in order to
decrease the number of distance calculations, which is
-2usually expensive for complex (multimedia) data. Global
representatives use a new approach which maintains the
tree dynamic. We will show that this approach leads to
amazing improvements for answering similarity queries.
Also, being a dynamic MAM is important because
applications usually continue inserting and deleting objects
after the creation of the index structure. We present the
results of experiments executed over three real datasets,
comparing the DF-tree with the state of the art MAMs.
The results show that the DF-tree performs 25% less
distance calculations and answers similarity queries up to
3 times faster. The results of the experiments also
demonstrate that this new MAM is scalable to very large
datasets, presenting a linear behavior over varying dataset
sizes.
We also defined a new measurement, called
"prunability,” to estimate how many distance calculations
can be pruned at each level of a tree. We have discovered
that the use of one node representative works well at the
upper levels of the existing MAMs, but is almost useless at
the lower levels. Using this measurement, we confirmed
that our new proposed MAM assures higher rates of
pruning at every level of the tree.
In addition to presenting the DF-tree and the improved
algorithms to accelerate answering similarity queries, this
paper also presents three support algorithms. The first one
decides when to select and use the global representatives,
the second one automatically detects if the current set is no
longer appropriate, and the last algorithm quickly selects a
new set of global representatives.
The remainder of this paper is organized as follows.
The next section summarizes the background for this work,
while section 3 explains why using more than one
representative per node to build a metric tree may lead to
static structures. Section 4 presents the proposed metric
access method DF-tree, as well as its supporting
algorithms. Section 5 presents the results of the
experiments performed on the DF-tree and compares it to
other recent MAMs. Section 6 discusses the conclusions of
this paper.
2 - Background
One of the underpinnings of a database system is the index
structure that supports it, so the design of efficient access
methods has long been pursued in the database area.
Because some complex data can be viewed as points in a
multidimensional domain, several methods to index data in
a multidimensional space have been proposed in the
literature [1] [6] [10] [11]. An excellent survey on
multidimensional access methods is presented in [9].
Unfortunately, these methods were developed only for
vector data. Note that many complex data do not have any
vectorial property, but their elements can be treated as
occupying a metric domain.
Previous work on metric methods focused on the static
case, for example: the structures proposed by Burkhard and
Keller [5], Uhlmann [15], and also the vp-tree [16], GNAT
[4] and mvp-tree [3], where no further insertions, deletions
and updates are handled. Overcoming this inconvenience,
dynamic MAM structures including M-tree [7], Slim-tree
[14] and OMNI-family [12] have been developed.
Multiple representatives help reducing the number of
distance calculations performed when answering similarity
queries. They were nicely used in the mvp-tree [3] and in
the Omni-family of index structures [12]. In contrast to
mvp-tree, the MAMs members of Omni-family are
dynamic, allowing for insertions and deletions. However,
the set of representatives used by the Omni-family is chosen
during the early construction phase of the tree and cannot
be changed afterwards. The main idea was that, given a
dataset S of objects in a metric domain, an object s i S can
be represented by a set of distances to a set of global
representative objects, the foci. A good algorithm to choose
these foci, the HF algorithm, is also presented in [12]. This
algorithm finds objects near the border of the dataset to be
used as foci, with linear cost on the size of the dataset.
The number of representatives required to maximize
the gain obtained from the reduced number of distance
calculations is associated with the intrinsic dimensionality
' of the dataset. As in [8], the intrinsic dimensionality is
defined as the exponent in the power-law:
'
nb(r ) ∝r
where nb ( r ) is the average number of neighbors within
distance r or less from an arbitrary object, and stands for
“is proportional to”. We will use this result and the HF
algorithm in this paper.
3 - Motivation
Metric access methods based on trees store data through
sets of fixed-size nodes, using reference objects in each
node to represent the other objects stored in that node and
in its sub-trees. Precalculated distances between stored
objects and these representatives are kept to speed up
insertions and deletions and to answer queries. This design
takes advantage of the triangular inequality property of
metric domains to prune many costly distance calculations
between the query centers and the stored objects, as is
explained following.
In a metric tree, each node stores a sub-set of the
objects. Let us first consider the objects sjS stored in a leaf
node i. One of these objects, tagged as sRi, is chosen as the
“representative” for the objects stored in the node i. The
distances from every object in this node to the representative are calculated, and the largest one is defined as the
covering radius rRi of that node. In this way any object
farther than rRi from sRi cannot be found in node i. In the
-3upper levels the same construct is employed, but the objects
stored in a non-leaf node are the representatives of their
descendant nodes. Again, one of these objects is chosen as
the representative of this node, and we will tag it as sRep.
The distances from every object stored in that node to its
representative are calculated and also stored in the node.
However, the covering radius of a non-leaf node is the
distance from the representative to the farthest object stored
in that node plus the covering radius of the node where this
object is the representative. Figure 1 shows a DF-tree
indexing 17 objects, considering nodes with a maximum
capacity of tree objects. Note that the root node does not
have a representative and the complete set S is stored in the
leafs. The global representatives are not depicted in this
figure because no query was asked yet. For example, any
two objects far from each other and on the “border” of the
dataset can be used as global representatives, such as the
objects K and P, in this figure. As the other illustrative
figures presented in this paper, figure 1(a) assumes a
dataset in a two-dimensional space with the Euclidean
distance function, although the same concepts apply to any
metric domain.
Figure 1 - A DF-tree with node capacity=3, indexing 17 objects.
Now, consider a range query asking for objects distant
up to rq from the query center sq, as shown in figure 2. The
query defines tree regions: in Region 1 are the objects that
are farther from sRep than its distance to the query center sq
plus the query radius rq. In Region 3 are the objects that
are closer to sRep than its distance to the query center sq
minus the query radius rq. In Region 2 are the objects that
do not satisfy any of these conditions. Objects in Regions
1 and 3 cannot be in the response set. Thus, nodes that are
completely in Regions 1 or 3 can be pruned from the search
process. In this way, by the triangular inequality, every
node i with representative sRi can be pruned if it satisfy one
of the following conditions:
a) d(sRep, sq)+rq < d(sRep, sRi)-rRi (Region 1) (Eq. 1)
b) d(sRep, sq)-rq > d(sRep, sRi)+rRi (Region 3) (Eq. 2)
The same pruning concept can be applied at the leaf nodes,
where the value of rRi is zero. Thus, the triangular
inequality enables pruning both the traversing of subtrees
in non-leaf nodes and distance calculations from the query
object to objects in the leaf nodes.
Figure 2 - Only objects in region 2 will
be in the response set of the range query
centered at sq with radius rq.
This mechanism leads to dynamic structures, because
if a new object needs to be inserted, it can be inserted in
any of the nodes that are able to “cover” it. If no node
covers it, the covering radius of an existing node is
enlarged. If the node capacity is surpassed, the node splits
and a new pair of representatives is chosen. This changes
one object in the upper level node and promotes one new
representative to this node, which will eventually require a
split too, in an iterative process. When the root node splits,
the tree grows one level. Thus, the construction of the tree
occurs in a bottom up approach, like the B-trees. Detailed
algorithms to perform those operations can be found in [14]
and [7].
Pruning can be enhanced by using more than one
representative (a reference object). In fact, the “amount of
information” embodied in a simple relation between three
objects (the query center, the target object sj and the node
representative) provided by the triangular inequality
property is rather low, particularly in pure metric or in
high-dimensional spaces [2]. The combined use of two or
more references eliminates larger portions of the dataset
than using each reference object individually. Figure 3
shows how much the region where the candidate set of
objects to answer the query shrinks by adding a global
representative. If only the node representative is used for
pruning, the shadowed region in figure 3(a) is where the
answer objects for a range query with center object sq and
radius rq could be found. In figure 3(b) one Global
Representative is also used and the darker shadowed region
indicates where the answer objects could be for the same
range query of (a). Note that the size of the resulting
region is much smaller.
-4the metric access method DF-tree presented in the next
section.
4 - The new DF-tree structure
Figure 3 - Regions that cannot be pruned: (a) using only the node
representative; (b) using also a global representative.
It is helpful to understand why multiple reference
objects lead to static structures, allowing us to overcome
this undesired effect, yet still being able to retain the strong
capacity of pruning achieved by a combination of many
representatives. In fact, if the set of reference objects at a
given node dictates where other objects should be stored at
lower levels (i.e., in which of its descending subtree), the
structure becomes static, because whenever a reference is
changed, the objects that were stored in a given subtree
should have to be moved to another subtree. Hence, in
order to be dynamic, a MAM must either allow for more
than one place to store each object or choose the
representatives in a bottom-up approach, changing the
reference objects based on the subtrees rather than on the
upper levels of the tree. Both approaches, however, have
their drawbacks. Allowing for more than one place to store
each object involves many attempts until the objects
required in a query are successfully retrieved. Choosing the
representatives in a bottom-up approach, on the other hand,
inhibits the combined use of the set of representatives of the
nodes whose path leads to that node.
The Slim-tree and the M-tree choose a compromise
between these two approaches. First, the representative at
each node restricts, rather than determines, where a given
object can be stored. Second, the representative of a node
is selected from among the representatives of its children
nodes, and the covering radius of the node is calculated
based on the covering radius of those children. However,
this compromise means that only one representative can be
used at each node. The Slim-tree improved the M-tree
tackling the first compromise, as minimizing the
intersection of nodes reduces the number of deep searches
needed to find an object. The DF-tree further improves the
Slim-tree allowing more representatives to be used to prune
distances, tackling also the second compromise. Although
we do not use the representatives of upper levels, as a
matter of fact, nothing hinders the use of more than one
object to prune subtrees, provided that only one is used to
guide the creation of the tree. That is what we do through
We propose here the use of Global Representatives (GR)
together with the fundamental concepts of a metric tree,
which aims to reduce the number of distance calculations
required to answer similarity queries. This paper
demonstrates that the proper use of GRs allows for fewer
distance calculations and thus to a shorter time to answer
similarity queries. The following definition is needed to
explain this new structure.
0 7
Definition 1 - Global Representative Set (* ): Let =< ,
d> be a metric space, where
is the domain of
features, which are the indexing keys, and d( ) is a
metric distance function. Given a dataset S G with
N objects, a Global Representative Subset of S (or a GR
set) is a set *={g1, g2, ..., gp | gk S, gkggj, pN}, where
each gj is chosen to be a Global Representative, and p
is the number of global representatives contained in *.
We call the single representative of each node of the
tree as the node representative in order to distinguish it
from the global representatives for the whole tree. Each
global representative is independent from the others,
including the node representatives, and is applied to every
object si inserted in the structure, so we say that each global
representative defines a distance field (DF) over the
domain . The distance field is represented by the distance
from each object to the corresponding global representative,
stored with that object. The main symbols used in this
paper are shown in Table 1.
The general idea of the DF-tree is to build the structure
combining a metric tree that uses one representative per
7
7
7
Symbols
7
S
'
sq
rq
k
h
*
gj
p
d( )
mdj
Ph(Q)
Definitions
domain of objects
set of objects in domain 7
intrinsic dimensionality of dataset S
a query object (or query center)
radius of a range query
number of neighbors in an NN query
level of the tree
set of global representatives
a global representative of *
cardinality of *
distance function
maximum distance from the global
representative gj to all the other global
representatives
Prunability at level h to answer a set Q of
queries
Table 1 - Summary of symbols and definitions.
-5node (e.g. the Slim-tree) with the distance fields generated
by each element of *. The distance fields do not interfere
with the creation of the tree structure, but can be used to
prune distance calculations when answering queries. The
selection of objects to act as global representatives and the
calculation of their corresponding distances to each stored
object can be postponed to any time before the first query is
answered. For example, the distances can be calculated
either when each object is being inserted or just after a bulk
loading operation is completed.
4.1 - The “prunability” property
The purpose of using global representatives is to increase
the pruning of distance calculations. The number of
distance calculations that can be pruned depends on the
relative sizes of the areas defined by each representative
(node or global), and the regions defined by the query
center and the representative radii (recall figure 2). These
sizes vary at different levels of a given tree, leading to
methods that are better to prune at some levels than at
others. Therefore, to gauge how efficiently a MAM can
prune distance calculations based on a set of reference
objects active in each node, we have defined the following
measurement.
Definition 2- “Prunability”: Given a large set Q of
similarity queries over a tree, the prunability Ph ( Q ) is the
average of the relation Nub(qi)/Ntb(qi) applied to each node
b accessed at a given level h to answer each query qi Q,
where:
Ntb(qi ) - is the total number of objects in node b
accessed to answer the query qi not pruned by the
references active in this node. Each access to a
node imposes a distance calculation.
Nub(qi) - is the number of objects in the nodes accessed
to answer the query qi that actually qualify, i.e.,
for which the distance calculation is unavoidable.
We consider that a distance calculation is unavoidable in a
given node at level h if at least one object in the children
nodes at level h+1 require further processing. Figure 4(a)
exemplifies the prunability when using just the node repres e n t a t i ve t o a n swer t h e q u e r y d e p i c t e d
(prunability=2/20=0.1), and figure 4(b) shows that when
using also one global representative to answer a range
query centered at sq (prunability=2/4=0.5). It can be seen
that the prunability increased five times.
4.2 - Structure of the DF-tree
The structure of the proposed DF-tree has two components:
the fields which are used to build the tree and the distance
fields. The tree component complies with the same
structure of the Slim-tree, and is used following its same
algorithms. The distance fields have valid values only after
the GR set * has been selected, which occurs when the first
query is asked. Each tree-node of the DF-tree structure
contains the number c of objects stored in this node, and an
array of c sub-structures. Leaf and non-leaf nodes have
slightly different format. The leaf-node substructure has the
following format: <si, Oidi, DFi, d(si, sRi) >, where si is the
indexed description of the object, Oidi is the identifier of
object si, and d(si, sRi) is the distance between object si and
the representative object sRi of this leaf node. The indexnode substructure has the following format:
<si, Ri, d(si, sRi), DFi, Ptr(Tsi), Nentries(Ptr(Tsi))> , where
si holds the object that is the representative of the sub-tree
indicated by Ptr(Tsi), and Ri is the covering radius of that
region. The distance between si and the representative of
this node sRi is given by d(si , sRi). The pointer Ptr (Tsi)
points to the root node of the subtree T rooted by si. The
current number of entries in the node indicated by Ptr(Tsi)
is stored in NEntries(Ptr(Tsi)).
Both leaf and index nodes store the distance field
component in DFi , an array of the distances of object si to
each global representative gj*. This component is used
only to answer queries, and not in the update operations of
the tree. Section 4.3 explains how they are used to answer
queries, and section 4.4 explains how to calculate them,
when to calculate, how many representatives constitute the
distance fields and how to determine if they need to be
recalculated after many updates in the tree had taken place.
The distance field component can be stored in a
separated file. We chose to store it together with the tree
component to reduce disk accesses. Storing all the data
together reduces the number of objects that fit in nodes of
a given size, which, in turn, reduces the fan-out of the tree.
We have assumed that this side effect is negligible when
the proposed structure is used with large objects, which is
the most common situation.
4.3 - Algorithms for range and nearest-neighbor queries
using global representatives
Figure 4 - Exemplifying how the prunability works.
Here we present the algorithms to execute both range and
k-nearest neighbor queries using global representatives on
the DF-tree. Let us first consider range queries, which are
represented as Rq=<sq, rq>. These queries begin looking at
-6the root node of the tree. When this node is read, its
representative is set as sRep, and the distance from sq to sRep
is calculated. In this way the three regions shown in figure
2 are generated for this representative, and the conditions
expressed by Equations 1 and 2 are evaluated. If none of
the conditions expressed by those equations holds, one of
the global representatives gj is set as sRep and both equations
are re-evaluated. If none of the equations holds, the next
global representative is used, and so on, until every
representative is used. If one equation holds for any
representative, the corresponding node can be pruned from
the searching process. Nodes that cannot be pruned must
be read, and processed recursively. This algorithm is shown
in figure 5.
Process a subtree to answer a range query
Input: a DF-tree node i and a range query Rq=<sq, rq>
Output: the query response set.
Begin
1 - Calculate the distance from the query object sq to the
representative of this node.
2 - For each object sj in node i:
3 - Set sRep as the representative of this node
4 - If Eq. (1) or Eq. (2) holds, get the next object. // object was
pruned
5else check if object sj can be pruned by the distance field
(see Fig. 4b)
If it can, get the next object. // object was pruned
6If this is a leaf node, put object sj in the response set
7else process the subtree rooted at this object.
End
D
E Check if an object can be pruned by the distance field
Input: the object sj to be checked, its covering radius rRi if it is
in a non-leaf node or zero otherwise, and its distances to the
global representatives
Output: true if it can be pruned, false otherwise.
Begin
1 - For each global representative gj *
2Set sRep as gj
3If Eq. (1) or Eq. (2) holds, return true, else get next
global representative
4 - return false
End
Figure 5 - The range query algorithm for the DF-tree.
The k-nearest neighbor queries kNN=<sq, k> use a
priority queue Pr of size k, where the candidate objects sj
are maintained sorted by its distance to the query center sq.
The distance of the farthest object currently in Pr is set as
the current query radius rc. The algorithm starts reading
the root node, with Pr empty, and until there are k objects
in the queue, rc is set to infinity. A new object is inserted
only if it is closer to sq than rc. Inserting a new object in a
full Pr replaces the farthest one, and the next farthest object
define the new value of rc. Whenever a node is read, every
object in it is stored in another priority queue Pw of
unlimited size, that holds the unprocessed objects, which
are also maintained sorted by its distance to the query
center sq. Objects in Pw are processed by an algorithm
similar to the one to process range queries, using rc as the
current query radius. The algorithm terminates when Pw
becomes empty. Figure 6 depicts this algorithm.
Process a subtree to answer a k-nearest neighbor query
Input: a DF-tree node i, a query kNN=<sq, k>, the response
queue Pr and the waiting queue Pw
Output: the query response set.
Begin
1 - If node i is a non-leaf node:
2For each object sj in node i:
3Set sRep as the representative of this node.
4If Eq. (1) or Eq. (2) holds, get the next object. // object
was pruned
5else if object sj can be pruned by the distance
field (see Fig 4b), get the next object // object
was pruned
6else insert sj in Pw with its distance from sq.
7 - While Pw is not empty, get sj and rj from Pw:
8Set sRep as sj
9If Eq.(1) or Eq.(2) holds, get the next object. // object
was pruned
10 else if object s j can be pruned by the distance field
(see Fig 4b), get the next object // object was pruned
11 else process the subtree rooted at object sj.
12 else // node is a leaf node
13 - For each object sj in node i:
14 Set sRep as the representative of this node.
15 If Eq. (1) or Eq.(2) holds, get the next object. //
node was pruned
16 else if object s j can be pruned by the distance
field (see Fig 4b), get the next object // node
was pruned
17 else insert sj in Pr with its distance from sq,
and get the new rc.
End
Figure 6 - The K-nearest neighbors query algorithm for the DFtree.
4.4 - Choosing the global representative set *
As discussed in the previous section, each representative
(either the node or a global representative) defines a ring in
the dataset domain where objects cannot be pruned using
this reference. When more than one representative is used,
only the objects in the intersection of the corresponding
rings cannot be pruned. The node representative is defined
by the tree construction algorithm, so it cannot be changed.
Hence, the flexibility in the choice of references falls on the
number and placement of the global representatives (GRs).
Determining the number of global representatives
It was shown in [12] that the maximum number p of
references worth considering is one plus the intrinsic
-7dimensionality M'N of the dataset. However, the way in
which DF-tree uses the GR involves different requirements,
so we have re-analyzed this number. As each node already
has a representative, the number of global representatives
can be only M'N.
If the distance of the intersecting regions of the rings
provided by two GRs is less than the distance between these
two GRs, this region tends to be smaller than the
intersection generated by the rings far from the GRs.
Therefore, among other requirements, the GR set must get
objects far from each other and close to the “border” of the
dataset. In this way, almost every distance from the objects
in the dataset to these GRs will be less than the distance
between the GRs. Figure 7 illustrates this idea. Figure 7(a)
shows a query occurring “between” the GRs, whereas the
queries shown in figures 7(b) and 7(c) occurring “outside”
the GRs. The intersections of the rings in figure 7(a) are
clearly smaller than those in figures 7(b) and 7(c).
Figure 7 - Where the GR set G is effective. (a) The most
effective: the range query is circumscribed by G={ A, B, C}. (b)
The intersection provided by G to solve a query centered on the
object sq2 which is not circumscribed by G, but close to at least
one GR. (c) The least effective: intersection provided by G to
solve a query centered on the object sq3, which is also not
circumscribed by G and is far from all the GRs.
It should be noted that increasing the number of GRs
reduces the fan-out of the tree, which, in turn, increases the
number of nodes and the height of the tree. Although the
triangular inequality allows for pruning of distance
calculations to objects in a previously read node, each disk
access involves at least one additional distance calculation from the query center to the node representative.
Therefore, increasing the number of nodes in a tree also
increases the need for distance calculations, leading to a
tradeoff between the increased prunability provided by a
greater number of representative and the resulting
decreased fan-out. This limits the useful number of global
representatives to M'N.
When to choose the first GR set *
When the tree is empty, it is impossible to choose a GR set.
As the objects are inserted, we have to decide at which
point to choose the GR set. The trade-off is clear: if we start
too early, we may be 'stuck' with a bad GR set. If we start
too late, we may not enjoy the speedups that the GR set
provides.
The DF-tree requires the GR only when a query is
issued. If the query is posted on a tree that has only a few
objects, the query can be answered without using this
resource. Therefore, we have determined that the first GR
set is calculated when the first query is answered, after the
DF-tree already has at least 2 levels (the root level and
another one). This ensures the presence of a reasonable
number of objects when the global representatives are
chosen.
Deciding when to update the GR set: the WU algorithm
An important consideration is the maintenance of a proper
set of global representatives. The idea is that the GRs are
chosen when the tree is first created. However, after the
tree has been created and many objects were inserted and
deleted, a possible change in data distribution may render
the original GR set * inadequate. Therefore, in addition to
having a fast algorithm to select the GR set in the current
dataset, an algorithm is required to determine when the
current * is no longer worthwhile. An ideal algorithm
must preclude new distance calculations. Considering that
the distance between each newly inserted object and each
GR must be calculated, the algorithm should rely only on
those distances.
Figure 8 enable to visualize how this algorithm works
through a two-dimensional
space using a Euclidean distance function. This figure
presents the regions delimited
by the paired distances between the three GRs A, B and
C. The shadowed area represents the region where the Figure 8 - The shadowed
GRs prune the objects more r e g i o n s h o w s t h e
effectively (see figure 7). The circumscribed area posed by
distance between each object the global representative A,
B, C.
inside the shadowed area to
each of the three GRs is less
than the maximum distance between the GRs. This can be
expressed by the following definition.
Definition 3 - Circumscribed objects: Let mdj=max
{ d(gj, gk) | gkggj, ~gj, gk * } be the maximum
distance from each global representative gj to all
the other global representatives gk. We say that an
object si that follows d(si, gj)mdj , ~gj * is
circumscribed by the GR set *.
The set * has a strong ability to prune its circumscribed
objects because the intersection of the corresponding rings
generated by queries in this region tends to be minimized.
This prunability gradually decreases as the distance of an
object outside this region increases. This point is
illustrated in figures 7(b) and 7(c), which show the same
-8space and the global representative set * as those of figure
8. The objects in the shadowed area of figure 8 are those
circumscribed by * = { A, B, C }. Figure 7(b) shows the
intersection provided by the GRs A, B and C to solve a
query centered on the object sq2 that is not circumscribed by
them, but close to at least one GR. Figure 7(c) shows the
intersection provided by * to solve a query centered on the
object sq3, which is also uncircumscribed and is far away
from every GR. As can be seen, the intersection of the
rings in figure 7(c) is much larger than the one in figure
7(b). It can be argued that this is true only for uniformly
distributed datasets. However, in practical situations, the
density of queries in a given region follows approximately
the density of the dataset in that region. Hence, the gross
result is independent from the density of the dataset over
the whole space. The desired algorithm can be built based
on this characteristic, and is called the WU (when to
update) algorithm.
Note that, because the cardinality p of * is at least 2,
the md set always can be calculated. The mdj value is
associated to the global representative gj, and can easily be
calculated as * is created. Since the distance of each object
to the global representatives is calculated, no extra distance
calculation is required to determine if it is circumscribed by
the current *. This allows for the development of an
inexpensive algorithm to determine when the current * is
no longer an efficient pruner, which is done by checking
every newly inserted object and verifying how many of
them are uncircumscribed by the current *. The algorithm
to change * is triggered whenever this number exceeds a
given limit. Objects that are uncircumscribed by the
current * and are far away from the circumscribed region
have stronger negative impact on prunability than do those
that are in closer proximity (as shown in figures 7(b) and
7(c)).
Therefore, we propose to count each
uncircumscribed object si by a weighted value, which
depends on the distance of the object to each GR gj,
p
calculated as
w =
p
∏
j =1
d ( si , g j )
md
.
As objects
j
close to GRs lead to rings with smaller intersections (there
are fewer false alarms) than do objects far away from all
GRs, this weighted value is a better choice. This weighted
counting of new objects inserted in the tree is restarted
whenever a new GR set is calculated, so it is associated
with the tree.
The trigger limit (threshold) used is left as a tuning
parameter of the DF-tree and can be experimentally
determined for each data domain. We found empirically
that using the number of objects already indexed when the
current * was chosen is a good starting point for the
threshold. Note that this value is independent from the
absolute scale of the dataset. Figure 9(a) presents the
algorithm to calculate the md set, while figure 9(b) shows
the WU algorithm to detect when the current * no longer
effectively prunes sub-trees.
D Calculate the md set (executed when a new GR set
*
is chosen)
input: the set of distances d(gj, gk), gj, gk new *
output: the set md
Begin
1 - For each Global Representative gj :
2Set mdj=0
3For each Global Representative gk distinct from
gj:
4If d(gj, gk) > mdj then set mdj=d(gj, gk)
end
E
WU Algorithm: Detect when to update * (executed
when an object is inserted in the tree)
input: the set md with cardinality p, the set of distances
d(gj, si) from the object si to each GR gj, the threshold
th, and the accumulated number of new
uncircumscribed objects (cnc). The value cnc is set to
null whenever a new * is calculated.
output: the updated number cnc. Returns true if * requires
changing, false otherwise
Begin
1 - Set w=1
2 - For each Global Representative gj :
3Set w = w*d(gj, si) / mdj
4 - Set cnc = cnc + (w)1/ p
5 - If cnc > th returns true; otherwise return false
end
Figure 9 - (a) Algorithm to calculate the md set. (b) The WU
algorithm.
How to choose a new GR set * : the HU algorithm
Here we describe an algorithm to update the set *, called
the HU (how to update) algorithm. Let us assume that a
new * with cardinality p has to be chosen from a set of
objects. As the algorithm computes p+1 distances for each
object in the candidate set, it is useful to reduce this set to
the minimum possible. The set * is composed of objects
close to the “border” of the dataset. Therefore, objects
already circumscribed by the current * are not candidates.
Thus, the candidate set includes the objects si in the dataset
whose distance to a global representative j is greater than
the mdj, i.e., d(si, gj)mdj to at least one global
representative gj.
If the uncircumscribed objects are concentrated in a
specific region and are unevenly distributed around the
current *, then the new * may put down uncircumscribed
the objects that are currently circumscribed. To overcome
-9this problem, the current * is also included in the
candidate set. The candidate set is then processed by the
usual HF algorithm to identify the representatives close to
the border of this set. The speed of this process is enhanced
when only the uncircumscribed objects plus the current *
are used as the candidate set instead of the full set of stored
objects, which would be the normal procedure. In fact, our
experiments have shown that the same objects are often
selected, whether the whole dataset or only this restricted
set is used. The HU algorithm is described in figure 10.
HU Algorithm: Update *
(executed when the WU algorithm triggers)
input: the current *, its cardinality p, and the DF-tree
output: the new *
Begin
1 - Set the “candidate set” to empty
2 - Traverse the DF-tree:
3For each object si stored at a leaf node:
4If for any i, 1<j<p, d(si, gj)mdj, then include si
in the candidate set
5 - Include * in the candidate set
6 - Execute the HF Algorithm on the candidate set
End
Figure 10 - The HU algorithm to update G.
4.5 - Properties of the DF-tree
In addition to optimizing the query answering performance,
the global representatives have the interesting
characteristics which lead to the following DF-tree
properties:
Property 1: The low costs involved in choosing and
changing the global representatives *.
Property 2: The GR set * can either be used or not. It
should be pointed out that, regardless of the existence
of * a query can be answered without using it. This is
an important feature in a concurrent environment,
because it allows the system to continue answering
queries even if a change of is in progress.
Property 3: Any number of the existing global
representatives can be used. The greater the number of
reference points used up to the limit given by M'N, the
higher structure’s prunability.
However, in
combination with the previous property,
can be
updated by incrementally changing the distances from
the objects to each GR one at a time, thereby taking
advantage of the idle time of the database manager.
Property 4: A can be replaced without affecting the tree
structure, because the global representatives are not
used to construct the tree. That is why the DF- tree
remains a dynamic structure even using global
representatives.
*
*
*
5 - Experimental results
This section discusses the results of the experiments
performed to evaluate the effectiveness of the DF- tree. The
DF-tree was implemented in C++ language, and the
experiments were ran in a Pentium III 800 Mhz with 256
Mbytes of RAM. The I/O device was an E-IDE Ultra
ATA/66 7200RPM hard disk. Although work was done on
a variety of both real and synthetic datasets, for reasons of
space limitations, in this paper we show measures obtained
with the following three real datasets.
EnglishWords - A set of 25,143 words from the
English dictionary, using the Ledit distance
function. As its intrinsic dimensionality is '=4.75
[13], the cardinality of used for this dataset is 5.
Faces - A set of 11,900 faces from the Informedia
project at Carnegie Mellon University. The faces
are described by 16-dimensional vectors, using the
Euclidean distance function. With '=5.27 , the
cardinality of is 6.
PortugueseWords - A set of 214,717 words from the
Portuguese dictionary. The distance function used
is Ledit, '=6.69, and the cardinality of used for
this dataset is 7.
The number of distance calculations and the number of
disk accesses shown in this section represents the average
of the values obtained from sets of 500 queries. For
example, distance plots of range queries represent the
average number of distance calculations for 500 queries in
each query radius or database size shown in the plot. Time
measurements show the total time (in seconds) spent to
answer each set of 500 queries. The selection of each query
set is biased, i.e., objects sampled from the full dataset were
used as query centers. The node size of every structure is
4 kbytes.
*
*
*
5.1 - A comparison between the DF-tree, the Slim-tree,
the M-tree and the Omni-sequential
This section describes the results obtained comparing the
DF-tree with the Slim-tree, the M-tree, the Omni-sequential
and the sequential scanning. Figure 11 shows the results
from the comparison of the performance of the four MAMs
for the EnglishWords dataset to answer 500 range queries
with a radius equal to one, as the dataset is stored in ten
percentile increments of the full dataset. This figure shows
the plots of (a) the average number of distance calculations,
(b) the average number of disk accesses, and (c) the total
time. Figure 11(c) does not include a plot for the M-tree
because it is implemented on another framework with
timing much slower rendering a comparison of time unfair
to the M-tree. As can be seen, all the MAMs display a
linear behavior; however, the slope representing the
number of distance calculations in the DF-tree plot is
significantly less steeper. In fact, the difference in the
- 10 number of distance calculations performed by the DF-tree
is quite significant compared to the number performed by
the others. The number of disk accesses of the DF- tree and
the Omni-sequential are equivalent and both show a better
performance here than the M-tree, whereas, from this
standpoint, the Slim-tree excels, confirming its original
purpose. However, considering the overall time factor,
which ultimately summarizes disk, distance and internal
logic, the DF-tree is clearly the winner, performing almost
twice as fast as the Omni-sequential, which takes second
place. We performed the same experiments for radii 2,4
and 5 and got equivalent results. It must be noted that the
most common queries are usually on small radii.
Figure 11 - Comparing the performance to answer 500 range
queries on the EnglishWords dataset indexed on the DF-tree, the
Slim-tree, the Omni-sequential, the M-tree and Sequential Scan,
for increasing dataset sizes. (a) Average number of distance
calculations per query, (b) Average number of disk accesses per
query. (c) Total time (500 queries) in seconds.
5.2 - Comparing the DF-tree and the Slim-tree
This section discusses the experiments involving
comparisons between the DF-tree, the Slim-tree, and the
sequential scanning. It will be shown that the DF-tree
Figure 12 - Distance calculations and total time comparing
Sequential Scanning and Slim-tree with DF-tree for the following
datasets: EnglishWords: (a) Average number of distance
calculations for each range query. (b) Average number of
distance calculations for each k-NN query. (d) Total time for 500
range queries for varying range radii; (e) Total time for 500 k-NN
queries varying the number of neighbors. Faces: (c) Average
number of distance calculations for each range query. (f) Total
time for 500 range queries for varying range radii. The averages
are for 500 queries.
requires fewer distance calculations to answer both range
and k-NN queries. Figures 12 and 13 show measurements
comparing the average number of distance calculations and
the total time required by both Slim-tree and DF-tree to
answer the same set of range and k-NN queries over the full
dataset. Figure 12(a) shows the average number of distance
calculations for range queries on the EnglishWords dataset,
while figure 12(d) shows the total time for these
experiments at each given range radius. Figure 12(b)
shows the average number of distance calculations for kNN queries in the EnglishWords dataset, and figure 12(e)
shows the total time for these experiments at each given
number of neighbors. Figure 12(c) shows the average
number of distance calculations for range queries in the
Faces dataset, while figure 12(f) shows the total time for
these experiments at each given range radius. These
graphs show the meaningful improvement achieved by the
DF-tree in such situations. The results obtained are
corroborated by the measurements got from the prunability
of these MAMs, as is shown in Section 5.3.
Figure 13(a) presents the average number of distance
calculations for k-NN queries in the full Faces dataset,
regarding the Slim-tree and the DF-tree, emphasizing that
both trees present a sub-linear behavior; in figure 13(d) the
sequential scanning results are also given. Figure 13(c)
depicts the total time for these experiments for different
numbers of neighbors for both trees and the same sub-linear
behavior is achieved. Figure 13(f) includes the results for
the sequential scanning. Figure 13(b) and (e) shows the
average number of disk accesses for k-NN queries on the
Faces dataset. As predicted, the average number of disk
accesses required by the DF-tree is greater than that
required by the Slim-tree. However, the gain in number of
distance calculations offsets this increase, and the total time
shown by figure 13(c) and (f) confirms the superior overall
performance of the DF-tree.
Figures 12 and 13 show the impressive drop in the
Figure 13 - Comparing the Slim-tree, the DF-tree and the
sequential scan, for 500 k-NN queries on the Faces dataset, for
varying numbers of neighbors. (a), (d) average number of
distance calculations, (b), (e) average number of disk accesses,
(c), (f) total time (in seconds).
- 11 average number of distance calculations and the total time
required to answer similarity queries, confirming the fact
that the DF-tree outperforms the Slim-tree by requiring less
than 25% of the number of distance calculations as required
by the Slim-tree. The DF-tree also processes small radii in
range queries up to 3 times faster than the Slim-tree does.
The same behavior is obtained for few neighbors in k-NN
queries over small portions of the stored dataset. Queries
over small portions of the stored dataset are usual in
practical situations. Moreover, the graphs show that the
sub-linear behavior of the DF-tree with increasing radii
sizes or number of neighbors is maintained.
5.3 - Prunability
Table 2 shows the prunability of the DF-tree compared to
that of the Slim-tree. Each measurement represents the
average pruning obtained by the application of 500 range
queries with a radius equal to 1% of the dataset diameter
for the Faces dataset and one letter of distance for the
EnglishWords dataset. Both radii are examples of small
values, which are the most common in real queries. We
tested other small values and the results were similar, so
they are not presented herein. The measurements also
included 500 k-NN asking for 5 neighbors. It must be
noted that the DF-tree requires one level more than the
Slim-tree to store the 25,143 objects of the EnglishWords
dataset. From this table one can see that, using only the
node representative, the Slim-tree does almost no pruning
at the leaf level (level 2). Nonetheless, the DF-tree
maintains considerable prunability.
Range queries
English
Words
Slim- DFtree tree
0 (root) 1.00 1.00
1
0.71 0.94
2
0.004 0.56
3
0.18
level
Faces
Slimtree
0.73
0.33
0.005
DFtree
0.99
0.76
0.51
k-Nearest neighbors
queries
English
Faces
Words
Slim- DF- Slim- DFtree tree tree tree
1.00 1.00 0.75 0.82
0.89 0.98 0.35 0.76
0.001 0.80 0.02 0.31
0.07
Table 2 - Prunability of the Slim-tree compared with
prunability of the DF-tree.
Consider, for example, the prunability at the leaf level
of the tree built over the Faces dataset when range queries
are executed. In this case, for each set of 200 distance
calculations that the single node representative of the Slimtree cannot prune (prunability=0.005), an average of only
one will find an object that is really part of the answer. In
contrast, for every two distance calculations that the node
representative and the global representatives of the DF-tree
cannot prune (prunability=0.56), an average of more than
one is part of the answer. The difference in prunability is
less dramatic at the upper levels of the tree; however, since
pruning of whole branches provided at the upper levels is
more valuable, the impact of increased prunability is always
worthwhile.
5.4 - Scalability
The DF-tree is linearly scalable regarding the dataset size,
the number of distance calculations, the number of disk
accesses and time to answer range queries. Figure 14
illustrates this statement for our largest dataset, the
PortugueseWord, with increments of 10% up to the full
dataset. The values shown represent the average number of
distance calculations, the average number of disk accesses,
and the total time to answer 500 range queries with a radius
rq=1.
Figure 14 - Linear behavior of the DF-tree when answering 500
range queries with radius 1 on the PortugueseWords dataset,
regarding: (a) average number of distance calculations, (b)
average number of disk accesses, (c) total time.
5.5 - Automatic fine-tuning provided by the WU
algorithm
The algorithm WU detects the downgrading of the after
considerable changes on the tree. Our real datasets are
randomly distributed over its entire size. Selecting at the
beginning of the tree construction, usually brings a poor
set, thus the WU algorithm should find a better as the tree
grows. Figure 15(a) shows the plots for the average
number of distance calculations with increasing size of the
database, when answering range queries of radius 1 on the
DF-tree for the EnglishWords dataset. The first plot shows
the average number of distance calculations using the earlychosen , and the second plot shows this number when the
algorithm is allowed to update the set with a threshold of
2000. In this second plot the algorithm triggered two
*
*
*
*
*
Figure 15 - Range queries of radius 1 on the EnglishWords
dataset without changing representatives, and changing
representatives using a threshold equal to 2000. (a) average
number of distance calculations, and (b) total time.
- 12 times: after 1,083 objects were inserted (before the first
point in the plots of figure 15(a) was measured), and after
24,510 objects were inserted (before the last point in the
plots of figure 15 was measured). The time spent to update
at the first time was 0.2 sec and the second time was 1.88
sec. As we can see, even for a dataset that has no emphatic
change in its tendency, the improvement is steady over a
large range of dataset size, both in number of distance
calculations and time. Note that the time measurements,
shown in figure 15(b), follow the graphs in figure 15(a).
data. The concept of the use of global representatives to
prune distance calculations accelerates query processing to
make it practical, without, however, increasing the
complexity of the characteristics yet to be developed.
*
6 - Conclusions
This paper presented new techniques aiming to improve the
efficiency of MAMs to answer similarity queries. Based on
them, we developed the DF-tree, which is dynamic and
takes advantage of using multiple global representatives.
The DF-tree offers tremendous speed-ups, being up to 3
times faster than the state of the art, in wall-clock time. The
gain in the number of distance calculations is also
impressive, requiring less than a quarter of the
computations to answer similarity queries.
This
improvement is achieved by taking advantage of a set of
global representatives, which increase the prunability
without interfering with the construction algorithm of the
tree, because they only have to be calculated when the first
query is asked. Moreover, the set of global representatives
can be changed at any time without ever disrupting the
response to ongoing queries over the tree. Additional
contributions of this paper are:
- an algorithm to automatically detect when the global
representatives require updating;
- an inexpensive algorithm to update the set of global
representatives.
We have also presented the new resource, called
“prunability”, whose purpose is to evaluate how efficient a
set of representatives is to prune distance calculations at
each level of the tree. Using this resource, we have found
that the use of a single representative works well at the
upper levels of the existing MAMs, but it is less effective at
lower levels. However, the proposed DF-tree is capable of
continuously pruning at high rates at every level of the tree.
Metric access methods have been intensely developed
in recent years and now, with the proposed DF-tree, they
have reached a level of performance that qualifies them for
inclusion among the indexing methods used in current
commercial database management systems, broadening
their support to include metric datasets. The DF-tree
makes possible to support queries by content over large sets
of images, time sequences and genetic data. This support
still requires further development to enable MAMs to
operate in the open transactional environment of
commercial DBMS, such as concurrent operations,
generation of and recovery from logs, and to support
interactions with selection clauses involving non-metric
7 - References
[1] S. Berchtold, D. A. Keim, H.-P. Kriegel, “The X-tree: An
Index Structure for High-dimensional data,” in VLDB 1996,
pp. 28-39.
[2] K. Beyer, J. Godstein, R. Ramakrishnan, U. Shaft, “When is
"Nearest Neighbor" Meaningful?,” in ICDT 1999, pp. 217235.
[3] T. Bozkaya and Z. M. Özsoyoglu, “Distance-Based Indexing
for High-Dimensional Metric Spaces,” in ACM SIGMOD
1997, pp. 357-368.
[4] S. Brin, “Near neighbor search in large metric spaces,” in
VLDB 1995, pp. 574-584.
[5] W. A. Burkhard and R. M. Keller, “Some Approaches to BestMatch File Searching,” CACM, vol. 16, pp. 230-236, 1973.
[6] K. Chakrabarti and S. Mehrotra, “The Hybrid Tree: An Index
Structure for High Dimensional Feature Spaces,” in IEEE
ICDE 1999, pp. 440-447.
[7] P. Ciaccia, M. Patella, P. Zezula, “M-tree: An efficient access
method for similarity search in metric spaces,” in VLDB
1997, pp. 426-435.
[8] C. Faloutsos, B. Seeger, A. J. M. Traina, C. Traina, Jr.,
“Spatial Join Selectivity Using Power Laws,” in ACM
SIGMOD 2000, pp. 177-188.
[9] V. Gaede and O. Günther, “Multidimensional Access
Methods,” ACM Computing Surveys, vol. 30, pp. 170-231,
1998.
[10] N. Katayama and S. i. Satoh, “The SR-tree: An Index
Structure for High-Dimensional Nearest Neighbor Queries,”
in ACM SIGMOD 1997, pp. 369-380.
[11] K.-I. D. Lin, H. V. Jagadish, C. Faloutsos, “The TV-Tree: An
Index Structure for High-Dimensional Data,” VLDB Journal,
vol. 3, pp. 517-542, 1994.
[12] R. F. Santos, Filho, A. J. M. Traina, C. Traina, Jr., C.
Faloutsos, “Similarity Search without Tears: The OMNI
Family of All-purpose Access Methods,” in ICDE 2001, pp.
623-630.
[13] C. Traina, Jr., A. J. M. Traina, C. Faloutsos, “Distance
exponent : a new concept for selectivity estimation in metric
trees,” Research Paper CMU-CS-99-110, March 1999 1999.
[14] C. Traina, Jr., A. J. M. Traina, B. Seeger, C. Faloutsos,
“Slim-Trees: High Performance Metric Trees Minimizing
Overlap Between Nodes,” in EDBT 2000, pp. 51-65.
[15] J. K. Uhlmann, “Satisfying General Proximity/Similarity
Queries with Metric Trees,” Information Processing Letter,
vol. 40, pp. 175-179, 1991.
[16] P. N. Yianilos, “Data Structures and Algorithms for Nearest
Neighbor Search in General Metric Spaces,” in
ACM/SIGACT-SIAM - SODA 1993, pp. 311-321.