Controllable Data Generation
Generates high-dimensional and large-scale vector data while preserving the original similarity distribution as much as possible.
SIGMOD 2026 · Filtered Vector Search Benchmark
VecBench: A Controllable Benchmark for Filtered Vector Search
A benchmark suite for evaluating vector databases under structured scalar predicates, controllable vector distributions, adjustable query conditions, and dynamic end-to-end workloads.
Renmin University of China · Tsinghua University
Overview
Why VecBench?
Filtered vector search combines nearest-neighbor retrieval with scalar predicates such as equality, range, and containment filters. Existing benchmarks often use fixed datasets, random filters, or query-only measurements. VecBench is designed to expose how vector databases behave under realistic, controllable, and dynamic workloads.
Adjustable Query Synthesis
Controls selectivity and filter correlation to stress pre-filtering, post-filtering, in-filtering, and expanded-filtering strategies.
Holistic Vrank Metric
Aggregates ranking across initialization, querying, concurrency, incremental loading, updates, and deletion to compare end-to-end behavior.
Framework
Three components, one evaluation loop.
VecBench separates benchmark construction from system execution. The data synthesizer controls vector dimensionality and scale; the query generator builds filtered search workloads with target properties; the executor runs database-specific pipelines and collects metrics.
This structure makes it easier to analyze whether a system is sensitive to vector dimension, dataset scale, filter selectivity, local filter correlation, or dynamic maintenance costs.
Data Synthesizer
Filtered Query Generator
Workload Executor
ConfigurationDatasetsDimensionScale
SchemaTemplateSelectivityCorrelation
RulesMetricsRankingReports
Data Loader
Pre-processing
Call Method
Execution Metrics
Leaderboard
End-to-end results on YFCC 10M
Click a metric to sort the table. Lower is better for T0, latency, T1, T2, T3, and Vrank; higher is better for recall and QPS. Green cells indicate the best value in the column.
N/A indicates that the corresponding result is not reported or not directly comparable in the current setting.
Workflow
Six-phase holistic evaluation
VecBench evaluates both static search quality and dynamic maintenance behavior, then summarizes the result with Vrank.
Initialization
Insert initial data and build the vector index.
T0Query Execution
Run filtered search queries on indexed data.
Recall · LatencyConcurrent Query
Execute multiple filtered search queries concurrently.
QPSIncremental Load
Insert remaining data and maintain indexes.
T1Update
Modify vectors and scalar attributes.
T2Deletion
Delete a portion of the dataset and measure delay.
T3
Team
Authors and contributors
The paper is authored by researchers from Renmin University of China and Tsinghua University. The project implementation also includes two undergraduate contributors.
Paper Authors
Project Contributors
UG
Wenbin Zhu
Renmin University of China
Implementation contributorUG
Bozheng Wen
Renmin University of China
Implementation contributor
Citation
Cite VecBench
Use the following BibTeX entry when referring to the benchmark, paper, or leaderboard.
@article{zhang2026vecbench,
title = {VecBench: A Controllable Benchmark for Filtered Vector Search},
author = {Zhang, Xiang and Zhang, Chao and Fan, Ju and Li, Guoliang and Du, Xiaoyong},
journal = {Proceedings of the ACM on Management of Data},
volume = {4},
number = {3},
year = {2026},
doi = {10.1145/3802125}
}