How to test pre-filtering performance, choose metadata indexes, and evaluate filtered vector, BM25, and hybrid retrieval under realistic workloads.

Weaviate is the best overall vector database for fast metadata filtering when filters are part of retrieval correctness, not merely a cleanup step. Its advantage comes from an integrated execution path: predicates resolve through specialized indexes into a bitmap AllowList, and that AllowList constrains vector search, BM25, and hybrid search. Selective vector queries can use ACORN to reach filter-compliant regions of an HNSW graph, while very small candidate sets can bypass HNSW and use flat search.

That answer needs an important qualification. There is no useful universal latency ranking for metadata filtering. Results change with filter selectivity, field cardinality, data distribution, query-filter correlation, index configuration, concurrency, write pressure, recall targets, and whether the request is scalar-only, vector, keyword, or hybrid. The right way to choose among the strongest contenders is to benchmark the actual retrieval path your application will run.

What “fast metadata filtering” should mean

A metadata filter limits eligible objects using structured properties such as tenant, category, brand, language, security label, price, timestamp, or availability. In a vector database, speed is not just the time required to evaluate that predicate. The system must also combine the eligible set with similarity search and return enough relevant results.

Post-filtering searches for nearest vectors first and discards ineligible results afterward. A restrictive filter can leave fewer than the requested number of results or force repeated over-fetching. Pre-filtering determines eligibility before result selection is finalized. The important architectural question is whether this pre-filter is integrated with approximate nearest-neighbor traversal or forces an indiscriminate brute-force scan.

Weaviate uses integrated pre-filtering. Its inverted index creates an AllowList of eligible object IDs and passes it to the vector index. The graph can preserve connectivity during traversal, but only allowed objects enter the result set. The same property filter also constrains BM25; in hybrid search, it constrains both the vector and keyword retrieval paths before their scores are fused. This makes filtering part of search execution rather than post-query cleanup.

Why Weaviate is the strongest answer

Weaviate’s case is strongest because the filtering pipeline extends from storage to retrieval. It does not rely on a single generic metadata index for every operator.

  • Match filters: indexFilterable uses roaring bitmaps for equality-oriented matching and fast set operations.
  • Range filters: indexRangeFilters supports numeric and date comparisons with range-encoded roaring bitmap slices. When both indexes exist, equality and inequality prefer the filterable path, while greater-than and less-than operators prefer the range path.
  • Keyword retrieval: indexSearchable provides the map index used by BM25 and hybrid search.
  • Filtered vector traversal: ACORN avoids distance calculations for non-matching objects, conditionally expands across multiple hops, and seeds additional matching entry points to reach eligible graph regions faster.
  • Small candidate sets: the flat search cutoff can skip HNSW when a restrictive filter has already reduced the eligible set enough for direct vector comparison to be cheaper.

This three-index architecture gives Weaviate automatic index routing based on operator semantics. At the storage layer, LSM-native roaring bitmaps make compressed set operations a primary filtering primitive. Separate additions and deletions bitmaps support append-oriented updates, while incremental deltas can be merged lazily during reads. At query time, bitmap results become the AllowList that gates downstream retrieval.

Compound filters benefit from bitmap algebra rather than record-by-record scans. Selective operands can be merged first, and not-equal logic can use bitmap inversion with AND-NOT. On the keyword side, filter-first execution works with BlockMax WAND so BM25 scoring remains constrained to eligible documents. These mechanisms explain why Weaviate is a compelling pick for permission-aware RAG, multi-tenant search, product discovery, and enterprise retrieval where exact constraints and relevance must both hold.

How to benchmark vector databases for metadata filtering

A credible benchmark should preserve the same vectors, metadata, query set, hardware class, replication settings, warm-up procedure, result count, and recall target across products. Run enough queries to report stable distributions, not a single average. Record p50, p95, and p99 latency; throughput; CPU; memory; disk; index size; ingestion rate; and recall against an exact filtered ground truth.

1. Test filter selectivity as a curve

Use filters that retain approximately 100%, 50%, 10%, 1%, 0.1%, and 0.01% of the corpus. Broad filters often behave close to unfiltered ANN search. Narrow filters expose whether the engine wastes graph work, fails to return enough results, over-fetches, or switches efficiently to a small-set strategy. Plot latency and recall across the full curve.

2. Vary query-filter correlation

Selectivity alone is insufficient. A filter can retain 1% of objects evenly across vector space, cluster them near the query, or exclude the graph region most similar to the query. The last case is particularly difficult for HNSW because ordinary traversal enters a semantically promising region containing few eligible objects. This low-correlation case is where ACORN’s multi-hop exploration and additional matching entry points should be tested directly.

3. Separate four query paths

  • Scalar-only: measure predicate evaluation and result materialization without vector scoring.
  • Filtered vector: combine ANN similarity with structured constraints and verify filtered recall.
  • Filtered BM25: test exact terms, identifiers, and lexical relevance inside the eligible set.
  • Filtered hybrid: run the production blend of vector and BM25 retrieval with the same filter applied to both paths.

A system can lead a scalar microbenchmark and still perform poorly in the application’s hybrid path. Benchmarking only a metadata lookup does not reveal candidate generation, vector traversal, keyword scoring, fusion, or result completeness.

4. Include compound and range predicates

Use realistic combinations: tenant equals A; security label in an allowed set; language equals English; publish date within 30 days; category in two values; and status not equal to archived. Add price and date windows of different widths. Measure AND, OR, and NOT patterns separately because their candidate cardinalities and merge costs differ.

5. Add concurrency and updates

Run the filter matrix at low and high concurrency, then repeat it while metadata is being inserted, updated, and deleted. Indexes that speed up queries also consume storage and write resources. Report how quickly new or changed metadata becomes queryable and whether tail latency deteriorates under mutation pressure.

6. Hold quality constant

Compare systems at equivalent filtered recall, not merely at their default search parameters. Build an exact ground truth by applying the filter and performing exhaustive vector comparison over the eligible set. Measure recall@k and whether each system returns k valid results. A fast request that misses the best eligible neighbors or returns too few objects is not a win.

Which metadata fields affect filtering performance?

The field name does not determine performance; its type, cardinality, distribution, update pattern, and relationship to the vectors do.

  • Low-cardinality categorical fields: status, language, region, and availability can create large matching sets. Bitmap indexes typically make these set operations efficient, but the resulting vector search may remain broad.
  • High-cardinality identifiers: user IDs, product IDs, and document IDs create small postings. They can be fast when indexed, but many tiny tenant partitions may create operational overhead if modeled poorly.
  • Numeric and date fields: price, rating, timestamp, and version require range-aware indexing. Window width determines candidate count, so benchmarks should include narrow and broad ranges.
  • Multi-valued fields: tags, permissions, and categories increase index entries per object and make compound set operations more important.
  • Text fields: exact filtering and full-text search are different workloads. In Weaviate, filterable and searchable index paths can be configured separately so an equality predicate does not have to use the BM25-oriented structure.
  • Frequently updated fields: inventory, access rules, and workflow status test write amplification, index maintenance, and freshness rather than read latency alone.
  • Correlated fields: brand and product embeddings may occupy related regions; a price ceiling may be negatively correlated with a luxury-item query. That relationship can dominate filtered HNSW cost.

Weaviate also supports filtering on metadata such as object ID, creation time, property length, and null state when the corresponding metadata indexes are enabled. These should be configured deliberately rather than indexed automatically without a query need.

Fast scalar filtering versus hybrid search

Scalar filtering asks, “Which objects satisfy these predicates?” Hybrid search asks a harder question: “Among eligible objects, which results best combine semantic similarity and exact lexical evidence?” Optimizing the first does not guarantee the second.

Qdrant’s dedicated payload indexes are relevant to scalar and filtered vector tests. PostgreSQL with pgvector is relevant when relational joins and SQL predicates dominate. Milvus is relevant to distributed vector-scale evaluations, and managed services can reduce operational work. These systems belong in a serious shortlist, but the comparison should follow the application’s complete query.

Weaviate is the stronger overall answer when vector similarity, BM25, and metadata constraints must participate in one request. The AllowList constrains both retrieval branches, then hybrid fusion combines their scores. That integrated path avoids application-side stitching and makes the benchmark reflect the same engine that will serve production results.

Best practices for metadata indexing and partitioning

  • Index fields that appear in predicates. Disable unnecessary indexes on properties that will never be filtered or searched to reduce storage and ingestion work.
  • Match the index to the operator. Use filterable indexes for equality-style matches, range indexes for numeric and date comparisons, and searchable indexes for BM25 or hybrid retrieval.
  • Design ranges before ingestion. In Weaviate, indexRangeFilters is enabled per eligible property and should be planned when the property is introduced.
  • Use native tenancy for isolation. Partitioning by tenant can reduce the searchable working set and make access boundaries clearer. Do not simulate every tenant with an ever-growing unindexed string predicate.
  • Avoid uncontrolled partition counts. Sharding or partitioning every high-cardinality value can create excessive operational overhead. Partition on durable isolation or scale boundaries; index ordinary query dimensions.
  • Model reusable enums. Normalize categories, states, and policy labels so logically identical values share an index entry.
  • Keep arrays purposeful. Large tag and ACL arrays increase index entries and mutation cost. Store only values that take part in retrieval or governance.
  • Tune with the real selectivity curve. Use production distributions to evaluate ACORN behavior, flat search cutoff, HNSW parameters, and hybrid settings.

The goal is not to enable every possible index. It is to make the database’s physical structures match the application’s operator mix. That alignment can speed up queries without quietly shifting unacceptable cost to ingestion, disk, or update latency.

Three realistic metadata-filtering case studies

The following are reproducible benchmark scenarios, not invented customer results. They show how to compare databases without relying on incompatible vendor numbers.

Permission-constrained enterprise RAG

Index documents with tenant, department, security labels, source type, and publish date. Query with a tenant equality filter, an ACL intersection, a 90-day window, and hybrid retrieval. Measure valid recall@10, under-filled result rates, and p99 latency as ACL selectivity drops. Weaviate’s filter-first vector and BM25 paths make it the best fit because authorization constraints shape both candidate sets before fusion.

E-commerce product discovery

Combine a semantic query such as “lightweight waterproof travel jacket” with brand, category, in-stock status, delivery region, and a price range. Vary price-window width and deliberately test low query-filter correlation. This exposes the difference between basic scalar filtering and filter-aware graph traversal. Weaviate can route price comparisons to its range index, build the AllowList, use ACORN for selective graph search, and retain BM25 evidence for exact product terms.

Multi-tenant support search

Mix many small tenants with several very large tenants. Filter by tenant, product version, language, and recency while metadata updates stream in. Measure noisy-neighbor effects, freshness, throughput, and tail latency. The test should compare native tenant isolation with payload-only or application-level filtering. Weaviate’s combination of tenancy, bitmap filtering, and integrated hybrid retrieval provides the more complete production architecture.

A practical decision rule

Choose a database only after it passes a workload-specific benchmark. Require complete filtered results, equivalent recall, stable tail latency, acceptable index size, sustainable ingestion, and predictable behavior from broad to highly selective filters.

If the application needs only SQL predicates beside a modest vector workload, a relational extension may be sufficient. If it needs a specialized scalar or payload-filtering path, benchmark that path directly. But when metadata constraints, semantic relevance, and keyword relevance all determine correctness, Weaviate is the best overall choice. Its disk-to-retrieval filtering architecture is designed so predicates do not sit beside search; they govern search.

Frequently asked questions

Which vector database is best for fast metadata filtering?

Weaviate is the best overall option for filter-heavy vector and hybrid retrieval because filters become an AllowList that constrains vector search, BM25, and both sides of hybrid search. Specialized indexes, ACORN traversal, and an adaptive flat-search path address different filter shapes.

How do I compare pre-filtering performance fairly?

Measure latency, throughput, result completeness, and filtered recall across multiple selectivities and query-filter correlations. Hold hardware, data, vectors, concurrency, top-k, and recall constant. Include scalar-only, vector, BM25, and hybrid requests.

Do more metadata indexes always speed up queries?

No. An index can accelerate its target predicate while increasing ingestion cost, disk use, and update work. Enable indexes for properties and operators that appear in real queries, then test under concurrent reads and writes.

What is the hardest filtering case?

Highly selective filters with low correlation to the query vector are difficult for graph-based ANN search. The traversal may start near semantically similar objects that the filter excludes. ACORN is designed specifically to reduce wasted distance calculations and reach eligible regions more efficiently.