Vector Databases: What They Are and How They Power AI

Vector databases are often presented as the part that makes an AI application understand your data. That is only partly true.

A vector index can find content with similar meaning. It cannot decide whether a document is current, whether the user is allowed to see it, or whether the retrieved passage actually answers the question. In 2026, the useful conversation is therefore not “which vector database should we buy?” It is “how do we build retrieval that people can trust?”

What a vector database does

An embedding model turns text, images, audio, or other content into a list of numbers called a vector. Items with related meaning tend to land near one another in that vector space.

When someone asks a question, the application embeds the question and searches for nearby vectors. This is semantic search: it can connect “parental leave” with a policy titled “family absence” even when the words do not match.

The distinction is simple:

• Keyword search is strong at exact names, identifiers, dates, and error codes.
• Vector search is strong at paraphrases, intent, and conceptual similarity.
• Hybrid search combines both signals and is usually the safer default.

The retrieval stack that matters in 2026

Production RAG is more than placing document chunks in an index. A practical request passes through several stages:

flowchart LR
  Q[Question] --> A[Identity and access filters]
  A --> K[Keyword retrieval]
  A --> V[Vector retrieval]
  K --> M[Merge candidates]
  V --> M
  M --> R[Rerank]
  R --> C[Relevant context with citations]
  C --> L[Model response]

Each stage solves a different problem:

1. Ingestion cleans content, removes duplicates, preserves source metadata, and splits material into useful units.
2. Access filtering limits retrieval to documents the current user may read.
3. Hybrid retrieval finds both exact and semantically related candidates.
4. Reranking applies a stronger model to a small candidate set and moves the most relevant passages to the top.
5. Grounded generation gives the model a small amount of relevant context and enough source information to cite it.

If answers are poor, changing the vector database is rarely the first fix. Inspect chunk boundaries, stale content, metadata, filters, query rewriting, and reranking first.

Approximate search without the jargon

Comparing a query with every stored vector works for a small collection but becomes expensive as the index grows. Most systems use approximate nearest-neighbour search to inspect a promising part of the collection instead.

You will commonly encounter:

• HNSW, a graph index with a strong speed-and-recall trade-off, at the cost of memory and index build time.
• IVF, which groups vectors and searches the most relevant groups.
• Quantization, which stores compressed representations to reduce memory and cost, then may rerank results using the original vectors.

There is no universally best configuration. Measure recall and latency with your own queries. A fast index that consistently misses the right policy or support article is not performing well.

When vectors are the wrong tool

Vector search is useful for fuzzy discovery over unstructured content. It should not replace systems that already express the question precisely.

Use SQL or a semantic data model for totals, joins, and business metrics. Use ordinary full-text search when exact language matters. Use APIs for live account state. Consider graph retrieval when the answer depends on relationships across many entities rather than a few relevant passages.

A common mistake is to embed database rows and ask a model to calculate from retrieved samples. That produces incomplete and difficult-to-audit answers. Let the database calculate; let retrieval locate explanatory context.

Can PostgreSQL or an existing search engine be enough?

Often, yes.

PostgreSQL with pgvector supports exact search, HNSW, IVFFlat, filtering, sparse and half-precision vectors, quantization, and hybrid patterns. It is a sensible starting point when the source data already lives in PostgreSQL and the team wants one operational system.

Search platforms such as Azure AI Search, Elasticsearch, and OpenSearch are attractive when hybrid retrieval, text analysis, filters, and search operations already matter. Document databases and managed vector services can also be good fits when they align with the team’s existing data and operational model.

Choose with a representative benchmark, not a feature checklist. Include realistic filters, update frequency, index size, concurrency, regional requirements, backup and recovery, and total cost.

Security and freshness are retrieval features

Enterprise content usually comes from several places: document stores, tickets, source repositories, databases, and internal sites. Copying all of it into one index without its permissions creates a data leak waiting to happen.

Store source identity, tenant, owner, access rules, timestamps, and deletion state with every searchable unit. Enforce authorisation during retrieval, not only when content is ingested. Test that revoked access and deleted content disappear promptly.

Freshness also needs a clear policy. Record when a source was read, when an embedding was created, and which embedding model produced it. Re-embedding an entire collection after a model change can be expensive, so version the vectors and plan the migration.

How to evaluate a retrieval system

Start with a small set of real questions and identify which passages should be returned. Include easy questions, ambiguous language, exact identifiers, permission boundaries, and questions the system should refuse to answer.

Track at least:

• Recall at k: did the candidate set contain a relevant passage?
• Ranking quality: did useful passages appear near the top?
• Groundedness: does the answer follow from the retrieved sources?
• Citation accuracy: does each citation support the nearby claim?
• Latency and cost: can the complete path meet the product’s budget?

Evaluate retrieval separately from generation. Otherwise, a fluent model can hide weak search, and strong retrieval can be blamed for a poor prompt.

A practical starting point

For a first production slice, keep the design modest:

1. Pick one domain with clear ownership and access rules.
2. Build keyword and vector retrieval over the same content.
3. Merge the results and add a reranker only if evaluation shows a benefit.
4. Return citations and log the retrieved document IDs.
5. Test permission changes, deletions, and stale content.
6. Expand only after the quality measurements are repeatable.

The vector database is infrastructure. The product value comes from retrieving the right evidence, for the right person, at the right time.

References

• pgvector documentation
• Hybrid search with Elasticsearch
• Vector search overview in Azure AI Search