Databricks adds direct AI Search sources beyond Delta tables
Databricks announced native streaming-table, materialized-view and managed-Iceberg sources, eliminating an intermediate copy.
The October 26 announcement includes managed Iceberg version 3 or later. Developers create a synchronized search index against the source instead of maintaining a separate Delta copy and custom update pipeline. Streaming tables hold arriving events; materialized views combine, filter or aggregate records. The company presents these as foundations for fresher inventory search and personalized results, rather than reporting a customer benchmark.
The service documentation explains the retrieval mechanism: embeddings encode the meaning of text or images as numbers. The index holds these representations and associated metadata, allowing an application to retrieve documents similar to a query. Deployment still requires a workspace with Unity Catalog governance and serverless computing enabled. Creating an index also requires permission to create tables in its destination schema. For a materialized view, external metadata must first be enabled on the view or its defining pipeline.
The setup guide requires standard-endpoint sources to use a change data feed (CDF). With row tracking enabled, CDF works without manual configuration for Delta tables, materialized views with external metadata enabled, and managed Iceberg v3+. Continuous synchronization has seconds of latency and higher costs because a processing cluster stays provisioned. Triggered synchronization starts through the interface, SDK or API. Storage-optimized endpoints allow only triggered synchronization, partially rebuilding the index each time; managed indexes reuse embeddings for unchanged source rows.