Probabilistic Cardinality Estimation for Complex Predicates Using Neural Sketches in Distributed Query Planners

Abstract

Modern distributed query optimizers rely on cardinality estimates to choose join orders, access paths, and parallelization strategies, yet complex predicates routinely break traditional assumptions of attribute independence and stationarity. This mismatch is most visible when predicates combine correlated filters, user-defined functions, semi-structured attributes, and join-dependent conditions, where errors cascade into misallocated resources and unstable runtimes. This paper develops a probabilistic cardinality estimation framework that couples compact data summaries with neural inference, targeting complex predicates in distributed planners. The approach represents each relation and join boundary through a neural sketch: a learned, communication-efficient embedding that fuses classical sketching intuitions with representation learning over predicate and data features. Cardinality is treated as a random variable conditioned on query structure, predicate text, and relation sketches, enabling calibrated uncertainty estimates alongside point predictions. The estimator is trained from a mixture of data-driven supervision and execution feedback, with objectives that align with planner costs under resource constraints. Integration is addressed end-to-end, including sketch maintenance under updates, distributed inference placement, and risk-aware planning that accounts for estimate variance. The paper analyzes computational complexity, communication budgets, and approximation error, and presents an evaluation methodology emphasizing reproducibility, workload shift, and planner-level outcomes. Results are discussed in terms of robustness to correlations, improved plan stability, and predictable resource usage under realistic distributed execution mechanics.