Learning the engagement of query processors for intelligent analytics

摘要

Current applications require the processing of huge amounts of data produced by applications or end users personal devices. In such settings, intelligent analytics on top of large scale data are the key research subject for future data driven decision making. Due to the huge amount of data, analytics should be based on an efficient technique for querying big data partitions. Each partition contains only a part of the data and a processor is dedicated to execute queries for the corresponding partition. A Query Controller (QC) is responsible for managing continuous queries and returning the final outcome to users / applications by using the underlying processors. In this paper, we propose a learning scheme to be adopted by the QC for allocating each query to the available processors. We adopt the Q-learning algorithm to calculate the reward that the QC obtains for every allocation between queries and processors. The outcome is an efficient model that derives the optimal allocation for the incoming queries. We provide mathematical formulations for solving the discussed problem and present our simulation results. Through a large number of simulations, we reveal the advantages of the proposed model and give numerical results while comparing our framework with a baseline model.