Advertising Impression Resource Allocation Strategy with Multi-Level Budget Constraint DQN in Real-Time Bidding

How to allocate advertising impressions under budget constraint is one of the leading research issues in Real-Time Bidding (RTB). Traditional methods are mostly grounded on ‘the highest bidder rule’ to optimize revenues of AD Exchange (ADX), which causes Demand Side Platforms (DSPs) to over-consume budget in the early stage and eventually lead to bad revenue of all trading parties. In addition, the long auction sequence in large-scale RTB environments may also bring challenges to existing algorithms. To those problems, we propose a Multi-Level Budget Constraint DQN (MLBC-DQN) framework, which divides the long sequence in RTB environment into several short sequence environments with different budgets’ level and use a deep Q network (DQN) to learn optimal strategy for environments of each level. The final strategy to interact with the original RTB environment of MLBC-DQN is weighted by strategies of all DQNs mentioned above. By weighing various strategies from different perspectives, an MLBC-DQN agent can quickly learn effective strategies to overcome the over-consume budget problem. We evaluate MLBC-DQN in two datasets comparing with traditional allocation strategies and DQN. Results show that the MLBC-DQN can achieve higher revenues of DSPs without prejudice to the income of ADX, and promote the marketing effect of advertising resources.