A hybrid graphics/video rate control method based on graphical assets for cloud gaming

In Hybrid Cloud Gaming (HCG), as long as the graphical assets are available at the client side, rendering can be performed locally. However, if the client is not able to render all the frames in time, this may result in the frame rate to drop under the target value. This paper presents an Asset-based frame-level hybrid graphics/video rate control method for HCG, referred to as AHCG, which aims at improving the Quality of Experience (QoE) of players by tapping into the available processing power at the client side, while keeping a steady frame rate for thin clients and taking full advantage of the user’s tolerable delay. In the proposed client/server model, graphics data are intercepted and streamed in an asset-based approach. This approach handles the rate control issue per asset which is defined to be 3D object model data, textures, and shader programs. Rendering a frame on the client side not only maintains original quality for that frame, but it also reduces bandwidth requirements of the entire service by reusing the same assets for different frames. In the proposed method, if the accumulated rendering delay at the client side violates the tolerable delay set by the user, video streaming is used to compensate for the client device’s lack of processing power. Furthermore, quality fluctuation between graphics and video frames is addressed to provide a seamless experience when switching between graphics and video streaming. Several objective and subjective tests are conducted and the experimental results show a 20-fps increase in frame rate while maintaining a minimum value of 58 fps, with a minimum of 0.25 units improvement in the pooled standard deviation of SSIM values, compared to existing HCGs. Also, the subjective tests suggest an average 5.62 percent improvement in MOS compared to best HCG methods.