On Threshold Correlation with Application to Studying the Relationship of Temperature and Relative Humidity

Finding thresholds for continuous variables is an important problem in many applications. For example, when studying the relationship of air pollution and lung diseases it is important to obtain findings such as: "PM2.5 concentrations above 40 mu g/m(3) are associated with increased occurrence of Bronchial Asthma". Finding such thresholds is also critical to come up with governmental regulations and laws to alleviate the health impacts of PM2.5. Developing knowledge discovery frameworks which find such thresholds is therefore important. The key contribution of this paper is the introduction of thresholds to classical correlation analysis; it proposes a novel generalization of classical correlation, called threshold correlation, and assesses the merit of threshold correlation analysis in a case study which centers on understanding the relationship between temperature and relative humidity in locations in five different cities. In contrast to classical correlation, threshold correlation restricts observations for which the correlation is computed using pairs (th1, th2) of thresholds and introduces four forms of threshold correlations named High/High, Low/Low, High/Low, Low/High correlations, excluding observations whose values are above/below the thresholds th1 and th2 before computing the correlation of the remaining observations. For example in the case study, threshold correlation analysis was able to reveal for a Houston location that in the year 2022 there was almost no correlation between humidity and temperature (the correlation is -0.11) but there is a strong negative correlation of -0.80 between relative humidity above 53% and temperatures above 80.98. Finally, the paper discusses the computational challenges of threshold correlation mining and describes work in progress.