Linear Regression - Regression Coefficients and their Properties

Regression Lines and the Point of Intersection: There are two regression lines, the line of $Y$ on $X$ and the line of $X$ on $Y$. These lines represent the best linear fit for predicting one variable given the other. Visually, these two lines always intersect at the point $(\bar{x}, \bar{y})$, which represents the arithmetic means of the two variables.

Nature of Regression Coefficients: The regression coefficients, $b_{yx}$ and $b_{xy}$, are the slopes of the regression lines of $Y$ on $X$ and $X$ on $Y$ respectively. $b_{yx}$ measures the change in $Y$ for a unit change in $X$, while $b_{xy}$ measures the change in $X$ for a unit change in $Y$. On a coordinate plane, the steeper the regression line of $Y$ on $X$, the larger the absolute value of $b_{yx}$.

Sign Consistency Property: The correlation coefficient ($r$) and both regression coefficients ($b_{yx}$ and $b_{xy}$) must always have the same sign. If $r$ is positive, both regression lines slope upwards from left to right; if $r$ is negative, both lines slope downwards. It is impossible for one coefficient to be positive and the other negative.

Geometric Mean Property: The correlation coefficient $r$ is the geometric mean of the two regression coefficients. This is mathematically expressed as $r^2 = b_{yx} \cdot b_{xy}$. When calculating $r = \pm\sqrt{b_{yx} \cdot b_{xy}}$, the sign of $r$ is determined by the sign of the regression coefficients.

The Magnitude Property: The product of the two regression coefficients cannot exceed 1 ($b_{yx} \cdot b_{xy} \le 1$). This implies that if one regression coefficient is greater than 1 in absolute value, the other must be less than 1. This ensures that the correlation coefficient $|r|$ never exceeds 1.

Angle Between the Lines: The angle $\theta$ between the two regression lines indicates the strength of the correlation between the variables. If $r = \pm 1$, the angle is $0^\circ$ and the lines coincide. If $r = 0$, the angle is $90^\circ$ and the lines are perpendicular. As $r$ moves from 0 toward 1, the visual 'gap' between the two lines closes as they rotate toward each other.

Property of Origin and Scale: Regression coefficients are independent of the change of origin but are dependent on the change of scale. If we transform $X$ to $u = \frac{X-a}{h}$ and $Y$ to $v = \frac{Y-c}{k}$, the new coefficient is related to the old one by the ratio of the scales: $b_{yx} = \frac{k}{h} b_{vu}$.