One way to think about regression is as a tool that takes a set of predictors and creates a weighted, linear composite that maximally correlates with the response variable. It finds a way to combine multiple predictors into a single thing, using regression weights, and the weights are chosen such that, once the single composite is formed, it maximally correlates with the outcome.

Here’s a simulation to punch that point home.

500 people.

N <- 500

The correlation matrix for three variables, x1, x2, and the outcome, y. The correlation between x1 and x2 is 0.1, the correlation between x1 and y is 0.4, and the correlation between x2 and y is 0.4.

sigma <- matrix(c(1.0, 0.1, 0.4,
0.1, 1.0, 0.4,
0.4, 0.4, 1.0), 3, 3, byrow = T)

The mean for each variable is 0.

mu <- c(0,0,0)

Use the correlation matrix and mean specifications to generate data.

library(MASS)

df <- mvrnorm(N, mu, sigma)

Turn it into a data frame and label it.

df <- data.frame(df)
names(df) <- c('x1', 'x2', 'y')
df$id <- c(1:N) Run regression and print the output. summary(lm(y ~ x1 + x2, data = df)) ## ## Call: ## lm(formula = y ~ x1 + x2, data = df) ## ## Residuals: ## Min 1Q Median 3Q Max ## -2.34415 -0.56538 -0.02533 0.56705 2.47427 ## ## Coefficients: ## Estimate Std. Error t value Pr(>|t|) ## (Intercept) 0.01756 0.03629 0.484 0.629 ## x1 0.31746 0.03660 8.675 <2e-16 *** ## x2 0.35819 0.03784 9.465 <2e-16 *** ## --- ## Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1 ## ## Residual standard error: 0.8091 on 497 degrees of freedom ## Multiple R-squared: 0.282, Adjusted R-squared: 0.2792 ## F-statistic: 97.62 on 2 and 497 DF, p-value: < 2.2e-16 Here’s the kicker: you can think of those weights as optimal functions telling us how to create the composite. Create a composite using the regression weights. library(tidyverse) df <- df %>% mutate(composite_x = 0.33*x1 + 0.4*x2) Those weights provide the maximum correlation between our composite and the outcome. cor(df$y, df\$composite_x)
## [1] 0.5308375

In other words, the above correlation could not be higher with any other set of weights. Regression found the weights that makes the correlation above as large as it can be.

summary(lm(y ~ composite_x,
data = df))
##
## Call:
## lm(formula = y ~ composite_x, data = df)
##
## Residuals:
##     Min      1Q  Median      3Q     Max
## -2.3552 -0.5743 -0.0312  0.5700  2.4773
##
## Coefficients:
##             Estimate Std. Error t value Pr(>|t|)
## (Intercept)  0.01792    0.03625   0.494    0.621
## composite_x  0.92420    0.06612  13.978   <2e-16 ***
## ---
## Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
##
## Residual standard error: 0.8085 on 498 degrees of freedom
## Multiple R-squared:  0.2818, Adjusted R-squared:  0.2803
## F-statistic: 195.4 on 1 and 498 DF,  p-value: < 2.2e-16

Bo$$^2$$m =)