5  Inference for simple linear regression

This chapter is a work in progress.

Learning goals

• Describe how statistical inference is used to draw conclusions about a population slope
• Construct confidence intervals using bootstrap simulation
• Conduct hypothesis tests using permutation
• Describe how the Central Limit Theorem is applied to inference for the slope
• Conduct statistical inference on the slope using mathematical models based on the Central Limit Theorem
• Interpret results from statistical inference in the context of the data
• Understand the connection between hypothesis tests and confidence intervals

R Packages

• library(tidyverse) (Wickham et al. 2019)

• library(patchwork) (Pedersen 2022)

• library(skimr) (Waring et al. 2022)

• library(broom) (Robinson, Hayes, and Couch 2023)

5.1 Introduction: Access to playgrounds

The Trust for Public Land is a non-profit organization that advocates for equitable access to outdoor spaces in cities across the United States. In the 2021 report Parks and an Equitable Recovery (The Trust for Public Land 2021), the organization stated that “parks are not just a nicety—they are a necessity”. The report details the many health, social, and environmental benefits of having ample access to public outdoor space in cities along with the various factors that impede the access to parks and other outdoor space for some residents.

One type of outdoor space the authors study in their report is playgrounds. The report describes playgrounds as one type of outdoor space that “bring children and adults together” (The Trust for Public Land 2021, 13) and a place that was important for distributing “fresh food and prepared meals to those in need, particularly school-aged children” (The Trust for Public Land 2021, 9) during the global COVID-19 pandemic.

Given the impact of playgrounds for both children and adults in a community, we will focus on understanding variability in the access to playgrounds in this chapter. In particular, we want to (1) investigate whether local government spending is useful in understanding variability in playground access, and if so, (2) quantify the true relationship between local government spending and playground access.

The data in parks.csv includes information on 97 of the most populated cities in the United States (US) in the year 2020. The data were originally collected by the Trust for Public Land and was a featured as part of the TidyTuesday weekly data visualization challenge. The analysis in this chapter will focus on two variables:

• per_capita_expend: Total amount the city spends per resident in 2020 (in US dollars). This is referred to as a city’s “per capita expenditure”.

• playgrounds : Number of playgrounds per 10,000 residents in 2020

Your turn!

Which of the following do you to be true about the relationship between per capita expenditure and the number of playgrounds per 10,000 residents?

• The relationship is positive.
• The relationship is negative.
• There is no relationship.

5.1.1 Exploratory data analysis

The visualizations and summary statistics for univariate and bivariate exploratory data analysis are in Figure 5.1 and ?tbl-parks-unviariate-eda .

Figure 5.1: Univariate exploratory data analysis

Table 5.1: Summary statistics for per capita expenditure and number of playgrounds per 10,000 residents

Variable	Mean	SD	Min	Q1	Median (Q2)	Q3	Max	Missing
per_capita_expend	113.0	72.0	15	65.0	89.0	142.0	399	0
playgrounds	2.8	1.1	1	1.9	2.6	3.6	7	0

The distribution of playgrounds per 10,000 residents (the response variable) is unimodal and right-skewed. The center of the distribution is the median of about 2.6 playgrounds per 10,000 residents, and the the spread of the middle 50% of the distribution (the IQR) is 1.7. There appear to be two potential outlying cities with more than 6 playgrounds per 10,000 residents, indicating high playground access relative to the other cities in the data set.

The distribution of city expenditures (the predictor variable) is also unimodal and right-skewed. The center of the distribution is around 89 dollars per resident, and the middle 50% of the distribution has a spread of about 77 dollars per resident. Similar to the response variable, there are some potential outliers. There are 5 cities with spending greater than 300 dollars per resident.

Figure 5.2: Bivariate exploratory data analysis

From Figure 5.2 there appears to be a positive relationship between a city’s per capita expenditure and the number of playgrounds per 10,000 residents. The correlation is 0.206, indicating the relationship between playground access and city expenditure is not very strong. This is partially influenced by the outlying observations in which there is relatively low city per capita expenditure but high numbers of playgrounds per 10,000 residents.

Linear regression model

To better explore this relationship, we fit a simple linear regression model of the form

$$\begin{array}{}\text{(5.1)}& \text{playgrounds}={\beta }_0+{\beta }_1\text{per\_capita\_expend}+ϵ,ϵ\sim N(0,{\sigma }_{ϵ}^2)\end{array}$$

The model can be fit using the following R code:

parks_model <- lm(playgrounds ~ per_capita_expend, data = parks)

parks_model_tidy <- tidy(parks_model) 

parks_model_tidy |> 
  kable(digits = 3)

Table 5.2: Linear regression model of per capita expenditure and number of playgrounds per 10,000 residents

term	estimate	std.error	statistic	p.value
(Intercept)	2.418	0.214	11.282	0.000
per_capita_expend	0.003	0.002	2.057	0.042

The estimated regression equation from the output in Table 5.2 is

$$\begin{array}{}\text{(5.2)}& \widehat{\text{playgrounds}}=2.418+0.003\times \text{per\_capita\_expend}\end{array}$$

Your turn!

Interpret the slope and intercept in the context of the data.

• Interpret the slope in terms of an additional $100 in per capita expenditure. Which interpretation is more meaningful in practice?

• Does the intercept have a meaningful interpretation? ¹

From the sample of 97 cities in 2020, the estimated slope of 0.003. This estimated slope is likely close to but not the exact value of the true population slope that we would obtain if we had data on every city in the US. Based on the equation alone, we are also not sure if this slope indicates an actual meaningful relationship between the two variables, or this slope is due to random variability in the data. Therefore, we will use statistical inference methods to help answer these questions and use the model to draw conclusions beyond these 97 cities.

5.2 Objectives of statistical inference

Based on the regression output in Table 5.2, for each additional dollar in per capita expenditure, the number of playgrounds per 10,000 residents is expected to be greater by 0.003, on average.

The estimate 0.003 is the “best guest” of the relationship between per capita expenditure and the number of playgrounds per 10,000 residents; however, this is likely not the exact value of the relationship in the population of all US cities. Therefore, we will use statistical inference, the process of drawing conclusions about the population parameters based on the analysis of the sample data.

There are two different types of statistical inference procedures:

• Hypothesis tests: Test a specific claim about the population parameter
• Confidence intervals: A range of values that may reasonably contain the value of the population slope.

In this chapter focuses on statistical inference for the slope ${\beta }_1$, but the general concepts can be applied to a variety of population parameters.

As we’ll see throughout the chapter, a key component of statistical inference is quantifying the sampling variability, sample-to-sample variability in the statistic that is the “best guest” estimate for the parameter. For example, when we conduct statistical inference on the slope of per capita expenditure ${\beta }_1$, we need to quantify the sampling the variability of the statistic ${\hat{\beta }}_1$, the estimated (sample) slope. This is the amount of variability in ${\hat{\beta }}_1$ that would be expected if we repeated the following process many times (1) collect a new sample of data the same size as our sample size ( 97 in this analysis), and (2) use the new sample to fit a model using the per capita expenditure to predict playgrounds per 10,000 residents and obtain an estimate of the slope. The idea is that ${\hat{\beta }}_1$ would not be the same for each new data sample, so we need a way to quantify the variability in these estimated slopes to understand the natural variation. The distribution of these ${\hat{\beta }}_1$ values make up the sampling distribution.

It is not feasible to collect a lot of new samples for this process. Instead, there are two approaches to obtain the sampling distribution, quantify the variability in the estimated slopes, and conduct statistical inference.

• Simulation-based methods: Quantifying the sampling variability by generating a sampling distribution directly from the sample data

• Theory-based methods: Quantify the sampling variability using mathematical models based on the Central Limit Theorem

Section 5.4 and Section 5.6 introduce statistical inference using simulation-based methods, and Section 5.8 introduces inference using theory-based methods. Before we get into those details, however, let’s introduce more of the foundational ideas underlying simple linear regression and how they relate to statistical inference.

Terminology

• Statistical inference: Using sample data to draw conclusions about a population.

• Population parameter: Quantitative measure of a feature of the population (for example, the slope for the population)

• Sample statistic: Estimate of the population parameter using the sample data (also called the point estimate)

• Sampling distribution: Distribution of the sample statistic across many samples.

• Sampling variability: Variability in the sampling distribution. This can be measured using the standard deviation, variance, IQR, etc.

5.3 Foundations of simple linear regression

In Section 4.3.1, we introduced the statistical model for simple linear regression $$\begin{array}{}\text{(5.3)}& Y={\beta }_0+{\beta }_{1}X+ϵϵ\sim N(0,{\sigma }_{ϵ}^2)\end{array}$$

such that $Y$ is the response variable, $X$ is the predictor variable, and $ϵ$ is the error term. Equation 5.3 can be rewritten in terms of the distribution of the response variable $Y$ given the predictor $X$. It is represented as $Y|X$ ( “$Y$ given $X$”)

$$\begin{array}{}\text{(5.4)}& Y|X\sim N({\beta }_0+{\beta }_{1}X,{\sigma }_{ϵ}^2)\end{array}$$

We conduct simple linear regression assuming Equation 5.4 is true. Equation 5.4 is the assumed distribution of the response variable conditional on the predictor variable under the simple linear regression model. Based on the equation we specify the assumptions that are made when we do simple linear regressions.

1. The distribution of the response $Y$ is normal for a given value of the predictor $X$.

2. The mean of the distribution of $Y$ given $X$ is ${\beta }_0+{\beta }_{1}X$. There is a linear relationship between the response and predictor variable.

3. The variance of the distribution of $Y$ given $X$ is ${\sigma }_{ϵ}^2$. This variance does not depend on $X$. It is equal for all values of $X$.

4. The error terms $ϵ$ in Equation 5.3 are independent of one another. This also means the observations are independent of one another.

Whenever we fit linear regression models and conduct inference on the slope, we do so under the assumption that some or all of these four statements hold. In Chapter 6, we will discuss how to check if these assumptions hold in a given analysis. As we might expect, these assumptions do not always perfectly hold in practice, so we will also discuss circumstances in which each assumption is necessary versus when some can be relaxed. For the remainder of this chapter, however, we will proceed as if all these assumptions hold.

(section title)

5.4 Bootstrap confidence intervals

A confidence interval is a range of values that may reasonably contain the value of the population slope. This range is computed from the sample data. By computing this range, we are more likely to capture the value of the true population parameter than if we only used the estimated sample statistic (called a point estimate). We will focus on confidence intervals for ${\beta }_1$, the population slope. The confidence level $C\mathrm{\%}$ will help determine the width of this range of values. .

In order to obtain this range of values we must understand the sampling variability of the statistic. Suppose we repeatedly take samples of size $n$ (the same size as the sample data) and fit regression models to compute ${\hat{\beta }}_1$, the estimated slope. The sampling variability is the variability in these estimated slopes. In practice, it is generally not feasible to collect multiple samples from the population, so we will use the one sample we’ve collected to simulate the process of obtaining new samples. We generate these new samples by bootstrapping, a simulation process in which we generate a sample of size $n$ by sampling with replacement from the current data set.

We can compute ${\hat{\beta }}_1$ for each bootstrap sample, then analyze the distribution of these estimated slopes. The variability in this distribution is the sampling variability we need to construct the confidence intervals.

Your turn!

Why do we sample with replacement when getting the bootstrap sample? What would the boot strap sample looks like sampling is done without replacement?²

5.4.1 Constructing a bootstrap confidence interval for ${\beta }_1$

A bootstrap confidence interval for the population slope, ${\beta }_1$, is constructed using the following steps:

1. Generate $n_{iter}$ bootstrap samples, where $n_{iter}$ is the number of iterations. We typically want to use at least 1000 iterations in order to construct a sampling distribution that is close to the theoretical distribution of ${\hat{\beta }}_1$ defined in Section 5.8.
2. Fit the linear model to each of the $n_{iter}$ bootstrap samples to obtain $n_{iter}$ values of ${\hat{\beta }}_1$, the estimated slope. There will also be $n_{iter}$ values of the estimated intercept, ${\hat{\beta }}_0$, but we will ignore those for now because we are not focusing on inference for the intercept.
3. Collect the $n_{iter}$ values ${\hat{\beta }}_1$ from the previous step to obtain the bootstrapped sampling distribution. It is an approximation of the sampling distribution of ${\hat{\beta }}_1$, and thus provides information about the to sampling distribution of ${\hat{\beta }}_1$.
4. Use the distribution from the previous step to calculate the $C\mathrm{\%}$ confidence interval. The lower and upper bounds of the interval are the points distribution that mark the middle $C\mathrm{\%}$ of the distribution.

Using these four steps, we can construct the 95% confidence for the population slope ${\beta }_1$.

1. Generate 1000 bootstrap samples (97 observations in each sample) by sampling with replacement from the current sample data of 97 observations. The first 10 observations from the first bootstrapped sample are shown in Table 5.3.

Table 5.3: First 10 rows of the first bootstrap sample. The replicate column identifies which bootstrap sample.

replicate	playgrounds	per_capita_expend
1	2.1	320
1	1.8	65
1	2.2	67
1	1.0	33
1	2.6	42
1	2.2	149
1	3.3	73
1	2.2	35
1	1.8	89
1	1.3	65

Your turn!

Why are there 97 observations in each bootstrap sample?³

2. Next, we fit a linear model of the form in Equation 5.1 to each of the 1000 bootstrap samples. The estimated slopes and intercepts for the first three bootstrap samples are shown in Table 5.4.

   Table 5.4: Estimated slope and intercept for the first two bootstrap samples.

   replicate	term	estimate
   1	intercept	2.3825970
   1	per_capita_expend	0.0049071
   2	intercept	2.5451734
   2	per_capita_expend	0.0023708
   3	intercept	2.3860538
   3	per_capita_expend	0.0045305

3. We are focused on inference for the slope of per_capita_expend, so we collect estimated slopes of per_capita_expend and make the bootstrap distribution. This is the approximation of the sampling distribution of ${\hat{\beta }}_1$ . A histogram and summary statistics for this distribution are shown in Figure 5.3 and Table 5.5, respectively.

   

   Figure 5.3: Bootstrap distribution of the slope

   Table 5.5: Summary statistics for bootstrap distribution of the slope

   Min	Q1	Median	Q3	Max	Mean	Std.Dev.
   -0.001	0.002	0.003	0.002	0.01	0.004	0.002

Your turn!

How many values of ${\hat{\beta }}_1$ make up the bootstrap sampling distribution shown in Figure 5.3 and summarized in Table 5.5 ?⁴

4. As the final step, we use the bootstrap distribution to calculate the lower and upper bounds of the 95% confidence interval. These bounds are calculated as the points that mark off the middle 95% of the distribution. These are the points that the ${2.5}^{th}$ and ${97.5}^{th}$ percentiles as shown by the vertical lines in Figure 5.4.

Figure 5.4: Bootstrap confidence interval for the slope

Table 5.6: 95% confidence interval for the slope

Lower bound	Upper bound
0.001	0.007

The 95% bootstrapped confidence interval for ${\beta }_1$, the slope of per_capita_expend is 0.001 to 0.007.

Your turn!

The points at what percentiles in the bootstrap distribution mark the lower and upper bounds for a

• 90% confidence interval?⁵

• 98% confidence interval?⁶

5.4.2 Interpreting the interval

The basic interpretation for the 95% confidence interval for ${\beta }_1$, the slope of per_capita_expend is

We are 95% confident that the interval 0.001 to 0.007 contains the population slope for per capita expenditure in the model of the relationship between city per capita expenditure and number of playgrounds per 10,000 residents.

Though this interpretation indicates the range of values that may reasonably contain the true population slope for per_capita_expend, it still requires the reader to further interpret what it means about the relationship between per capita expenditure and playgrounds per 10,000 residents. It is more informative to interpret the confidence interval in a way that also utilizes the interpretation of the slope from Section 5.1.1 , so it is clear to the reader exactly what the confidence interval means. Thus, a more complete and informative interpretation of the confidence interval is as follows:

We are 95% confident that for each additional dollar a city spends per resident, the number of playgrounds per 10,000 residents is greater by 0.001 to 0.007, on average.

This interpretation not only indicates the range of values as before, but it also clearly describes what this range means in terms of the average change in playgrounds per 10,000 residents as per capita expenditure increases.

5.4.3 What does “confidence” mean?

The beginning of the interpretation for a confidence interval is “We are $C\mathrm{\%}$ confident…”. What does “$C\mathrm{\%}$ confident” mean? The notion of “confidence” refers to the statistical process used construct the confidence interval. This means if we replicate the process thousands of times - obtain a sample of 97 cities, construct a bootstrap distribution for ${\hat{\beta }}_1$, calculate the bounds that mark the middle $C\mathrm{\%}$ of the distribution, , the intervals defined by the upper and lower bounds would contain the value of the true population slope ${\beta }_1$ $C\mathrm{\%}$ of the time.

In reality we don’t know the value of the population slope (if we did, we wouldn’t need statistical inference!), so we can’t definitively conclude if the interval constructed in Section 5.4.1 is one of the $C\mathrm{\%}$ that contains the population slope or not. Though we aren’t certain that our interval contains the population slope, we can conclude with some level of confidence, $C\mathrm{\%}$ confident to be exact, that we think it does based on our process.

Thus far, we have used a confidence interval to produce a plausible range of values for the population slope. We can also test specific claims about the population slope using another inferential procedure called hypothesis testing.

5.4.4 Bootstrap confidence intervals in R

The bootstrap distribution and confidence interval are computed using the infer package (Couch et al. 2021). Because bootstrapping is a random sampling process, the code should include set.seed() to ensure the results are reproducible. Any integer value can go inside the set.seed function.

1. Set a seed to make the results reproducible.
2. Define the number of bootstrap samples (iterations). Bootstrapping can be computing intensive when using large data sets and a large number of iterations. We recommend using a small number of iterations ( 10 - 100) when testing code, then increasing the iterations once the code is finalized.
3. Specify the data set and save the bootstrap distribution in the object boot_dist.
4. Specify the response and predictor variable.
5. Specify the type of simulation (“bootstrap”) and the number of iterations.
6. For each bootstrap sample, fit the linear regression model.

We can use ggplot to make a histogram of the bootstrap distribution.

boot_dist |>
  filter(term == "per_capita_expend") |>
  ggplot(aes(x = estimate)) +
  geom_histogram() +
  labs(x = "Estimated coefficient", 
       title = "Bootstrapped sampling distribution of slope")

Finally, we can compute the lower and upper bounds for the confidence interval using the quantile function. Note that the code includes ungroup() , so that the data are not grouped by replicate.

# A tibble: 1 × 2
        lb      ub
     <dbl>   <dbl>
1 0.000735 0.00711

5.5 Hypothesis testing

Hypothesis testing is used to evaluate a claim about about a population parameter. The claim could be based on previous research, an idea a research or business team wants to explore, or a general statement about the parameter. We can use the process detailed in this section to test a claim about the intercept ${\beta }_0$ or any population parameter; however, we continue to focus on the slope.

Before getting into the details of simulation-based hypothesis testing, we’ll illustrate the steps for a hypothesis test based on a common used analogy, the general procedure of a court trial in the United States (US) judicial system.

Define the hypotheses
The first step to any hypothesis test (or trial) is to define the hypotheses to be evaluated. These hypotheses are called the null and alternative. The null hypothesis $(H_0)$ is the baseline condition typically indicating no relationship between the response and predictor, and the alternative hypothesis $(H_a)$ is defined by the claim being tested - typically that there is some relationship between the two variables.

In the US judicial system, a defendant is deemed innocent unless proven otherwise. Therefore, the null and alternative hypotheses are $H_0$: the defendant is not guilty, and $H_a$: the defendant is guilty. We say that a person is “innocent until proven guilty beyond a reasonable doubt.” Therefore, the trial proceeds assuming the null hypothesis of innocence is true and the objective is to evaluate the strength of evidence against this hypothesis. The same is true for hypothesis testing in statistics. The test is conducted under the assumption that the null hypothesis, $H_0$, is true, and we use statistical methods to evaluate the strength of the evidence against $H_0$.

Evaluate the evidence

The primary component of trial (or hypothesis test) is a presenting and evaluating the evidence. In a trial, this is the point when the evidence is presented and it is evaluated under the assumption the null hypothesis (defendant is not guilty) is true. Thus, the lens in which the evidence is being evaluated is “given the defendant is not guilty, how likely is it that this evidence would exist?”

For example, suppose an individual is on trial for a robbery at a jewelry store. The null hypothesis is that they are not guilty and did not rob the jewelry store. The alternative hypothesis is they are guilty and did rob the jewelry store. If there is evidence that the person was in a different city during the time of the jewelry store robbery, the evidence would be more in support of the null hypothesis of innocence. It seems plausible the individual could have been in a different city at the time of the robbery if the null hypothesis is true. Alternatively, if some of the missing jewelry was found in the individual’s car, the evidence would seem to be strongly in support of the alternative hypothesis. If the null hypothesis is true, it does not seem likely that the individual would have the missing jewelry in their car.

In hypothesis testing, the “evidence” being assessed is the analysis of the sample data. Thus we are considering the question “given the null hypothesis is true, how likely is it to observe the results seen in the sample data?” We will introduce approaches to address this question using simulation-based methods in Section 5.6 and theory-based methods in Section 5.8.3.

Make a conclusion

There are two typical conclusions in a trial in the US judicial system - the defendant is guilty or not guilty based on the evidence. The criteria to conclude the alternative that a defendant is guilty is that the strength of evidence must be “beyond reasonable doubt”. If there is sufficiently strong evidence against the null hypothesis of not guilty, then the conclusion is the alternative hypothesis that the defendant is guilty. Otherwise, the conclusion is that the defendant is not guilty, indicating the evidence against the null was not strong enough to otherwise refute it. Note that this is the not the same as “accepting” the null hypothesis but rather indicating that there wasn’t enough evidence to suggest otherwise.

Similarly in hypothesis testing, we will use a predetermined threshold to assess if the evidence against the null hypothesis is strong enough to reject the null hypothesis and conclude the alternative, or if there is not enough evidence “beyond a reasonable doubt” to draw a conclusion other than the assumed null hypothesis.

5.6 Permutation tests

Now that we have explained the general process of hypothesis testing, let’s take a look at hypothesis testing using a simulation-based approach, called a permutation test.

The four steps of permutation test for a slope ${\beta }_1$ are

1. State the null and alternative hypotheses.
2. Generate the null distribution.
3. Calculate the p-value.
4. Draw a conclusion.

These steps are described in detail in the context of the hypothesis test for the slope in Equation 5.1.

5.6.1 State the hypotheses

As defined in Section 5.5 the null hypothesis ($H_0$) is baseline condition, and the alternative hypothesis ($H_a$) is defined by the claim being tested. Recall from Section 5.1 that one objective for the analysis in this chapter is to investigate whether per capita expenditure is useful in understanding variability in playground access in US cities. In terms of the linear regression model, the claim being tested is whether there is a linear relationship between the city’s per capita expenditure and the number of playgrounds per 10,000 residents. We use this claim to define the alternative hypothesis. The null hypothesis is the baseline condition of there being no linear relationship between the two variables.

• Null hypothesis: There is no linear relationship between playgrounds per 10,000 residents and per capita expenditure. The slope of per_capita_expend is equal to 0. $(H_0:{\beta }_1=0)$
• Alternative hypothesis: There is a linear relationship between playgrounds per 10,000 residents and per capita expenditure. The coefficient of per_capita_expend is not equal to 0. $(H_a:{\beta }_1=0)$
  • Note that we have not hypothesized whether the slope is positive or negative.

The hypotheses are defined specifically in terms of the linear relationship between the two variables, because we are ultimately drawing conclusions about the slope ${\beta }_1$.

Tip 5.1: Mathematical statement of hypotheses

Suppose there is a response variable $Y$ and a predictor variable $X$ such that

$$Y={\beta }_0+{\beta }_{1}X+{ϵ}_i,ϵ\sim N(0,{\sigma }_{ϵ}^2)$$

The hypotheses for testing whether there is truly a linear relationship between $X$ and $Y$ in the population are

$$\begin{array}{rl}& H_0:{\beta }_1=0\\ & H_a:{\beta }_1\ne 0\end{array}$$

5.6.1.1 One vs. two-sided hypotheses

The alternative hypothesis defined in Note 5.1 and ?eq-slr-hypotheses is “not equal to 0”. This is the alternative hypothesis corresponding to a two-sided hypothesis test, because it includes in the scenarios in which ${\beta }_1$ is less than or greater than 0. There are two other options for defining the alternative hypothesis, “${\beta }_1$ is less than 0” ($H_a:{\beta }_1<0$) and “${\beta }_1$ is greater than 0” ($H_a:{\beta }_1>0$). These are one-sided hypothesis tests, as they only consider the alternative scenario in which ${\beta }_1$ is either less than or greater than 0, respectively.

A one-sided hypothesis test imposes some information about the direction of the parameter, that is positive ($>0$) or negative ( $<0$). Given this additional information imposed by the direction of the alternative hypothesis, it requires less evidence to reject the null hypothesis in favor of the alternative. Therefore, it is best to only use such a one-sided hypothesis if (1) there is some indication from previous knowledge or research that the relationship between the response variable and the predictor being tested is in a particular direction, or (2) only one direction of the relationship between the response and predictor being tested is relevant for the research. Outside of these two scenarios, it is not advisable to use the one-sided hypothesis, as there could appear to be a statistically significant relationship between the two variables merely by chance of how the hypotheses were constructed.

A two-sided hypothesis test makes no assumption about the direction of the relationship between the response variable and predictor being tested. Therefore, it is a good starting point for drawing conclusions about the relationship between a given response and predictor variable. From the two-sided hypothesis, we will conclude whether there is or is not sufficient statistical evidence of a linear relationship between the response and predictor. With this conclusion, we cannot determine if the relationship between the variables is positive or negative without additional analysis. We can use a confidence interval (Section 5.4) to make specific conclusions about the direction (and magnitude) of the relationship.

5.6.2 Generate the null distribution

Recall that hypothesis tests are conducted assuming the null hypothesis $H_0$ is true. Based on the hypotheses defined in Section 5.6.1, a hypothesis test for the slope is conducted under the assumption ${\beta }_1=0$ , that there is no linear relationship between the response and predictor variables.

To assess the evidence, we will use a simulation-based method to approximate the sampling distribution of the sample slope ${\hat{\beta }}_1$ under the assumption that $H_0:{\beta }_1=0$ is true. This distribution, called the null distribution, allows us to understand the sample-to-sample variability under the scenario in which the true population slope equals 0. The distribution is similar to the bootstrap distribution in Section 5.4 in that we use it to understand sample-to-sample variability in the estimated slopes. The difference between the two distributions is where the center of each distribution lies. The center for the bootstrap distribution in Section 5.4 was close to the value of ${\hat{\beta }}_1$ estimated from the data. The center of the null distribution, however, is the null hypothesized value of 0. Therefore, to get the null distribution for hypothesis testing, we will use a different simulation method, called permutation sampling.

In permutation sampling the values of the predictor variable are randomly shuffled and paired with values of the response, thus generating a new sample of the same size as the original data. The process of randomly pairing the values of the response and the predictor variables simulates the null hypothesized condition that there is no linear relationship between the two variables.

The steps for generating the null distribution using permutation sampling are the following:

1. Generate $n_{iter}$ permutation samples, where $n_{iter}$ is the number of iterations. We ideally use at least 1,000 iterations in order to construct a distribution that is close to the theoretical null distribution defined in Section 5.8.
2. Fit the linear model to each of the $n_{iter}$ permutation samples to obtain $n_{iter}$ values of ${\hat{\beta }}_1$, the estimated slope. There will also be $n_{iter}$ values of ${\hat{\beta }}_0$ the estimated intercepts; we will ignore those for now because we are focused on inference for the slope.
3. Collect the $n_{iter}$ values of ${\hat{\beta }}_1$ from the previous step to make the simulated null distribution. This is an approximation of the distribution of ${\hat{\beta }}_1$ values if we were to repeatedly take samples the same size as the original data and fit the linear model to each sample, under the assumption that the null hypothesis is true.

Let’s look at an example and generate the the null distribution to test the hypotheses in ?eq-slr-hypotheses.

1. First we generate 1000 permutation samples, such that in each sample, we permute the values of per_capita_expend, randomly pairing each to a value of playgrounds. This is to simulate the scenario in which there is no linear relationship between per_capita_expend and playgrounds. The first 10 rows of the first permutation sample are in Table 5.7.

Table 5.7: First 10 rows of the first permutation sample. The replicate column identifies which permutation sample.

replicate	playgrounds	per_capita_expend
1	2.1	319
1	1.8	307
1	2.2	219
1	1.0	301
1	2.6	190
1	2.2	250
1	3.3	215
1	1.8	399
1	1.8	162
1	1.9	179

2. Next, we fit a linear regression model to each of the 1000 permutation samples. This gives us 1000 estimates of the slope and intercept. The first 10 rows of the first permutation sample are in Table 5.8 .

Table 5.8: Estimated slopes from first 10 permutation samples.

replicate	term	estimate
1	per_capita_expend	-0.0002980
2	per_capita_expend	0.0005629
3	per_capita_expend	0.0012841
4	per_capita_expend	-0.0015942
5	per_capita_expend	-0.0021873
6	per_capita_expend	0.0036558
7	per_capita_expend	-0.0024923
8	per_capita_expend	-0.0017146
9	per_capita_expend	-0.0012171
10	per_capita_expend	0.0014289

3. Next, we collect the estimated slopes from the previous step to obtain the simulated null distribution. We will use this distribution to assess the strength of the evidence from the original sample data against the null hypothesis.

   

   Figure 5.5: Null distribution generated from permutation sampling

Table 5.9: Null distribution generated from permutation sampling

Min	Q1	Median	Q3	Max	Mean	Std.Dev.
-0.005	-0.001	0	-0.001	0.006	0	0.002

Note that the distribution visualized in Figure 5.5 and summarized in Table 5.9 is approximately unimodal, symmetric, and looks similar to the normal distribution. As the number of iterations (permutation samples) increases, the simulated null distribution will be closer and closer to a normal distribution.

You may also notice that the center of the distribution is approximately 0, the null hypothesized value. The standard error of this distribution 0.002 is an estimate of the standard error of ${\hat{\beta }}_1$, the sample-to-sample variability in the estimates of ${\hat{\beta }}_1$ when taking random samples of size 97, the same size as the original data.

Your turn!

How does the estimated variability in the simulated null distribution in Table 5.9 compare to the variability in the bootstrapped distribution in Table 5.5? Is this what you expected? Why or why not?⁷

5.6.3 Calculate p-value

We use the null distribution to understand the values we expect ${\hat{\beta }}_1$, the slope of per_capita_expend , to take if we repeatedly take random samples and fit a linear regression model, assuming the null hypothesis ${\beta }_1=0$ is true. To evaluate the strength of evidence against the null hypothesis, we will compare the estimated slope in Table 5.2, ${\hat{\beta }}_1=$ 0.003 (the evidence) to what we would expect ${\hat{\beta }}_1$ to be based on the null distribution.

This comparison is quantified using a p-value. The p-value is the probability of observing estimated slopes at least as extreme as the value estimated from the sample data, given the null hypothesis is true. In the context of the parks data, the p-value is the probability of observing values of the slope that are at least as extreme as ${\hat{\beta }}_1=$ 0.003 in the null distribution.

In the context of statistical inference, the phrase “more extreme” means the area between the estimated value ( ${\hat{\beta }}_1$ in our case), and the outer tail(s) of the simulated null distribution. The alternative hypothesis determines which tail(s) to include when calculating the p-value.

• If $H_a:{\beta }_1>0$, the p-value is the probability of obtaining a value in the null distribution that is greater than or equal to ${\hat{\beta }}_1$.

• If $H_a:{\beta }_1<0$, the p-value is the probability of obtaining a value in the null distribution that is less than or equal to ${\hat{\beta }}_1$.

• If $H_a:{\beta }_1\ne 0$, the p-value is the probability of obtaining a value in the null distribution whose absolute value is greater than or equal to ${\hat{\beta }}_1$ . This includes values that are greater than or equal to $|{\hat{\beta }}_1|$ or less than or equal to $-|{\hat{\beta }}_1|$.

Recall from Section 5.6.1 that we are testing a two-sided alternative hypothesis. Therefore, we will calculate the p-value corresponding to the alternative hypothesis $H_a:{\beta }_1\ne 0$. As illustrated in Figure 5.6, this p-value is the probability of observing the slope that is 0.003 or more extreme, given the null hypothesis is true. In this case, it is the probability of observing a value in the null distribution that is greater than or equal to |0.003| or a value that is less than or equal to -|0.003|.

The p-value for this hypothesis test is 0.046 and is shown by the dark shaded area in Figure 5.6.

Figure 5.6: Simulated null distribution with p-value represented by the shaded area

Your turn!

Use the definition of the p-value at the beginning of this section to interpret the p-value of 0.046 in the context of the data.⁸

5.6.4 Permutation test in R

The null distribution and p-value are computed using the infer package (Couch et al. 2021). Much of the code to generate the null distribution is similar to the code to conduct bootstrap sampling. Because permutation sampling is a random process, the code should include set.seed() to ensure the results are reproducible. Any integer value can go inside the set.seed function.

1. Set a seed to make the results reproducible.
2. Define the number of bootstrap samples (iterations). Permutation sampling can be computing intensive when using large data sets and a large number of iterations. We recommend using a small number of iterations ( 10 - 100) when testing code, then increasing the iterations once the code is finalized.
3. Specify the data set and save the null distribution in the object null_dist.
4. Specify the response and predictor variable.
5. Specify the null hypothesis of “independence”, corresponding to no linear relationship between the response and predictor variables.
6. Specify the type of simulation (“permute”) and the number of iterations.
7. For each permutation sample, fit the linear regression model.

We can use ggplot to make a histogram of the null distribution.

null_dist |>
  filter(term == "per_capita_expend") |>
  ggplot(aes(x = estimate)) +
  geom_histogram() +
  labs(x = "Estimated coefficient", 
       title = "Simulated null distribution of slope")

Finally, we can compute the the p-value using the get_p_value function.

# A tibble: 1 × 2
  term              p_value
  <chr>               <dbl>
1 per_capita_expend   0.054

Warning

When using permutation tests, the software may output a p-value of 0, as in this example. Note that the true theoretical p-value is not exactly 0; it is just so small that we did not observe a slope at least as extreme as the slope estimated from the sample data in the 1000 permutation samples used to generate the null distribution.

We will calculate the exact p-value in Section 5.8.3 when we conduct hypothesis testing using theory-based inference methods.

5.6.5 Draw conclusion

Recall that the goal is to evaluate the strength of evidence against the null hypothesis. The p-value is a measure of the strength of that evidence and is used to draw one of the following conclusions:

• If the p-value is “sufficiently small”, there is strong evidence against the null hypothesis. We reject the null hypothesis, $H_0$, and conclude the alternative $H_a$.
• If the p-value is not “sufficiently small”, there is not strong enough evidence against the null hypothesis. We fail to reject the null hypothesis, $H_0$, and stay with the null hypothesis.

We use a predetermined decision-making threshold called an $\mathit{\alpha }$-level to determine if a p-value is sufficiently small enough to reject the null hypothesis.

• If $p-value<\alpha$, then reject $H_0$

• If $p-value\ge \alpha$, then fail to reject $H_0$.

A commonly used threshold is $\alpha =0.05$. If stronger evidence is required to reject the null hypothesis , then a lower threshold $\alpha <0.05$ can be used to make a conclusion. If such strong evidence is not required (this may be the case of analyses with very small sample sizes), then a threshold $\alpha >0.05$ can be used. It is general convention to use a threshold 0.1 or less, so any p-value 0.1 is considered large enough to draw the conclusion to fail to reject the null hypothesis.

Back to parks analysis. We will use the common threshold of $\alpha =0.05$. The p-value calculated in the previous section is 0.046, which is less than 0.05. We reject the null hypothesis $H_0$. The data provide sufficient evidence of a linear relationship between the amount a city spends per resident and the number of playgrounds per 10,000 residents.

5.6.6 Type I and Type II error

Regardless of the conclusion that is drawn (reject or fail to reject the null hypothesis), we have not determined that the null or alternative hypothesis are definite truth. We have just concluded that the evidence (the data) has provided more evidence in favor of one conclusion versus the other. As with any statistical procedure, there is the possibility of making an error - more specifically a Type I or Type II error. Because we don’t know the value of the population slope, we will not know for certain whether we have made an error; however, understanding the potential errors that can be made can help inform the decision-making threshold $\alpha$ and make a more informed assessment about the implication of these results in practice.

Table 5.10 shows how Type I and Type II errors correspond to the (unknown) truth and the conclusion drawn from the hypothesis test.

Table 5.10: Type I and Type II error

		Truth
		$H_0$ true	$H_a$ true
Hypothesis test decision	Fail to reject $H_0$	Correct decision	Type II error
	Reject $H_0$	Type I error	Correct decision

A Type I error has occurred if the null hypothesis is actually true, but the p-value is small enough to reject the null hypothesis. The probability of making this type of error is the decision-making threshold $\alpha$. This is because $P(\text{reject }{H}_0|H_0\text{ true})=\alpha$, the probability of rejecting the null hypothesis given the null is true is $\alpha$.

A Type II error has occurred if the alternative hypothesis is actually true, but we fail to reject the null hypothesis, because the p-value is large. Computing the probability of making this type of error is less straightforward. It is calculated as $1-Power$ , where the $Power=P(\text{reject }{H}_0|H_a\text{ true})$.

In the context of the parks data , a Type I error is concluding that there is a linear relationship between per capita expenditure and playgrounds per 10,000 residents in the model, when there actually isn’t one in the population. A Type II error is concluding there is no linear relationship between per capita expenditure and playgrounds per 10,000 residents when in fact there is.

Your turn!

Given the conclusion in Section 5.6.5, is it possible we’ve made a Type I or Type II error?⁹

5.7 Relationship between CI and hypothesis test

We have described the two main types of inferential procedures: hypothesis testing and confidence intervals. At this point you may be wondering whether there is any connection between the two. Spoiler alert: there is!

Testing a claim with the two-sided alternative $H_a:{\beta }_1\ne 0$ and decision-making threshold $\alpha$ is equivalent to using the $C\mathrm{\%}$ confidence interval to evaluate the claim, where $C=(1-\alpha )\times 100$. This means we can also use confidence intervals to evaluate two-sided hypotheses. When using a confidence interval to draw conclusions about a claim, we use the following guide:

• If the null hypothesized value ( $0$ based on the tests defined in Section 5.6.1 ) is within the range of the confidence interval, fail to reject $H_0$ at the $\alpha$ - level.

• If the null hypothesized value is not within the range of the confidence interval, reject $H_0$ at the $\alpha$- level.

This illustrates the power of confidence intervals; they can not only be used to draw a conclusion about a claim (reject or fail to reject $H_0$), but they also give the range values that reasonably contains the value of the population parameter. Therefore, it is good practice to always report the confidence interval, even if the outcome of the hypothesis test is of primary interest, because the confidence interval provides more detail about a population slope ${\beta }_1$ beyond the reject/fail to reject conclusion of the hypothesis test.

Statistical vs. practical significance

When we reject a null hypothesis, we conclude that there is a statistically significant linear relationship between the response and predictor variables. Concluding there is statistically significant relationship between the response and predictor, however, does not necessarily mean that the relationship is practically significant. The practical significance, how meaningful the results are in the real world, is determined by the magnitude of the estimated slope of the predictor on the response and what an effect of that magnitude means in the context of the data and analysis question.

5.8 Theory-based inference

Thus far we have approached inference using simulation-based methods (bootstrapping and permutation) to generate sampling distributions and null distributions. When certain conditions are met, however, we can use theoretical results about the sampling distribution to understand the variability in ${\hat{\beta }}_1$. In this section, we present that theory, then use it to conduct statistical inference for the population slope. One thing you’ll notice as we go through this section is that the inferential procedures and conclusions are very similar as before. The primary difference is in how we understand the sampling variability in ${\hat{\beta }}_1$ and obtain the null distribution.

5.8.1 Central Limit Theorem

The Central Limit Theorem (CLT) is a foundational theorem in statistics about the distribution of a statistic and the associated mathematical properties of that distribution. For the purposes of this text, we will focus on what the Central Limit Theorem says the distribution of an estimated slope ${\hat{\beta }}_1$, but note that this theorem applies to statistics other than the slope. We will also focus on the results of the theorem and less so on derivations or advanced mathematical details of the Central Limit Theorem.

By the Central Limit Theorem, we know under certain conditions (more on these conditions in the Chapter 6 .) $$\begin{array}{}\text{(5.5)}& {\hat{\beta }}_1\sim N({\beta }_1,S{E}_{{\hat{\beta }}_1})\end{array}$$

Equation 5.5 means that by the Central Limit Theorem, we know that the sampling distribution of ${\hat{\beta }}_1$ is (1) normal, (2) with a mean at the true slope ${\beta }_1$, and (3) a standard error of $S{E}_{{\hat{\beta }}_1}$¹⁰. The center of this distribution ${\beta }_1$ is unknown; the purpose of statistical inference is to draw conclusions about ${\beta }_1$. The standard error of this distribution $S{E}_{{\hat{\beta }}_1}$ is

$$\begin{array}{}\text{(5.6)}& S{E}_{{\hat{\beta }}_1}={\hat{\sigma }}_{ϵ}\sqrt{\frac{1}{(n-1)s_X^2}}\end{array}$$

where $n$ is the number of observations, $s_X^2$ is the variance of the predictor variable $X$, and ${\hat{\sigma }}_{ϵ}$ is the regression standard error, the variability of the observations around the regression line. It is introduced in more detail in Section 5.8.2. The details of how this formula is derived is beyond the scope of this text, but let’s take a moment to think about what we know about the sampling variability of ${\hat{\beta }}_1$ from Equation 5.6.

The regression standard error ${\hat{\sigma }}_{ϵ}$ is in the numerator, indicating one would expect more variability in ${\hat{\beta }}_1$ when there is more variability in the data about the regression line. The variability in the predictor variable $s_X^2$ is in the denominator, indicating we expect more variability in ${\hat{\beta }}_1$ when there is less variability in the values of the predictor. Therefore, $S{E}_{{\hat{\beta }}_1}$ is the balance between the variability about the regression line and the variability in the predictor variable itself.

As you will see in the following sections, we will use this estimated of the sampling variability in the estimated slope to draw conclusions about the true relationship between the response and predictor variables based on hypothesis testing and confidence intervals.

5.8.2 Estimating ${\sigma }_{ϵ}$

As discussed in Section 4.3.1, there are three parameters that need to be estimated for simple linear regression ${\beta }_0$, ${\beta }_1$, and ${\sigma }_{ϵ}$. Section 4.4 introduced least squares regression, a method for deriving ${\hat{\beta }}_0$ and ${\hat{\beta }}_1$ the estimates for intercept and slope, respectively. Now, we turn to the estimation of ${\sigma }_{ϵ}$, also known as the regression standard error.

By obtaining the estimates ${\hat{\beta }}_0$ and ${\hat{\beta }}_1$, we have the equation of the regression line and therefore can estimate the mean value $Y$ given a value of the predictor $X$. We can then use the residuals ($e_i=y_i-{\hat{y}}_i$) to estimate the variability of the distribution of the response variable about the regression line (see Section 5.3). The regression standard error is the estimate of this variability as it is the estimated standard deviation of the distribution of the errors (Equation 5.3). The equation for the regression standard error is shown in Equation 5.7 ¹¹.

$$\begin{array}{}\text{(5.7)}& {\hat{\sigma }}_{ϵ}=\sqrt{\frac{\sum _{i=1}^n{e}_i^2}{n-2}}=\sqrt{\frac{\sum _{i=1}^n(y_i-{\hat{y}}_i{)}^2}{n-2}}\end{array}$$

Why divide by $n-2$?

You may have noticed that the denominator in Equation 5.7 is $n-2$ not $n$ or $n-1$. The value $n-2$ is called the degrees of freedom. The degrees of freedom are how many observations are available to understand variability about the regression line. We need at a minimum two observations to estimate the equation for the simple linear regression line - one observation for ${\hat{\beta }}_0$ and one observation for ${\hat{\beta }}_1$. Another way to think about this is that it takes at a minimum two points to make a line. The remaining $n-2$ observations help us understand variability about the line. We will talk more about how we use degrees of freedom as we define the distribution to compute confidence intervals and conduct hypothesis tests.

Recall that the standard deviation is the average distance between each observation and the mean of the distribution. Therefore the regression standard error can be thought of as the average distance the observed value the response is from the regression line. The regression standard error ${\hat{\sigma }}_{ϵ}$ is used to quantify the sampling variability in the estimated slope ${\hat{\beta }}_1$ (Equation 5.6), so we will use this value as we conduct inference on the slope.

5.8.3 Hypothesis test for the slope

The overall goals of hypothesis tests for a population slope are the same when using theory-based methods as previously described in Section 5.5. We define a null and alternative hypothesis, conduct testing assuming the null hypothesis is true, and draw a conclusion based on an evaluation of the strength of evidence against the null hypothesis. The main difference from the simulation-based approach in Section 5.6 is in how we quantify the variability in ${\hat{\beta }}_1$ and thus obtain the null distribution. In Section 5.6.2 we used permutation sampling to generate the null distribution. We’ll see that by the Central Limit Theorem we have results that specify exactly how the null distribution is defined.

The steps for conducting a hypothesis test based on the Central Limit Theorem are the following:

1. State the null and alternative hypotheses.
2. Calculate a test statistic.
3. Calculate a p-value.
4. Draw a conclusion.

As in Section 5.6 the goal is to use hypothesis testing to determine whether there is evidence of a statistically significant linear relationship between a response and predictor variable, corresponding to the two-sided alternative hypothesis of “not equal to 0”. Therefore, the null and alternative hypotheses are the same as defined in ?eq-slr-hypotheses $$\begin{array}{r}{H}_0:{\beta }_1=0\\ H_a:{\beta }_1\ne 0\end{array}$$

The next step is to calculate a test statistic. Similar to a $z$-score in the Normal distribution, the test statistic tells us how far the observed slope is from the hypothesized center of the distribution. The general form of a test statistic is

$$T=\frac{\text{estimate }-\text{ hypothesized}}{\text{standard error}}$$

More specifically, in the hypothesis test for ${\beta }_1$, the test statistic is

$$T=\frac{{\hat{\beta }}_1-0}{S{E}_{{\hat{\beta }}_1}}$$

To calculate the test statistic, the estimated slope is shifted by the mean, and then rescaled by the standard error. Let’s consider what we learn from the test statistic. Recall that by the Central Limit Theorem, the distribution of ${\hat{\beta }}_1$ is $N({\beta }_1,S{E}_{{\hat{\beta }}_1})$. Because we conduct hypothesis testing under the assumption the null hypothesis is true, we are assuming that the mean of this distribution of ${\beta }_1=0$.

The test statistic is calculated by shifting the observed slope by the hypothesized mean 0 and rescaling it by $S{E}_{{\hat{\beta }}_1}$, defined in Equation 5.6. Thus, the test statistic is the number of standard errors the estimated slope is from the hypothesized mean of the sampling distribution. The magnitude of the test statistic $|T|$ provides a measure of how far the observed slope is from the center of the distribution, and the sign of $T$ indicates whether the observed slope is above (positive sign) or below (negative sign) the hypothesized mean of 0.

Your turn!

Consider the magnitude of the test statistic, $|T|$. Do you think test statistics with small magnitude provide evidence in support or against the null hypothesis? What about test statistics with large magnitude?¹²

Next, we use the test statistic to calculate a p-value and we will ultimately use the p-value to draw a conclusion about the strength of the evidence against the null hypothesis, as before. The test statistic, $T$, follows a $t$ distribution with $n-2$ degrees of freedom, denoted as $T\sim t_{n-2}$ . Similar to the simulated null distribution in Section 5.6.2, we use this $t_{n-2}$ distribution to evaluate how far the estimated slope is from what we would expect given the null hypothesis is true.

Though the sampling distribution of ${\hat{\beta }}_1$ is normal by the Central Limit Theorem (Equation 5.5), the test statistic follows a $t$ distribution. The $t$ distribution is used, because the value $S{E}_{{\hat{\beta }}_1}$ in the test statistic is calculated using the regression standard error, ${\hat{\sigma }}_{ϵ}$ (see Equation 5.6) We know the estimates ${\hat{\sigma }}_{ϵ}$ and $S{E}_{{\hat{\beta }}_1}$ are likely not equal to the true population values, so we need a distribution that allows for a bit more variability when calculating the p-value. The $t$ distribution better accounts for this extra variability compared to the standard normal distribution.

$t_{df}$ vs. $N(0,1)$ distribution

Figure 5.7 shows the standard normal distribution $N(0,1)$ and the $t$ distribution for different degrees of freedom. The $t$ distribution is very similar to the standard normal distribution: they are both centered at 0 and have a shape that is unimodal and symmetric. In other words, they both look like “bell curves” centered at 0. The difference is that the $t$ distribution allows for more variability than what is expected in the standard normal distribution. This is also referred to as having “heavier tails”. The $t$ distribution has more area under the curve in the tails (or more extreme values) of the distribution, meaning that more extreme values are more likely under the $t$ distribution than under the $N(0,1)$ distribution. As the degrees of freedom increase, the $t$ distribution becomes closer to the the standard normal distribution.

Figure 5.7: Standard normal vs. t distributions

As explained in Section 5.6.3, because the alternative hypothesis is “not equal to”, the p-value is calculated on both the high and low extremes of the distribution as shown in Equation 5.8.

$$\begin{array}{}\text{(5.8)}& \begin{array}{rl}& p-value=Pr(|t|>|T|)=Pr(t<-|T|\text{ or }t>|T|)\\ & \text{where }t\sim t_{n-2}\end{array}\end{array}$$

We compare the p-value to a decision-making threshold $\alpha$ to draw final conclusions. If $p-value<\alpha$ , we reject the null hypothesis and conclude the alternative. Otherwise, we fail to reject the null hypothesis. See Section 5.6.5 for more detail about using the $\alpha$-level and p-value to draw conclusions.

Now let’s apply this process to test whether there is evidence of a linear relationship between per capita expenditure and the number of playgrounds per 10,000 residents. As before, the null and and alternative hypotheses are

$$\begin{array}{rl}& H_0:{\beta }_1=0\\ & H_a:{\beta }_1\ne 0\end{array}$$

where ${\beta }_1$ is the true slope between per_capita_expend and playgrounds. From Table 5.2, we know the observed slope ${\hat{\beta }}_1$ is 0.0033 and the estimated standard error $S{E}_{{\hat{\beta }}_1}$ is 0.0016. The test statistic is

$$T=\frac{0.0033-0}{0.0016}=2.063$$

This test statistic means that given the true slope is per_capita_expend in this model is 0 and thus the mean of the distribution of ${\hat{\beta }}_1$ is 0, the observed slope of 0.003 is 2.063 standard errors above this hypothesized mean. It’s difficult to determine whether or not this is really “far enough” away from the center of the distribution, but we can calculate a p-value to determine the probability of observing a slope at least this far given the null hypothesis is true.

Given there are $n$ = 97 observations, the test statistic follows a $t$ distribution with $97-2=$ 95 degrees of freedom. The p-value, is $Pr(t<-|2.063|\text{ or }t>|2.063|)=$ 0.042.

Using a decision-making threshold $\alpha =0.05$, the p-value $0.042$ is sufficiently small, so we reject the null hypothesis. The data provide sufficient evidence that the coefficient of per_capita_expend is not 0 in this model and that there is a statistically significant linear relationship between per capita expenditure and playgrounds per 10,000 residents in US cities.

Note that this conclusion is the same as in Section 5.6.5 using a simulation-based approach. This is what we would expect, given these are the two different approaches for conducting the same process - hypothesis testing for the slope. We are also conducting the tests under the same assumptions that the null hypothesis is true. The difference is in the methods available - simulation versus theory-based - to quantify $S{E}_{{\hat{\beta }}_1}$, the sampling variability in the estimated slopes ${\hat{\beta }}_1$.

5.8.4 Confidence interval

As with simulation-based inference, a confidence interval calculated based on the results from the Central Limit Theorem is an estimated range of the values that reasonably contain ${\beta }_1$. The purpose, interpretation, and conclusions drawn from confidence intervals are the same as described before in Section 5.4 . What differs, however, is how the interval is calculated. In the simulation-based approach, we used bootstrapping to construct a sampling distribution to understand sample-to-sample variability in ${\beta }_1$. By the Central Limit Theorem, we know exactly how to quantify the sample-to-sample variability in ${\hat{\beta }}_1$ using theoretical results.

The equation for a $(C\times 100)\mathrm{\%}$ confidence interval for ${\beta }_1$ is

$${\hat{\beta }}_1\pm t^{\ast }\times S{E}_{{\hat{\beta }}_1}$$

where $t^{\ast }\sim t_{n-2}$.

From earlier sections, we know about the estimated slope ${\hat{\beta }}_1$ and the standard error $S{E}_{{\hat{\beta }}_1}$. Therefore, let’s focus on $t^{\ast }$, known as the critical value.

The critical value is the point on the $t_{n-2}$ distribution such that the probability of being between $-t^{\ast }$ and $t^{\ast }$ is $C\mathrm{\%}$. Thinking about this visually, this is the point such that the $C\mathrm{\%}$ of the area under the curve is between $-t^{\ast }$ and $t^{\ast }$ is . Note that we are still using a $t$ distribution with $n-2$ degrees of freedom, the same distribution used to calculate the p-value in hypothesis testing. The critical value can be calculated from modern statistical software or using online apps.

Let’s calculate the 95% confidence interval for ${\beta }_1$ . The critical value on the $t_{95}$ distribution is 1.985. You may notice that this value is close to the critical value of 1.96 on the standard normal distribution. Putting everything together, the 95% confidence interval is

$$\begin{array}{r}0.0033\pm 1.985\times 0.0016\\ 0.0033\pm 0.0032\\ [0.0001,0.0065]\end{array}$$

The interpretation is the same as before: We are 95% confident that the interval 0.0001 to 0.0065 contains the true slope for per_capita_expend. This means we are 95% confident that for each additional dollar increase in per capita expenditure, the number of playgrounds per 10,000 residents is greater between 0.0001 and 0.0065, on average.

Theory-based inference in R

The output from the lm function contains the statistics discussed in this section to conduct theory-based inference. The p-value in the output corresponds to the two-sided alternative hypothesis $H_a:{\beta }_1\ne 0$.

The confidence interval does not display by default, but can be added using the conf.int argument in the tidy function. The default confidence level is 95%; it can be adjusted using the conf.level argument in tidy().

parks_fit <- lm(playgrounds ~ per_capita_expend, data = parks)

tidy(parks_fit, conf.int = TRUE, conf.level= 0.95)

# A tibble: 2 × 7
  term              estimate std.error statistic  p.value conf.low conf.high
  <chr>                <dbl>     <dbl>     <dbl>    <dbl>    <dbl>     <dbl>
1 (Intercept)        2.42      0.214       11.3  3.11e-19 1.99       2.84   
2 per_capita_expend  0.00330   0.00160      2.06 4.24e- 2 0.000115   0.00648

5.9 Summary

In this chapter we introduced two approaches for conducting statistical inference and drawing conclusions about a population slope - simulation based methods and methods based on the Central Limit Theorem. You may have noticed that the standard error, test statistic, p-value and confidence interval we calculated using the mathematical models from the Central Limit Theorem align with what it seen from the output produced by statistical software in Table 5.2. Modern statistical software will produce these values for you, so in practice you will not typically derive these values “manually” as we did in this chapter. As the data scientist your role will be to interpret the output and use it to draw conclusions. It’s still valuable, however, to have an understanding of where these values come from in order to interpret and use them accurately.

We have these two approaches for inference, so which one do we use for a given analysis? In the next chapter, we will discuss the model assumptions from Section 5.3 in more detail along with conditions to check if the assumptions hold for a given data set. We will use these conditions in conjunction with other statistical and practical considerations to determine when we might prefer simulation-based methods or those based on the Central Limit Theorem.

Couch, Simon P., Andrew P. Bray, Chester Ismay, Evgeni Chasnovski, Benjamin S. Baumer, and Mine Çetinkaya-Rundel. 2021. “Infer: An r Package for Tidyverse-Friendly Statistical Inference” 6: 3661. https://doi.org/10.21105/joss.03661.

Pedersen, Thomas Lin. 2022. “Patchwork: The Composer of Plots.” https://CRAN.R-project.org/package=patchwork.

Robinson, David, Alex Hayes, and Simon Couch. 2023. “Broom: Convert Statistical Objects into Tidy Tibbles.” https://CRAN.R-project.org/package=broom.

The Trust for Public Land. 2021. “Parks and an Equitable Recovery.” Online.

Waring, Elin, Michael Quinn, Amelia McNamara, Eduardo Arino de la Rubia, Hao Zhu, and Shannon Ellis. 2022. “Skimr: Compact and Flexible Summaries of Data.” https://CRAN.R-project.org/package=skimr.

Wickham, Hadley, Mara Averick, Jennifer Bryan, Winston Chang, Lucy D’Agostino McGowan, Romain François, Garrett Grolemund, et al. 2019. “Welcome to the Tidyverse” 4: 1686. https://doi.org/10.21105/joss.01686.

---

1. Slope: For each additional dollar a city spends per resident, the number of playgrounds per 10,000 residents is expected to increase by 0.003.
   
   Intercept: The expected number of playgrounds per 10,000 residents for a city that spends $0 on its residents is 2.418. We would not expect a city to spend $0 on its residents, so this interpretation is not meaningful in practice.

2. We sample with replacement so that we get a new sample each time we bootstrap. If we sampled without replacement, we would always end up with a sample that looks exactly like our existing sample data.

3. Each bootstrap sample is the same size as our current sample data. In this case, the sample data we’re analyzing has 97 observations.

4. There are 1000 values, the number of iterations, in the bootstrapped sampling distribution.

5. The points at the $5^{th}$ and ${95}^{th}$ percentiles make the bounds for the 95% confidence interval.

6. The points at the $1^{st}$ and ${99}^{th}$ percentiles mark the lower and upper bounds for a 98% confidence interval.

7. The variability is approximately equal in both distributions.

8. Given there is no linear relationship between spending per resident and playgrounds per 10,000 residents ( $H_0$ is true), the probability of observing a slope of 0.003 or more extreme in a random sample of 97 cities is 0.046 .

9. It is possible we have made a Type I error, since we rejected the null hypothesis.

10. Standard error is the term used for the standard deviation of a sampling distribution.

11. Note its similarities to the general equation for sample standard deviation, $s=\sqrt{\frac{\sum _{i=1}^n(y_i-\overline{y}{)}^2}{n-1}}$

12. Test statistics with small magnitude provide evidence in support of the null hypothesis, as they are close to the hypothesized value. Conversely test statistics with large magnitude provide evidence against the null hypothesis, as they are very far away from the hypothesized value.