Tutorial for the Shiny App, Bayesian and Frequentist Side by Side

This document might be revised in the future. Please note the document date above and check back later for an updated version. Re-load this document in your browser to be sure you’re not viewing an old version cached in your browser.

Getting oriented

This tutorial guides you through a Shiny app that puts frequentist and Bayesian analysis side by side.

This tutorial is best viewed in a wide window so the dynamic table of contents (TOC) appears on the left of the text. With the TOC visible, you can click in it to navigate to any section you like. In a narrow window, however, the TOC appears at the top of the screen and disappears when you scroll down.

Core structure of the app

The app is organized as a 2 \(\times\) 2 table: There is one column for frequentist analysis and a second column for Bayesian analysis; there is one row for estimation with uncertainty and a second row for null hypothesis tests. The cells of the 2 \(\times\) 2 table indicate the typical information provided by each type of analysis, as noted in the figure below:

The app’s 2 x 2 table of analyses.

Why this structure?

The framework is explained in the article:
Kruschke, J. K. and Liddell, T. M. (2018). The Bayesian New Statistics: Hypothesis testing, estimation, meta-analysis, and power analysis from a Bayesian perspective. Psychonomic Bulletin & Review, 25, 178-206. DOI: https://doi.org/10.3758/s13423-016-1221-4
This article will be referred to in this tutorial as The Bayesian New Statistics. Readers who are familiar with the layout of analyses in The Bayesian New Statistics will notice that the rows of the app are reversed relative to the table in that article. This reversal is intentional, to emphasize that hypothesis testing is not the default analysis and is not necessary to do at all.

Interactive sliders

The app has lots of interactive sliders with which you can specify aspects of the data and assumptions of the analyses, as suggested by the schematic figure below. The sliders are great for learning what happens to all of the analyses simultaneously. The sliders are also great for focusing what to think about when translating a real-world situation to analysis assumptions.

Schematic of sliders in the app.

Learning objectives

The interactive sliders provide a natural framework for your learning objectives, and for how to assess your learning. If you have mastered the ideas, you should be able to predict the (qualitative) effect of every slider on every cell in the app, and be able to explain why the effect happens. Moreover, you should be able apply the analyses to the real world, which means you should be able to translate real-world information to appropriate settings of the sliders. These objectives are summarized in the figure below:

Learning objectives suggested by arrows to and from sliders.

Layout of the app

The previous figures emphasized the structure of the app without displaying all of its details. The actual appearance of the app when first invoked is shown below. The app may have changed since this tutorial was written, but its basic layout should be similar.

Appearance of app when opened.

Notice the left edge of the screen is where you specify the data. To the right of the data edge, the screen shows aspects of the analyses. The analysis part of the screen is arranged as a 2 \(\times\) 2 table, as described in the previous sections.

Organization of this tutorial

This tutorial steps through all the zones of the app. The figure below shows the app (with all its features displayed) with this tutorial’s sequence of topics and their location on the app screen.

Sequence of topics in this tutorial.

The sequence of topics is

1. Data
2. Analysis model: A normal distribution with its parameters
3. Frequentist estimation: The maximum likelihood estimate
4. Bayesian estimation with uncertainty: Posterior distribution on the parameters
5. Bayesian hypothesis testing: The Bayes factor
6. Frequentist hypothesis testing: The p value
7. Frequentist uncertainty: The confidence interval

This tutorial provides many interactive examples en route. The interactive sections are marked by “Try It!” headers.

Opening the Shiny App

You can open the active app in your web browser by clicking this link.

You will need a large window to view the full app. A smart-phone screen will not be comfortable. On a computer screen, you might need to maximize the size of the window in which the app appears. You might also need to reduce your font magnification to see it all at once; if so use <ctrl>+\(-\) (i.e., control key with minus key) to decrease the app size within the window.

Data

Prerequisites: This section assumes you already know the meanings of arithmetic mean, standard deviation, and histogram.

For this app, the data are a single group of metric values. That is to say, the data are numerical values from a scale like height, temperature, or time. To make the values concrete, we will suppose the values are intelligence quotient (IQ) scores from a group of people, and therefore the values are in the general vicinity of 100 with a standard deviation in the vicinity of 15.

The figure below highlights the data panel of the app. The app generates random data values with characteristics specified by you. Within the data panel are three sliders with which you specify the data mean, standard deviation, and sample size (i.e., the number of values in the sample). The sample of data is constructed so it has exactly the mean, standard deviation, and sample size specified by the slider values. A histogram of the data is shown below the sliders.

The data panel is highlighted.

(1) Try It!

In the active version of the app, which you opened via the link at the beginning of this tutorial, try moving the sliders and watch what happens. The examples below guide you through some some options.

Delayed Response. When you move the sliders, you will notice a delay in the app’s response. A delay is intentionally built into the app so that quick consecutive adjustments by the user are not all responded to. Instead, the app waits for the slider position to be stationary before doing the computations. Moreover, the computations churning “under the hood” take a few seconds.

• (1.1) Try sliding the data mean. Find the slider for the data mean. Move it to a different value, such as 85. Watch what happens to the histogram below the sliders; it will change and will show data values near the mean you selected.

In terms of IQ scores, you can imagine that different types of groups would have different means. Perhaps one group is students from an advanced-placement high-school economics course, and so you might think their mean IQ score would be above the general population mean of 100. Or perhaps a group of people from the general population has been given a “smart drug” that is supposed to increase IQ scores (at least while under the influence of the drug). Or perhaps a group has been given a depressant (such as alcohol) that might decrease IQ scores.

In this app, the data are generated randomly but are designed always to have exactly the sample mean and standard deviation selected by the user. In other words, if you set the slider to a mean of 110, then the data will have a sample mean of 110. This feature makes it easy to understand the influence of data properties on downstream analysis results.

• (1.2) Try sliding the data standard deviation. Make the standard deviation bigger or smaller, and watch the width of the histogram change accordingly.

In terms of IQ scores, you can imagine that different types of groups would have different standard deviations. For example, a group of students from a particular advanced-placement high-school economics course might have IQ scores less spread out than scores in the general population. Or consider a group of people from the general population who are taking an IQ test while being watched by other people. Under this social stressor, some people might be facilitated but other people might “choke” and perform less well, and therefore the standard deviation of the group would increase.

• (1.3) Try sliding the data sample size. Try manipulating the slider for the sample size. The histogram will become smoother for larger sample sizes, chunkier for smaller sample sizes.

The default slider settings are a mean of 111.0, a standard deviation of 20.0, and a sample size of 18. You might want to re-set the sliders to their default values. An easy way to re-set the app is by clicking the re-load button in your browser.

Analysis Model

Data are described by mathematical models

The foundation of data analysis is describing data with mathematical models. You can think of a mathematical model as a machine that generates patterns of random data values. The machine has control knobs that determine details of the pattern of data. The control knobs are called parameters and a knob’s position is called the value of the parameter.

One simple example of such a machine is an ordinary bathroom shower head. Think of each water droplet’s location on the floor as a datum generated by the machine. One “control knob” on the machine is the direction of the shower head. If you point the shower head leftward, the data (i.e., droplets) land on the left. If you set the shower head rightward, the data (i.e., droplets) land on the right. The angle of the shower head is one parameter that controls the location of droplets on the floor. Many shower heads also have a knob that controls the spread of the spray. When set narrow, the droplets land close together on the floor, but when set wide the droplets cover a wide area of the floor. These ideas are illustrated in the figure below. The location knob is labeled μ (mu) and the spread knob is labeled σ (sigma).

A shower head as machine that generates random data.

The bathroom shower head is just one device for generating a pattern of random water droplets. There are others, such as different types of lawn sprinklers. Each type of sprinkler generates a different pattern of droplets, and each type of sprinkler has different control knobs.

Different mathematical models of data are like different shower heads or sprinklers, each with their own control knobs. Each mathematical model generates a particular type of data, and each mathematical model has particular knobs – called parameters – that control the specific details of the pattern of data.

The essential idea of data analysis is that we try to imitate the pattern of data generated by the world with a pattern of data generated by a model. If the model imitates the world’s data well, then we can summarize and describe the data in terms of the model and its parameters. Moreover, we can understand the data in terms of the model and its parameters.

For example, suppose we come across a wet area of the lawn. There are different models for what may have created the wet zone. There’s the oscillating sprinkler, the cicular spray sprinkler, and the rotary sprinkler. Each of those models has parameter values including direction and spread. When we infer that the wet zone was made by an oscilating sprinkler set to one side, we are summarizing the data in terms of a particular sprinkler type (the model) and its settings (the parameter values) and we understand the data in those terms.

Using a different metaphor, suppose the data we’re trying to understand are paw prints in the mud. There are different models for what may have created the paw prints. There’s the rabbit model, the cat model, the dog model, and the deer model. And each of those models has parameter values including speed (e.g., walking, running, jumping, galloping) and direction. When we infer that the prints were made by a dog walking north, we are summarizing and describing the data in terms of a dog (the model) and it’s speed and direction (the parameters) and we understand the data in those terms.

In data analysis, the data are not water marks or paw prints, but are measurements of some sort. For example, IQ scores are numerical values. Such data are modeled by mathematical distributions, not by shower heads or roaming animals.

The model: A normal distribution with parameters μ and σ

The app uses a normal distribution to describe the data. A normal distribution specifies the probability of each possible data value. When plotted, a normal distribution is a symmetric bell-shaped curve, indicating that the data value under the peak of the curve is the most probable, and data values less than or greater than that central value are less probable. The figures below show examples of a normal distribution with different settings of its parameters. Notice that the horizontal axis is data values, and the vertical axis is the probability of those data values. The settings of μ (mu) and σ (sigma) are specified in the title of each plot:

As marked in the plots above, the μ parameter controls the location of the central tendency of the normal distribution, and the σ parameter controls the width of the normal distribution.

Technical digression: Probability density. The vertical axis of the plots above is labelled “probability density.” You can just think of it intuitively as probability, because it’s not crucial for us here to carefully define the mathematical notion of “density.” But if you’d like a very simple explanation of probability density, see Chapter 4 of Doing Bayesian Data Analysis, 2nd Edition.

The figure below highlights the region of the app screen that displays the model. The app shows a normal distribution superimposed on the histogram of data:

The analysis model is highlighted: a normal distribution.

The key idea is that we’re using the normal distribution to describe the data. The data are a heap of numbers, like water droplets on the ground. The normal distribution is a model of the data, like a sprinkler that might have made a similar pattern of water droplets. We would like to infer what settings of μ and σ are plausible descriptions of the data. There may be lots of settings of μ and σ that plausibly mimick the data. Statistical methods quantify the plausbility across all possible values of the parameters, so we know not only the most plausible values but also the range of other reasonably plausible values. As we’ll see, the more data we have, the tighter is our estimate of the parameter values.

Pedagogical digression: Why did I use a normal model in the app? In designing the app, one key goal was to illustrate frequentist corrections for multiple tests. This requires a model with at least two parameters (to conduct a test of each parameter). The normal distribution is the most commonly known model with two parameters.

Frequentist estimation

There are lots of settings of μ and σ that may make the normal distribution mimick the data reasonably well, but what values of μ and σ are the best? The classic answer to this question in the frequentist framework is the values of μ and σ that maximize the probability of the data. Another technical term for probability is “likelihood,” and so the values that maximize the probability of the data are called the maximum likelihood estimate or MLE. It turns out that for the normal distribution, the MLE value for μ is just the arithmetic mean of the data, and the MLE value for σ is just the standard deviation of the data.

Technical digression: Formula for σ_MLE. The MLE for σ uses the formula for standard deviation that divides by \(N\). However, most software packages compute standard deviation using a formula that divides by \(N-1\). Such software packages presume the user wants the “unbiased” estimate of SD, which is a frequentist concept that assumes a fixed-\(N\) sampling distribution for SD. The MLE makes no assumptions about sampling distributions.

Philosophical digression: Is the MLE frequentist? I have chosen to put the MLE in the frequentist column of the app because the MLE is clearly not Bayesian: The MLE does not take into account any prior probabilities of parameter values and the MLE does not refer to the posterior probabilities of the parameter values. On the other hand, the MLE might not be claimed by frequentists either because the MLE does not take into account sampling intentions (as mentioned in the previous paragraph). In particular, the MLE is not always an unbiased estimator of a parameter, where bias is determined by considering a sampling distribution of the estimator. In other words, the notion of bias is inherently frequentist, and the MLE ignores any consideration of bias. I put the MLE in the frequentist column because, if desired, frequentists can derive sampling distributions of MLE values and determine their bias.

To illustrate the MLE of parameter values in the normal distribution, consider a sample of data that has mean of 111.0 and standard deviation of 20.0, shown in the histogram in the figures below. Superimposed on the histogram are normal distributions with different candidate values of μ and σ. Visual inspection reveals that the best mimickry of the data is achieved when μ equals the sample mean and σ equals the sample standard deviation.

The figure below highlights where the app displays the values of the MLE:

Frequentist estimation: The maximum likelihood estimate.

(2) Try It!

In the active version of the app, which you opened via the link at the beginning of this tutorial, do the following:

• (2.1) Try moving the slider for the data mean and watch the MLE’s. The data histogram will change, of course. Notice that the MLE of μ changes too! In fact, the MLE of μ always matches the data mean because, as was mentioned above, it turns out that the MLE of μ is the mean of the data. Also watch what happens to the MLE of σ: nothing. Nothing happens to the MLE of σ when you change the data mean because the specification of the data standard deviation has not changed.

• (2.2) Try moving the slider for the data standard deviation and watch the MLE’s. Notice that the MLE of σ changes too! In fact, the MLE of σ always matches the data standard deviation because, as was mentioned above, it turns out that the MLE of σ is the standard deviation of the data. Also watch what happens to the MLE of μ: nothing. Nothing happens to the MLE of μ when you change the data standard deviation because the specification of the data mean has not changed.

• (2.3) Try moving the slider for the data sample size and watch the MLE’s. The sample size does not affect the MLE values of μ and σ because the specification of the data mean and standard deviation have not changed.

The initial slider settings for the data are a mean of 111.0, a standard deviation of 20.0, and a sample size of 18, in case you want to reset to the initial values. An easy way to reset the app is to click the re-load button in your web brower.

Why are no confidence intervals shown? The default display of the app does not show confidence intervals around the MLE; that is, the values are shown as “NA” (not available). Why? Because a confidence interval is tantamount to a hypothesis test: Any value outside the confidence interval is rejected and any value inside the confidence interval is not rejected. In fact, that’s the most coherent and general way to define a confidence inteveral: a confidence interval is the range of parameter values not rejected by a hypotheses test. Thus, you can’t really specify a confidence interval without testing hypotheses, and the default mode of the app does not test hypotheses. Later in this tutorial we will address hypothesis testing and confidence intervals.

Bayesian estimation and uncertainty

Bayesian reasoning is just re-allocation of credibility across possibilities. In Bayesian data analysis, the possibilities are values of parameters.

Sherlock Holmes was using Bayesian reasoning when he said, “How often have I told you, when you have eliminated the impossible, whatever remains, no matter how improbable, must be the truth?” Sherlock started with a set of possible explanations for a crime, with each explanation having some initial credibility. He gathered data that made all but one explanation extremely unlikely. He therefore re-allocated credibility to the remaining possible explanation, even though the initial credibility of that explanation may have been small.

We do the same sort of reasoning in Bayesian data analysis. We start with a set of possible parameter values in a model, with initial credibilities of those parameter values. We gather data that make some parameter values more or less credible. We re-allocate credibility to the parameter values that are more consistent with the data, and re-allocate credibility away from parameter values that are less consistent with the data.

In our present application using a normal model, there are two parameters, μ and σ. Each combination of μ and σ values represents a possible description of our set of data. Recall the figures, above, that superimposed different normal distributions onto a histogram of data. Each normal distribution was a specific combination of μ and σ, and some combinations were more consistent with the data than other combinations. The figures showed only three candidate combinations; there are in fact an infinity of other combinations of μ and σ. The set of all combinations of μ and σ is a two-dimensional “parameter space”.

Just like Sherlock starts an investigation with prior credibilities of the possible explanations, in Bayesian analysis we start with prior credibilities of all combinations of the parameters. The prior credibilities are defined mathematically as a probability distribution over the parameter space. The figure below shows where the prior probability distribution is specified in the app:

Bayesian prior probability distribution.

The app shows two small graphs of probability distributions over the parameters μ and σ. Notice that these are probabilities of parameter values; they are not probabilities of data values. The horizontal axis of each plot is labeled with its parameter. The upper plot has μ on its horizontal axis and the title “Prior p(μ)” to indicate that it displays the prior probability of μ values. The lower plot has σ on its horizontal axis and the title “Prior p(σ)” to indicate that it displays the prior probability of σ values.

The prior probability distributions default to being very broad and noncommittal, meaning that a very wide range of values for μ and σ are roughly equally plausible at the beginning. The plots only show a limited range of the distribution which extends far beyond both sides of the plot window. The breadth implies that the prior distribution has little impact on the subsequent re-allocation of probabilities, as you’ll see below.

Philosophical digression: For Bayesians, the prior is part of the model. In the app layout, the upper left corner shows the normal distribution superimposed on the data histogram. And this tutorial has, so far, described “the model” as being (merely) the normal distribution with its parameters. Bayesians, however, think of the prior distribution as also being part of “the model.” This perspective becomes important later when doing model comparison. When comparing the ability of different models to describe data, the analyst must specify the priors assumed by each model, so the prior becomes an inherent part of the model.

When the data are taken into account, the probabilities of parameter values are re-allocated toward values that are consistent with the data. The re-allocated probabilities are called the posterior distribution. The figure below indicates where in the app the posterior distribution is displayed:

Bayesian posterior distribution.

Within the cell that displays the posterior distribution, the upper plot has μ on its horizontal axis and the title “Posterior p(μ|D)” to indicate that it displays the posterior probability of μ values. The notation “p(μ|D)” means the probability of μ given the data D. The lower plot has σ on its horizontal axis and the title “Posterior p(σ|D)” to indicate that it displays the posterior probability of σ values given the data.

Mode of the posterior distribution: Take a close look at the posterior distributions of μ and σ. You’ll see that they are annotated with their modal value, that is, the value of the parameter that is most probable. The mode of the posterior distribution is one way of defining the “best” estimate of the parameter.

Uncertainty of the posterior distribution: The highest density interval (HDI). One attraction of Bayesian methods is that the posterior distribution inherently reveals the uncertainty of the estimated parameter value. When the posterior distribution is wide, the estimate is uncertain. When the posterior distribution is narrower, the posterior estimate is more certain. One way of quantifying the width of a distribution is with the 95% highest density interval (HDI). Any point inside the HDI has higher probability density than any point outside the HDI, and the 95% HDI covers 95% of the distribution. Thus we can say the 95% HDI includes the 95% most probable or most credible values.

(3) Try It!

For these exercises, leave the prior distributions broad (as they are by default when the app is invoked).

• (3.1) Slide the data mean to a different value and watch the posterior mode of μ. The posterior mode of μ should be close to the data mean. This makes sense because credible values of μ should be near the data mean when there is not strong prior information to suggest otherwise.

• (3.2) Slide the data standard deviation to a different value and watch the posterior mode of σ. The posterior mode of σ should be close to the data standard deviation. This makes sense because credible values of σ should be near the data SD when there is not strong prior information to suggest otherwise.

• (3.3) Slide the data standard deviation to a different value and watch the posterior HDI of μ. When the data SD gets smaller, the HDI on μ gets smaller. Does that make sense? Answer: Yes, because when there is less “noise” in the data, our estimate of μ becomes more stable and more precise.

• (3.4) Slide the data sample size to different values and watch the width of the posterior HDIs. When the sample size is smaller, the HDI’s are wider. When the sample size is larger, the HDI’s are narrower. This makes intuitive sense: With more data, we are more certain about the estimated values of the parameters.

The displayed numerical values of the mode and HDI limits are a bit wobbly. This wobble happens because “under the hood” the posterior distribution is represented by an algorithm called Markov chain Monte Carlo (MCMC). MCMC represents a probability distribution over a parameter with an enormous random sample of values from that distribution. MCMC merely makes a picture of the probability distribution, with higher resolution the longer it runs. The app runs MCMC just long enough to get reasonably stable numerical values. For research applications, the MCMC can be run as long as desired to get a more stable and highly accurate representation of the posterior distribution. For our purposes all you need to know about MCMC is that it creates a picture of the posterior distribution, and its numerical values have some wobble. You can learn more about MCMC in Chapter 7 of Doing Bayesian Data Analysis, 2nd Edition.

(4) Try It!

• (4.1) Notice MCMC wobble in the results. As mentioned above, the algorithm running under the hood uses MCMC methods to represent the posterior distribution, thereby creating random wobble in the numerical details of the posterior distribution. To get a feeling for how much MCMC wobble there is, try the following. First note the current numerical details of the mode and HDI for the posterior distribution of μ. Then quickly slide the Data Sample Size to any other value and back to its starting value. The app will re-run its analysis because it has detected slider movement, even though there is actually no change in the data mean, standard deviation, and sample size. Notice in the posterior distribution of μ there are slightly different numerical values for the mode and HDI. The underlying posterior distribution is unchanged; only its MCMC representation has wobble.

There are some applications in which you may have strong prior knowledge about the likely values of the parameters. For instance, suppose you are measuring IQ scores and you know you are sampling from the general population which, by definition of IQ scores, should have a μ near 100 and a σ near 15. You might want to take into account the prior knowledge when estimating μ and σ for your new sample. Therefore you would set the prior distribution on μ to be centered on 100.0 and very narrow to reflect your prior certainty, and you would set the prior distribution on σ to be centered on 15.0 and very narrow to reflect your prior certainty. What do you think the posterior distribution will look like? Will the posterior distribution depend on the amount of data?

(5) Try It!

To make this set of experiments most dramatic, start with these settings:

• Set the data mean to 120 (which is much different than 100).
• Set the data standard deviation to 25 (which is much different than 15).
• Set the data sample size to 5 (which is small).

With the prior distribution broad, notice the form of the posterior distribution. The modal estimates of μ and σ are near the data mean and SD, but the posterior distributions have wide HDI’s because of the sample size is small.

• (5.1) Find the pair of sliders for the μ Prior Mode & SD. Leave the mode slider at 100.0, but move the SD slider down to 3. This change will make the prior distribution of μ be narrow around 100. What is the consequence for the posterior distribution of μ? Answer: The posterior mode is now much closer to the prior mode of 100 than it was when the prior was broad, and the posterior distribution of μ is also much narrower. In other words, the strong prior knowledge dominates the small amount of new information from the data.

• (5.2) Find the pair of sliders for the σ Prior Mode & SD. Leave the mode slider at 15.0, but move the SD slider down to 1. This change will make the prior distribution of σ be narrow around 15. What is the consequence for the posterior distribution of σ? Answer: The posterior mode is now closer to the prior mode of 15 than it was when the prior was broad, and the posterior distribution is also much narrower. In other words, the strong prior knowledge dominates the small amount of new information from the data.

• (5.3) Change the data sample size to its largest possible value. For this app, the slider for sample size only goes up to a modest value, so the largest possible value is not really very big. Nevertheless, you can see that with a larger sample, the posterior distributions for μ and σ move away from the prior modes closer to the data mean and SD. In other words, even strong priors can be overcome by sufficient amounts of data.

While manipulating the prior distribution you may have noticed there was no change of the MLE in the frequentist column. This is because the maximum likelihood estimate does not take into account prior knowledge.

(6) Try It!

• (6.1) Try implementing some strong prior knowledge. Suppose you’re investigating the short-term effect of alcohol consumption on IQ test performance. In many studies you’ve had volunteers drawn at random from the general population consume three alcoholic drinks in a hour and then take an IQ test. Suppose sample after sample has shown a mean near 90 (a fictitious number). Now you’re collecting data from a new alcohol-dosed sample and you want to use your prior knowledge to help estimate the mean of the new sample. Use the sliders to set the prior on μ so it’s centered on 90 and is fairly narrow to reflect your prior certainty. Play around with different settings of the data to see the effect of the strong prior knowledge on the posterior estimate of μ. Notice, however, there is no effect of your strong prior knowledge on the MLE.

• (6.2) Change the prior SD’s of μ and σ back to large values such as 90. You’ll see that the posterior modes are now very close to the data mean and SD. Play around with values for the SD’s of the priors. You’ll see that as long as the prior is fairly broad relative to the scale of the data, the posterior is virtually unaffected by exactly how broad the prior is.

After playing around with the sliders, if you want to re-set the default values, simply re-load the page in your browser window.

The posterior distribution inherently reveals the uncertainty of the parameter estimates. The 95% HDI is one way of quantifying the uncertainty. In the frequentist column, however, again notice that there is no confidence interval (CI) displayed. The absence of CI is the result of not (yet) doing any hypothesis tests. The CI is the range of parameter values not rejected by a hypothesis test, and without hypothesis testing there is no CI. (CI’s defined by coverage are tantamount to, and less general than, CI’s defined by tests.)

For further explanation and discussion, see Bayesian Data Analysis for Newcomers.

Decision using ROPE and HDI

Sometimes researchers are interested in assessing particular candidate parameter values, to decide if the candidate parameter value can be rejected as implausible or even accepted as a good-enough representation of the data. For example, consider investigating a new “smart drug” and wanting to know if the mean of the test sample is clearly above the general population mean of 100. You should check if all the most credible values of μ are sufficiently far above 100 for you to be sure that there is a clinically important effect. That is, checking all the most credible values is about being sure, and checking that they are sufficiently far above 100 is about clinical importance.

One way to specify clinical importance is by defining a region of practical equivalence (ROPE) around the target value of the parameter. The ROPE specifies values that are only negligibly different from the target value. For example, with IQ scores, we might say that IQ scores within 2 points of 100 are negligibly different from 100. Therefore, for us to declare that a sample mean is non-negligibly different from 100, we need to know that all the most credible values of μ are at least 2 points away from 100.

This reasoning leads to the following decision rule: If the 95% HDI falls outside the ROPE around the target value, then reject the target value — it’s not a credible description of the data because the 95% most credible values are all non-negligibly different from the target value. On the other hand, the target value can be accepted for practical purposes if the 95% HDI falls completely inside the ROPE, because all the 95% most credible values are only negligibly different from the target value. If the HDI falls partially inside the ROPE and partially outside the ROPE, then no decision about the target value can be made.

The app does not display ROPEs and does not implement the HDI+ROPE decision rule. But it is easy for you to do visually; just specify your ROPE and check where the HDI falls relative to your ROPE. (The ROPE is a decision threshold, and the app avoids options for setting decision thresholds because the app layout would get even more cluttered. The only decision threshold explicitly used by the app is the conventional threshold for frequentist p values, which we will encounter later.)

The HDI+ROPE decision rule is not called “hypothesis testing”, which is terminology reserved for different techniques discussed in subsequent sections. To learn more about the HDI+ROPE decision rule, see the article:
Kruschke, J. K. (2018). Rejecting or Accepting Parameter Values in Bayesian Estimation, Advances in Methods and Practices in Psychological Science, 1, 270-280. DOI: https://doi.org/10.1177/2515245918771304
with important Supplementary Material here: https://osf.io/fchdr/

Hypothesis Testing

It’s not necessary to test hypotheses, and the app defaults to not displaying hypothesis tests. There’s a classic book titled, “What if there were no significance tests?” (Harlow, L. L., Mulaik, S. A., & Steiger, J. H. 2016 Routledge; 1997 Psychology Press.). The app’s default view shows what a world without significance tests would look like!

But if you want to test hypotheses about the parameters, you can specify which parameter values you want to test. The bottom row of the app holds the controls and results of hypothesis testing, as shown in the figure below:

Controls for selecting hypotheses to test.

(7) Try It!

• (7.1) Click the buttons under “Null Hyp. Test” to select which parameters you want to test. You’ll see corresponding results panels appear under the frequentist and Bayesian columns. You’ll also see values appear for the confidence intervals (CIs) for frequentist uncertainty.

All of the details will be explained in subsequent sections of this tutorial. In the next section, we’ll examine Bayesian hypothesis testing. After that we’ll consider frequentist hypothesis testing, and then frequentist confidence intervals.

Bayesian hypothesis testing

Like Bayesian parameter estimation, Bayesian hypothesis testing involves re-allocation of credibility across possibilities. But for hypothesis testing the possibilities are different hypotheses about the parameters. One hypothesis restricts a parameter to its “null” value, and the hypothesis is called a “null hypothesis”. Another hypothesis allows all parameters to range more widely, and this hypothesis is called the “alternative” hypothesis. Bayesian hypothesis testing re-allocates crdibility across the null hypothesis and the alternative hypothesis.

To make the ideas concrete, suppose we administer a “smart drug” to a group of people and then have them take an IQ test. If there were no effect of the smart drug on the central tendency of IQ scores, then the mean IQ of the group should be the same as the general population mean of 100.0. In other words, the null value of μ is 100 and the null hypothesis claims that μ is set at 100. But the null hypothesis about μ doesn’t care about the value of σ and puts a broad prior on σ. The null hypothesis for μ therefore puts a prior distribution on the <μ,σ> parameter space that looks like a sharp ridge as shown in the left panel below:

Priors for Bayesian hypothesis test of μ.

The ridge in the left panel above indicates that only the null value of μ has non-zero probability in the prior distribution, while a broad range of σ values has non-zero prior probability.

The right panel above shows the prior distribution for the alternative hypothesis. That prior is broadly spread out over both parameters, indicating that lots of values of μ have non-zero prior probability.

(The app does not show plots like the figure above because there is not enough screen space. The app does not show the null-hypothesis prior distribution at all. The app does show aspects the alternative-hypothesis prior, but not in perspective like the figure above. The app shows the marginal distributions of μ and σ in the alternative hypothesis.)

In Bayesian hypothesis testing, the two priors in the figure above are conceived as alternative models. Each model has a prior probability. For example, if we think that the null hypothesis is unlikely, that is, if we think that the smart drug really will have an effect, then we might say that the null-hypothesis model has only a 20% prior probability and the alternative-hypothesis model has a 80% prior probability. A display of the entire prior probability distribution, across model indices and parameters within models, is shown below:

Priors for Bayesian hypothesis test of μ with prior probabilities of the hypotheses.

In the figure above, the upper panel displayes the probabilities of the model indices (in this case labeled “Null” and “Alternative”). The model-index variable is itself a parameter an overarching model of the data. The model-index variable has values that correspond to each model, and there is a probability distribution over those values, as displayed in the upper panel of the figure above.

When data are taken into account, Bayesian reasoning re-allocates probabilities across all the parameters in the overarching model simultaneously. Probabilities are shifted across μ and σ parameters within the alternative-hypothesis model, and across parameters within the null-hypothesis model, and across the model-index parameter.

In Bayesian hypothesis testing, the focus is on the model-index parameter: How much do the probabilities on the model indices shift? If the data are not consistent with the null-hypothesis prior, then probability shifts toward the alternative-hypothesis index. Interestingly, if the data are consistent with the null-hypothesis prior, then probability shifts toward the null-hypothesis index.

The degree of shift across hypotheses, from prior probabilities to posterior probabilities, is called the Bayes factor. Usually the Bayes factor is framed as a shift toward the null hypothesis away from the alternative hypothesis, and is denoted BF_null. When BF_null is greater than 1.0 there has been a shift of probability toward the null hypothesis away from the alternative hypothesis, and when BF_null is less than 1.0 there has been a shift of probability away from the null hypothesis toward the alternative hypothesis. (The Bayes factor is always greater than zero.)

(8) Try It!

• (8.1) In the app, under the header Null Hyp. Test specify your Test Intention by selecting the button for “mu only”. After clicking the button, new information appears in the bottom row of the app.

Bayesian hypothesis testing with ‘mu only’ selected.

• Under the header “Null Hyp. Test” there appears a slider for specifying the null value of μ, which is denoted μ₀. It defaults to μ₀=100.0, but you can set it to other values.

• In the Frequentist column there appears information for frequentist null hypothesis significance tests. We’ll cover this information later.

• In the Bayesian column there appears information about the Bayes factor. In particular, the panel displays μ: BF_null = …. As mentioned before, when BF_null>1.0 then credibility has shifted toward the null, but when BF_null<1.0 then credibility has shifted away from the null. The numerical value of the BF_null has MCMC wobble.

• In the Bayesian column there also appears a slider for specifying the prior probability of the null hypothesis. The prior-probability distribution is called a null “model” to distinguish it from the value of μ₀ in the null hypothesis. Beneath the information about the BF_null is shown the posterior probability of the null model. When BF_null>1.0 then the posterior probability of the null model is greater than its prior probability; when BF_null<1.0 then the posterior probability of the null model is less than its prior probability.

(9) Try It!

Manipulate the Data sliders and watch what happens to the BF_null’s and posterior probabilities of the models.

• (9.1) Manipulate the Data Mean; observe effect on BF_null for μ₀. When the data mean is close to the value of μ₀, then BF_null for μ₀ gets large (greater than 1.0). When the data mean is far from the value of μ₀, then BF_null for μ₀ gets small (less than 1.0). In other words, when the data mean is near the null value then the BF suggests a shift of credibility toward the null hypothesis, but when the data mean is far from the null value then the BF suggests a shift of credibility toward the alternative hypothesis.

• (9.2) Manipulate the Data SD; observe effect on BF_null for σ₀. Again, when the data SD is near the null value σ₀ then the BF suggests a shift of credibility toward the null hypothesis, but when the data SD is far from the null value σ₀ then the BF suggests a shift of credibility toward the alternative hypothesis.

• (9.3) Manipulate the Data Mean; observe the effect on BF_null for μ₀ when the data SD is large vs when the data SD is small. When the data SD is small, that is when there is less noise in the data, there is a bigger change in the BF when the data mean is changed.

(10) Try It!

Manipulate the Null Hyp Values and watch what happens to the BF_null’s and posterior probabilities of the models.