#169 Bayesian Workflow for Beginners, Alex at Princeton
New episodes, blog posts and Bayesian modeling resources — roughly once a month.
Featuring Alexandre Andorra & Princeton University
This episode is a bit different: instead of an interview, it's the talk I gave at Princeton Psychology's visiting research seminar in September. The question at the heart of it is one every practitioner running repeated-measures studies has faced: a participant's accuracy moves from one session to the next, so is that variability real, or just noise?
To answer it, I take the mixed model most people in the room fit with lme4, and rebuild it from its generative story. When you fit a model with default frequentist software, you're making a stack of assumptions -- they're just made for you, silently. In contrast, the Bayesian workflow forces you to write your assumptions down, priors included, and then gives you the tools to check them.
The story goes like this: there's a population of people, participants are drawn from it, they come back for several sessions, and within each session they do a bunch of trials. That's already a hierarchy, and the model has to mirror it. Each participant gets their own latent skill, drawn from a population with a mean and a standard deviation, and a condition effect shifts everyone when they switch to the hard version of the task.
Written out in PyMC, each of those choices becomes visible, and each is debatable. One ability per participant means you assume it's stable across sessions. One condition effect means everyone reacts to the hard condition by the same amount -- no interaction. Independent trials means no momentum, which anyone who's watched a tennis player unravel after losing a few points knows isn't always true.
That's why my rules of thumb are boring on purpose:
Because a Bayesian model is generative, it can produce data before it has seen any. That's what a prior predictive check is: draw parameters from the priors, simulate fake studies, and look at whether those worlds are absurd.
The classic trap is the innocent-looking Normal(0, 10). Put it on the logit scale of a success probability and the model expects most participants to get either zero or a hundred successes out of a hundred. No experiment I know of expects that.
Now, enough data will eventually overwhelm a bad prior, but you're paying for it in participants. Better to fix the prior -- with elicitation tools like preliz, or by asking the domain expert, the way I used to ask baseball coaches what they'd expect from a player before seeing him play.
If you run those simulations in a loop, with an effect size you care about baked in... you get a power analysis for free! You can now say how many participants you need to recover the true effect three times out of four. That also answers the question I got at the end of the workshop about pre-registration -- the model, the story, the expected results and the power analysis all exist before a single participant walks in.
Once you fit the model, the payoff of the hierarchy is partial pooling. After only twenty trials per person, the model pulls extreme participants hard toward the population mean -- it's not ready to call someone exceptional on so little evidence -- and lets go as trials accumulate.
The real test came from comparing two models. The first is the standard binomial mixed model, which assumes a participant's skill doesn't change between sessions. The second is session-aware: it lets each participant's success probability wobble from session to session through a beta-binomial.
Their posterior predictive rootograms looked equally fine, but the calibration plots don't lie: a 90% interval should contain the truth 90% of the time, and the session-blind model fell well short -- it was not just wrong, it was confidently wrong.
All those steps are easy to forget, so I encoded them into an open-source agent skill. Then, I gave the same coding assistant this exact same problem and dataset, with and without the skill.
Without, it did a decent job, but it never reviewed its own work or checked calibration, and never made the switch from the binomial to the beta-binomial. With it, the assistant reviewed its own first draft, caught diagnostics it had glossed over, and rebuilt the model as a beta-binomial.
That doesn't make the human optional though: to be the human in the loop, you still need to know what the model is doing!
Check out the full episode above. The slides, the code and the workshop notebook are all open, and linked in the show notes.
You can also interact with the episode on NotebookLM! Ask questions, generate flashcards, and more.
Hope you enjoyed it, and see you in two weeks, my dear Bayesians!
02:45 What assumptions hide under the hood of the mixed models psychologists fit?
05:58 What does Bayes' theorem say about updating your beliefs with data?
09:14 What is the generative story of your data and how does it become a model?
11:59 How do lme4, brms and PyMC express the same mixed model?
15:02 How does a hierarchical model encode participants and their population?
18:03 Which hidden assumptions does a standard mixed model make?
27:17 What is a prior predictive check and why are default priors like Normal(0, 10) risky?
36:39 How do you read posterior distributions and credible intervals?
40:53 What is partial pooling and why do hierarchical models shrink extreme participants?
46:02 How do you check whether a Bayesian model is well calibrated?
52:20 How can an AI agent skill help you run the Bayesian workflow?
Alex Andorra: This episode is a bit special, my dear Bayesians, because it's not an interview, it's the talk I gave at Princeton in September to open the psychology department's visiting
research seminar series.
A big thank you to Suyog Chandramouli for the invitation.
By the way, I had a blast.
The question I wanted to answer for the folks there is one that anyone running repeated measures experiments has faced.
You probably know that a participant's accuracy moves from one session to the next.
So how do I know if that variability is real or if it's just noise?
And how do I estimate that?
Well to get there, I take the mixed model most psychologists fit with lme4, and I rebuild it from its generative story in PyMC.
You'll see that every line of that model is an assumption, and that your default software is making these assumptions for you.
Under the hood.
Then we do the fun stuff: prior predictive checks, power analysis for free, partial pooling, and calibration plots with the new ArviZ 1.0 that catch a model being confidently
wrong.
And to close, I show how I package this whole workflow into an open source AI agent skill and what changes when an assistant uses it.
There are a lot of slides and plots here, folks, so you may want to watch this one on YouTube.
The slides, the code, the workshop notebook are all free, open source, and in the show notes, as usual.
This is Learning Bayesian Statistics, episode 169, recorded September 16, 2026.
And change your predictions after taking new
Let's adjust those expectations.
What's a Bayesian?
It's someone who cares about evidence.
Alex Andorra: Welcome to Learning Bayesian Statistics, a podcast about Bayesian inference, the methods, the projects, and the people who make it possible.
I'm your host, Alex Andorra.
You can follow me on Twitter at Alex underscore Andorra, like the country.
For any info about the show, learnbayesstats.com is Laplace to be.
Show notes.
Becoming a corporate sponsor, unlocking Bayesian merch, supporting the show on Patreon, everything is in there.
That's learnbayesstats.com.
If you're interested in one-on-one mentorship, online courses, or statistical consulting, feel free to reach out and book a call at topmate.io slash alex underscore andorra.
See you around, folks, and best Bayesian wishes to you all.
Suyog Chandramouli: Delighted to welcome Alex Andorra to Princeton.
Alex is a senior data scientist in the industry, and many of you may also know him as the creator and host of the Learning Bayesian Statistics Pod podcast.
It features uh the who's who from Bayesian world, including many from psychology.
Alex will be talking about the Bayesian workflow and uh connected to questions we kind of um often encounter in psychology.
Thank you.
Alex Andorra: Yeah, thank you, Suyog, for uh
The invitation, very happy to be here.
Hi everybody.
My name is Alex.
I'm working in the tech industry right now, but I was working before in the sports industry.
And before that, I was doing statistical modeling consulting, mostly related and centered around causal inference, Bayesian inference.
So that's what we're gonna talk about today.
And to make sure I don't uh bore my wife with nerdy topics.
I have a podcast where I talk about all of that and I can, you know, nerd out.
That's called Learning Bayesian Statistics.
I've been doing that for um nine years now, so we have a huge back catalog.
I do think we have a playlist about psychology modeling, so feel free to to check this out.
I'm sure you'll be you'll be able to find some interesting nuggets.
but so today we're gonna focus on mixed models, because I've heard
I've heard you folks do a lot of uh of mixed modeling in in psychology.
but I will peek under the hood and show you what you're actually doing when you're using probably a frequentist software.
By default, you're making some assumptions, and today we're gonna peek at these assumptions.
And the easiest way to do that is from the Bayesian perspective.
So
Just a quick show of hands, even if you have your fork in your in their hand, that's fine, you can um pick it up.
Uh anybody has heard about Bayesian stats?
How like how comfortable are you with that topic?
You know, have you heard about it?
Okay, awesome.
How many of you have actually sampled from a Bayesian model?
Okay, okay, that's decent.
That's a decent conversion rate because uh often people know about Bayesian stats but they don't know where to start.
So that's great that you have done that already.
today we'll focus mostly on PyMC, which is one of the main uh core package to to do that in Python.
But that's gonna be mostly concepts.
Everything you learn here is applicable if you use R, if you use Julia, if you use any other package we're gonna
We're gonna talk about concept.
For the people who don't know about Bayes yet, I'm gonna give you a quick refresher.
It's actually a very simple formula.
That's what I love about Bayesian stats.
Like if you if you do frequentist stats, it's like it turns into a zoo of tests that you have to pair with every situation.
Each time I have to do that, I wanna like pull my hair out.
What I love in Bayesian stats is that you have one estimator.
That's the posterior distribution.
The only thing you need and you care about.
Once you have that, you can do any test you want.
And you can tailor your test and even come up with your own test statistics.
So it's all based on Bayes Theorem, which here is a simplified version.
Um, and Bayes Theorem just tells you, well, I have a prior before seeing any data, I know stuff before I even observe something in an experiment I'm making.
So what's that?
I'm forced to encode.
That knowledge.
It can be knowledge you have, or it can be knowledge the department has, or knowledge that science has through papers, for instance.
And then I need to update that prior, that's called the likelihood in the theorem.
And that just tells you, well, okay, I'm observing data now, like I'm putting my priors to the test of the experiment.
And then the formula just tells you if I'm a reasonable human being.
Which I hope you are.
How am I supposed to update to optimally update my beliefs based on what I just observed?
That's what the posterior distribution gives you.
And so, actually I had some uh some some cards here to to help you memorize that.
But that's pretty simple, I think, and it's it's a very intuitive formula.
and you will notice that I don't talk about
Numbers or point estimates.
I talk about distributions all the time because each time you deal with these models, you're gonna get distributions on that.
So that means you're gonna get uncertainty estimation out of the box.
You don't have to do anything special, not have to deal with bootstrap or all that stuff afterwards.
Okay?
Um I like humor and I think it's a great teaching tool.
So here is an illustration.
Of Bayes formula.
We'll see if you're able to identify the three elements of Bayes formula in that twenty second clip.
We'll see if the clip works first because of course it worked when I tried, but now we'll see if that works live.
I was offered a summer research fellowship at Princeton.
Alex Andorra: Can you listen?
Yeah?
Okay, awesome.
Let me start again.
I was offered a summer research fellowship at Princeton.
Princeton a fine institution.
The place where Albert Einstein taught and where Leonard got his PhD, so it may have gone downhill.
Alex Andorra: So it's humor of course, right?
Nothing is true with me.
Uh at least the conclusion is not true.
But there you had the three elements of of Bayes formula, right?
The prior was Princeton's a fine institution where Albert Einstein taught, then likelihood comes in.
Leonard did his PhD there.
So then obvious posterior is that, well, Princeton may have gone downhill.
So that's just that.
Prior um Bayes formula through a great
Series.
So today we're gonna talk about a lot of different topics, but first, and that's one of the most important concepts in the Bayesian world, is the generative story of your data.
This is literally how you come up with your models, and you'll hear that all the time.
What's the generative story of your model in the Bayesian world?
And so I'm I'm pretty sure you fit that kind of model already.
Uh
Uh I th I think I haven't to be honest.
Uh I started with Bayesian stats from the get-go, so but I know people use I think it's from lme4, right?
So when you do mixed models in lme4, that's what that's what you'll do.
basically you have a model where you try to infer the probability of being correct and it depends on the condition, whether it's the let's say standard or hard condition, and also
it depends on the participant.
So that's why it's a mixed model, because it's not like a fixed estimate, but it's I think also some people come that call that random effects.
I'm not a fan of saying that way, but a lot of people do.
So I'll I'll use that also.
So um that will run, that will give you a coefficient, a standard error, a p-value, but
Do you know what it no what did it just say?
You know, like what is that model actually telling you?
I think it's at the core of understanding what you're doing and and why even do that kind of model.
The point is, in the Bayesian framework, every model is a story about how the data you have came to be.
And so here, the story is that we have a population of people, okay, which are from which are drawn some participants who are in our study, and these participants they do
Some tests that you give them at each session, and they come back from one session to another.
So you observe multiple sessions per participant, and in each session they do several trials.
So here you have already a hierarchical structure of your world.
Basically, each model is a small world, and your role is to try and pack that world into the model.
And here what you're doing when you're inputing this uh weird term, I don't even really know how to pronounce it, but like in my mind, it's like one given participant, just
because I know probabilities and when I see a uh line like that for me it's like conditioning on on the thing that was before.
But that's just mixed models in a lot of software, so lme4, uh
You may have heard of a Bayesian package that's called BRMS.
If you use R, that's a really amazing package to do Bayesian stats out of the box.
It comes with a lot of things that's gonna make your life easier.
If you're in the Python world, something called Bambi, it's gonna help you do the same thing BRMS, BRMS does, but in the Python world.
And BRMS is based on Stan, which is what's called a probabilistic programming language, PPL.
Bambi is based on PyMC, which is the PPL that I'm a co core developer of.
Right.
Both are great choices.
Okay.
Just like pick pick whatever you prefer.
Don't get uh paralysis by choice.
Okay, just like pick whatever you you're most comfortable with right now and then get from there.
So getting back to my point, when you do that, when you write that, that's actually here what you're doing.
These are the assumptions that you're encoding in your model by default.
And so I just mentioned the the three um the three different packages.
So here you have the three different dialects if you want.
Okay, so you are you must be very familiar with that one.
BRMS you can see is like very, very, very similar.
And then PyMC.
Here I'm showing you PyMC and not Bambi because PyMC is gonna force you actually to write down the model and the assumptions, which is what I want you to get out of today.
Because you won't always have to do that, but it's great that you can do it because you can peek under the machine and when you hand that off to, for instance, an AI coding
assistant, which now you have to know how to do, you can actually know how to evaluate your assistant.
Because otherwise if you don't really know what the model is about, like what are you doing?
Okay.
So you'll need to be the human in the loop here, and to be the human in the loop, you need to know what the AI agent has been doing.
So here you can see in PyMC, that story is here you have the likelihood that we talked about, that we mentioned with Bayes' theorem.
So the likelihood here is a Bernoulli.
It can be a binomial too.
You can see that in lme4 it's the binomial.
Binomial is just the accumulation.
of Bernoulli trials.
So it's really literally the same thing.
Just Bernoulli is gonna be binary result, zero or one, whereas binomial is like you throw a dice ten times and you ac aggregate the results.
You aggregate the counts.
So it's gonna be like if it's a fair dice and you throw it 100 times, you expect more or less fair um let let's take a coin.
It's gonna be easier.
So yeah it's a fair coin, you throw it 1000 times
You expect about 500 um heads.
Okay, that's the binomial.
And the Bernoulli is just disaggregation of all the trials.
Depending on the use case, you want one or the other.
So that's the likelihood.
And then Bernoulli has just one parameter that you're trying to infer.
Here, that's your let's say conversion rate, probability of success, that's usually called.
But here I've been talking about
um an experiment you're doing with participants, but can be like that could be an A B test.
If you're in industry, you often do A B tests, that would literally be the probability that your A B test is better than the placebo.
Okay?
So that's a probability of success.
And here that's kind of like the only math part of that model.
But the one you need to know is theta.
Here is the participant skill, let's say, in that experiment.
Okay?
And theta, we know that the participants come from the population, the general population.
That's what we just saw before in the story.
How do I translate that in the model?
Well, the general population has a tendency, a general tendency, an average.
Here it's mu.
It's often called mu, so sorry for the Greek letters.
But yeah, mu is us usually the general tendency of the population.
So here that's how you do that.
And then the population has a general tendency, but it also varies.
People are different, but they cluster.
So the way we as statisticians try and estimate the way people cluster, but not so much, is with the standard deviation or the variance of the population.
Which is what sigma here is.
And the z is just telling you how far each participant is from the variance of the population.
And so that these two parameters are coming from the population.
These parameters, the z tells you how the participant, that's why you have that index here, PIDs for participant index.
You have that participant.
situation location in the global population that's all encoded into theta, which then goes into the equation for P.
And we could be done here, but we know in our experiments that we have two conditions.
We have the standard one and we have the hard conditions.
And we have to account for that because people are expected to perform better in the standard uh in the standard condition.
So how do we do that?
We just have one more variable which is here called hard.
So it's just like the observation for each participant's session's trials.
Was she were yeah, was she in the hard or in the standard condition?
So it's gonna be one if it's the hard condition, zero if it's standard.
So that means by default the set the theta is
The participant skill in the standard condition.
And then when it's in the hard condition, we're gonna have that B parameter that tells you, okay, actually we have an offset.
Like they tend to perform that way better or that way worse in the hard condition.
Okay, so that's like that shouldn't be too new for for you.
It's like I'm I'm sure you fit mixed models already, and that's that's what it's doing.
Okay, so here I'm showing you PyMC again because it forces you to put your assumptions out in the open.
You won't always have to do that, but I recommend at the beginning try and do that.
And then afterwards you can use tools like Bambi, BRMS to help you just abstract away all that stuff.
And then when when you need to do uh more complex models again, then you'll get back to PyMC because
You can customize everything.
So the the title of my talk contains these words: "already believes".
And so that's that's because every line of the model I just saw you, show you, showed you, sorry, is in a is an assumption, right?
Again, uh one of the assumptions is well, participants come from one shared population.
That's what we saw.
Um they have one we have one ability per participant, and that ability is stable across sessions.
So that's actually an assumption, right?
We'll need to test that depending on our experiments.
We have a condition that shifts everyone by the same amount.
Might not be true.
We might want to have an interaction depending on the participant, meaning that an interaction is just like that participant reacts more to a change in condition in
condition than the average participant, let's say.
And that usually happens a lot.
So here, sometimes you'll see if you're like a rule of thumb, if you start so first rule of thumb, always start with the simplest model, the dumbest model you can find.
Okay.
Um you always wanna try and and have the fanciest model at the beginning, don't.
Because it's not gonna work.
And so when the model doesn't work, doesn't sample, it's gonna be much easier to debug it if you started with something simple.
And then you add on top of that once you know the model samples well, but you know it's too simple for your case.
Well then you just add one layer of complexity after the other.
Second rule of thumb related to that, don't start with the interactions first.
Start with just your main effects.
And usually if you see that some of the main effects have very big standard deviation, posterior standard deviation, or very big posterior average, that may be a sign that you
need an interaction somewhere.
Because usually, especially in the social science, the effects are not that big.
So usually if you have a main effect that's really big, unless there is a difference of scale in the underlying data, that may be because the model is lacking an interaction an
interaction term.
But don't start with that.
So and fourth uh fourth assumption, we assume that the trials within a session are exchangeable.
Which again might not be true.
It like if you're thinking about sports, for instance, there is a huge psychological component to performance.
And a player, like for instance in tennis, psychology is extremely important.
If if the player starts to lose a lot of points, you will see that usually there is an autocorrelation because they start getting in their head.
And so here you don't have independent independent samples.
Often that's an assumption you can get away with in practice, honestly.
But sometimes it you won't and then you have to remember, wait, that's true.
We're making this assumption with that.
So now
Yeah, so I made that point already.
When you're using most of the time a frequentist software, you're making these assumptions.
It's just you don't see them because they are made for you silently.
Here, when you're using the Bayesian framework, you have to lay out your assumptions, which can intimidate people.
I would say it is actually one of the best things you can do, especially as researchers, because then it also makes reviewing your work much easier.
And sometimes you'll see that.
People end up having different conclusions about the same data, but it's because they have different priors.
And so that's interesting when you see people ending up like having different conclusions, but they observe the same data, then ask them about their priors.
And that's usually what's gonna what's gonna help you make sense of the situation.
So now, part two, the priors.
yeah, the priors are a big part.
of of the Bayesian workflow.
I love them.
Um but a lot of people are very intimidated by them.
So let's talk about priors.
How how do we do that actually in the wild?
I'm not gonna do any philosophy epistemology talk today.
not that I don't love that, I do, but I it's not the most practical way to introduce people uh to to the Bayesian framework.
I think practically showing you how you can do that in your models is much more interesting.
So
Priors is just another assumption.
And the great thing is that you can just inspect your assumptions in a Bayesian model.
And the way the way you do that, one of the main ways we do that is by doing something that's called prior predictive checks.
So that's a very intimidating term that you get you that you you can use, you know, at cocktail parties or stuff like that.
But it's very simple.
Like it's coming the the actual
Concept behind it is called forward sampling.
And that's because a Bayesian model, as I showed you, is generative.
And we we don't need the data to make the model.
I just I I didn't show any data yet, and we already have a model.
That's because in the Bayesian framework you come with that generative story.
And so that means your model can generate data.
It's not only here to infer the latent parameters.
But it can actually also generate potential data.
So that means you have two kinds of sampling in a Bayesian model.
You have the classical one, the one that you've heard about, which is like the backward sampling.
So that means we're coming from the likelihood, we observe the data and we push that backward into the model and we try to infer the latent parameters.
Here we just have one in this model, that's the P, the probability of success that we saw.
That's the actual quantity we're after.
And that's the hard sampling part.
That's where you need all the MCMC sampling and fancy stuff that you don't have to care about because we do that for you in PyMC or in Stan.
And then you have forward sampling, which is a great intuitive tool because it's just taking the parameters and then fixing them, pushing that forward into the model, and that
means you can get data before you actually observe data.
So that's awesome because you can already see if your model is assuming something stupid before you spend time and money in recruiting people for a study.
And you can already know if your model makes sense even before people come.
And that's for free.
So the way you do that is you draw parameters from the prior and then you simulate a fake study based on these fixed.
parameters.
Okay.
So here for instance, that would be okay, let's say p equals 0.45.
What would that what kind of world would that give me in reality?
What kind of data would I observe if the actual world was the probability of success in my experiment equals 45%?
And I'm saying 45%, but it's just like random.
What the software is gonna do for you under the hood is gonna sample that a lot of times, a lot of times, and then you're gonna get a lot of prior samples.
And then you're gonna look at these samples.
We're gonna do that just now, so don't worry.
Um, you're gonna look at these samples and you're gonna see, you're gonna check if it's absurd or not.
And if it's absurd, you can already make your model more compact and more.
And and smarter even before observing any data.
So that's what a prior predictive check is for.
Concretely, what does it look like?
So um a prior that's very common, like you'll see a lot of these priors like normal zero ten.
So normal distribution means zero, standard deviation ten.
I mean, who knows what a standard deviation 10 means, right?
It's just like it's I don't know.
I just know the variance is gonna be 100 in that case, because I I take the square to get the variance.
So yeah, maybe that's big.
I don't know.
It depends on the data.
If we're modeling rates, so that means quantities between zero and one, yeah, it's probably too big.
But if we're modeling distances in kilometers, well, you know, could be too small.
So the only way you can know that is to do prior predictive checks.
So here what I've done is I've written the model we just saw in PyMC and then did prior predictive checks.
And what I get afterwards is these kind of plots, which are called rootograms, um, which are basically like this is um this is actually very new, so that's that's cool.
Like you have actually researchers sh who like, you know, their their job is to come up with better plots for us afterwards.
So um you know them.
So you're you've been there in Helsinki, there is a big uh a big group of people who do amazing amazing work.
Uh and thanks to them we have that in the open source software.
And here, what that kind of prior gives us, it looks innocent, you know, when you see that in a model, but of like a lot of people will l will use these default priors, like zero
ten, looks looks good enough.
If you do that for that model that we just saw, that means that the counts of successes
that we expect.
So people coming into our lab, doing the the the the the trials that we gave them across sessions before seeing them, that model is expecting most of the people to either have
zero successes or one hundred successes out of one hundred.
I don't know a lot of experiments which e expect that kind of result.
Like that you that's probably a very stupid prior.
Because that means you're like, here all these quantities are very, very small.
And that's kind of fine in the Bayesian framework, because even if your priors are bad, if you get enough data and if none of the quantities here are literally zero, then Bayes
formula will do
its job and then afterwards you don't have any choice.
You have to update your priors.
But that means you need the here you'll need a lot of data to actually center around the posterior it's the true posterior distribution.
So it's better to actually have a better prior.
So how do we do that?
We can just simulate.
Well, there are actually some rules of of thumb now, you don't have to guess, okay?
Um and there is very good software to help you do that now and to do prior elicitation.
As you know, Suyog.
Um, so now it's super easy.
You can go into the software and tell it, well, I'm looking for a normal distribution whose 90% or 95% of the mass is between say two and six.
Just optimize that and give me the actual parameters that I would need for that prior.
So that's super easy.
And the great thing is that you don't have to do that yourself.
You can go to an expert and be like, hey, you know.
Like when I was working in baseball, that was literally what I was doing because when I was starting I had no idea what baseball was.
So I was there for the statistical models.
Thankfully, there was a lot of people who knew a lot about baseball in the team.
So I would just go to the coach and be like, Hey, you know, uh, for strikeout rates, what do you think the best like what do you think it's gonna look like before you see a player?
And then based on that, I would put that into the software and that would give me distributions that would look much more like that.
So a priori, if your model doesn't know a lot, you need that kind of distribution that you have here on the screen.
Okay.
That's a much more reasonable prior because it's expecting basically more or less a flat distribution across the probability probability of success.
Okay.
Actually I didn't
Tell you how to interpret these plots, even though they are pretty intuitive.
But so blue dots are the means, right?
Prior means of the distribution.
And the blue bars, remember, these are all distributions, so these are on the 94% uh confidence intervals.
I mean it's called credibility intervals in the Bayesian framework.
And that's great because you can actually interpret it the way you wanna interpret it in the frequentist framework, but you can't.
But here that literally means.
There is a 94% chance that all the probability mass, the true parameter, is gonna be in these in these blue bars.
Okay, that's just what it's giving you.
And uh why ninety-four percent?
it's just trolling the frequentists.
So you can you can like there is nothing magic about 94% intervals.
Um you can use other numbers, that's awesome.
Um
And mainly it's just one change of the call in the software, right?
Sometimes you will need other other um intervals.
And you can overlay them also.
You can have like the 50% interval, and then you can have a bigger one, like 95% or 90, or something that works well with people who don't know a lot of statistics is if you do like
two in three or five in six, they will have more intuition about that than ninety-five percent.
Statisticians will be like, give me ninety five percent intervals.
They don't know what.
W what are you talking about five in six?
But for non-technical people, that's great because you can tailor that to their intuition, which is what you want, because otherwise nobody's gonna use our model.
So that's that's a good prior predictive check basically, uh in a nutshell.
I would argue actually maybe that's too wide.
Like maybe that's too agnostic.
Because probably before people come in, we would expect that most of the people would be around here.
Whereas here that model is basically flat.
And like you would argue maybe I I expect if it's a classic test, I expect most of the people to be between twenty and eighty percent.
So I would want like a bell curve like that.
That's definitely doable.
Um I can show you that this afternoon.
But basically that that's what a prior predictive check is doing.
And if you do that a lot, that's a power analysis.
If you do that, if you repeat that sampling a lot of times, you get a power analysis for free.
So that's awesome because you can just set the effect that you would care about, and then you simulate a lot of study a lot of times, and you compute how often you would recover
the true effect.
Because here you set it like you're playing God, basically.
And so you're setting your effect, and you're like, okay, I care about that in particular.
So I want to know how many participants I would need to actually recover the true effect.
So you do that in a loop, basically, and then you compute how often do I recover the true parameter?
And then it's a trade-off, right?
If you can get more participants easily and you see that you get the true effect only fifty percent of the time, and that's not acceptable for you, then you can compute okay,
how many participants would I need to get to Yeah, my threshold here is three and four.
I need to be able to have the true effect three three times out of four.
So I'll I'll bump up the the number of participants.
So that's a pretty cool thing, you can do that without any data.
So now posterior.
That's uh that's the fun stuff.
Once you have your model, then you handle everything to the machine, and uh and well, you get your posterior distributions.
And that's what it looks like.
Uh here in the model I just showed you, we had three parameters.
You remember the population mean, that's the mu we had in the model.
The participant standard deviation, so that's how
Different people are in the actual population.
And then we had the condition effect.
Like how worse are people doing when we switch them to the hard condition?
So you can see that the model, that model in particular, in that particular simulation, the model was able to recover all the true parameters with reasonable uncertainty.
Okay, so again, this plot, pretty intuitive, but I'll I'll show you how to read it.
Uh here if we focus on the condition effect, that's telling us that on average people are worse by 0.55 when we switch them to the condition effect, the hard condition.
Okay, makes sense.
Like that's what we're expecting.
And the true value was 0.5.
And you can see that the distribution here, so this is the full distribution, okay.
That's all the posterior samples we got from the model.
But usually you summarize the distribution for people, and that's again the credible the credible interval.
So that's a 94% credible interval that you get out of the box from the software.
But again, you can customize it.
So here you can see that most of the time, like there is a 94% chance that the actual true parameter is between 0.6 and 0.52, something like that, 0.53.
So here, actually, for the condition effect, if that's something that's very important to you to get that right, I would argue that model is not doing a very good job.
Because that's in the tail, it it gets it, but it gets it in the tail of the distribution.
Like for that model, the cool thing of the Bayesian thing is that here, if you wanted the probability of that, actually the software compute that for you out of the box.
Here, if you compute the number of samples that are here to the left.
of the condition effect, of the true condition effect.
And you divide that by the total number of samples, posterior samples, that gives you the probability that the model expects that the condition effect was worse than it actually
was.
And so here it's telling us that basically there's like something like a 97-98% chance that the tr the true condition effect was worse according to the model than what was
observed.
But we know that what was observed was actually the truth.
So that model is not very doing a very good job on the condition effect.
And so if that's very important to you, then you need to improve your model here.
But if what's important to you is mostly to recover the population parameters, so mu, population mean, and participant sd, standard deviation, then the model is doing a very
good job and you can be confident that it should it should work on the true data.
Okay?
So here
The model is simple enough, so I would argue it should be able to recover all three parameters.
It's not it's not a complicated model.
Okay, so we'd we'd go back to the drawing board.
Here it's probably a matter of making the prior a bit better.
Do you remember the prior is a bit is a bit stupid.
Um so like yeah, we'll would make the prior a bit better and then we should be good.
So uh yeah, it's just what I told you, reiterating that again we don't get point estimates, right?
We get posterior distributions.
You can get point estimates after once, don't worry.
Like you see we have the means, you can you can compute any point estimates you want afterwards.
Like these are just posterior samples, you just you just count.
So if you want the median, you just look at what's in the middle of the distribution.
But you can get the percentiles also, and you can get that super easily, and the software will do that for you anyway.
So, yeah, that's that's a very good uh that's a very fun part of the Bayesian workflow.
Something that's awesome is that you get hierarchies for free in the Bayesian workflow, which is something very, very hard usually to get from frequentist models.
So these are called hierarchical uh models most of the time, or random effects models, stuff like that.
What you get from these models is that they model not only the participants.
But they will model the population also at the same time.
So that's literally what I just showed you.
But what you get from that is that you get something for free that's called partial pooling or shrinkage.
What does that mean?
So here in yellow you have the inferred population mean by the model.
So that's the mu that we saw before.
And in red, you have the data per participant per session.
So you see we have five uh six actually, sorry, session per participant.
no, it depends.
So we have a number of sessions per participant.
Okay.
And in blue, you have the partial pooling estimates, the mean partial pooling estimates from the model.
And this is based on just the first 20 trials of each participant.
But they do many more trials than that.
But let's say that you want some results at the very beginning and you want to have an idea of where you're going.
That is the thing you would see in your model.
And what does that mean?
You can see that the blue points are much closer to the yellow line than the red points, which is the data.
And that's the model, basically, hedging.
Because we don't have a lot of data yet.
We have only 20 trial, 20 trials per person.
That's not a lot.
We know they'll do much more.
So we need
be more careful.
And what the model is doing is you can see that the participants that are close already to the population mean, the shrinkage of the parameters is very small.
Okay?
But as the participants go to the extreme, whether bad or good, you can see that the model is much, much more aggressive.
And it's like, wait, wait, Mike, I only have 20 trials about that person.
I don't know if I can already call them that bad or that good.
So I'm gonna shrink their estimate
To the population mean.
So it's basically encoding the idea that extraordinary claims require extraordinary evidence.
If somebody is in the extreme, I need to have more evidence that it's actually the case.
And so hierarchical models do that for you for free.
And that's all depending on the population standard deviation.
Because if the population is much more di it's very diverse, then the model is gonna allow for much more extremes.
If it's very common,
then the model is going to be much more aggressive on the partial pooling.
I'm gonna start getting a bit faster because I talk a lot, but I want you to be able to ask questions.
So here is what I was telling you.
You see that here is the plot that I just showed you.
With 20 trial, the shrinkage is much more aggressive.
And as we increase the number of trials, you can see that basically now the mean estimate of the model is converging towards the observed data.
So that's what you want.
That's a good sign that the model is doing is doing a good job.
here, difference between the two kind of intervals.
Okay, I won't go too much into that, but the talk is available afterwards, so you can you can go into that.
But basically I I just talked to you about that already.
What I wanna emphasize here is that the blue intervals are the Bayesian credible intervals or posterior intervals based on the partial pooling model.
The good thing is that since you can interpret them as like here it's 90%, so
Since you can interpret that as there is a 90% chance that the true parameter is in there, you can actually test that assumption.
Because if you do that over the whole range of the probabilities, so 0 to 100%, you should be able to see if the model actually recovers the true parameters, well, 90% of the time.
Or if you're looking at a 50% interval, you expect the model to be right 50% of the time.
If that's the case, that means your model.
well calibrated.
And that's one of the most important checks you can you can do.
So that's the last part.
Checking your models.
So this is the posterior rootogram.
You remember we looked at the rootograms in the prior predictive checks.
Now we do the same thing but posterior predictive predictive checks.
So that means the universe is much smaller now because the model was updated, the model's parameters, latent parameters, were updated based on real data.
And so now, so these are two models, one that's blind to the session, and another the one that encodes the session.
So it's letting participant have a different skill per session.
Whereas that one has the strong assumption that participant skill doesn't vary by session.
It's like it's fixed.
And on that diagnosis, you're gonna really tell the two models apart.
Both models are fine.
You can see that both models have contracted.
The prior was very wide and now it's very much contracted.
The blu the um white dots are uh are the data points, by the way.
Okay.
And so yeah, like that's a good that's a good rootogram.
Okay.
I cannot distinguish the two models from that.
But I need to do other checks.
And what I care about is actually well, does my model actually recover the between session standard deviation?
And I simulated the data so I know that.
People vary by session.
And that makes sense most of the time.
You know, people get like they can be more tired, they didn't sleep well that day, or they are hungry, or they're distracted.
Lots of things can happen that don't depend on their actual latent skill.
So you probably want your model to be able to model that too.
And that model actually does recover this extremely well, whereas that one is expecting way less variation per session.
So that one is very probably overconfident and not that well calibrated.
And the way we can diagnose that is with actual calibration plots.
So that's exactly what I told you.
It's looking for each equal tailed interval, right?
So that's ETI, that's the technical term for posterior credible intervals in the Bayesian world.
So ETI for each number between zero and one hundred, I'm computing the ETI from the posterior of the model.
And comparing to the reality to what actually happened.
And if a model is well calibrated, you should expect it to be around zero here, around the zero line.
You shouldn't observe any big differences between what the model expects in probabilities and what the model actually sees in the data.
Here, that model, that session blind is very bad.
And if that goes down like that, that means your model is overconfident.
So not only is the model wrong, but it's like
Confidently wrong.
So this is not good.
uh The yellow dots are very influential.
I'll go into that later in the workshop, but basically that's what you need to know right now.
This is another way of looking at that.
This is not calibration, but it's called PIT.
We'll talk about that later.
And yes, so these are the two different models, right?
This is the one we had, which is binomial.
And P participant, one fixed per session.
And that one that was much better.
It just changed that no, actually, we have P of each participant, but on top of that, each participant is allowed to vary their probability of success per session, which comes from
another distribution that's called the beta distribution.
Uh don't worry about that, but yeah, that's just like a prior on that on that probability.
And
That probability then goes into the binomial.
And that's what's allowing the model to not be session blind anymore.
And that works much better as we as we just saw.
If you're in BRMS, that's just one call.
That's instead of having family equals binomial, you just do beta binomial.
And I shall show you a good naming convention for once in statistics.
It's literally we have a beta binomial.
So now, if I look at the calibration of the session aware model, that's what a good calibration looks like.
It's around zero.
And here, actually, that's the only p-value you'll see in that talk.
That's an actual test that's trying to reject the hypothesis that the model distribution is different from the uniform distribution.
Here you can see that at the 1% threshold, the
The hypothesis hypothesis cannot be rejected.
So it's a roundabout way to say it's calibrated.
Okay.
Uh and you can see that here also.
And no yellow dots, so that's great.
So when do you stop?
That there is no there is no magic number.
I'm sorry.
There is a bunch of diagnostics.
Uh Suyog, you know a lot about ELPD, so that's why you have here on the screen.
That's that's called LOO.
Leave one out cross-validation.
That's a way to estimate your model's out of sample prediction performance without actually doing it.
That's an algorithm, so that's not perfect, but that's like a good ballpark.
But don't stop there.
Like this is not telling you if the model is right.
It's not telling you if the model makes sense.
It's just telling you okay, which model is expected to do better out of sample.
doesn't mean it's a better causal model, just pure prediction.
So here, unsurprisingly, the session aware model is much better.
But again, Mike, don't stop at just one one number.
Okay.
It's a bunch of checks that we just ran and you're looking for red flags.
If you cannot find red flags, that's good sign.
You cannot ever know that your model is right, that your model is not uh very problematic.
I mean very problematic you'll see it very fast.
But like
You cannot prove that your model is right.
You can only prove that it's not very bad.
And the way you do that is you look for red flags and you never see them.
Well, that's very good sign.
And for that you need to run a bunch of different s different diagnostics, which is if we put that all together, this is basically the Bayesian workflow in in a nutshell.
And what I did is actually uh now that we have AI assistants, this is very like a workflow is not that
good for humans, right?
We we forget the steps and so on.
So what I did is that I encoded that into an agent skill that I open sourced.
And here is just um like I I I gave it to the same agent with the skill and without the skill and then I checked what it was doing differently.
I'll leave you read that I'll also talk about that later in in the workshop.
But basically with the skill it was doing a better job, which is great because it's what the skill is about.
but without the skill it was doing already a pretty good job, but it was failing on very important things like checking its own work and for instance switching from the plain
binomial to have a session aware model, which we saw was here actually the best move to make the model better.
So here and yes, here's the the skill.
I'll put that again in the in the final slide, so don't worry, but
it's open source so you can you can use that in your own agents, but the good thing is that you know the model that's under hood under the hood, so you can check actually what
the agents did, because otherwise you cannot.
Okay, awesome.
Thank you so much.
This has been another episode of Learning Bayesian Statistics.
Be sure to rate, review, and follow the show on your favorite podcatcher and visit learnbayesstats.com for more resources about today's topics, as well as access to more
episodes to help you reach true Bayesian state of mind.
That's learnbayesstats.com.
Our theme music is Good Bayesian by Baba Brinkman.
feat. MC Lars and Mega Ran.
Check out his awesome work at bababrinkman.com.
I'm your host.
Alex Andorra.
You can follow me on Twitter at Alex underscore Andorra like the country.
You can support the show and unlock exclusive benefits by visiting patreon.com slash learnbayesstats.
Thank you so much for listening and for your support.
You're truly a good Bayesian change your predictions after taking information.
And if you're thinking I'll be less than amazing, less adjust those expectations.
Let me show you how to be.
Good daisy change calculations after taking fresh dating Those predictions that your brain is making Let's get them on a solid foundation
A: Written as a generative story, a standard repeated-measures model says participants are drawn from a shared population, each has one latent ability that stays stable across sessions, a condition effect shifts everyone by the same amount, and trials within a session are independent. Each of those is testable and can be wrong: abilities drift between sessions, some participants react more to a hard condition than others (an interaction), and performance can be autocorrelated, the way a tennis player losing several points in a row starts getting in their own head. Frequentist software like lme4 makes these assumptions silently; writing the model in PyMC puts them in the open.
A: The generative story is a description of how the data you observe came to be. In a typical psychology experiment: a population of people, participants drawn from it, several sessions per participant, and several trials per session. That nesting already tells you the model needs a hierarchy. In the Bayesian workflow, you build the model by translating that story line by line into distributions, which is why you can write the whole model before seeing any data, and why the model can simulate data on its own.
A: Start with the simplest, dumbest model you can find. A complex model that doesn't sample is very hard to debug, while a simple model that samples well gives you a base to add one layer of complexity at a time. Likewise, don't start with interactions: fit the main effects first, and if one comes out with a very large posterior mean or a very wide posterior, that can be a sign an interaction is missing (unless there's a difference of scale in the data), especially in the social sciences where effects are usually modest.
A: A prior predictive check is forward sampling: draw parameter values from your priors, push them through the model to simulate fake studies, and look at whether the resulting data are plausible. Because a Bayesian model is generative, it can produce data before any participant is recruited. If the simulated worlds are absurd, you fix the model before spending time and money on data collection, essentially for free.
A: Repeat the prior predictive simulation many times with a fixed effect size you care about, fit the model to each simulated study, and count how often it recovers the true effect. Then trade off recovery rate against participants: if you only recover the effect half the time and need three times out of four, you increase the sample size until you get there. The same simulations feed a pre-registration: the model, its generative story, the expected results and the probability of detecting the effect, all before collecting any data.
A: A hierarchical model estimates participants and the population at the same time, so each participant's estimate borrows strength from the others. With only twenty trials per person, estimates for participants near the population mean barely move, while extreme participants get pulled strongly toward it, since the model isn't ready to call someone exceptional on so little evidence. As trials accumulate, the estimates converge to each participant's observed data. How strongly the model shrinks depends on the population standard deviation it infers.
A: A 90% credible interval says there's a 90% probability the value is inside it, given the model. If the model is calibrated, such intervals contain the truth about 90% of the time, and you can test that across the whole range of interval widths, comparing expected and observed coverage. A well-calibrated model stays around the zero line on a calibration plot. If the curve drops below it, the model is overconfident: not only wrong, but confidently wrong.
A: When a participant's success probability can change from session to session for reasons unrelated to their latent skill, such as sleeping badly or being hungry. In the talk's example, a session-blind binomial model and a session-aware beta-binomial model had equally good posterior predictive rootograms, but only the beta-binomial recovered the between-session variation and stayed calibrated, while the binomial was overconfident. In brms, the switch is just changing the family from binomial to beta-binomial.
A: No. LOO estimates a model's out-of-sample predictive performance without refitting it on held-out data, and it's a good ballpark, but it doesn't tell you whether the model makes sense or is a better causal model. There is no magic number in the Bayesian workflow: you run many diagnostics and look for red flags. You can never prove a model is right; you can only hunt for red flags and fail to find them.
A: The workflow has many steps that humans tend to forget, so Alex encoded it in an open-source agent skill in baygent-skills. Given the same data with and without the skill, the assistant did a decent job either way, but without the skill it never reviewed its own work or checked calibration, and never switched from the binomial to the beta-binomial; with the skill, it reviewed its own draft, caught diagnostics it had glossed over, and rebuilt the model as a beta-binomial. You still need to understand the model to be the human in the loop and evaluate what the agent did.