The Many Choices for Estimating Treatment Effects in Blocked, Cluster-Randomized Trials

Dr. Luke Miratrix

Harvard University

Friday, January 30, 2026, from 11am ET

Gentry 144

https://tinyurl.com/rmme-Miratrix2

Blocked, cluster-randomized controlled trials are a common tool in education and other fields due to the natural nested structure of data (e.g., students in schools, in districts). Such trials present unique challenges in estimating overall average treatment effects, especially in the face of possible impact heterogeneity and variation in block and cluster sizes. Researchers must decide whether to target the average effect across individuals or clusters and, given that choice, select from a wide range of estimators (we have identified more than 30) that make different bias-variance tradeoffs and rely on various assumptions. These choices are further complicated by degrees-of-freedom constraints and the instability of nominally unbiased estimators. To assess the consequence of these choices in practice, we examine over 30 field trials in the social sciences, applying 32 estimators, including various design-based estimators, aggregation approaches, linear models, and random-effects models to each associated dataset for all primary outcomes. We then evaluate the extent to which different methodological choices impact estimated effects and estimated standard errors. Point estimates can vary substantially, and estimated standard errors can easily differ by a factor of 3 or more. We supplement our findings with a calibrated simulation and theoretical inquiry to identify what drives found differences. We close with practical guidance to researchers navigating these issues.

Loading...

Taking too long?

Reload document

| Open in new tab

Item Response Theory (IRT), Introduction to Bayesian Statistics, Bayesian IRT, & Machine Learning in STATA

Dr. Chuck Huber

StataCorp

Friday, October 24, from 10am - 3pm ET

https://tinyurl.com/rmme-Huber

On Friday, October 24th, Dr. Chuck Huber will give a series of talks focused on statistical analyses using the popular statistical software package, STATA. The day will include a series of 60-minute talks, one on each of the following topics:

Item Response Theory: In this talk, I introduce the concepts and jargon of item response theory including latent traits such as ability, item characteristic curves, difficulty, discrimination, guessing, and differential item functioning. I also demonstrate how to use Stata's -irt- commands to fit 1PL, 2PL, and 3PL models for binary items as well as partial credit, generalized partial credit, rating scale, and graded response models for ordinal outcomes.

Introduction to Bayesian Statistics Using Stata: Bayesian analysis has become a popular tool for many statistical applications. Yet many data analysts have little training in the theory of Bayesian analysis and software used to fit Bayesian models. This talk will provide an intuitive introduction to the concepts of Bayesian analysis and demonstrate how to fit Bayesian models using Stata. No prior knowledge of Bayesian analysis is necessary, and specific topics will include the relationship between likelihood functions, prior, and posterior distributions, Markov Chain Monte Carlo (MCMC) using the Metropolis-Hastings algorithm, and use of Stata's Bayes prefix to fit Bayesian models.

Bayesian Item Response Theory: In this talk, I briefly review item response theory (IRT) and introduce the concepts and jargon of Bayesian statistics. Then, I demonstrate how to use Stata's -bayesmh- command to fit 3PL, 4PL, and 5PL IRT models which cannot be fit using maximum likelihood. I finish by showing how to compare the fit of the Bayesian IRT models.

Introduction to Machine Learning and AI Using Stata:This talk will briefly review the history of machine learning (ML) and artificial intelligence (AI), introduce relevant concepts and language, and demonstrate how to use these tools in Stata. Specific examples may include Lasso and elasticnet methods, Bayesian methods and MCMC, support vector machines (SVM) using Python integration, random forests and gradient boosting machines using H2O, and the user-written commands "chatgpt", "claude", "gemini", and "grok".

Loading...

Taking too long?

Reload document

| Open in new tab

RMME/STAT Joint Colloquium

Power Analysis Has Lost Its Way: New Methods To Bring It Back Home

Dr. Gregory Hancock

University of Maryland

Friday, September 26, at 11 AM ET

https://tinyurl.com/rmme-Hancock

In a time when the alarms of research replicability are sounding louder than ever, mapping out studies with statistical and inferential integrity is of paramount importance. Indeed, funding agencies almost always require grant applicants to present compelling a priori power analyses to justify proposed sample sizes, in an effort to ensure a sound investment. Unfortunately, even researchers’ most sincere attempts at sample size planning are fraught with the fundamental challenge of setting numerical values not just for the focal parameters for which statistical tests are planned, but for each of the model’s other more peripheral or contextual parameters as well. As we plainly demonstrate, regarding the latter parameters, even in very simple models, well-intentioned numerical guesses that are even slightly off can undermine power for the assessment of the more focal parameters that are of key theoretical interest. Toward remedying this all-too-common but seemingly underestimated problem in power analysis, we adopt a hope-for-the-best-but-plan-for-the-worst mindset and present new methods that attempt (1) to restore appropriate conservatism and robustness, and in turn credibility, to the sample size planning process, and (2) to greatly simplify that process. Derivations and suggestions for practice are presented using the framework of measured variable path analysis models as they subsume many of the types of models (e.g., multiple linear regression, ANOVA) for which sample size planning is of interest.

Loading...

Taking too long?

Reload document

| Open in new tab

RMME/STAT Joint Colloquium

Bayesian Longitudinal (Non)Linear Mediation Models

Dr. Nidhi Kohli

University of Minnesota, Twin Cities

Friday, May 2, at 11 AM ET

https://tinyurl.com/rmme-Kohli

This study develops Bayesian (non)linear random effects mediation models (B(N)REMM) to directly estimate both linear and nonlinear longitudinal mediation effects, overcoming limitations in existing structural equation modeling (SEM) approaches. We propose two models: a linear trend model (L-BREMM) and a segmented trend model using linear-linear piecewise functions with random changepoints (P-BREMM). We also examine the impact of omitting confounders in (non)linear mediation models using data from the Early Childhood Longitudinal Study-Kindergarten Cohort (ECLS-K). Furthermore, we extend this framework to develop Bayesian (nonlinear) growth mixture mediation models (B(N)GMMM), which assess heterogeneous treatment effects (HTE) of the intervention variable X on the longitudinal dependent variable Y, mediated by longitudinal variable M. To evaluate the robustness of these methods, we conducted a comprehensive Monte Carlo simulation study for all the models.

Loading...

Taking too long?

Reload document

| Open in new tab

RMME/STAT Joint Colloquium

Adventitious Error Everywhere

Dr. Paul De Boeck

The Ohio State University

Friday, February 7, at 11 AM ET

https://tinyurl.com/rmme-PDeBoeck

Adventitious error is a concept introduced by Wu and Browne (Psychometrika, 2015) to explain imperfect goodness of fit of covariance structure models (CSMs, i.e., factor models, SEM). The paper was published together with critical remarks from the reviewers. In my presentation I will discuss, illustrate, and speculate about the potential of adventitious error beyond CSM, (a) as a unitary framework to understand and deal with underestimated inferential uncertainty regarding relations between variables, heterogeneity in meta-analysis, violations of measurement invariance, individual differences in the validity of tests, and (b) as a joint framework for reliability and validity. The presentation is partly based on De Boeck, DeKay, and Pek (Psychometrika, 2024, 89, 1055-1073).

Loading...

Taking too long?

Reload document

| Open in new tab