The Many Choices for Estimating Treatment Effects in Blocked, Cluster-Randomized Trials

Dr. Luke Miratrix

Harvard University

Friday, January 30, 2026, from 11am ET

Gentry 144

https://tinyurl.com/rmme-Miratrix2

Blocked, cluster-randomized controlled trials are a common tool in education and other fields due to the natural nested structure of data (e.g., students in schools, in districts). Such trials present unique challenges in estimating overall average treatment effects, especially in the face of possible impact heterogeneity and variation in block and cluster sizes. Researchers must decide whether to target the average effect across individuals or clusters and, given that choice, select from a wide range of estimators (we have identified more than 30) that make different bias-variance tradeoffs and rely on various assumptions. These choices are further complicated by degrees-of-freedom constraints and the instability of nominally unbiased estimators. To assess the consequence of these choices in practice, we examine over 30 field trials in the social sciences, applying 32 estimators, including various design-based estimators, aggregation approaches, linear models, and random-effects models to each associated dataset for all primary outcomes. We then evaluate the extent to which different methodological choices impact estimated effects and estimated standard errors. Point estimates can vary substantially, and estimated standard errors can easily differ by a factor of 3 or more. We supplement our findings with a calibrated simulation and theoretical inquiry to identify what drives found differences. We close with practical guidance to researchers navigating these issues.

Loading...

Taking too long?

Reload document

| Open in new tab

Item Response Theory (IRT), Introduction to Bayesian Statistics, Bayesian IRT, & Machine Learning in STATA

Dr. Chuck Huber

StataCorp

Friday, October 24, from 10am - 3pm ET

https://tinyurl.com/rmme-Huber

On Friday, October 24th, Dr. Chuck Huber will give a series of talks focused on statistical analyses using the popular statistical software package, STATA. The day will include a series of 60-minute talks, one on each of the following topics:

Item Response Theory: In this talk, I introduce the concepts and jargon of item response theory including latent traits such as ability, item characteristic curves, difficulty, discrimination, guessing, and differential item functioning. I also demonstrate how to use Stata's -irt- commands to fit 1PL, 2PL, and 3PL models for binary items as well as partial credit, generalized partial credit, rating scale, and graded response models for ordinal outcomes.

Introduction to Bayesian Statistics Using Stata: Bayesian analysis has become a popular tool for many statistical applications. Yet many data analysts have little training in the theory of Bayesian analysis and software used to fit Bayesian models. This talk will provide an intuitive introduction to the concepts of Bayesian analysis and demonstrate how to fit Bayesian models using Stata. No prior knowledge of Bayesian analysis is necessary, and specific topics will include the relationship between likelihood functions, prior, and posterior distributions, Markov Chain Monte Carlo (MCMC) using the Metropolis-Hastings algorithm, and use of Stata's Bayes prefix to fit Bayesian models.

Bayesian Item Response Theory: In this talk, I briefly review item response theory (IRT) and introduce the concepts and jargon of Bayesian statistics. Then, I demonstrate how to use Stata's -bayesmh- command to fit 3PL, 4PL, and 5PL IRT models which cannot be fit using maximum likelihood. I finish by showing how to compare the fit of the Bayesian IRT models.

Introduction to Machine Learning and AI Using Stata:This talk will briefly review the history of machine learning (ML) and artificial intelligence (AI), introduce relevant concepts and language, and demonstrate how to use these tools in Stata. Specific examples may include Lasso and elasticnet methods, Bayesian methods and MCMC, support vector machines (SVM) using Python integration, random forests and gradient boosting machines using H2O, and the user-written commands "chatgpt", "claude", "gemini", and "grok".

Loading...

Taking too long?

Reload document

| Open in new tab

RMME/STAT Joint Colloquium

Power Analysis Has Lost Its Way: New Methods To Bring It Back Home

Dr. Gregory Hancock

University of Maryland

Friday, September 26, at 11 AM ET

https://tinyurl.com/rmme-Hancock

In a time when the alarms of research replicability are sounding louder than ever, mapping out studies with statistical and inferential integrity is of paramount importance. Indeed, funding agencies almost always require grant applicants to present compelling a priori power analyses to justify proposed sample sizes, in an effort to ensure a sound investment. Unfortunately, even researchers’ most sincere attempts at sample size planning are fraught with the fundamental challenge of setting numerical values not just for the focal parameters for which statistical tests are planned, but for each of the model’s other more peripheral or contextual parameters as well. As we plainly demonstrate, regarding the latter parameters, even in very simple models, well-intentioned numerical guesses that are even slightly off can undermine power for the assessment of the more focal parameters that are of key theoretical interest. Toward remedying this all-too-common but seemingly underestimated problem in power analysis, we adopt a hope-for-the-best-but-plan-for-the-worst mindset and present new methods that attempt (1) to restore appropriate conservatism and robustness, and in turn credibility, to the sample size planning process, and (2) to greatly simplify that process. Derivations and suggestions for practice are presented using the framework of measured variable path analysis models as they subsume many of the types of models (e.g., multiple linear regression, ANOVA) for which sample size planning is of interest.

Loading...

Taking too long?

Reload document

| Open in new tab

RMME/STAT Joint Colloquium

Bayesian Longitudinal (Non)Linear Mediation Models

Dr. Nidhi Kohli

University of Minnesota, Twin Cities

Friday, May 2, at 11 AM ET

https://tinyurl.com/rmme-Kohli

This study develops Bayesian (non)linear random effects mediation models (B(N)REMM) to directly estimate both linear and nonlinear longitudinal mediation effects, overcoming limitations in existing structural equation modeling (SEM) approaches. We propose two models: a linear trend model (L-BREMM) and a segmented trend model using linear-linear piecewise functions with random changepoints (P-BREMM). We also examine the impact of omitting confounders in (non)linear mediation models using data from the Early Childhood Longitudinal Study-Kindergarten Cohort (ECLS-K). Furthermore, we extend this framework to develop Bayesian (nonlinear) growth mixture mediation models (B(N)GMMM), which assess heterogeneous treatment effects (HTE) of the intervention variable X on the longitudinal dependent variable Y, mediated by longitudinal variable M. To evaluate the robustness of these methods, we conducted a comprehensive Monte Carlo simulation study for all the models.

Loading...

Taking too long?

Reload document

| Open in new tab

RMME/STAT Joint Colloquium

Adventitious Error Everywhere

Dr. Paul De Boeck

The Ohio State University

Friday, February 7, at 11 AM ET

https://tinyurl.com/rmme-PDeBoeck

Adventitious error is a concept introduced by Wu and Browne (Psychometrika, 2015) to explain imperfect goodness of fit of covariance structure models (CSMs, i.e., factor models, SEM). The paper was published together with critical remarks from the reviewers. In my presentation I will discuss, illustrate, and speculate about the potential of adventitious error beyond CSM, (a) as a unitary framework to understand and deal with underestimated inferential uncertainty regarding relations between variables, heterogeneity in meta-analysis, violations of measurement invariance, individual differences in the validity of tests, and (b) as a joint framework for reliability and validity. The presentation is partly based on De Boeck, DeKay, and Pek (Psychometrika, 2024, 89, 1055-1073).

Loading...

Taking too long?

Reload document

| Open in new tab

RMME/STAT Joint Colloquium

Challenges in Time-varying Causal Effect Moderation Analysis in Mobile Health

Dr. Walter Dempsey

University of Michigan

Friday, January 24, at 11 AM ET

https://tinyurl.com/rmme-Dempsey

Twin revolutions in wearable technologies and smartphone-delivered digital health interventions have significantly expanded the accessibility and uptake of mobile health (mHealth) interventions in multiple domains of health sciences. Sequentially randomized experiments called micro-randomized trials (MRTs) have grown in popularity as a means to empirically evaluate the effectiveness of mHealth intervention components. MRTs have motivated a new class of causal estimands, termed “causal excursion effects”, that allow health scientists to answer important scientific questions about how intervention effectiveness may change over time or be moderated by individual characteristics, time-varying context, or past responses. In this talk, we present two new tools for causal effect moderation analysis. First, we consider a meta-learner perspective, where any supervised learning algorithm can be used to assist in the estimation of the causal excursion effect. We will present theoretical results and accompanying simulation experiments to demonstrate relative efficiency gains. Practical utility of the proposed methods is demonstrated by analyzing data from a multi-institution cohort of first year medical residents in the United States. Second, we will consider effect moderation with tens or hundreds of potential moderators. In this setting, it becomes necessary to use the observed data to select a simpler model for effect moderation and then make valid statistical inference. We propose a two-stage procedure to solve this problem that leverages recent advances in post-selective inference using randomization. We will discuss asymptotic validity of the conditional selective inference procedure and the importance of randomization. Simulation studies verify the asymptotic results. We end with an analysis of an MRT for promoting physical activity in cardiac rehabilitation to demonstrate the utility of the method.

Loading...

Taking too long?

Reload document

| Open in new tab

RMME/STAT Joint Colloquium

Recent Lessons on Mixing Mail and Web Data Collection Modes in Household Surveys

Dr. Kristen Olson

University of Nebraska-Lincoln

Friday, December 6, at 11 AM ET

https://tinyurl.com/rmme-Olson

Survey designs are increasingly incorporating self-administered modes – namely mail and web – to counter decreasing response rates and increasing costs in interviewer-administered surveys. In this talk, I will cover recent research related to mixing modes of self-administered data collection in household surveys, with a focus on designing for survey participation. I will explore theory for why sample members may select one self-administered mode over another, how modes affect response rates, representation and costs, and design decisions related to recruitment materials for mixed-mode surveys.

Loading...

Taking too long?

Reload document

| Open in new tab

RMME/STAT Joint Colloquium

Effects Across Time to Modeling States of a System: A Dynamical Systems Perspective on Modeling Social Science Data

Dr. Pascale Deboeck

University of Utah

Friday, November 15, at 11 AM ET

https://tinyurl.com/rmme-Deboeck

Modeling of repeated observations across time generally falls into one of two categories. The first is modeling observations as they change across time; that is, as a function of time. The second is to model the observations as they relate to prior observations; many common models in the social sciences are represented in this second category, including auto-regressive models, cross-lagged panel models, and latent difference score models. A less-common approach in this latter category are differential equation models. These models express relations between the state of observations and how they are changing. These models offer the opportunity to imagine change relations that may be commonly overlooked when modeling repeated measures. This talk will introduce the application of differential equation modeling to intensive, longitudinal data. Specific examples will include estimating derivatives from noisy data and the possibility of testing novel models of intraindividual dynamics.

Loading...

Taking too long?

Reload document

| Open in new tab

RMME/STAT Joint Colloquium

Assessment of Fit of Item Response Theory Models: Full-information and Limited-information Methods, Item and Person Fit Analysis, and Beyond

Dr. Sandip Sinharay

Educational Testing Service (ETS) Research Institute

Friday, October 11, at 11:15 AM ET

AUST 110

https://tinyurl.com/rmme-Sinharay

Item response theory (IRT) is one of the central methodological pillars supporting many large and high-profile assessment programs globally. IRT analysis is essentially a type of discrete multivariate analysis and is performed using IRT models that are latent variable models for discrete data. However, IRT models involve multiple assumptions like conditional independence, monotonicity etc. and the results obtained from IRT models may not be accurate if one or more of the assumptions are not met, that is, if there is IRT model misfit. This presentation will include a comprehensive review of the literature on the assessment of fit of IRT models. The presenter will discuss various approaches and concepts regarding IRT model fit including full-information and limited-information methods, residual analysis, item and person-fit analysis, Bayesian methods, analysis for differential item functioning, and assessment of practical significance of misfit. A real data example will be used to illustrate some of the approaches. One goal of the presentation is to stimulate discussions involving the audience members regarding IRT model-fit assessment.

Loading...

Taking too long?

Reload document

| Open in new tab