Skip to main content

Appendix B — Statistical Analysis Plan (mock): clinical performance study SIM-217-001

Two-cut-off plasma pTau217-like test vs amyloid PET. Simulated data only.

Status of this document. A teaching SAP. The study, device, data and numbers are simulated; the structure follows ICH E9 by analogy and the content items of IVDR Annex XIII Part A 2.3.2 (statistical design, endpoints, population size, method for determining true clinical status). No claim is made that a regulator accepts this plan. Tags: [S] public source, [A] assumption, [Y] my synthesis.

Field Entry
Document ID / version SAP-SIM217-001 v1.0 (DRAFT)
Protocol reference CPSP-SIM217-001 (placeholder)
Author / independent reviewer / approver names and dates: placeholders
Applicable SOPs SOP-STAT-xxx (programming), SOP-QC-xxx (QC), SOP-DM-xxx (data management): placeholders
Version history v0.1 outline; v1.0 locked before unblinding of reference-standard data

1. Objectives and hypotheses

Intended use (simulated). Aid in identifying amyloid pathology in adults aged 55 years and older with signs, symptoms or complaints of cognitive decline; used with other clinical information; not stand-alone; not for monitoring. [A]

Primary objective. Show that a positive result carries a high probability of amyloid-PET positivity and a negative result a low one.

Co-primary endpoint Performance goal (H0) Expected value Success rule
P1: PPV of the positive category PPV ≤ 0.75 0.88 [A] lower limit of two-sided 95% Wilson CI > 0.75
P2: NPV of the negative category NPV ≤ 0.90 0.95 [A] lower limit of two-sided 95% Wilson CI > 0.90

Success requires both (intersection-union): no alpha adjustment is needed because each test is run at the one-sided 0.025 level and both must reject. [Y]

Secondary objectives. Intermediate-zone fraction; PPA and NPA with intermediates counted as non-agreement; PPV/NPV of each category with likelihood ratios; performance by pre-specified subgroups.

2. Design, reference standard and cut-offs

  • Design. Prospective-retrospective, banked K2-EDTA plasma, multi-site, primary and secondary care. [A]
  • Reference standard. Amyloid PET visual read by a blinded reader panel (≥ 3 readers, majority rule; unreadable scan = missing). The interval between blood draw and PET is recorded; per-protocol requires ≤ 180 days. Readers are blind to the index test; index-test operators are blind to PET. [A]
  • Cut-offs (locked). Derived on a separate simulated derivation cohort (n = 700) by derive_two_cutoffs() with targets sensitivity ≥ 94% (rule-out) and specificity ≥ 93% (rule-in); stored in data/locked_cutoffs.csv and not re-estimated on performance-study data. Convention as in the public decision summary [S]: negative if ≤ lower cut-off, positive if > upper cut-off.
kable(data.frame(Category = c("Negative (rule-out)", "Intermediate", "Positive (rule-in)"),
                 `Rule (pg/mL)` = c(sprintf("<= %.3f", LO), sprintf("> %.3f to <= %.3f", LO, HI), sprintf("> %.3f", HI)),
                 check.names = FALSE))
Category Rule (pg/mL)
Negative (rule-out) <= 0.229
Intermediate > 0.229 to <= 0.418
Positive (rule-in) > 0.418

3. Analysis populations

Population Definition Role
Enrolled all subjects with a plasma sample disposition
Evaluable (primary) valid assay result and readable PET primary analysis, the diagnostic analogue of the ITT/full-analysis set [Y]
Per-protocol evaluable, PET interval ≤ 180 days, no major deviation sensitivity analysis

4. Endpoints and statistical methods

  • Proportions: Wilson 95% CIs (primary); Clopper–Pearson as sensitivity. PPV = positive-category subjects with PET+ / all positive-category subjects; NPV analogously. Both are conditional on category and prevalence-dependent; observed prevalence reported with CI.
  • Likelihood ratios per category with log-scale 95% CI (asymptotic, ratio of two proportions).
  • PPA = (positive result among PET+); NPA = (negative result among PET−); intermediates count as non-agreement (as in the public decision summary [S]). Rule-out step sensitivity and rule-in step specificity as additional descriptors.
  • No imputation of PET. Subgroup CIs are descriptive; interaction tests are exploratory and not used for claims.

5. Sample size

Planning assumptions [A]: positive-category frequency 0.336 and negative-category frequency 0.465, taken from the public pivotal data of a comparable cleared test (FDA K261686, p.27 [S]) and used here only as planning inputs; 8% loss to non-evaluable.

n_pos <- n_exact_goal_stable(0.88, 0.75, alpha = 0.025, power = 0.90)   # subjects with a positive result
n_neg <- n_exact_goal_stable(0.95, 0.90, alpha = 0.025, power = 0.90)   # subjects with a negative result
N_pos <- ceiling(n_pos / 0.336); N_neg <- ceiling(n_neg / 0.465)
N_eval <- max(N_pos, N_neg); N_enr <- ceiling(N_eval / 0.92)
kable(data.frame(Endpoint = c("P1 PPV (exact one-sided 0.025, power 0.90)", "P2 NPV (exact one-sided 0.025, power 0.90)"),
                 `Subjects needed in category` = c(n_pos, n_neg), `Evaluable subjects needed` = c(N_pos, N_neg),
                 check.names = FALSE))
Endpoint Subjects needed in category Evaluable subjects needed
P1 PPV (exact one-sided 0.025, power 0.90) 104 310
P2 NPV (exact one-sided 0.025, power 0.90) 316 680 
c(evaluable_required = N_eval, enrolled_required = N_enr, planned_enrolled = nrow(dat))
evaluable_required  enrolled_required   planned_enrolled 
               680                740               1100

Joint power of the two co-primaries is approximately the product of the individual powers when the two are nearly independent (≈ 0.81 for 0.90 each) [Y]; the planned N exceeds the requirement, which raises it. Cross-check method: the exact calculation matches MKmisc::power.diagnostic.test (Chapter 7, tested).

6. Missing, invalid and unreadable results

  • Invalid assay results are repeated once per the instructions for use (not simulated); results still invalid are excluded from the evaluable set and reported in the flow table.
  • Unreadable PET = missing reference; excluded from evaluable. Sensitivity analysis S2 (worst case): invalid assay results among PET+ are counted as negative, among PET− as positive.
  • A STARD flow diagram (Table 1) documents all exclusions. [S: STARD 2015 has 30 items]

7. Subgroups (descriptive)

Age (55–70, 71–85), sex, care setting (primary, secondary), MMSE (15–24, 25–27, 28–30), kidney function by eGFR (< 30, 30–59, ≥ 60), with a minimum of 20 subjects per category for a CI to be reported. Kidney function is pre-specified because the public decision summary of the comparable test contains no eGFR-stratified analysis (only a small kidney-disease comorbidity subgroup) [S].

8. Multiplicity, interim analyses, sensitivity analyses

  • Co-primary success rule above; secondary and subgroup results are not adjusted and are labelled descriptive. No interim analysis (fixed sample).
  • S1 per-protocol; S2 worst-case for invalid assay results; S3 Clopper–Pearson intervals.

9. Data standards

CDISC publishes SDTMIG for Medical Devices (v1.1, 2019) and ADaMIG for Medical Devices (v1.0, 2021) [S: cdisc.org]; no IVD-specific CDISC standard was found, and nothing verified says CDISC is required for IVDR submissions. This mock uses ADaM-style analysis datasets (subject-level ADSL, performance ADPERF) by analogy, with variable names and derivation notes in the programming specification. [Y]

10. Software and quality control

Software, validation and QC are in the companion note SAP in a regulated R environment. The primary endpoints are double-programmed; the QC comparison is executed below.

11. Table shells and dry run on simulated data

d <- dat
d$assay_valid <- !is.na(d$PTAU217); d$pet_valid <- !is.na(d$PET_VISUAL)
d$EVAL <- d$assay_valid & d$pet_valid
d$CAT <- ifelse(d$EVAL, as.character(classify_two_cut(d$PTAU217, LO, HI)), NA)
d$CAT <- factor(d$CAT, levels = c("rule-out", "intermediate", "rule-in"))
ev <- d[d$EVAL, ]

Table 1. Disposition (STARD flow).

kable(data.frame(Step = c("Enrolled", "Valid assay result", "Readable PET", "Evaluable (primary set)", "Per-protocol subset"),
                 N = c(nrow(d), sum(d$assay_valid), sum(d$pet_valid), sum(d$EVAL), sum(d$EVAL & d$PPFL == "Y"))))
Step N
Enrolled 1100
Valid assay result 1078
Readable PET 1067
Evaluable (primary set) 1046
Per-protocol subset 1028

Table 2. Baseline characteristics of the evaluable set by PET status.

b <- function(f) c(f(ev[ev$PET_VISUAL == 0, ]), f(ev[ev$PET_VISUAL == 1, ]), f(ev))
kable(data.frame(Characteristic = c("n", "Age, mean (SD)", "Female, n (%)", "Primary care, n (%)", "MMSE, mean (SD)", "eGFR < 60, n (%)"),
  `PET-` = c(sum(ev$PET_VISUAL == 0), sprintf("%.1f (%.1f)", mean(ev$AGE[ev$PET_VISUAL == 0]), sd(ev$AGE[ev$PET_VISUAL == 0])),
             sprintf("%d (%.0f)", sum(ev$SEX[ev$PET_VISUAL == 0] == "F"), 100 * mean(ev$SEX[ev$PET_VISUAL == 0] == "F")),
             sprintf("%d (%.0f)", sum(ev$SETTING[ev$PET_VISUAL == 0] == "primary"), 100 * mean(ev$SETTING[ev$PET_VISUAL == 0] == "primary")),
             sprintf("%.1f (%.1f)", mean(ev$MMSE[ev$PET_VISUAL == 0]), sd(ev$MMSE[ev$PET_VISUAL == 0])),
             sprintf("%d (%.0f)", sum(ev$EGFR[ev$PET_VISUAL == 0] < 60), 100 * mean(ev$EGFR[ev$PET_VISUAL == 0] < 60))),
  `PET+` = c(sum(ev$PET_VISUAL == 1), sprintf("%.1f (%.1f)", mean(ev$AGE[ev$PET_VISUAL == 1]), sd(ev$AGE[ev$PET_VISUAL == 1])),
             sprintf("%d (%.0f)", sum(ev$SEX[ev$PET_VISUAL == 1] == "F"), 100 * mean(ev$SEX[ev$PET_VISUAL == 1] == "F")),
             sprintf("%d (%.0f)", sum(ev$SETTING[ev$PET_VISUAL == 1] == "primary"), 100 * mean(ev$SETTING[ev$PET_VISUAL == 1] == "primary")),
             sprintf("%.1f (%.1f)", mean(ev$MMSE[ev$PET_VISUAL == 1]), sd(ev$MMSE[ev$PET_VISUAL == 1])),
             sprintf("%d (%.0f)", sum(ev$EGFR[ev$PET_VISUAL == 1] < 60), 100 * mean(ev$EGFR[ev$PET_VISUAL == 1] < 60))),
  check.names = FALSE))
Characteristic PET- PET+
n 640 406
Age, mean (SD) 70.7 (7.3) 71.9 (6.6)
Female, n (%) 340 (53) 219 (54)
Primary care, n (%) 204 (32) 67 (17)
MMSE, mean (SD) 25.9 (2.9) 25.6 (2.8)
eGFR < 60, n (%) 142 (22) 78 (19)

Table 3. Result category by amyloid PET.

t3 <- table(ev$CAT, ev$PET_VISUAL); colnames(t3) <- c("PET-", "PET+")
kable(addmargins(t3))
PET- PET+ Sum
rule-out 462 26 488
intermediate 144 77 221
rule-in 34 303 337
Sum 640 406 1046

Table 4. Primary endpoints and decision.

pp <- two_cut_performance(ev$PTAU217, ev$PET_VISUAL, LO, HI)
ppv <- pp$rulein_ppv; npv <- pp$ruleout_npv
kable(data.frame(
  Endpoint = c("P1 PPV positive", "P2 NPV negative"),
  `n/N` = c(sprintf("%d/%d", t3["rule-in", "PET+"], sum(t3["rule-in", ])), sprintf("%d/%d", t3["rule-out", "PET-"], sum(t3["rule-out", ]))),
  `Estimate % (95% Wilson CI)` = c(pct(ppv), pct(npv)),
  Goal = c("> 75%", "> 90%"),
  Met = c(ppv[["lower"]] > 0.75, npv[["lower"]] > 0.90), check.names = FALSE))
Endpoint n/N Estimate % (95% Wilson CI) Goal Met
P1 PPV positive 303/337 89.9 (86.2-92.7) > 75% TRUE
P2 NPV negative 462/488 94.7 (92.3-96.3) > 90% TRUE 
cat("Study success (both co-primaries met):", ppv[["lower"]] > 0.75 && npv[["lower"]] > 0.90, "\n")
Study success (both co-primaries met): TRUE

Table 5. Secondary endpoints.

lr <- function(cat) {
  n1 <- sum(ev$PET_VISUAL == 1); n0 <- sum(ev$PET_VISUAL == 0)
  a <- sum(ev$CAT == cat & ev$PET_VISUAL == 1); b0 <- sum(ev$CAT == cat & ev$PET_VISUAL == 0)
  p1 <- a / n1; p0 <- b0 / n0; est <- p1 / p0
  se <- sqrt((1 - p1) / (n1 * p1) + (1 - p0) / (n0 * p0))
  sprintf("%.2f (%.2f-%.2f)", est, exp(log(est) - 1.96 * se), exp(log(est) + 1.96 * se))
}
kable(data.frame(
  Endpoint = c("Intermediate fraction", "PPA (positive among PET+; intermediates = disagreement)", "NPA (negative among PET-)",
               "Rule-out step sensitivity (not-negative among PET+)", "Rule-in step specificity (not-positive among PET-)",
               "LR positive category", "LR intermediate", "LR negative category"),
  `Estimate % (95% CI)` = c(pct(pp$intermediate), pct(pp$ppa_incl), pct(pp$npa_incl), pct(pp$ruleout_sens), pct(pp$rulein_spec),
                            lr("rule-in"), lr("intermediate"), lr("rule-out")), check.names = FALSE))
Endpoint Estimate % (95% CI)
Intermediate fraction 21.1 (18.8-23.7)
PPA (positive among PET+; intermediates = disagreement) 74.6 (70.2-78.6)
NPA (negative among PET-) 72.2 (68.6-75.5)
Rule-out step sensitivity (not-negative among PET+) 93.6 (90.8-95.6)
Rule-in step specificity (not-positive among PET-) 94.7 (92.7-96.2)
LR positive category 14.05 (10.08-19.58)
LR intermediate 0.84 (0.66-1.08)
LR negative category 0.09 (0.06-0.13)

Table 6. Subgroups (descriptive; n < 20 in a category suppressed).

sg <- function(var, label) do.call(rbind, lapply(sort(unique(as.character(ev[[var]]))), function(l) {
  v <- ev[as.character(ev[[var]]) == l, ]; q <- two_cut_performance(v$PTAU217, v$PET_VISUAL, LO, HI)
  np <- sum(v$CAT == "rule-in"); nn <- sum(v$CAT == "rule-out")
  data.frame(Subgroup = paste(label, l), N = nrow(v), `Intermediate %` = pct(q$intermediate),
    `PPV positive %` = if (np >= 20) pct(q$rulein_ppv) else "n<20", `NPV negative %` = if (nn >= 20) pct(q$ruleout_npv) else "n<20",
    check.names = FALSE) }))
kable(rbind(sg("AGEGR", "Age"), sg("SEX", "Sex"), sg("SETTING", "Setting"), sg("MMSEGR", "MMSE"), sg("EGFRGR", "eGFR")), row.names = FALSE)
Subgroup N Intermediate % PPV positive % NPV negative %
Age 55-70 470 21.5 (18.0-25.4) 86.6 (79.8-91.3) 94.9 (91.3-97.1)
Age 71-85 576 20.8 (17.7-24.3) 92.1 (87.6-95.1) 94.5 (90.9-96.7)
Sex F 559 18.1 (15.1-21.5) 89.5 (84.4-93.1) 94.4 (90.9-96.6)
Sex M 487 24.6 (21.0-28.7) 90.4 (84.5-94.2) 95.0 (91.3-97.2)
Setting primary 271 18.1 (14.0-23.1) 85.2 (73.4-92.3) 93.5 (88.7-96.3)
Setting secondary 775 22.2 (19.4-25.3) 90.8 (86.9-93.7) 95.3 (92.4-97.1)
MMSE 15-24 362 20.7 (16.9-25.2) 90.6 (84.2-94.5) 95.6 (91.2-97.9)
MMSE 25-27 351 20.5 (16.6-25.0) 92.7 (86.2-96.2) 92.4 (87.4-95.5)
MMSE 28-30 333 22.2 (18.1-27.0) 86.1 (78.1-91.6) 96.2 (92.0-98.2)
eGFR <30 22 22.7 (10.1-43.4) n<20 n<20
eGFR >=60 826 21.1 (18.4-24.0) 92.2 (88.3-94.9) 93.7 (90.8-95.7)
eGFR 30-59 198 21.2 (16.1-27.4) 83.1 (72.7-90.1) 100.0 (95.7-100.0)

Table 7. Sensitivity analyses.

pps <- ev[ev$PPFL == "Y", ]; q_pp <- two_cut_performance(pps$PTAU217, pps$PET_VISUAL, LO, HI)
# S2 worst case: invalid assay among PET+ -> negative, among PET- -> positive
inv <- d[!d$assay_valid & d$pet_valid, ]
pos_w <- t3["rule-in", "PET+"]; pos_n <- sum(t3["rule-in", ]) + sum(inv$PET_VISUAL == 0)
neg_ok <- t3["rule-out", "PET-"]; neg_n <- sum(t3["rule-out", ]) + sum(inv$PET_VISUAL == 1)
cpf <- function(x, n) pct(ci_clopper_pearson(x, n))
cp_ppv <- cpf(t3["rule-in", "PET+"], sum(t3["rule-in", ])); cp_npv <- cpf(t3["rule-out", "PET-"], sum(t3["rule-out", ]))
kable(data.frame(
  Analysis = c("S1 per-protocol: PPV", "S1 per-protocol: NPV", "S2 worst case invalid assay: PPV", "S2 worst case invalid assay: NPV",
               "S3 Clopper-Pearson: PPV", "S3 Clopper-Pearson: NPV"),
  `Estimate % (95% CI)` = c(pct(q_pp$rulein_ppv), pct(q_pp$ruleout_npv),
    pct(ci_wilson(sum(pos_w), pos_n)), pct(ci_wilson(neg_ok, neg_n)), cp_ppv, cp_npv), check.names = FALSE))
Analysis Estimate % (95% CI)
S1 per-protocol: PPV 89.8 (86.1-92.6)
S1 per-protocol: NPV 94.7 (92.4-96.4)
S2 worst case invalid assay: PPV 86.6 (82.6-89.7)
S2 worst case invalid assay: NPV 93.1 (90.6-95.1)
S3 Clopper-Pearson: PPV 89.9 (86.2-92.9)
S3 Clopper-Pearson: NPV 94.7 (92.3-96.5)

12. Double programming: independent QC comparison

A second programmer re-derives the primary endpoints from the raw CSV without using the project helpers (different packages, different code path); the lead programmer’s results must agree.

raw <- read.csv("data/sim_ptau217.csv"); raw <- raw[!is.na(raw$PTAU217) & !is.na(raw$PET_VISUAL), ]
neg <- raw$PTAU217 <= LO; pos <- raw$PTAU217 > HI
w_ppv <- binom.confint(sum(pos & raw$PET_VISUAL == 1), sum(pos), methods = "wilson")
w_npv <- binom.confint(sum(neg & raw$PET_VISUAL == 0), sum(neg), methods = "wilson")
qc <- data.frame(item = c("PPV est", "PPV lower", "NPV est", "NPV lower"),
  lead = c(ppv[["est"]], ppv[["lower"]], npv[["est"]], npv[["lower"]]),
  independent = c(w_ppv$mean, w_ppv$lower, w_npv$mean, w_npv$lower))
qc$difference <- qc$lead - qc$independent
kable(qc, digits = 10)
item lead independent difference
PPV est 0.8991098 0.8991098 0
PPV lower 0.8623223 0.8623223 0
NPV est 0.9467213 0.9467213 0
NPV lower 0.9230796 0.9230796 0 
stopifnot(all(abs(qc$difference) < 1e-10))

13. Audit trail

data.frame(file = c("data/sim_ptau217.csv", "data/locked_cutoffs.csv"),
           md5 = unname(tools::md5sum(c("data/sim_ptau217.csv", "data/locked_cutoffs.csv"))))
                     file                              md5
1    data/sim_ptau217.csv 90cd60854699626d99347758402f292b
2 data/locked_cutoffs.csv 276554c1cbdbc45e1beeca762df583cc

The cut-off file hash is recorded before the performance data are analysed; any later change is a deviation and must be documented. [Y]

References

IVDR (EU) 2017/746 Annex XIII Part A 2.3.2; ISO 20916:2019; ICH E9 (1998) and E9(R1) (2019) by analogy; FDA K261686 decision summary (public pivotal data used as planning context); STARD 2015 (BMJ 2015;351:h5527); FDA 2007 statistical guidance. See claims_ledger.csv.