# (Non-)Linear Tests for Null Hypotheses, Joint Hypotheses, Equivalence, Non Superiority, and Non Inferiority Uncertainty estimates are calculated as first-order approximate standard errors for linear or non-linear functions of a vector of random variables with known or estimated covariance matrix. In that sense, `hypotheses` emulates the behavior of the excellent and well-established car::deltaMethod and car::linearHypothesis functions, but it supports more models; requires fewer dependencies; expands the range of tests to equivalence and superiority/inferiority; and offers convenience features like robust standard errors. To learn more, read the hypothesis tests vignette, visit the package website, or scroll down this page for a full list of vignettes: * [https://marginaleffects.com/chapters/hypothesis.html](https://marginaleffects.com/chapters/hypothesis.html) * [https://marginaleffects.com/](https://marginaleffects.com/) Warning #1: Tests are conducted directly on the scale defined by the `type` argument. For some models, it can make sense to conduct hypothesis or equivalence tests on the `"link"` scale instead of the `"response"` scale which is often the default. Warning #2: For hypothesis tests on objects produced by the `marginaleffects` package, it is safer to use the `hypothesis` argument of the original function. Using `hypotheses()` may not work in certain environments, in lists, or when working programmatically with *apply style functions. Warning #3: The tests assume that the `hypothesis` expression is (approximately) normally distributed, which for non-linear functions of the parameters may not be realistic. More reliable confidence intervals can be obtained using the `inferences()` function with `method = "boot"`. ```r hypotheses( model, hypothesis = NULL, vcov = NULL, conf_level = 0.95, df = NULL, equivalence = NULL, joint = FALSE, joint_test = "f", numderiv = "fdforward", ... ) ``` ## Arguments - `model`: Model object or object generated by the `comparisons()`, `slopes()`, or `predictions()` functions. - `hypothesis`: specify a hypothesis test or custom contrast using a numeric value, vector, or matrix; a string equation; string; a formula, or a function. * Numeric: * Single value: the null hypothesis used in the computation of Z and p (before applying `transform`). * Vector: Weights to compute a linear combination of (custom contrast between) estimates. Length equal to the number of rows generated by the same function call, but without the `hypothesis` argument. * Matrix: Each column is a vector of weights, as describe above, used to compute a distinct linear combination of (contrast between) estimates. The column names of the matrix are used as labels in the output. * String equation to specify linear or non-linear hypothesis tests. If the `term` column uniquely identifies rows, terms can be used in the formula. Otherwise, use `b1`, `b2`, etc. to identify the position of each parameter. The `b*` wildcard can be used to test hypotheses on all estimates. If a named vector is used, the names are used as labels in the output. Examples: * `hp = drat` * `hp + drat = 12` * `b1 + b2 + b3 = 0` * `b* / b1 = 1` * String: * "pairwise": pairwise differences between estimates in each row. * "reference": differences between the estimates in each row and the estimate in the first row. * "sequential": difference between an estimate and the estimate in the next row. * "meandev": difference between an estimate and the mean of all estimates. * "meanotherdev": difference between an estimate and the mean of all other estimates, excluding the current one. * "revpairwise", "revreference", "revsequential": inverse of the corresponding hypotheses, as described above. * Formula: * `comparison ~ pairs | group` * Left-hand side determines the type of comparison to conduct: `difference` or `ratio`. If the left-hand side is empty, `difference` is chosen. * Right-hand side determines the pairs of estimates to compare: `reference`, `sequential`, or `meandev` * Optional: Users can supply grouping variables after a vertical bar to conduct comparisons withing subsets. * Examples: * `~ reference` * `ratio ~ pairwise` * `difference ~ pairwise | groupid` * Function: * Accepts an argument `x`: object produced by a `marginaleffects` function or a data frame with column `rowid` and `estimate` * Returns a data frame with columns `term` and `estimate` (mandatory) and `rowid` (optional). * The function can also accept optional input arguments: `newdata`, `by`, `draws`. * This function approach will not work for Bayesian models or with bootstrapping. In those cases, it is easy to use `get_draws()` to extract and manipulate the draws directly. * See the Examples section below and the vignette: https://marginaleffects.com/chapters/hypothesis.html - `vcov`: Type of uncertainty estimates to report (e.g., for robust standard errors). Acceptable values: * FALSE: Do not compute standard errors. This can speed up computation considerably. * TRUE: Unit-level standard errors using the default `vcov(model)` variance-covariance matrix. * String which indicates the kind of uncertainty estimates to return. * Heteroskedasticity-consistent: `"HC"`, `"HC0"`, `"HC1"`, `"HC2"`, `"HC3"`, `"HC4"`, `"HC4m"`, `"HC5"`. See `?sandwich::vcovHC` * Heteroskedasticity and autocorrelation consistent: `"HAC"` * Mixed-Models degrees of freedom: "satterthwaite", "kenward-roger" * Other: `"NeweyWest"`, `"KernHAC"`, `"OPG"`. See the `sandwich` package documentation. * One-sided formula which indicates the name of cluster variables (e.g., `~unit_id`). This formula is passed to the `cluster` argument of the `sandwich::vcovCL` function. * Square covariance matrix * Function which returns a covariance matrix (e.g., `stats::vcov(model)`) - `conf_level`: numeric value between 0 and 1. Confidence level to use to build a confidence interval. - `df`: Degrees of freedom used to compute p values and confidence intervals. A single numeric value between 1 and `Inf`. When using `joint_test="f"`, the `df` argument should be a numeric vector of length 2. - `equivalence`: Numeric vector of length 2: bounds used for the two-one-sided test (TOST) of equivalence, and for the non-inferiority and non-superiority tests. See Details section below. - `joint`: Joint test of statistical significance. The null hypothesis value can be set using the `hypothesis` argument. * FALSE: Hypotheses are not tested jointly. * TRUE: All parameters are tested jointly. * String: A regular expression to match parameters to be tested jointly. `grep(joint, perl = TRUE)` * Character vector of parameter names to be tested. Characters refer to the names of the vector returned by `marginaleffects::get_coef(object)`. * Integer vector of indices. Which parameters positions to test jointly. - `joint_test`: A character string specifying the type of test, either "f" or "chisq". The null hypothesis is set by the `hypothesis` argument, with default null equal to 0 for all parameters. - `numderiv`: string or list of strings indicating the method to use to for the numeric differentiation used in to compute delta method standard errors. * "fdforward": finite difference method with forward differences * "fdcenter": finite difference method with central differences (default) * "richardson": Richardson extrapolation method * Extra arguments can be specified by passing a list to the `numDeriv` argument, with the name of the method first and named arguments following, ex: `numderiv=list("fdcenter", eps = 1e-5)`. When an unknown argument is used, `marginaleffects` prints the list of valid arguments for each method. - `...`: Additional arguments are passed to the `predict()` method supplied by the modeling package.These arguments are particularly useful for mixed-effects or bayesian models (see the online vignettes on the `marginaleffects` website). Available arguments can vary from model to model, depending on the range of supported arguments by each modeling package. See the "Model-Specific Arguments" section of the `?slopes` documentation for a non-exhaustive list of available arguments. ## Joint hypothesis tests The test statistic for the joint Wald test is calculated as (R * theta_hat - r)' * inv(R * V_hat * R') * (R * theta_hat - r) / Q, where theta_hat is the vector of estimated parameters, V_hat is the estimated covariance matrix, R is a Q x P matrix for testing Q hypotheses on P parameters, r is a Q x 1 vector for the null hypothesis, and Q is the number of rows in R. If the test is a Chi-squared test, the test statistic is not normalized. The p-value is then calculated based on either the F-distribution (for F-test) or the Chi-squared distribution (for Chi-squared test). For the F-test, the degrees of freedom are Q and (n - P), where n is the sample size and P is the number of parameters. For the Chi-squared test, the degrees of freedom are Q. ## Equivalence, Inferiority, Superiority $\theta$ is an estimate, $\sigma_\theta$ its estimated standard error, and $[a, b]$ are the bounds of the interval supplied to the `equivalence` argument. Non-inferiority: * $H0$: $\theta \<= a$ * $H1$: $\theta \> a$ * $t=(\theta - a)/\sigma_\theta$ * p: Upper-tail probability Non-superiority: * $H0$: $\theta \>= b$ * $H1$: $\theta \< b$ * $t=(\theta - b)/\sigma_\theta$ * p: Lower-tail probability Equivalence: Two One-Sided Tests (TOST) * p: Maximum of the non-inferiority and non-superiority p values. Thanks to Russell V. Lenth for the excellent `emmeans` package and documentation which inspired this feature. ## Examples ```r library(marginaleffects) mod <- lm(mpg ~ hp + wt + factor(cyl), data = mtcars) hypotheses(mod) # Test of equality between coefficients hypotheses(mod, hypothesis = "hp = wt") # Non-linear function hypotheses(mod, hypothesis = "exp(hp + wt) = 0.1") # Robust standard errors hypotheses(mod, hypothesis = "hp = wt", vcov = "HC3") # b1, b2, ... shortcuts can be used to identify the position of the # parameters of interest in the output of hypotheses(mod, hypothesis = "b2 = b3") # wildcard hypotheses(mod, hypothesis = "b* / b2 = 1") # term names with special characters have to be enclosed in backticks hypotheses(mod, hypothesis = "`factor(cyl)6` = `factor(cyl)8`") mod2 <- lm(mpg ~ hp * drat, data = mtcars) hypotheses(mod2, hypothesis = "`hp:drat` = drat") # predictions(), comparisons(), and slopes() mod <- glm(am ~ hp + mpg, data = mtcars, family = binomial) cmp <- comparisons(mod, newdata = "mean") hypotheses(cmp, hypothesis = "b1 = b2") mfx <- slopes(mod, newdata = "mean") hypotheses(cmp, hypothesis = "b2 = 0.2") pre <- predictions(mod, newdata = datagrid(hp = 110, mpg = c(30, 35))) hypotheses(pre, hypothesis = "b1 = b2") # The `hypothesis` argument can be used to compute standard errors for fitted values mod <- glm(am ~ hp + mpg, data = mtcars, family = binomial) f <- function(x) predict(x, type = "link", newdata = mtcars) p <- hypotheses(mod, hypothesis = f) head(p) f <- function(x) predict(x, type = "response", newdata = mtcars) p <- hypotheses(mod, hypothesis = f) head(p) # Complex aggregation # Step 1: Collapse predicted probabilities by outcome level, for each individual # Step 2: Take the mean of the collapsed probabilities by group and `cyl` library(dplyr) library(MASS) library(dplyr) dat <- transform(mtcars, gear = factor(gear)) mod <- polr(gear ~ factor(cyl) + hp, dat) aggregation_fun <- function(x) { predictions(x, vcov = FALSE) |> mutate(group = ifelse(group %in% c("3", "4"), "3 & 4", "5")) |> summarize(estimate = sum(estimate), .by = c("rowid", "cyl", "group")) |> summarize(estimate = mean(estimate), .by = c("cyl", "group")) |> rename(term = cyl) } hypotheses(mod, hypothesis = aggregation_fun) # Equivalence, non-inferiority, and non-superiority tests mod <- lm(mpg ~ hp + factor(gear), data = mtcars) p <- predictions(mod, newdata = "median") hypotheses(p, equivalence = c(17, 18)) mfx <- avg_slopes(mod, variables = "hp") hypotheses(mfx, equivalence = c(-.1, .1)) cmp <- avg_comparisons(mod, variables = "gear", hypothesis = "pairwise") hypotheses(cmp, equivalence = c(0, 10)) # joint hypotheses: character vector model <- lm(mpg ~ as.factor(cyl) * hp, data = mtcars) hypotheses(model, joint = c("as.factor(cyl)6:hp", "as.factor(cyl)8:hp")) # joint hypotheses: regular expression hypotheses(model, joint = "cyl") # joint hypotheses: integer indices hypotheses(model, joint = 2:3) # joint hypotheses: different null hypotheses hypotheses(model, joint = 2:3, hypothesis = 1) hypotheses(model, joint = 2:3, hypothesis = 1:2) # joint hypotheses: marginaleffects object cmp <- avg_comparisons(model) hypotheses(cmp, joint = "cyl") ```

hypotheses function