F-test

PyAstronomy.pyasl.ftest(chi1, chi2, dof1, dof2, compute_ratio_of_variance=False)

Performs an F-test.

Parameters

chi1float

The chi-squared statistic or variance for model 1.

chi2float

The chi-squared statistic or variance for model 2. Should be better (lower) than chi1.

dof1integer

The number of degrees of freedom of model 1

dof2integer

The number of degrees of freedom of model 2. Should be lower than dof1.

compute_ratio_of_varianceboolean, optional

Determines how the F-statistics is computed. If False (default), the “ANOVA F-test in regression analysis of nested nonlinear models” is applied. If True, “Fisher’s classical test for the equality of two variances” is used.

Returns

Test resultsdictionary

• “F statistic” - Value of F statistic.

• “p-value” - Probability to obtain an F-value as large as or larger than observed assuming H_0 to be true

• “Gaussian sigma level” - Same probability converted to Gaussian standard deviations (“sigmas”)

Notes

The “method of least squares” is widely used in parameter estimation. Much of its appeal lies in its simplicity: Minimizing the sum of squared differences between the model and the data.

The backbone of least squares is the classical multiple regression analysis using linear models to relate several independent variables to a dependent variable. However, many physical models are nonlinear; this is the realm of nonlinear least squares regression methods.

The aim of model fitting is to explain as much of the variation in the dependent variable as possible from information contained in the independent variables. The contributions of the independent variables to the model are measured by partitions of the total sum of squares of the difference of data and model (“analysis of variance”, ANOVA).

ANOVA F-test in regression analysis for nested nonlinear models

The sum of squares for any hypothesis can be determined from the difference between the residual sums of squares (\(RSS\)) of two models: The so-called “full” model, in the context of which the null hypothesis is to be tested, and the “reduced” model, which is derived from the full model by imposing additional constraints specified by the null hypothesis; setting one parameter to zero is a common example. The reduced model is a special case of the full model and, hence, its residual sum of squares must not be lower than the residual sum of squares for the full model.

Accounting for \(k\) independent constraints on the full model and \(N\) data points, we compute

\[Q = RSS(reduced) - RSS(full)\]

which has \(k\) degrees of freedom. Here, \(k\) corresponds to the difference in the number of parameters—or equivalently, the difference in the degrees of freedoms—between the full and reduced model. The \(F\)-test of the null hypothesis can then be computed as

\[\hat{F} = (Q/k) / s^2\]

where \(s^2\) is an unbiased estimate of \(\sigma^2\), e.g., derived from the full model. In the case of error-weighted data, the \(F\) statistic reads

\[\hat{F} = \frac{(\chi^2_{reduced}-\chi^2_{full})/(\nu_{reduced}-\nu_{full})}{\chi^2_{full}/\nu_{full}}\]

where \(\nu\) denotes the number of degrees of freedom, which may be calculated as \(N-p-1\) given a model with \(p\) parameters and an additional constant. The expectation value of \(F\) is \(1\), thus, if \(\hat{F}=1\) there is no significant difference between the RSS of both models. If, however, \(F\) deviates from \(1\), we can compute the probability for \(F\) to equal or exceed the obtained value \(\hat{F}\) by

\[Prob(F\geq \hat{F}) = CDF(F( \nu_{reduced}-\nu_{full}, \nu_{full} ))\]

For details on the principles of the formalism, see Sect. 4.5.4 in Rawlings’ “Applied Regression Analysis”.

In case of nonlinear models, the distribution of the least-square parameter estimates only approximately follows the normal distribution. Then, the \(F\) statistic (in this case called “Wald statistic”) also holds approximately. The validity of the method depends on how well the model is represented by a linear approximation in the parameters (“parameter effects curvature”). For details, we refer to Sect. 15.3 in Rawlings’ “Applied Regression Analysis” and references therein.

Fisher’s classical test for the equality of two variances

If the two vying models cannot be treated as nested, in the sense that the full model encompasses the reduced model, we can directly compare the model variances with an F-test.

Let \(s_1\) and \(s_2\) denote unbiased estimates of the variances of two independent, normally distributed populations of size \(N_1\) and \(N_2\), respectively. Under the null hypothesis both variances are equal:

\[H_0: s_1^2=s_2^2, \hspace{20pt} H_1: s_1^2>s_2^2\]

Then the quantity

\[\hat{F} = \frac{s_1^2}{s_2^2}\]

is \(F\)-distributed with \(N_1-1\) and \(N_2-1\) degrees of freedom. In the case of weighted errors we have

\[\hat{F} = \frac{\chi_1^2/\nu_1}{\chi_2^2/\nu_2}\]

and \(\hat{F}\) is distributed according to an \(F\)-distribution with \(\nu_1\) and \(\nu_2\) degrees of freedom.

References

John O. Rawlings, Sastry G. Pantula, David A. Dickey Applied Regression Analysis: A Research Tool, Second Edition Springer, New York, NY, 1998