statistical optimality

[J P Morgan]

Hello All,

Below is a proposal regarding how statistical optimality criteria should be 
incorporated into the external representation. First a few comments on the 
breadth of allowable optimality functions. In the talk on 3rd April I showed 
the definition of a type-I optimality function, essentially a convex function 
of the eigenvalues plus a few extra conditions. The point of this is not that 
any particular function in that family would actually be used or indeed would 
hold any substantive statistical interest. Rather, the notion is employed 
because

a) It provides a minimal mathematical framework within which general 
optimality results can be obtained. The generality can help clarify (even 
simplify) the underlying mathematical problem. For instance, useful bounds 
for criteria values can be derived using only the type-I conditions. The 
bounds can then be applied to determine optimality for any particular 
criterion of interest. In the work with irregular BIBDs, this approach 
results in conditions for A- and D-optimality in a unified framework.

b) Occasionally a design or class of designs can be shown optimal w.r.t. all 
type-I criteria. Such a strong result is highly desirable - not just because 
it shows the tremendous statistical "strength" of the designs in question, 
but also in precluding the need for separate arguments for each criterion of 
genuine interest. It is not, however, an endorsement of more than cursory 
statistical relevance for each and every individual criterion in the family.  

As another example, statisticians also consider Schur-optimality: optimality 
w.r.t. all Schur-convex functions of the eigenvalues. One may be able to show 
that the vector of eigenvalues for a certain information matrix is majorized 
by those of all others, and thus it corresponds to a Schur-optimal design. 
But again this is no endorsement of all Schur-convex functions as 
individually interesting criteria on their own. It is simply reflective of 
the fact that many of the interesting criteria can be embedded in larger 
families for which we can sometimes determine quite general results. When, as 
is often the case, optimality w.r.t. such a general family does not hold, 
then we turn to individual criteria of interest and try to separately 
establish results for each.

What are these individual criteria of interest? On the one hand, the formal 
optimality theory, starting in earnest from the mid 1970s, has largely (not 
exclusively) concentrated on these four: (A,D,E,MV). Each corresponds to a 
desirable statistical property. On the other hand, earlier statistical block 
design work concentrated on various notions of balance as reflected in BIBDs 
and PBIBDs. Interpretability of results is also an important consideration, 
usually aligned with balance notions such as (for example) number of distinct 
variances, range of variances, etc. Another related issue concerns the 
pattern of variances as it relates to the structure of the treatment set.

A full notion of optimality should reflect the statistical usefulness of a 
design, and consequently should incorporate all of these ideas: efficiency 
w.r.t. alphabetic optimality (such as the four horsemen above) combined with 
notions of balance and structure as is appropriate.

That's my brief take on the background from which we proceed. For the 
external representation we face two main issues: what criteria to include, 
and in what form to express them. A proposal follows.

We currently have a category named efficiency_factors. I propose that we add 
a category under that heading named "optimality_functions_of_efficiency 
_factors" (or something similar), to include geometric mean, harmonic mean, 
etc as Rosemary skecthed out last Wednesday, but limited solely to functions 
of the efficiency factors.  Each is optimized if maximal.

We also include a category at the same level as efficiency_factors named 
optimality_criteria_values. Here we put functions that are minimized for 
optimality. Included are selected functions of the eigenvalues \mu_i of the 
information matrix and other selected criteria not based on eigenvalues. Here 
is my suggested list:

D = \phi_0
A = \phi_1
\phi_2
E = \phi_\infty = 1/\mu_1 = E1
E2 = E1+1/\mu_2 
E3 = E2 + 1/\mu_3
 .
 .
 .
E(v-1) = E(v-2) + 1/\mu_{v-1}
MV = max pairwise variance
# distinct  \mu_i
var_ratio = \mu_{v-1}/\mu_1
# distinct variances for pairwise comparisons
pairwise_var_ratio = MV/min pairwise variance

A,D,E,MV are all there, of course, along with several other ways of thinking 
about dispersion of variances and extremal behavior. The Ej's, which are 
components of a Schur-optimality argument, have also been studied on their 
own, albeit not extensively.


Rationale: 

1. For equireplication, criteria in terms of efficiency factors have the best 
intuitive appeal, so we should have them. They are also what many 
practitioners are more likely to look for.

2. For reasons including 

(i) ruling out nonequireplcate designs as optimality competitors when 
equireplication is possible, 
(ii) robustness studies of equireplicate designs under losses of plots and/or 
blocks, and 
(iii) direct optimality studies of settings (v,b,k) where v does not divide bk

we must include optimality criteria expressed in ways that seemlessly 
encompass both equireplicate and nonequireplcate designs, which is not the 
case when they are based on efficiency factors. This formulation will appeal 
less to practitioners, but holds great appeal to many statisticians working 
in design optimality.

3. There is redundancy between the two proposed categories, but this is not 
unlike redundancy we have incorporated elsewhere, and in this case is 
additionally supported by serving communities of users that only partially 
overlap.

4. The resource we are creating has the potential to make a significant 
impact on the design optimality community. It is thus very important, I 
believe, that we formally consider measures of statistical balance as 
optimality criteria. Why? So to nudge this community away from attitudes like 
"here is A-best and that's the end of story" and towards those like "after 
sifting through all the reasonably efficient designs, the best for this 
experiment has been selected based on a judicious balancing of efficiency, 
dispersion in the variances of estimated contrasts, the number of distinct 
variances, and the structure of the treatment set."  In doing so we enrich 
block design optimality as a field of study while bringing the mathematical 
optimality theory closer to the pragmatic optimality considerations of real 
experiments. Essentially I am saying that optimal design theory is more 
relevant to practice if the class of criteria considered is sufficiently 
rich, and I do not think A,D,E,MV alone make a sufficiently rich class. Why 
not? Because they do not incorporate balance notions that are important from 
multiple perspectives in so many actual experiments. 

Comments/discussion welcome.

JP