To higher understand this issue, we have now render theoretic knowledge. In what comes after, i basic model the fresh new ID and you can OOD investigation distributions and obtain statistically the newest design productivity out of invariant classifier, where model aims to not ever have confidence in environmentally friendly has to own prediction.
Configurations.
We consider a binary classification task where y ? < ?>, and is drawn according to a fixed probability ? : = P ( y = 1 ) . We assume both the invariant features z inv and environmental features z e are drawn from Gaussian distributions:
? inv and you can ? 2 inv are exactly the same for everyone surroundings. In contrast, environmentally friendly variables ? elizabeth and you will ? 2 elizabeth will vary round the age , in which the subscript is used to indicate the importance of the latest environment plus the index of environment. As to what comes after, we introduce the outcomes, with outlined proof deferred in the Appendix.
Lemma step one
? elizabeth ( x ) = Yards inv z inv + Meters age z e , the suitable linear classifier to have an atmosphere elizabeth comes with the associated coefficient 2 ? ? step 1 ? ? ? , where:
Note that new Bayes maximum classifier uses ecological provides which can be educational of the term however, low-invariant. Instead, develop to help you rely just to the invariant keeps if you are ignoring ecological have. Such as for example a beneficial predictor is also also known as optimum invariant predictor [ rosenfeld2020risks ] , which is given about pursuing the. Keep in mind that this really is a different sort of case of Lemma step one having Yards inv = I and you may Meters age = 0 .
Proposition 1
(Max invariant classifier playing with invariant has actually) Assume new featurizer recovers the fresh invariant ability ? elizabeth ( x ) = [ z inv ] ? age ? E , the optimal invariant classifier has got the related coefficient 2 ? inv / ? dos inv . step 3 3 step three The ceaseless term regarding classifier loads was diary ? / ( step 1 ? ? ) , and this i exclude here along with the sequel.
The optimal invariant classifier clearly ignores environmentally friendly features. not, a keen invariant classifier read will not necessarily depend just for the invariant features. Next Lemma implies that it could be you’ll be able to to know an invariant classifier one relies on the environmental features while reaching straight down risk than the maximum invariant classifier.
Lemma 2
(Invariant classifier using non-invariant features) Suppose E ? d e , given a set of environments E = < e>such that all environmental means are linearly independent. Then there always exists a unit-norm vector p and positive fixed scalar ? such that ? = p T ? e / ? 2 e ? e ? E . The resulting optimal classifier weights are
Keep in mind that the optimal classifier pounds dos ? is actually a stable, which does not believe the environmental surroundings (and you may neither does the optimal coefficient to own z inv ). This new projection vector p will act as a great «short-cut» the learner are able to use to produce a keen insidious surrogate code p ? z age . The same as z inv , that it insidious code may also produce an invariant predictor (round the environment) admissible from the invariant reading methods. Put differently, inspite of the differing analysis shipping across environments, the optimal classifier (using low-invariant provides) is the identical per environment. We now reveal all of our head results, in which OOD identification is also falter around such a keen invariant classifier.
Theorem step 1
(Failure of OOD detection under invariant classifier) Consider an out-of-distribution input which contains the environmental feature: ? out ( x ) = M inv z out + M e z e , where z out ? ? inv . Given the invariant classifier (cf. Lemma 2), the posterior probability for the OOD input is p ( y = 1 ? ? out ) = ? ( 2 p ? z e ? + log ? / ( 1 ? ? ) ) , where ? is the logistic function. Thus for arbitrary confidence 0 < c : = P ( y = 1 ? ? out ) < 1 , there exists ? out ( x ) with z e such that p ? z e = 1 2 ? log c ( 1 ? ? ) ? ( 1 ? c ) .