... different set of feature weights for each case: wv,1when the non-verbal features are included, and wv,2otherwise. The formal definition of the potentialfunction for conditional modality fusion ... w)−Xy,mp(m, y|xi; w)∂∂wjψ(y, m, xi; w)The form of the potential function ψ is where ourintuitions about the role of the hidden variable areformalized. Our goal is to include the non-verbalfeatures ... becauseindependent training of the modality-specific classi-fiers forces them to account for data that they can-not possibly explain. For example, if the speaker isnot gesturing meaningfully, it...