Facial expressions contribute to nonverbal communication, social coordination, and interaction. Facial expressions may reflect the emotional state of the expressor, but they may be modulated by the presence of others, for example, by facial mimicry or through social display rules. We examined how deliberate facial expressions of happiness and anger (smiles and frowns), prompted by written commands, are modulated by the congruency with the facial expression of background faces and how this effect depends on the age of the background face (infants vs. adults). Our main interest was whether the quality of the required expression could be influenced by a task-irrelevant background face and its emotional display. Background faces from adults and infants displayed happy, angry, or neutral expressions. To assess the activation pattern of different action units, we used a machine classifier software; the same classifier was used to assess the chronometry of the expression responses. Results indicated slower and less correct performance when an incongruent facial expression was in the background, especially when distractor stimuli showed adult faces. Interestingly, smile responses were more intense in congruent than incongruent conditions. Depending on stimulus age, frown responses were affected in their quality by incongruent (smile) expressions in terms of the additional activation or deactivation of the outer brow raiser (AU2), resulting in a blended expression, somewhat different from the prototypical expression for anger. Together, the present results show qualitative effects on deliberate facial expressions, beyond typical chronometric effects, confirming machine classification of facial expressions as a promising tool for emotion research.