What is it about your face that tells me what you want from me? Emotional appeals are associated with specific mental images

ABSTRACT Emotional facial expressions have a communicative function. Besides information about the internal states (emotions) and the intentions of the expresser (action tendencies), they also communicate what the expresser wants the observer to do (appeals). Yet, there is very little research on the association of appeals with specific emotions. The present study has the aim to study the mental association of appeals and expressions through reverse correlation. Using reverse correlation, we estimated the observer-specific internal representations of expressions associated with four different appeals. A second group of participants rated the resulting expressions. As predicted, we found that the appeal to celebrate was uniquely associated with a happy expression and the appeal to empathize with a sad expression. A pleading appeal to stop was more strongly associated with sadness than with anger, whereas a command to stop was comparatively more strongly associated with anger. The results show that observers internally represent appeals as specific emotional expressions.

Suppose that a colleague during a zoom meeting tells you that the paper she had been working on for quite some time was finally accepted for publication in a top journal.At that very moment, the audio system stops working.But you still want to send her the message that you want to celebrate this achievement with her.How would you send this message?Smiling may be a good option, as it appears to tell her "let us celebrate!"without using any words, and it would most likely lead to her smiling in return.The fact that you may show a smile to deliver this celebratory message and that such message would be immediately understood suggests that people have a mental representation of the kinds of appeals that specific emotional expressions like smiling transmit to others in a given context.
Generally speaking, there is relative agreement that one of the core functions of emotion expressions is communicative, even though there is disagreement on what exactly is communicated (Darwin, 1872(Darwin, /1965;;Fox et al., 2018;Fridlund, 1994).An especially contentious debate concerns whether emotional expressions convey information about internal states, i.e. what the person is feeling, and if they do so universally across cultures (Ekman 1994;Fridlund, 1994, Russell 1994).Less contentious communicative functions of emotional expressions are that they convey information about what the person might do next, i.e. action tendencies (Frijda, 1987;Frijda et al., 1989), about stimulus conditions implied by appraisal patterns associated with specific emotions (e.g.Fontaine et al., 2013;Roseman, 1991;Roseman et al., 1990), and about the expresser's values and motivations (Hess & Hareli, 2019).
Emotions also convey what the expresser wants the perceiver to do (Fridlund, 1994;Scarantino, 2017).However, until recently, little empirical research focused on this type of information.From an evolutionary perspective, this is a puzzling oversight.In the 1970s, Dawkins and Krebs (Dawkins & Krebs, 1978;Krebs & Dawkins, 1984) made a powerful case that the evolutionary point of communicating is to get recipients to do things that are advantageous to the signaller.This view was initially contrasted with information-based approaches, according to which the point of communicating is to transfer information from sender to recipient (Otte, 1974;Seyfarth et al., 1980).The two approaches eventually converged, leading to the contemporary view that communication is an attempt to influence a recipient through information transfer, where both the signal and the response to the signal are adaptations (Maynard Smith & Harper, 2003;Scarantino, 2013).
This view of communication demands that we pay special attention to how signals affect the behaviour of recipients to the advantage of both parties (on average) by virtue of the information they carry.A central piece of information for the purpose of influencing is information about what the signaller wants the recipient to doif communication is about changing the behaviour of the recipient in advantageous ways, then information about what the signaller wants the recipient to do is surely of the essence.
Several theoretical frameworks have been developed to study this imperative dimension of emotional expressions.Scherer (1988) applied to emotional expressions a framework originally developed by Bühler (1934) to describe what he considered to be the three primary functions of language: to represent states of affairs (Darstellungsfunktion), to express inner states (Ausdrucksfunktion) and to make appeals (Appellfunktion) to others.Scherer proposed that emotional expressions work as symbols of the eliciting event, as symptoms of the emotional state of the sender, and as appeals of what the expresser is trying to get the recipient to do (see also, Hess et al., 1995).
Taking a broader perspective on the types of information that emotion expressions communicate, Scarantino's (2019) Theory of Affective Pragmatics (TAP) proposes that expressing emotions, either automatically or deliberately, makes possible a variety of sophisticated communicative moves that qualify as analogues of speech acts.This approach shares with Fridlund's (1994) behavioural ecology's view the assumption that facial displays are tools for negotiating social transactions which involve communicating "intentions" and "requests", and it shares with Ekman's basic emotion theory the assumption that facial displays express emotions, while adding that they do so in a highly context-dependent fashion rather than automatically.TAP's core proposal is that emotional expressions are ways of manifesting one's emotions, but also of representing states of affairs, directing other people's behaviours, and committing to future courses of actions.
Despite these theoretical developments, empirical research focused directly on what specific appeals are associated with specific emotions has been largely absent.This is the very topic Scarantino et al. (2022) addressed in two empirical studies which showed that appeals are systematically related to facial emotional expressions in a context-dependent way, and that recipients manifest willingness to act according to the appeals they detect.In particular, we assessed the extent to which expressions of anger, fear, disgust, happiness and sadness are associated with appeals to help the expresser/side with them, repair the relationship with them, celebrate/affiliate with them, stop what they are doing, comply with their demands, empathize with them, and be warned about the situation.
A distinction emerged between appeals associated predominantly with one emotion, and appeals shared by several emotions.For example, the appeal to celebrate/affiliate was strongly associated with happy expressions, and the appeal to empathize with sad expressions.By contrast, the appeal to stop what one is doing was strongly associated with several emotions, including anger, sadness and fearthree emotions that share the feature of being produced in situations where something unwelcomed happens (Scherer, 1987).Scarantino et al. (2022) left a central question open for investigation.Because appeals are not directly observable, participants must infer them, presumably by matching facial features to mental representations.Which mental representations do participants use to infer the appeals associated with facial expressions?One option is that what drives the appeal judgment is the mental representation of the emotional state expressed by the face.On this view, when participants are asked to determine to what extent a face conveys, say, the demand to celebrate, they represent demanding to celebrate as being happy, and the happier the face looks, the more they judge it to convey an appeal to celebrate.
Alternatively, there may be no visual mental representation of a specific emotional expression mediating appeal inferences.Since in the research by Scarantino et al. (2022) participants were presented with expressions of emotions and were asked to rate the degree to which each expression in their opinion is associated with each appeal, it is in principle possible that subjects generated an implicit association between the emotion concept (e.g. the concept of happiness) and the appeal concept (e.g. the concept of celebrating).The presence of an implicit association between previously labelled concepts does not tell us what mental representations subjects spontaneously use when presented with unlabelled instances of the two concepts.Further, to the degree that a specific mental representation is involved in the process in question, it is also unclear if the mental representation of the facial features associated with a specific appeal are necessarily facial features that reflect a specific emotion rather than multiple emotions or no emotion at all.
One of the most helpful techniques currently available for investigating the visual mental representations that drive inferences of social characteristics is reverse correlation.Reverse correlation is a procedure that allows to identify the visual features of the internal mental representations that drive social judgements (Brinkman et al., 2017).When used to assess facial features associated with specific social judgments, participants are presented with a neutral face to which random noise is added.Participants are then asked to classify a pair of face + noise combinations in terms of similarity with the judgment of interest.Figure 1 presents an example for a pair of stimuli used in our experiment, where we added and subtracted the same random noise pattern consisting of light and dark patches to a neutral face, resulting in two different images (see Figure 1).This task is repeated several hundred times and then the faces associated with the judgment are averaged.Random noise that is not associated with the judgments cancels out.Noise that happens to be associated with the judgment is selected across many faces and correspondingly changes the original image systematically (see Brinkman et al., 2017, for more detail).What is obtained through this process is a so-called classification image, which shows the facial features that drive the social judgement of interest.This approach has so far been successfully applied to obtain classification images underlying judgments of race, gender, dominance, trustworthiness, ethnicity, and expressions of emotions among others (Brinkman et al., 2017).
Here, we apply the reverse correlation technique for the first time to judgments of appeals.The reverse correlation approach is well suited to assess whether specific appeals are associated with mental representations of distinctive emotional expressions without presenting these expressions to the participants, and without even mentioning to them the idea that an emotion might be expressed by the face.An advantage of the reverse correlation method is that it is entirely open with respect to the visual features that drive the judgment of interest.Any information encoded into the visual image that results from the application of the technique is spontaneously selected by participants in the process of deciding which of two faces conveys a given appeal more than the other.This allows in principle for the discovery of completely unexpected features of the classification images that drive appeal judgments.The resulting classification images can then be subjected to evaluation by other participants to determine what these faces express.In this case, we Figure 1.An example of a pair of stimuli presented to participants in the reverse correlation study, who were asked which of the two faces conveys a certain appeal more than the other.
asked whether the resulting faces express a specific emotion.
We focused on three appeals out of the seven we used in Scarantino et al. (2022): the appeal to celebrate/affiliate, the appeal to empathize and the appeal to stop.Based on our findings from this research, we predicted that when asked to select images that represent the appeal to celebrate participants would select images that contained features of a happy expression, and when asked to select images that represent the appeal to empathize participants would select images that contained features of a sad expression.Thus, we predicted that the classification image for the appeal to celebrate is a happy face, and the classification image for the appeal to empathize is a sad face.
The appeal to stop is interesting, in the sense that Scarantino et al. (2022) found it to be associated with several distinct negative emotions.It seems that the appeal to stop can be interpreted in at least two different ways.First, as a plea to stop by someone who feels sad or fearful, i.e. someone who does not have control over the situation.Second, as a command to stop by someone who feels anger and has the power to force obedience.
We therefore used two variants of the appeal.In one we asked participants to select the face that asks them to stop what they are doing (pleading appeal) and in the other to select the face that commands them to stop (commanding appeal).We expected that the pleading appeal would be associated more strongly with sad facial features and less control/dominance and the commanding appeals more strongly with angry features and more control/dominance, as the ability to command someone requires the sort of power over the other typical of anger rather than of sadness (Cuddy et al., 2009;Tiedens, 2001).

Participants
A total of 67 participants (21 men) with a mean age of 24.8 (SD = 5.5) participated in the reverse correlation task.An additional 148 participants (61 women, 1 non binary) with a mean age of 38.9 (SD = 11.9) were recruited via MTurk and rated the resulting classification images for emotional content.The power of the reverse correlation task is driven by the number of trials.Dotch (2016) suggests 300 trials, we opted for 600 given the complexity of the judgment.The sample size for the rating study was based on considerations regarding average interrater reliability for emotion judgments.Based on data from our laboratory approx.35 participants per condition are needed for an effective interrater reliability >.80.Participants in the reverse correlation task were given course credit.MTurk participants received $.75 for the approx.5 min task.

Stimuli
For the reverse correlation task, we created a grey scale, gender and emotion neutral base image (see A in Figure 2) by averaging 70 photos (35 for each gender) showing neutral expressions from the Karolinska Directed Emotional Faces (Lundqvist et al., 1998).Figure 2A shows the base face.Using rcicr (Reverse correlation image classification toolbox, Dotsch, 2016) in R (R Core Team, 2021), we created 600 pairs of faces by either adding or subtracting random noise from the base image.Figure 1 shows two example faces with noise added and subtracted.
The R package rcicr uses noise based reverse correlation as described in Dotsch and Todorov (2012).We obtained four classification images from the reverse correlation study, which we then used as stimuli in the rating study.To obtain classification images we used rcicr (Dotsch, 2016) in R (R Core Team, 2021).This r package averaged the unscaled noise patterns across participants and then reapplied these to the base image.The resulting images for all participants in a given condition were combined in the same way to create four "grand mean" images, one for each of the four appeals.These images were assessed in the rating study in terms of the emotion they expressed.

Reverse correlation study
Due to pandemic conditions, the experiment was conducted online during a zoom session.The experimenter made sure that the participants were using a computer screen (not a small tablet or smartphone) and provided the participant with a link which started the programme.Participants were informed about the nature of the experiment and their task as well as their right to withdraw from the experiment at any time or to ask for deletion of their data following the experiment.None of the participants availed themselves of these options.Once participants had given informed consent, the experiment was started.
Each of the 600 trials of the reverse correlation study consisted of two images presented side by side, with participants choosing the image that best reflected the target appeal.Each participant was given a short description of one appeal and asked to select the image that matched that appeal.Specifically, they were instructed to select which face better matched asking the observer to "celebrate with the expresser" (appeal to celebrate), asking the observer "to stop what they are doing" (pleading appeal to stop), commanding the observer "to stop what they are doing" (commanding appeal to stop) and asking the observer to "empathize with the expresser" (appeal to empathize).The complete session took approx.25-30 min.

Rating study
The rating study was conducted online via MTurk.Raters saw only one image each and were asked to select the emotion label that best describes the image among the following list: happiness, sadness, anger, disgust, contempt, fear, shame or none.Participants then rated the face for valence, arousal, and control using the Self-Assessment Manikin (SAM) (Bradley & Lang, 1994).SAM is a scale that measures the dimensions of pleasure, arousal and dominance using a series of graphic abstract characters horizontally arranged according to a 9-points scale.For pleasure, participants are asked to indicate the degree to which the person in the image is happy vs. unhappy.The graphic characters for this rating range from a frowning to a smiling figure.For arousal, characters span from a sleepy to a widely awake figure showing an incremental explosion at the centre, and participants are instructed to determine how excited vs. calm the person is by selecting the character the best describes the person in the image.Lastly, dominance ranges from a very small to a very large character and participants are asked to select the character that best represents the degree to which the person in the image is controlled vs. in control.We added these measures because they are an accepted alternative to describe emotional expressions and can provide convergent evidence for establishing the link between appeals and facial expressions.
Participants further indicated how masculine, feminine, and friendly the face appeared, and to what degree the face seemed to communicate the appeal for which it was created.These latter ratings were not analysed for this report.

Reverse correlation study
Each trial of the reverse correlation study consisted of two images presented side by side, with participants choosing the image that best reflected the appeal.
To obtain the classification images of each group of participants, we used rcicr (Dotsch, 2016).We obtained four images which are the visual proxies of the mental representations that correspond to, respectively, the appeal to celebrate, the pleading appeal to stop, the commanding appeal to stop and the appeal to empathize (see Figure 2).

Discrete emotion ratings
Figure 3 shows the participants' choice of emotion labels.About 68% rated the classification image associated with the appeal to celebrate as happy (no other emotion label received more than 6%), 69% rated the classification image associated with the pleading appeal to stop as sad (with 11% rating it as angry and 9% rating it as contemptuous); 44% rated the classification image associated with the commanding appeal to stop as sad and 34% as angry, and 90% rated the classification image associated with the appeal to empathize as sad.A Χ 2 test confirmed that the distribution of the labels differs significantly between appeals, Χ 2 (21) = 150.19,p < .001.Details of the analysis can be found in the R Markdown in the supplemental materials.
The appeal to celebrate and the commanding appeal to stop were rated higher on arousal and dominance than the appeal to empathize and the pleading appeal to stop.Details of the analyses can be found in the R Markdown in the supplemental materials.

Discussion
Scarantino's (2019) Theory of Affective Pragmatics posits that one of the communicative functions of emotion expressions is to provide information about what the expresser wants the observer to do.
If emotion expressions have this imperative or appeal function, then people should have a visual mental representation of a given appeal that matches the emotion expression associated with this appeal.As in the zoom meeting example that introduces this research, people ought to associate an invitation to celebrate with a happy facial expression.
As predicted, participants who performed the reverse correlation task based on a description of appeals produced internal representations that corresponded to the expected emotional expressions.Specifically, participants associated the appeal to celebrate with facial features for which the label happy was chosen by the majority of participants in the rating study (68%).Furthermore, subjects rated this classification image as highest on valence as well as high on dominance and arousal.This fits with the notion that the appeal to celebrate is associated with happiness, an emotion that is positive and also perceived as a signal of dominance (Hess et al., 2000;Knutson, 1996).
By contrast, participants' internal representations for the appeal to empathize were associated with facial features almost uniformly rated as sad (90%).The classification image for the appeal to empathize was also rated as low in dominance, which is expected for a sad face since sad faces are rated as relatively low on dominance (Knutson, 1996).
Most interesting were the internal representations for the two variants of the appeal to stop.The classification images associated with the pleading appeal to stop (asking to stop) and the commanding appeal to stop were both labelled as predominantly sad and somewhat angry.In the case of the pleading appeal to stop, 69% labelled the classification image as sad, with negative valence, relatively low dominance and low arousal.This makes for a compelling case that the mental image associated with this appeal is that of sadness.In the case of the commanding appeal to stop, 34% described the classification image as angry and 44% as sad.This would seem to indicate that the commanding appeal to stop is at best ambiguous, in that almost an equal number of subjects evaluated it as sad and angry.Two elements, however, speak in favour of a different interpretation.The first is that participants in the rating study chose anger three times more often for the classification image of the commanding appeal to stop (34%) than for the classification image of the pleading appeal to stop (11%).The second is that the classification image of the commanding appeal to stop was rated as negative in valence, high in dominance and high in arousal, a SAM profile much more typical of anger than sadness.Hence, although the evidence is not conclusive, we think the balance of evidence speaks in favour of considering the classification image for the commanding appeal to stop to be an angry face.
Overall, the present research shows that the associations between appeals and emotion expressions found by Scarantino et al. (2022) correspond closely to participants' internal visual representations of these appeals.Participants understood expressions of emotions as reflecting appeals.Whereas some appeals were represented by a specific emotional expression, other appeals were represented by more than one emotional expression.Scarantino et al. (2022) found that for some appeals the association between the appeal and a specific emotion depends on context.For example, the appeal to stop what one is doing was associated with anger, sadness and fear, with the strength of association varying as a function of context.A sad expression when elicited by the behaviour of the recipient was perceived by a majority of the recipients as an appeal to stop (their own behaviour) as well as an appeal to emphasize, whereas the same sad expression when not caused but only witnessed by the recipient was perceived as an appeal to help and to empathize with the expresser.Further, an anger expression was only associated with an appeal to stop when the recipient caused the behaviour, and not when the recipient witnessed the expression without having caused it.
The present findings show that the mental representations of appeals vary similarly.When participants think of a pleading appeal to stop, the resulting mental representation was predominantly associated with sadness, whereas when they think of a commanding appeal to stop it was more strongly associated with anger.Whether someone is in the position to command or has to take recourse to a plea would likely depend on the context of the interaction.This suggests that, depending on the specifics of the appeal an expresser is conveying, different emotional expressions are required.
Emotion expressions are complex signals in that they communicate several types of information.Sadness signals not only a goal obstruction but also that the expresser lacks the power or ability to address this obstruction successfully (Scherer, 1986;Tiedens, 2001)  and from this position only pleading with the interaction partner to stop what they are doing can be a successful appeal.By contrast, anger is associated with dominance (Knutson, 1996) and the ability or power to influence others (Scherer, 1986;Tiedens, 2001) and this allows for a commanding demand to stop.
The present research shows not only that there are specific mental representations of emotional expressions spontaneously associated with appeals, rather than simply an implicit semantic association between emotion concepts and appeal concepts, but also that it is not the case that "motives bear no necessary relation to emotion, and indeed, [that] a range of emotions can co-occur with any social motive" (Fridlund, 1994, p. 139).On the contrary, we found that certain appeals like the appeal to celebrate and the appeal to empathize are predominantly associated with a specific emotional expression.And even when the same appeal can be conveyed by multiple emotional expressions, it cannot be conveyed by just any emotional expressionthere are systematic and predictable correlations between emotional expressions and appeals, in line with the Theory of Affective Pragmatics (Scarantino, 2019).The emotion that is expressed is what drives the social inference regarding what the expresser demands.As such, emotional expressions do not become explanatorily superfluous when we embrace the view that facial displays are tools for social transactions.

Figure 2 .
Figure 2. (A) Base figure, (B) Classification image for the appeal to celebrate/affiliate, (C) Classification image for the pleading appeal to stop, (D) Classification image for the commanding appeal to stop, (E) Classification Image for the appeal to empathize.

Figure 3 .
Figure 3. Percent of emotion labels chosen by participants as a function of appeals.