From Diversity by Numbers to Diversity as Process: Supporting Inclusiveness in Software Development Teams with Brainstorming

Negative experiences in diverse software development teams have the potential to turn off minority participants from future team-based software development activity. We examine the use of brainstorming as one concrete team processes that may be used to improve the satisfaction of minority developers when working in a group. Situating our study in time-intensive hackathon-like environments where engagement of all team members is particularly crucial, we use a combination of survey and interview data to test our propositions. We find that brainstorming strategies are particularly effective for team members who identify as minorities, and support satisfaction with both the process and outcomes of teamwork through different mechanisms.


INTRODUCTION
Conversations about diversity are increasingly coming to the forefront within software engineering.Research is taking strides to understand and impact a variety of issues within this domain: unconscious bias within organizations and online technology communities [1], [2], challenges in the pipeline such as recruitment and education [3], as well as how to design software for diverse learning and interaction styles [4].Despite these efforts, software development teams have become less diverse rather than more.Gender distribution is an often cited example of this decline -though the US population is evenly split by gender, the proportion of women in IT related jobs has dropped from an already low 31% in 1990 to 25% in 2014 [5].
Every day stereotypical interactions within software engineering work gradually accumulate and contribute to greater intentions to leave the profession [6].Again, taking gender as an example, Seron et al. [6] document an "isolating culture" that often "assumes women are second class experts" and does not take their input seriously (pg.203).This pattern may persist across different dimensions of diversity, such as in terms of race or educational background [7].It is therefore critical to not only encourage the formation of more diverse teams, that is, to support diversity in numbers, but to ensure that all team members, particularly those in the minority, are equally engaged and satisfied in their everyday work interactions through processes designed to support this diversity.
In this paper we examine one increasingly prevalent type of everyday group interaction for software developers: collaborative coding events, sometimes called hackathons, sprints or codefests.The number of hackathon-like events held, as well as their participant numbers, have grown rapidly in the past years.In the 2015 academic year alone, some 45000 college students participated in hackathons [9].This statistic is likely an under-estimate as it only includes collegiate hackathons affiliated with Major League Hacking, and does not take into account events held by or within organizations, or at other educational levels.Hackathon-like events are rapidly becoming a major source of everyday group interactions for developers, offering learning, collaboration and career opportunities [8].
Research on the hackathon phenomenon is also expanding.However, research thus far has largely focused on describing different types of events [10], as well as supporting team outcomes, such as how these events can be employed towards software sustainability [11] and varieties of nonsoftware applications [12], [13].Relatively little work has focused on encouraging effective processes in such ad-hoc time-intensive contexts, particularly among diverse teams.
This is an important area to address because hackathons have the potential to alienate minority participants through stereotypical micro-encounters in diverse teams [6].Specifically, diverse teams are more likely to suffer from unconscious intergroup biases [14], [15]: favoritism among the majority, stereotypical role expectations of minority participants (such as being assigned less technical work) and less opportunities for those who are in the minority to express and integrate their ideas [16].As a result, diverse hackathon teams may report poorer performance and less satisfaction [17], particularly when teams are not evenly mixed [7].
The negative effects of this stereotyping behavior are particularly relevant in hackathon-like settings because they are short-term and time intensive, thus there is insufficient time to develop deeper understanding among team members, yet the contribution of each team member is critical to the outcome.Group processes in hackathons are also noteworthy because they resemble other team activity where ad-hoc teams form to accomplish time-intensive work and minority team members may require particular support, such as classroom group projects, or "tiger/red teams" in industry.
While a significant amount of work has identified these issues in diverse teams, less has focused on concrete strategies that can be used to improve the experience of minority team members in diverse teams, particularly in time-intensive settings [18].Doing so has important benefits for team affective health and member retention.
The present work reports on a group mechanismbrainstorming -that our results show to be effective at improving the participation and satisfaction of self-reported minority team members.In doing so, we aim to contribute to the existing body of work on developing diverse software engineering teams by evaluating theoretically driven strategies that can be practically used to support team participation.At the same time, as the next section demonstrates in more detail, we revisit and expand theory on brainstorming to examine its possible effects in improving the experience of teamwork for minority participants.
In the next section we develop an argument for why brainstorming may be one effective strategy for improving the interactions within diverse teams, with reference to its impact on satisfaction with various aspects of teamwork.We then describe our research methodology, present and discuss our research findings in the subsequent sections.

A. Brainstorming
The concept of brainstorming has grown to be synonymous with idea generation and exchange.In software engineering, brainstorming is often used in the early phases of software design [19], and has been incorporated into requirement engineering and agile software development as a common idea generation technique [20].Brainstorming offers a concrete set of principles that enable teams to generate ideas and develop them into avenues for action: 1) deferring judgement about ideas proposed to create a safe space for expression; 2) aiming to generate as many ideas as possible in the time given; 3) freewheeling, or contributing even the most off-beat ideas; and 4) working to build on and integrate ideas proposed [21].
Brainstorming is expected to support group productivity by improving the number and quality of ideas generated by a group [21], [22], [23].However, research over decades has found that the effect of brainstorming on group success varies with the specific task context [24], and team composition [25].Specifically, factors such as group cultural background [26], gender [27] and levels of communication apprehension in a group [28] determine the success of brainstorming techniques.Prior work has commonly examined these moderators by contrasting homogenous groups, such as individualist versus collectivist cultures, all male versus all female groups, or low versus high communication apprehension within the group.As a result, we know relatively little about the impact of brainstorming on heterogeneous groups, particularly with respect to subjective outcomes like satisfaction.
We take a step further to examine the role brainstorming plays in heterogeneous, or diverse, groups that include participants traditionally considered minorities in some respect, either in terms of race, gender, cultural or education background.Specifically, we argue that brainstorming techniques are particularly useful in diverse teams in stimulating participation and promoting better subjective perceptions of the team, such as satisfaction.
We examine satisfaction, a subjective outcome, rather than an objective performance outcome for three reasons.First, retention of minority developers is a primary concern of our work, and satisfaction with group processes has been shown to be an important predictor of minority retention in the profession [6].Specifically, it is a lack of satisfaction with every day team interactions that is associated with greater intentions to leave the software engineering profession [6].Thus we chose this as an outcome variable.
Second, prior work on brainstorming has found brainstorming to have differing effects on subjective outcomes like satisfaction and objective outcomes like performance [19].For example, supportive communication when moderating brainstorming was associated with greater satisfaction, but not greater performance [19].This is an important dynamic to validate in diverse teams where creating a supportive environment is critical.To extend prior work we propose that brainstorming will relate to group satisfaction in two distinct ways, as we describe below.
Third, satisfaction with teamwork is often indirectly associated with team performance because teams perform better when all members are engaged and all resources are utilized [29].This is particularly true in short-term time intensive environments, like hackathons, where it is necessary to engage every team member continuously in order to produce a complete and effective software outcome.
In the next section, we elaborate on our propositions concerning brainstorming and diversity with respect to two dimensions of satisfaction and with reference to prior work.

B. Satisfaction
Research on satisfaction with teamwork has traditionally distinguished between two dimensions: satisfaction with outcome and satisfaction with process [30], [31].Satisfaction with outcome involves a positive judgement by a team member that the work accomplished meets certain requirements and constraints [31].Satisfaction with process, on the other hand, refers to the degree to which a team member derives a sense of affective arousal or positive valence from the team processes, such as the procedures and tools used [30].In other words, while satisfaction with outcome refers to a sense of accomplishment with respect to the output produced, satisfaction with process refers to the sense of fun team members derive in producing the output.Therefore, it is possible that while a team member may be satisfied with the overall product of the group's work (such as receiving a prize at a Hackathon for a project), they may not necessarily have had fun in the process, and vice versa.
Consequently, we examine the differential impact of brainstorming on both dimensions of satisfaction separately, as we describe below.We also examine two confounds that are commonly associated with different dimensions of satisfaction to understand the extent to which brainstorming may support increased satisfaction in addition to the variables identified in prior work.

1) Satisfaction with Process
Research has shown that there are a number of related constructs influencing satisfaction with process.Voice refers to employees' beliefs that they can speak out in opposition to those in higher organizational roles [38].Psychological safety refers to shared beliefs among work unit members that they can safely take interpersonal communication risks [39].Participative decision-making refers to the level of input that employees have into decisions, from high-level strategic to routine day-to-day decisions about how to do their own jobs [40].What these concepts have in common is the sense that team members have both an avenue for and the ability to express their opinions.We refer to this related set of ideas as perceived participation and contrast this with effects of brainstorming on satisfaction with process.
If team members feel less able to express their opinions, they are less likely to speak up both during idea generation and other team interactions.They may not experience benefits like disclosing personal information, which support liking and attachment to teammates [41], as well as fewer opportunities for learning and useful feedback from others.Furthermore, group discussions may progressively diverge from their own interests, leading them to become bored and feel left out.These negative emotions will likely decrease the enjoyment they derive in working in the team.Therefore, we propose: Hypothesis 1. Perceived participation is positively and directly associated with hackathon participants' satisfaction with their teams' process.
While it is important that team members feel that they are able to speak up (perceived participation), it is equally important for team members to feel they are being heard.In other words, it is not enough for the group to support raising new ideas, it is also crucial for the group to acknowledge these ideas and their sources, and for team members to see their ideas integrated into the final product.This is because the feeling of being heard promotes additional positive emotions such as motivation and satisfaction, perceptions of fairness, as well as a sense of ownership that is particularly important for completing work in short-term teams [42].
Group brainstorming explicitly supports being heard by encouraging team members to acknowledge, build on, and integrate all proposed ideas.
Team members using brainstorming should therefore feel more positively about the process of working together, even if their ideas evolve significantly before becoming part of the final product.We therefore propose that group brainstorming explains variance in satisfaction with process beyond that which is explained by perceived participation.Specifically, we hypothesize that: Hypothesis 2. Group brainstorming is positively and directly associated with hackathon participants' satisfaction with their teams' process.
We expect group brainstorming to have a stronger impact on process satisfaction among minority team members because they are often more likely to feel they are not being heard.Unconscious social-categorization favors ideas and opinions matching those of the majority [16], thus participants may discard or dismiss different perspectives coming from minority team members.This may reinforce emerging doubts minority team members have about their ability to contribute to team [6], leading to negative feelings about the team's process.
When utilizing brainstorming strategies, team members will be encouraged to build upon and integrate all ideas into a common outcome, instead of discarding perspectives that deviate from the majority.In this way, brainstorming would provide a structure that supports explicitly acknowledging input from minority team members and visible integration of this input into the overall team product.Thus minority team members in particular will feel more positively towards the process of working together.We propose: Hypothesis 3. The relationship between group brainstorming and hackathon participants' satisfaction with process will be stronger for minorities than for nonminorities.

2) Satisfaction with Outcome
In the following section we propose that by contrast to its direct relationship with process satisfaction, brainstorming will have an indirect relationship with outcome satisfaction that will be moderated by goal clarity.We elaborate on our reasoning below.
Goal clarity refers to the extent to which members have a shared understanding of the goals and objectives they should pursue [32], [33].Team members who share an understanding of the team's goals are more likely to meet these goals because they are able to more precisely direct their efforts towards the outcome [34].As a result, teams with clearer goals tend to report higher levels of satisfaction with the work [30].
In the context of hackathons, the group goal is often the software solution a team is preparing to build.This goal is defined by the team at the beginning of the event, rather than being imposed by management or a project road map.We suggest that in time-intensive hackathons even more so than in other work contexts, having clear goals is critical to ensure team members remain focused on the tasks needed to achieve desired outcomes.Without clear goals, team members may have different interpretations of the outcome they are working towards, and unwittingly direct efforts towards different purposes, leading to inefficient resource allocation and poorer results.They may also feel less satisfied with the output produced because it is more fragmented and conforms less with the outcome they had in their mind [37].Therefore, we first propose: Hypothesis 4. Goal clarity is positively and directly associated with hackathon participants' satisfaction with their teams' outcomes.
Furthermore, we propose that the use of brainstorming techniques further supports satisfaction with outcome in hackathon teams indirectly by increasing the level of goal clarity in the group.
Hackathons are time-bounded spaces where teams aim to develop novel and innovative ideas.Successful teams need to not only generate creative and effective solutions to problems that others may not have considered, but to do so quickly while ensuring all team members are on the same page about executing these ideas [35], [36].
Group brainstorming helps teams accomplish this in two related ways.First, by supporting the expression of all ideas, no matter how freewheeling, brainstorming encourages all team members to contribute to the discussion.This supports not only greater idea diversity, but also greater overall goal clarity because it gives team members the opportunity to restate the goal in their own words and clarify any inconsistencies.Second, brainstorming provides structure to integrate the diverse ideas generated into a cohesive whole, thus leading to more a focused solution that is a product of all team members' input [22].Therefore, team members are more likely to see clearly both the output and how their work fits into this bigger picture.At the same time, having discussed and built upon an exhaustive set of possible solutions can promote greater confidence in the eventual decision made [37].Teams that brainstorm are therefore more likely to have clearer goals, and be more satisfied with what they have produced:

Hypothesis 5. The positive relationship between group brainstorming and satisfaction with hackathon team outcomes is mediated by goal clarity.
We further propose that brainstorming will have a stronger effect on improving goal clarity for minority participants than for non-minority participants.Without intervention, minority team members may be more likely to suffer unconscious categorization by other team members and find participating in group decision making more challenging [14], [15].These perceptions may prescribe certain roles and status to minority team members, such as assigning less technical work or not consulting on technical decisions about the group output [6], [16].Team members who identify as minorities may, in turn, internalize these stereotypes and be less likely to volunteer opinions on areas of the work that they may feel are outside of their expertise, potentially due to "impostor syndrome" [7].This may result in a reduced level of overview concerning the group goals, and how their activity aligns with these goals.
Brainstorming may be more beneficial in supporting goal clarity for minority participants in two ways.First, as we describe earlier, brainstorming creates a more supportive environment for proposing ideas.This would be particularly helpful for minority team members, who may otherwise miss out on opportunities to express their ideas, thereby indirectly clarifying their goal alignment with the rest of the group.They may also experience less engagement with the final group outcome and therefore report lower goal clarity.Second, by integrating all the ideas proposed, rather than selecting one or two promising ones, brainstorming enables minority team members to see clearly not only the group output, but their role in the bigger picture.Therefore, we propose: Hypothesis 6.The relationship between brainstorming and hackathon participants' goal clarity will be stronger for minorities than non-minorities.
In the following section, we describe the methodology we use to test the above hypotheses.

A. Research Setting and Design
We identified two hackathon-like events that occurred within a week of each other at the end of May 2016.Both were two-day events aimed at producing technical code output, that explicitly recruited diverse participants and further featured a mixture of participants with strong software development backgrounds, as well as participants with little to no software development background.Both events were also similar in theme: EVENT1 encouraged the development of open source projects connected with science, while EVENT2 supported the development of infrastructure for an open network of citations.We specifically chose two non-competitive events, as they are more likely to generalize to other environments such as classroom project work and "red/tiger" software teams in organizations.
Both events featured groups of between 2-10 participants working together at a time, and both events supported some fluidity in group membership -not all teams that started had finished with the same composition.For this reason, and because it was not feasible to obtain responses from all participants in every group (described below), our level of analysis is the individual participant throughout this study.This methodological decision was not expected to significantly impact the nature of our findings: because individual members' perceptions of group process may vary within the team, particularly between minority participants and those who do not identify as such, we were primarily interested in measuring individual perceptions, and linking them with individual participants' levels of satisfaction.
Another important note with respect to our methodology is that contrary to much work in the brainstorming domain thus far, we evaluate the applications of brainstorming techniques that are discovered and driven by the teams themselves in the field, rather than imposed by a research design in a lab setting.We did not prime any of the participants to use brainstorming approaches.Rather, we sought to observe natural variability across teams in their use of common brainstorming techniques, for greater ecological validity.
To address our research hypotheses, we designed a survey instrument and a set of semi-structured interview questions that were utilized across both events.
The survey methodology allows us to directly measure statistical associations among our variables of interest in a field context, thus addressing our hypotheses.The semi-structured interviews, on the other hand, allow us to probe more deeply into the relationships observed to understand the reasoning that gave rise to these associations, and providing some evidence of the nature of causal relationships hypothesized.The sections below describe our methodology for each.

B. Survey instrument 1) Instrument design
Where possible, we utilized survey measures that have been validated in prior literature.For latent constructs such as satisfaction, that is, constructs that are not directly observable, we employed multi-item scales that utilize several related statements to describe a complex idea.

a) Satisfaction with Process
To evaluate the extent to which participants were satisfied with the process of working in their group, we utilized Reining's [30] Satisfaction with Process scale.The scale consisted of 4 items, and was evaluated on a 5-point semantic differential scale.We asked, "Would you describe your group/session's work process as more:" and provided 4 response groups: Inefficient-Efficient, Uncoordinated-Coordinated, Unfair-Fair, Confusing-Easy to Understand.The inter-item reliability for this scale was acceptable with a Cronbach's α of 0.70, a Mean (M) of 3.86 and Standard Deviation (SD) of 0.62.

b) Satisfaction with Outcome
We also drew on Reining's [30] methodology to evaluate Satisfaction with Outcome, that is the extent to which participants were satisfied with the final product of their group/session.The scale consisted of 7 items, and was evaluated on a 5-point Likert scale, from "Strongly Disagree" to "Strongly Agree".We asked participants to indicate their level of agreement with statements such as "I am satisfied with the work completed in my group/session" and "I am satisfied with the quality of my group/session's output".The scale showed good inter-item reliability with a Cronbach's α of 0.84 (M=3.85,SD=0.63).

c) Perceived participation
To evaluate participants' perceived participation we modified an existing scale from Paul et al. [43] measuring perceived participation in group decision making.The scale consisted of 6 items, and was evaluated on a 5-point Likert scale, from "Strongly Disagree" to "Strongly Agree".We asked participants to indicate their level of agreement with statements such as, "I always felt free to voice my comments during the session" and "Everyone had a chance to express his/her opinion".The scale showed good inter-item reliability with a Cronbach's α of 0.80 (M=4.44,SD=0.50).

d) Goal Clarity
To evaluate the extent to which participants felt the goals of their group were clear to them, we modified Sawyer's [32] goal clarity scale.The scale we used consisted of 4 items most relevant to our study context, and was evaluated on a 5-point Likert scale, from "Strongly Disagree" to "Strongly Agree".We asked participants to indicate their level of agreement with statements such as "I was unclear about the goals and objectives for my work in this session/group" and "I was unsure how my work relates to the overall objectives of my group/session".Negatively worded statements were reverse coded and higher values on the resulting scale were associated with greater goal clarity.The scale showed very good interitem reliability with a Cronbach's α of 0.87 (M=2.72,SD=0.97).

e) Braintorming
To the best of our knowledge, a reliable measure of brainstorming processes aligned with Osborn's [21] original propositions and appropriate for our study context was not yet available at the time of study.Thus we designed our own scale that consisted of 7 questions.A full list of questions used in our scale is available in Appendix "A".We asked participants to what extent each of the statements in the scale reflected the way their group decided what to work on, and evaluated their responses on a 5 point Likert scale from "Not at all" to "Completely".
All the items in our scale worked together to produce sufficient inter-item reliability with the exception of one reverse coded question: "Group members criticized ideas proposed during the group/session".After verifying that the reverse coding was performed correctly, we dropped this question from our scale.The resultant inter-item reliability of the remaining 6 items was acceptable with a Cronbach's α of 0.76 (M=3.49,SD=0.72) f) Minority Identification In Hypotheses 3 and 6 we were interested to examine whether our proposed relationships were stronger for team members who were in the "minority" within diverse work groups.We operationalized minority identification subjectively as the extent to which participants felt they were a minority (in terms of gender, race, background or other characteristic) because minority self-perception is an important moderator in individuals' satisfaction with group interactions [7].We validated this with objective demographics during interviews.We asked, "Do you consider yourself a minority?(For example in terms of race, gender, expertise or in another way)".Answer options were either "Yes", "No" or "Rather not say".Of participants who definitively answered this question, 22% identified themselves as a minority in some way.
g) Covariates Finally, we included measures of potential covariates.We were interested to control for potential confounds that may cut across minority identification.To do so, we asked participants to report their level of software self-efficacy, that is, the extent to which participants were comfortable in learning and using new software tools.To evaluate self-efficacy, we used a scale modified from the work of Holcomb et al. ([44]; Cronbach's α 0.70, M=3.68, SD=0.76).We also asked participants to indicate the number of years of programming experience they had (M=6.89,SD=7.66).Finally, to control for potential confounding effects of different leadership styles in teams, we asked participants whether their team had a well-defined leader role (Yes or No).Among participants who answered this question, 82% stated their group had a leader.

2) Data collection procedure
We distributed surveys within a few days after the end of both events.We included an invitation on common mailing lists for the events, and where available, sent invitations directly to lists of participants.We also sent two follow-up reminders one week and two weeks after the first invitation, respectively.
The response rate for the EVENT2 survey was 80%, out of 55 participants attempting the survey.We collected 31 responses from EVENT2 participants that were complete and passed all our attention checks.By contrast, the response rate for the EVENT1 survey was 28%, with 101 out of 366 EVENT1 participants attempting the survey.We collected 70 complete responses from EVENT1 participants that passed all our attention checks.
There were relatively more attention check problems with the EVENT1 survey than EVENT2.Furthermore, the EVENT1 survey received proportionately fewer responses than EVENT2.We should note that our response rate for EVENT1 is based on e-mailing invitations to a list of all participants who registered to attend the event.EVENT1 organizers did not have data on how many of those registered actually took part in the distributed event.The number of actual EVENT1 participants invited to take our survey could be substantially lower, artificially lowering our survey response rate, since not all who were invited were able to take it.This is also a reason it was not feasible to collect grouplevel data, as information about exact groups and participant numbers was not available to organizers of EVENT 1.

C. Semi-Structured Interviews
We obtained a sample of interview participants from the pool of survey respondents who agreed to be contacted for follow-up interviews.First, we reached out to every respondent who identified as a minority, and consented to be contacted for follow-up.There were 2 EVENT2 participants and 5 EVENT1 participants who both identified as minorities and consented to follow-up.All but 1 EVENT1 participant was interviewed, with scheduling issues preventing us from interviewing the last participant.Next, we reached out, where possible, to respondents who participated in the same group or session as the initial pool of interviewees.We further interviewed 2 participants in this category between June and July 2016.
All interviews were conducted via video or audio call, with the conversations transcribed for further analysis.We did not ask about brainstorming directly.Instead, we asked more general questions, such as "Describe, in your own words, the goals of your group", as well as more specific questions related to participants' experience and based on their survey responses.For example, we asked participants who reported positive brainstorming scores to describe their group processes in greater detail, and how these activities supported their interaction.When interviewing group leaders, we specifically probed about awareness of team members' individual goals and how these fed into the direction of the group's work.Our aim with these questions was to gain more context on the causal links between the associations we observed in our survey data.Thus our analysis consisted of examining transcripts for further details that would shed light on the relationships observed.

IV. RESULTS
The previous section presented descriptive statistics and inter-item reliability for all the measures used in the study.Before combining the results from both events for analysis, we ran additional t-tests to ascertain there were no significant differences in the variance of the measures across the two events.We confirmed no significant differences across events in the scores for brainstorming (t(66)=-0.43,p > .05),perceived participation (t(74)=-0.37,p > .05),goal clarity (t(75)=-1.64,p > .05),satisfaction with outcome (t(79)=-0.28,p > .05)and satisfaction with process (t(78)=-1.94,p > .05).

A. Satisfaction with Process
First we examined whether brainstorming had an impact on process satisfaction over and above that of perceived participation and if this effect was particularly strong for minority participants.
To test our hypotheses, we constructed a series of hierarchical multiple regression models predicting individual participants' satisfaction with their group process.Table 1 presents the multiple models in detail, together with model fit, coefficients of individual variables and variance inflation factors (VIF) to demonstrate a lack of multicollinearity issues (all VIF values were well below 10).
First we introduced dummy control variables indicating whether or not participants reported having a team leader and minority identification.We initially also included selfefficacy, the number of years of programming experience and a dummy variable controlling for the event type (EVENT1 or EVENT2).Neither of these variables contributed significantly to variance in these or subsequent models, so we exclude them from our analysis reports for simplicity.
Next we separately entered perceived participation (Model 2) and brainstorming (Model 3), followed by an interaction term between brainstorming and minority identification (Model 4).The continuous variables involved in interaction terms have been centered to minimize multi-collinearity between main and interaction effects.
We find that the control variables had a significant association with process satisfaction (specifically, minority identification: β = 0.35, p < 0.05; Model 1), and accounted for 11% of the variance.Thus non-minority participants reported overall greater process satisfaction compared to minorities, keeping constant team leadership styles.Perceived participation explained an additional 9% of variance while brainstorming and the interaction term explained an additional 10%.
We find that all other things being equal, that is, when taking into account the effects of brainstorming and control variables, perceived participation (β = 0.28, p < 0.05; Model 3) is significantly and directly associated with process satisfaction.Thus our Hypothesis 1 is supported.
Brainstorming on the other hand, appears to have a strong and complex relationship with process satisfaction.We observed a significant, positive and strong direct effect on satisfaction with process (β = 0.71, p < 0.01; Model 4).In other words, keeping constant the effects of perceived participation and control variables, all participants whose teams used brainstorming techniques reported greater process satisfaction.Thus our Hypothesis 2 is supported.
Furthermore, we find a significant and strong interaction between brainstorming and minority identification in predicting satisfaction with process (β = -0.50,p< 0.05; Model 4), lending support to our Hypothesis 3. Figure 1 demonstrates this relationship in greater detail, showing that minorities experienced greater positive effects of brainstorming on process satisfaction compared to non-minorities.Therefore, our findings suggest that though non-minorities report higher process satisfaction overall, brainstorming techniques help to close this gap for minority participants.

B. Satisfaction with Output
Next, we examined whether brainstorming had an indirect impact on satisfaction with output through greater goal clarity, and if this effect was particularly strong for minority participants.
To test our hypotheses about indirect effects (5 and 6), we needed to construct separate models predicting both direct effects on satisfaction with outcome and indirect via goal clarity [45].We report on the models with direct effects on outcome satisfaction first.
Similar to the earlier section, we first constructed a series of hierarchical multiple regression models predicting individual participants' satisfaction with the output of their group (Table 2).Model 1 featured control variables (the presence of a team leader and minority identification), Model 2 introduced goal clarity, Model 3 introduced brainstorming and Model 4 introduced the interaction term between brainstorming and minority identification.
We find that the control variables had a marginal impact on output satisfaction: while minority identification had a significant effect (non-minorities were more likely than minorities to be satisfied with team outcomes, keeping constant team leadership style) (β = 0.35, p < 0.05; Model 1), the control model accounted for only about 9% of the variance.Goal clarity accounted for an additional 33% of variance in output satisfaction (Model 2), brainstorming only an additional 4% (Model 3), and the interaction term failed to improve model fit significantly (Model 4).
To investigate if goal clarity mediates the relationship between brainstorming and outcome satisfaction, we needed to show that the paths between brainstorming and goal clarity, as well as goal clarity and satisfaction with outcome (when controlling for brainstorming) are both significant.Model 4 in Table 2 shows that despite the addition of both brainstorming and an interaction effect to the model, goal clarity continued to have a significant and strong main effect on satisfaction with outcome (β = 0.59, p < 0.001; Model 4).Hypothesis 4 predicting the direct effect between goal clarity and satisfaction with outcome was supported.
We ran an additional series of hierarchical multiple regression models predicting goal clarity to demonstrate the remainder of the indirect effect.Again, we included the presence of a team leader as a control, a dummy variable for minority identification (Model 2), mean centered brainstorming (Model 3) and an interaction between brainstorming and minority identification (Model 4).Table 3  presents the full multiple regression models, alongside model fit, individual coefficients and VIF.We found a positive, significant and strong main effect of brainstorming on goal clarity (β = 0.68, p < 0.01; Model 4), supporting Hypothesis 5, as well as a strong and significant interaction between brainstorming and minority identification (β = -0.68,p < 0.01; Model 4), supporting Hypothesis 6.The model explained a significant amount of variance in goal clarity, 24%, with a bulk of the variance provided by the interaction term.Figure 2 demonstrates the interaction in greater detail, showing that the positive effect of brainstorming on goal clarity is stronger for participants who identify as minorities.
Because the paths between brainstorming and goal clarity, and goal clarity and satisfaction with outcome are significant, this suggests the presence of mediation [45].However, because the interaction term accounts for a large proportion of the variance in goal clarity, this suggests the possible presence of moderated mediation, that is, mediation that varies in strength between minority identification categories [46].Our sample size does not afford us the ability to test the moderated mediation relationship statistically [46].
Taken together, we show that brainstorming has a differential impact on satisfaction: a direct and positive impact on process satisfaction, and an indirect positive impact on outcome satisfaction.Both effects are stronger for minority participants.

C. Interview findings
As described in Section III, we conducted follow-up interviews to better understand the issues that minority team members faced at hackathons, and concrete ways in which brainstorming improved those experiences.We found evidence both of a lack of perceived ability to participate (speak up) and minority participants feeling like they were not being heard by other group members.We also saw elements of brainstorming activity addressing these concerns in various ways.

1) Not Speaking Up
Our interviews provide evidence supporting the idea that participating in hackathon-like events can be intimidating for participants who identify as minorities.For instance, one female participant who identified herself as one of the few non-technical persons in her team said: "

I just was basically listening for a good, I'd say, seveneighths of the process. […] I know it was a lot of these very important stakeholders, and it was very intimidating. So, I wasn't speaking up." (E)
N, a CS student who is also a racial minority, described a general feeling of intimidation just from observing that she was one of only a handful of women at the event.This lead to her over attributing greater levels of experience to other older and male participants she had not met: "At first it did feel more intimidating not only because we were the only females but also because most of the other participants were older and had a lot more experience […] I think sometimes it can be like, like intimidating, um, to be like around other guys."(N) Finally, one participant remarked that it was difficult to speak up about elements missing from the event, for fear of retribution from organizers: "it didn't seem like a very safe space" (M). 2

) Not Being Heard
In addition to reporting feeling intimidated and a reluctance at speaking up, some minority participants described difficulties being heard.Specifically, E and M who were in the minority in terms of gender as well as a selfprescribed less technical background, describe having their ideas "bulldozed over" by others in the team.For example, when E expressed an opinion about how others in a similar role as herself would have problems using a tool being designed, she was brushed aside: "But there was a whole flock of them [teammates] that were just like, 'oh, white noise,' be-

because I'm only a [nontechnical role]. And I'm, like, [laughs] -I kept saying, 'well, [others in this role] don't use information in this way, and they need information in this way.'" (E)
Furthermore, some minority participants reported direct altercations with other participants at their events.In one example, participants reported that they experienced "mansplaining" and said they found it difficult to speak up about the issue: "I can't tell you how disheartening that was.Um, both the fact that it happened, that he wouldn't listen to us.

" (M)
The incident prompted the participant to question their further participation in this community: "I just felt like wow, this is really, this is the last time I'm going to have any interaction with this community, because I've been deemed to be some kind of […] a rebel."(M) 3) Brainstorming Alongside some negative reports of group dynamics in the previous two sections, findings from our interviews also provide initial causal links between brainstorming principles and more positive affective team outcomes like satisfaction.
Consistent with Hypothesis 3, we saw evidence that employing brainstorming techniques supported team members feeling more satisfied with the process of working in the group, particularly from diverse team members.For example, one group used brainstorming to develop a user interface for a tool the team was prototyping.First, everyone made individual drawings on the whiteboard of what they thought the interface should look like, and then the group collectively discussed the ideas.Instead of picking only one idea, the group combined elements from two of the proposed drawings.N, a member of this group, reflects that the experience was "fun" because her efforts played a big part in the team's project actualization: "It was fun.We were just building something that we had designed … just doing something that we had drawn out and now we were just coding it and making it a real thing."(N) We also observed team leaders using chat and check-ins to acknowledge their members' input: "But I definitely like cheerleaded him, I don't know if that's the right grammar, um, in the [Chat] Room.So, like, when he made the pull request I was like woo, woo!" (K) "I tried really hard to kind of cheerlead people who were doing stuff, so my check in I had was like D's done this and E's done this." (K) Finally, we saw some initial evidence supporting the indirect relationship between brainstorming and satisfaction with output via goal clarity.When asked what actions are important to ensure an outcome the team can be happy with, D, a team leader, suggested that it was important to both ensure the goals were clear and that the problem was relatable to all team members: "Making [the goals] achievable, and just define the problem clearly [...] so that they can feel the problem relates to them" (D) D strategically employed brainstorming in her group to support both of these elements: working to integrate solutions provided an opportunity for all team members to clarify their understanding of the output they were working towards, while allowing every individual to propose an idea encouraged feeling that the problem related to them.

V. DISCUSSION
In the present work we set out to understand how techniques designed to stimulate idea generation, such as brainstorming, can support greater involvement of minority team members and their satisfaction in working within software engineering teams.We found that brainstorming directly supported stronger affective outcomes, such as a feeling of "fun" or satisfaction with the process of working in the team.We also found that brainstorming supported outcome satisfaction indirectly by improving clarity of team goals.Below we discuss the implications of these findings both for theory and software engineering practice.

A. Implications for theory
Though work within the domain of brainstorming spans several decades [22], the principles have been thus far applied with a somewhat narrow focus within software engineering, primarily as a means for generating more ideas in managerial practice and methods such as Agile [19], [20].Our results provide initial evidence that there may be elements of brainstorming that have broader and less direct effects on team success than initially imagined in prior work.
Firstly, we find that elements of brainstorming such as integrating and building on proposed ideas provide mechanisms for structured inclusion and support better affective outcomes.Specifically, our findings show that brainstorming techniques significantly improve satisfaction with working in the group over and above merely feeling able to express an opinion, and that this relationship appears to be particularly salient for participants who identify as a minority in their work group.Thus brainstorming techniques encourage more positive group processes beyond the freedom to voice out ideas that also encourage those ideas to be acknowledged and integrated into teams' eventual outcomes.
Second, we find that brainstorming elements such as encouraging freewheeling ideas and suspending judgement, indirectly facilitate satisfaction with team output.Specifically, our results suggest that brainstorming impacts satisfaction with outcome indirectly by increasing clarity of goals.This relationship also appears to be stronger for participants identifying as minorities.Brainstorming techniques work towards the development of a shared understanding of group goals and objectives, reducing social loafing in the group by aligning the goals towards those shared by all team members and enabling the team to work collectively and effectively toward that goal [34].These techniques are particularly relevant for addressing the experiences of minority participants in groups, who may otherwise have less opportunity to express their ideas, and as a result may find the overall outcomes being developed to be less aligned with their individual expectations.
The present work advances our existing understanding of brainstorming in several important ways.Firstly, by investigating minority identification in diverse teams, our work extends previous research that had either not considered moderating factors connected with diversity, or studied these factors by contrasting homogenous groups with different demographic properties.Second, we show that brainstorming has potential benefits that go beyond direct and objective measures of performance, idea quality and number of ideas generated.In fact, our findings suggest that the affective benefits of brainstorming, like encouraging satisfaction with process, may be relatively more powerful than cognitive benefits, like encouraging better outcomes.While we are not the first to look at satisfaction in connection with brainstorming (e.g.[24]), our findings diverge from earlier work that found no significant improvement in satisfaction.We believe the differences we are observing are in large part due to investigating two distinct dimensions of satisfaction, as well as the inclusion of minority identification as a moderator, in the same study.
Taken together, we show that brainstorming continues to have relevance when you take into account diverse team characteristics, and hope to see more work expanding both on the kinds of moderators examined as well as continuing to refine the concept of brainstorming itself.For example, our interview findings suggest that public recognition of individual group members' efforts may be an important additional element of successful work in skewed teams and software development teams more generally.This may be a potential extensions of Osborn's [21] proposition to integrate and build on the work of others, and warrant further empirical investigation.

B. Implications for practice
Firstly, the present study offers a test of theory driven, concrete and actionable strategies that may support greater participation of minority members in diverse teams.In particular, we show initial evidence that brainstorming techniques can support minority members in teams that have to produce software outcomes within very short time periods, not only designing but implementing the tasks they determine together, thus bringing this work closer to the field than experimental work in the lab.We also show that these findings appear to generalize across both teams with team leaders, as well as more self-directed teams.Thus our findings show the increasing relevance of brainstorming strategies as a process for supporting diversity, alongside efforts to increase the sheer numbers of diverse development teams.
Group brainstorming is one of several possible ways teams can support participation of minority team members.Another is having a professional facilitator, a neutral party trained in group dynamics and communication who advocates for fair, open, and inclusive procedures to accomplish the group's work [47].However, brainstorming might be more readily applied in time sensitive contexts that depend on groups generating their own ideas, such as hackathons, since it takes time for the facilitator to learn the culture of the group, and clearly understand their goals.Moreover, facilitators may be seen as top down, intrusive and expensive, whereas brainstorming can be conducted by participants themselves.
Second, the present study develops and tests a new measurement tool that can be used to reliably evaluate the extent to which teams in the field adhere to brainstorming principles.This may prove to have a significant impact on future work in the brainstorming domain, as a focus on experimental contexts in the lab had been a primary limitation of earlier work [22].Specifically, a reliable scale of items can allow researchers to examine larger numbers of real world teams and develop insights at greater scale than we have been able to previously.The measurement scale may also be of interest to industry practitioners looking for a concise way of measuring the impact of brainstorming techniques they introduce based on our recommendations.

C. Limitations
Of course, our study is not without limitations.As is common with studies in the field, there are potential confounding variables.One important confound is the design of the two separate events that we studied, that may lead to different team practices and thus systematic variance in some measured variables.We took steps to manage this potential confound by looking for significant differences in variance for all the variables in our study across the two different events.As we describe in Section IV, following a series of t-tests, we found no significant differences.We also examined individual variances in software engineering experience, and selfefficacy with regards to software and found these do not add significantly to our models.There may be other confounding variables that are outside of the scope of our study, and we encourage future work to address additional variables.
Second, the correlational nature of our study introduces potential causal ambiguity in our propositions.We attempted to address this limitation with pointed interviews that allowed us to more confidently support our causal interpretations.
Third, our study is situated in the context of short-term time intensive work during hackathons.Though our research design is focused on maximizing potential generalizability by looking at non-competitive events, and controlling for a number of variations in event and group design, the above findings would benefit from looking at a broader range of software development teams to understand the extent to which our findings may be generalized.
Furthermore, though we talk about team processes and outcomes, due to our focus on individual experiences and satisfaction, our models are intentionally built on individual level data and predict individual level outcomes.While we collected group information in our surveys, the response rates and small size of the events themselves preclude us from doing additional group-level statistical analyses.We hope that future work will be able to take our initial findings further by also taking into account group level variations such as distinct group norms that may have formed in the course of working together.
Finally, though we find initial evidence of potential moderated mediation, our sample size precludes us from conducting an appropriate statistical test of this effect using more recent techniques [46].Rather than measuring the significance of the mediation directly, we use the classic methodology by Baron and Kenny [45] that infers significance of a mediated relationship based on significant coefficients representing different pieces of the relationship.We hope to see future work take this design further using the brainstorming scale items we have developed.

VI. CONCLUSION
The present work set out to explore how brainstorming techniques may be used in different ways to support minority participation in diverse hackathon teams.Specifically, we found brainstorming to directly support minority participants sense of enjoyment with working in the team, and indirectly support a sense of achievement with respect to team outcomes.Our work contributes both a theoretical dimension in expanding our understanding of brainstorming, as well as practical advice for team leaders, participants and managers on navigating diversity in discussion.

Figure 2 .Figure 1 .
Figure 2. Interaction Between Brainstorming and Minority Identification in Predicting Goal Clarity

TABLE 2 .
MODEL PREDICTING SATISFACTION WITH OUTCOME

TABLE 1 .
MODEL PREDICTING SATISFACTION WITH PROCESS