A quick review of ethics, design thinking, gender, and AI development

ABSTRACT Ethics, artificial intelligence (AI), and design thinking are current buzz words. The threat of bias-riddled machine learning algorithms is represented media-wide. Research and development initiatives are endeavoring to ‘translate’ ethics into processes and machine logic and design thinking as a method is adopted to gauge the interests and values of a vast range of stakeholders. Gender, its framing, reflection, and critical evaluation in relation to design thinking as a means for developing ethical AI appear to be less represented in scholarly discourse. Against a background of critical theory and gender studies that describe and problematize gender, its construction and norms in socio-technological discourse, the authors of this article aim to generate insight into the current state of gender in design thinking research focused on ethics and AI. A review of scholarly articles revealed trends in popularity of concepts and prominence in the application of design thinking in specific fields (i.e., educational research). Repetition characterizes the more challenging topics or wicked problems. Provocation and investigation of gender from the perspectives of practitioners, creativity, and its influence in design thinking seem all but visible.


Introduction
'Design thinking' has been a popular term for the past few decades. It has been used to refer broadly to thinking and cognition within design-designer's thinking, cognition in the design of its processes and formations, problem-solving, and creativity (Buchanan, 1992;Cross, 2011Cross, , 2018. It has also been used to describe a co-creational methodology that engages multiple fields and disciplines to effectively problem-solve, imagine, and realize products and services from a wider stakeholder perspective (Kimbell, 2011;Leifer & Meinel, 2018). The scope includes user-centrism, humancomputer interaction, business, design, engineering, and more. Design thinking as a methodological approach is known for its multidisciplinary engagement in creative problem-solving and design realization. Stanford d.school's design thinking process model is perhaps the most well-known artifact relating to practical application of design thinking (Thoring & Müller, 2011). It was adapted from the consultancy company IDEO that had focused on developing systematic design methods for tackling wicked problems (Buchanan, 1992). In international health emergencies (pandemics), global warming, economic instability, and resource challenging climate, designers are faced with bountiful wicked problems. None-the-least, the race toward a smart society inherent with highly functioning artificially intelligent (AI) systems has introduced a new wave of challenges -ethical Stortzfus et al., 2011). Studies have shown no significant difference between levels of creativity and gender in children and adolescents (Baer & Kaufman, 2008). There were conflicting results shown regarding gender and creativity in adults. At the heart of the arguments explaining why differences exist rests a myriad of theories. These range from different prioritizations in the sensory-cognition system (Runco et al., 2010) to 'blanket environmental' ones (Baer & Kaufman, 2008). Despite the lack of difference between actual levels of creativity, there is less representation of women in creative fields in adulthood, especially in technical professions, possibly due to differences in gender expectations and environments during childhood (Baer & Kaufman, 2008;Stortzfus et al., 2011). The trajectory of this quick review and its broader research study is to ascertain what is already understood of the ways in which these categorizations live on in a design-oriented AI-aspiring society. This is coupled by further examination of how ethics is linked to discussions in design thinking, and if this in turn strongly connects to gender considerations of the practitioners in the field.
The study utilizes literature searches conducted through Google Scholar to show the resulting quantities of key search terms, how studies connect with one another, and how ethics, in relation to gender and AI development, are being represented in contemporary design thinking research's published discourse. In order to establish a common language for this paper, we begin with a concise account of ethics, followed by briefly describing design thinking as an IDEO and d.school derived methodology, including the conception of creativity in this process model from the perspective of collective cognition. Acknowledging the richness and diversity of ethics as a philosophical and sociological yet focusing on brevity, we touch upon the origins and some classifications, then move into greater detail from the perspective of modern design and design thinking practices in AI development. The literature search results are presented, which is followed by a reflective discussion interpreting what these results represent in terms of conditions related to gender, ethics, and design thinking at the dawn of a smart society.

Ethics
Ethics is an area of scholarship and philosophy that spans back through human history (see, e.g., Kant, 2001;Rivers, 2000). Not only is the field of ethics vast but as with all deeply philosophical and by nature, controversial scholarly disciplines, the character of ethics, what they involve, and how they may be defined are still much contested (Braunack-Mayer, 2001). During times of pandemics, economic instability, evolving workforce and intelligent technological transformation, and ethical questions move increasingly to the fore of public consciousness (Chandler, 2014;Hoffmann & Hahn, 2020;Mascaro et al., 2015).
There is no 'one size fits all' approach to understanding ethics, their philosophical and theological roots, and the factors that influence their construction within cultures and various cultural settings. Ethics, and the field of moral philosophy, entail the organization, standardization, recommendation, and defending of behavior that could be considered as either right (just) or wrong (unjust;Fieser, 2020). Ethics is commonly divided into three main approaches: 1) metaethics -the origins of ethics; 2) normative ethics -where moral principles are related to behavior and how they are permeated through socio-cultural systems, both discursively and systemically (Gewirth, 1960;Pellegrino, 1995); and 3) applied ethics -ethics put into practice through design, systems, and behavior (Fieser, 2020). Applied ethics go beyond utilitarian ethics (a part of normative ethics) in actionable endeavors for maximum benefit for many with minimum harm (see, e.g., Mill, 2010) through the acceptance of complexity. Applied ethics attempts to analyze phenomena and apply principles, standards, and guidelines to generate the greatest possible positive outcomes considering controversy and complexity. There is always knowledge of trade-offs and potential opposition, as the issues applied ethics deals with will never be fully and acceptably resolved on a societal level or otherwise (Fieser, 2020). These may be understood as moral dilemmas (Nichols & Mallon, 2006) or the wicked problems of the moral world.
The historical emergence of The Golden Rule (The 10 Commandments of the Bible and the Code of Hammurabi) to treat others the way you would like to be treated (M. G. Singer, 1963) -is an interesting topic for applied ethics. As it also falls into the category of both ruler/influencerinfluenced invention, and the necessity to provide moral structure to an otherwise chaotic society (see, e.g., Prince, 1904;Saariluoma & Rousi, 2020). Virtue ethics, as studied by the Ancient Greeks (e.g., Aristotle, etc.) relates more specifically to individual actors in connection to moral character and virtues held and represented by individuals or groups (Hursthouse & Pettigrove, 2018).
Virtue ethics have proven extremely popular within the modern corporate landscape, particularly in relation to technology-related issues, brand management, and corporate reputation (see, e. g., Audi, 2012). This is due to the social and cultural power of identifying values (i.e., equity, fairness, sustainability, etc.), aligning them with the business message and offerings, which in turn aligns potential customers (Murphy et al., 2007). Clear, consistent, and repetitive communication of these values in public discourse via websites, social media, and other marketing media, develops strong semantic associations between particular characteristics of an organization and recognition of what is deemed as 'good' or socially responsible (Van de Ven, 2008;Williams & Murphy, 1990). Thus, from a marketing and communication perspective, virtue ethics is about creating strong associations between what is good, just, and responsible, while at the same time othering and defining what is not (Ailon, 2015). From a design perspective, design teams and practitioners have the task of probing into the heart of the organization and its people's (multiple stakeholders') and embodying these through output (communication, form, logic, business models, etc.; Mortati et al., 2014). The harnessing of values, or value-creation, is not simply projected in the look and feel of products and services (Amit & Zott, 2001). Rather, the way they operate and are encountered, their existence and relationality within communities and societies (freedom of choice, voluntariness, satisfaction, benefits, etc.) additionally radiate a company's ethical stance (Rousi, 2021). This is where the mechanisms and qualities of design thinking as a cognitive method of inclusion -users, stakeholders, multidisciplinary -enter the picture, particularly in the era of intelligent digital transformation (Magistretti et al., 2021).

Design thinking
Design thinking in its various forms, has been a popular concept in industry and academia for several decades (Verhulsdonck et al., 2021;Von Thienen et al., 2021). Depending on one's scope, design thinking can be thought of as a movement to combine multiple disciplines and perspectives in design, development, and business processes (i.e., as seen in the d.school model; Kimbell, 2011) or can be traced further back through cognitive scientific studies on design cognition, design logic, and even problem-solving (see, e.g., Cross et al., 1992;Dorst, 2011;Newell & Simon, 1972;Rowe, 1991). If design refers to planning, processing and shaping phenomena through intentionality (Crilly, 2011;Dennett, 1987;Parsons, 2015), then design thinking can be seen as the explicit act of expressing and communicating thought and values through design. It can also be understood as a means of extending and embodying the human mind through designed objects, services, and systems (Saariluoma et al., 2021). This idea of human augmentation comes into play when engaging in design thinking as a method. For through multi-disciplinary teams, stakeholder inclusion, and then manifestation of the design project, the method can be seen as an actionable process of collective and extended cognition (Murty et al., 2010). This is particularly pertinent when considering that a great deal of design projects involve the development of AI -cognitive technologythat carries the logic of its human creators. Over or underrepresentation of particular cultural groups or types of designers instills a narrow logic within the systems that excludes diversity and the potential for broader considerations and alternatives.
The practical six-step design thinking process that was developed and promoted through Stanford's d.school can be understood along the lines of strategic design-focused communication that combines minds (knowledge) and articulate values from multiple viewpoints to create and sustain more effective design outcomes (Kimbell, 2011). Moreover, through engaging with experts across disciplinary and professional borders, there is an expectation not only that the outcomes will be more effective and effective, but sustainable (economic, socially responsible, and possibly even environmental) and ethical (doing right and being good in relation to broader fields of stakeholders; Birkeland, 2012). In fact, this ethical standpoint of attempting to maximize benefits for as many stakeholders as possible while increasing positive societal impact, has been a cornerstone of design thinking discourse over recent years (Andrews, 2015). This has been manifested in activities such as manifestos, workshops, reports, and toolkits (see, for instance, IDEO's DesignKit 1 or Sohail, 2017). Levels of ethical thinking or moral positioning can also be seen within current corporate strategies such as those that promote sustainability (environmental, economic, corporate, humane) and social responsibility (i.e., Google, IDEO, Gofore, etc.). This form of approach is also key to understanding many of the current development practices such as Agile, Lean, and other human-centered design and development initiatives.
By nature, design thinking aims at offering the opportunity to influence the direction of design from a human (humane) and stakeholder-centric perspective (Kolko, 2015). This is achieved through increasing the possibilities of positive impact through engagement, interaction, and communication, which in turn may be understood as an ethical and democratic approach to design (Saward, 2021). The more intensive the design thinking process, the more possibilities there are for opening the gates to ethical considerations (see, for instance, Dorst, 2011;Panke & Harth, 2019). The Ethical Design Thinking Toolkit (Sohail, 2017) for example, combines key tools in humancentric design from IDEO and considerations for personal values by Joe Edelman (co-founder of Time Well Spent, 2018). This combination provides a strategic approach for aligning personal and human values with tangible product design (product values, see, e.g., Friedman & Kahn Jr, 2003). In reflection of the virtue ethics description on value-alignment with business (image, vision, and marketing) and offerings mentioned above, this ensemble allows for value-based embodiment within the organization's design.
Questions such as, 'What is the problem I am trying to solve?' or 'Who am I trying to solve these problems for, and why?', as well as, broader questions including, 'Does my challenge drive toward ultimate impact?' are as much the foundational considerations of design thinking as they are for ethical thinking (Saariluoma & Rousi, 2020). Thus, design thinking from a practical design methodological perspective, entails deeper levels of human understanding in light of design, shifting the emphasis from, 'Will someone use it?' to 'Why and how will they use it, and what does this mean in the context of their lived experience?' (Dorst, 2011;Leikas et al., 2012). This deeper understanding is reinforced by multi-professional, or multidisciplinary, examination of the design problems in question in order to gain a holistic view for design and development. To build knowledge on design problems through multiple disciplines also entails that ethical questions arise that may vary in nature from one field to the next (Dwivedi et al., 2021;Norwood & Paterson, 2002). For innovation processes and practices in the space of complex data-intensive emerging systems such as AI, this characteristic is paramount (Dwivedi et al., 2021). For, in the case of AI, ethical considerations are not only implicated in the technology itself but amasses questions via the borders it spans and fields it entails (Leikas et al., 2019).
For this reason, the authors have sought to gain an enhanced understanding of the current state of ethics in design thinking scholarship and discourse. Numerous perspectives are relevant to this topic (i.e., privacy, equality, economics, ethnicity, culture, linguistic, etc.) the matter of gender in relation to ethics and design thinking was chosen due to its relevance and resonance with the authors' professional experience. Through taking the aspect of gender in design thinking and ethics, this quick review serves as a basis for extended empirical research in the field.

Materials and method
A literature review was performed to generate an understanding of how ethics have been studied and represented in design thinking research from the angle of gender. The literature searches progressed in a structured manner (Swanson & Santamaria, 2021) that began with the basic terms of 'ethics and design thinking' and accumulatively progressed through the themes: 'gender and design thinking'; 'gender and design thinking and ethics'; and 'AI and ethics'. The authors utilized online search using Google Scholar. To ensure integrity, two researchers performed the same search actions in the above-mentioned databases (Kitchenham, 2004). The search results were ordered by relevance. Data that arose from the search term 'gender and ethics' is excluded because the resulting articles rested outside the scope of this review (they were not related to design thinking or AI discourse).
To maintain a manageable sample, enabling a 'quick review' of the current state of research and discourse, the first two pages from search results of each category were documented in excel. These were then analyzed based on the article title, abstract, focus of the article and year of publication. For the purposes of this quick review, the year or period of publications is important as it gauges the rate of popularity in represented themes, topics, and issues during certain windows of time. To ascertain active and current discourse of research studies from the respective periods presented (based on publication date), possible books were rejected from results. Books are fruitful to examine, yet often possess a broader scope, theoretical contribution, and discursive framework while presenting thought that has developed and matured over a longer period (Tilburge University, 2021).
The overall search results are represented in Table 1. 'AI' and 'ethics' generated 1,980,000 hits (publication years ranged from 2003 to 2020). This is currently a popular field of research as researchers, technologists, and businesses alike strive to instill ethical principles and account for ethical issues within their AI development (Vakkuri et al., 2020). This is broader than strictly design thinking research, yet the researchers considered it important to ascertain the level of inclusion of human insight through so-called human-centered approaches to emerging technologies (AI ethics). AI ethics is maintained in the results as opposed to gender and ethics, due to the prominence of the topic within the discourse of design thinkingrelated literature. The search words 'ethics and design thinking' generated 555,000 hits (between 2011 and 2021). This shows that ethics and design thinking are relatively recent topics. 'Gender and design thinking' attracted 20,100 hits (ranging from 2001 to 2020) demonstrating a longer, yet perhaps not so popular tradition of study in design thinking research. Finally, '"gender and design thinking' and 'ethics' gained 2,280 hits (dated 2001 to 2021). The time range is like the gender and design thinking category, no doubt with overlaps, yet the inclusion of 'ethics' meant a dramatic decrease in hits (see, Figure 1).
The details (citation details, title, abstract, keywords, and focus of study) of the first two pages of search hits from each category were entered into an Excel spreadsheet. The authors then reviewed the document and coded the data according to topic and focus. The authors aimed to ascertain the scope of focus areas that have been covered within the sample of the literature to understand the current discourse and trends in each category. In total, 69 articles were included in the spreadsheet for closer review. The abstracts were read and analyzed, and subsequently 12 full texts (3 from each search term) with the highest number of citations were assessed in terms of deriving themes.

Results
Here, the findings of the excel-based data analysis are discussed in relation to the main categories and trends arising in the literature. The results of each search term are discussed sequentially. These results can be seen in the supplementary material (Appendices tables) for the article. The findings were analyzed in terms of keywords, focus, and how design thinking was approached. In addition to gaining insight on the most popular topics and range of perspectives represented, the authors also reflect on the citation amount to ascertain the level of influence the publications have had on other research.

'Ethics' and 'design thinking'
Fourteen publications out of the results emerging on the first two pages of the search on 'ethics' and 'design thinking' were chosen for this review (see Appendix 1). These publications were selected due to the direct inclusion of 'ethics' and 'design' and/or 'design thinking' being included in their title and their relevance -as evaluated from their abstracts -in relation to the design of emerging (AI) technological systems in practice and education. In total, at the time that this quick review was performed, these publications were cited 664 times (2011 to 2021). Zwijsen et al.'s (2011), 'Ethics of using assistive technology in the care for community-dwelling elderly people' was cited 263 times as being seemingly the most influential paper out of the 14. The article focused on the development of assistive technology and ethics in the context of aged care. Their study revealed three prevailing themes: personal living environment -issues of privacy, autonomy, and obtrusiveness; the outside world -stigma and human contact; and the design of assistive technology devices -individual approach, affordability, and safety. Zwijsen and colleagues discussed how while the above-mentioned concepts are vital for ethical approaches to design thinking grave problems rest in their application. They emphasize the term 'thick concepts' to denote constructs that lose the power to signify through careless overuse (see, e. g., Laclau, 2017). Their research showed tendencies for assumptions such as those pertaining to the ethical value of human autonomy and self-determination, without gaining full insight into the types of case-specific and contextual factors that impede possibilities for autonomy (i.e., cognitive decline, mobility issues, etc.).
Peter Dalsgaard's (2014) article, 'Pragmatism and design thinking', was the next most cited (165). Interestingly, this takes a philosophical pragmatist perspective to understanding design thinking through attaching experience to ethics. In this article, Dalsgaard focuses on the relationship between pragmatism, or the position that individuals experience and know the world through agency (doing), and design thinking. Dalsgaard draws on John Dewey (2004) to cite the similarities between pragmatism and design thinking. In his article, he elevates the pragmatist principles of psychology, logic, experience, art, democracy, morals, and ethics. No doubt based on this inclusion of morals and ethics, the paper emerged in the search results. Dalsgaard compares design thinking and pragmatism through learning and emergence -understanding that the world is never a complete picture or fully finished product. Thus, pragmatism, like design thinking, is about understanding constantly evolving processes. This theoretical and anecdotal paper 2 interweaves philosophy with practice, and as seen by the citation amount, the theory meets practice way of uniting design thinking to ethical questions has been popular. This article promotes an understanding that everything happens in interaction. Design thinking is posed as a site of interaction between disciplines and stakeholders. Shapira et al.'s (2017), 'The integration of design thinking and strategic sustainable development' (88 citations) taps into the burning topic of sustainability. While the citation rate is still relatively small considering work on other topics, this article attempts to establish a framework for strategic sustainable development. This can be considered an applied ethics approach to instilling sustainability within human-centered design and innovation practice through design thinking. The study presented in this article was influenced by IDEO's understanding and promotion of design thinking -very much from the perspective of promoting understanding among educators. While the article differs in approach to Dalsgaard's article, it also emphasizes the importance of learning and education. Here, the authors argue that ethics (and sustainability) are conditions within an ongoing process, for which we need the tools to learn. Similarly, some papers addressed education explicitly. Kirkman et al.'s (2017), 'Teaching Ethics as Design' observes the factors influencing learning and skill acquisition (including thinking skills) in the context of design and engineering education. Kirkman et al.'s (2017) article focuses on understandings of ethical values in design. This paper presents a study in which problem-based learning is used as a vehicle to systematically identify and analyze ethical and design values in every decision made within a design process. Ethical thinking skills and moral imagination are examined via Latent Semantic Analysis of short answers to ethical design questions. In the same vein, Spiekermann and Winkler's (2020) much less cited article (six citations) 'Value-based engineering for ethics by design' focuses on values in the design process and how these should guide the deliberation of priorities. A key notion within the paper is emphasis on respect for users from design to implementation. Jensen and Vistisen's (2013) article, 'The ethics of user experience design discussed by the terms of apathy, sympathy and empathy' (one citation) explores the constructs of empathy in relation to sympathy and apathy. Empathy and sympathy are often confused with one another, and the notion of apathy is used to expand understandings of ethics toward social responsibility and accountability for the designs that have been produced. The remaining articles in the list by Madson (2021), Ericson (2021) and a book by Rousi et al., 2020) have not yet been cited according to the search date. Key issues within these publications include creativity, design thinking, innovation, education (medical and design/engineering), entrepreneurship, emotions, and culture.
Several key themes emerged through this literature search category. Firstly, emotions and empathy, is an area that is recognized as contributing to our understanding of ethics, as well as the incentive to design with ethics in mind. In other words, as designers (engineers, developers) we should aim to include consideration for the users and potentially the users themselves within the design process to ascertain the best possible solution, supporting physical, social, economic, and psychological needs to promote wellbeing (on many levels) and minimize negative impacts.
Secondly, design thinking, learning (problem-based), and education, are common categories in which design thinking is instrumental for both the development of ethical design processes and products, as well as probing ethical aspects of design. Thirdly, values are important. The distinction between design and ethical values is often made, and the importance of aligning design, designer, and user and ethical values is integral. Interestingly, while articles representing sustainability do not arise as much within this search category, the citation amounts show that it is a topical subject in design thinking ethics. This stands true also for the theme of aging and returning to the roots of understanding what ethics are considering philosophy and design (pragmatism).

'Gender' and 'design thinking'
The act of searching articles with the search term, 'gender' and 'design thinking' was fascinating, and the results can be seen in Appendix 2. Altogether 17 publications are reviewed from this category (see Appendix 2). The overall citation amount at the time of retrieval was 6ʹ109 from 2001 to 2020. At the time of the literature search, no articles emerging in Google Scholar had specifically focused on gender in design thinking processes. Gender instead seemed to be the result and byproduct of data analyses in which gender was just one of the independent variables.
The article with seemingly the most influence in terms of citation quantity (2924) was Dym et al.'s (2005) 'Engineering design thinking, teaching, and learning'. The article focuses on the role of design thinking in engineering education, how it has been utilized and how its role should be emphasized. The starting point of article was that the goal of engineering education is to produce engineers who can design. Here, design thinking is treated as a complex field. The paper briefly reviews the history of design thinking and its role in engineering education from different perspectives. Gender was mentioned in their review of related work, then included within their description of collected background information. The article mentioned previous studies regarding gender issues in HCI, in addition to research regarding females in engineering, yet gender was not otherwise dealt with in this article.
The second most cited article (1797 citations) was 'Design thinking for social innovation' (Brown & Wyatt, 2010). This article focused on how to utilize design thinking in social innovation projects. The authors present an example of innovation work on a local water supply in Ghana. Three pages in this article focus on women, their aspirations, priorities, and design initiatives that were intended to support access to resources and foster family wellbeing. This can be seen in the authors' references to a free fishing net initiative for pregnant women and mothers of small children in Ghana, as well as some text discussing internships. While females were prominent within the article, they are framed as receivers of the initiatives of the project. Gender issues per se in terms of the influence of designer gender or prevailing gender-based ideologies, were not addressed in this piece.
The third most cited (928) article, 'Design thinking: past, present and possible futures' (Johansson-Sköldberg et al., 2013) focuses on two design thinking perspectives: managerial and design. The article critically examines the discourse related to design thinking and its different context-related meanings. In managerial area, design thinking is seen as an effective means for instilling creativity and innovation. Regarding design, the paper identifies various discourses. The authors state that different views should not compete against each other, instead they should be developed in parallel. In this article, gender is mentioned in terms of students' changing views toward gender, and as a personal research interest. Once more, the analysis does not delve deeper in relation to how gender may be read within design thinking practice.
Most of the articles focused on either 'design thinking' (five articles) or 'education and learning' (five articles) (see Appendix 2). Articles in the group 'design thinking' highlighted different applications of design thinking, for example, team-based design thinking, childhood design thinking, and history of design thinking. Articles related to 'education and learning' focus on various applications of design thinking in education and learning, similarly to what is stated above. Two articles focus on design 'studying design thinking'. One focuses on studying different approaches of design thinking, and another on measuring design thinking ability. One publication focuses on gender equality and diversity (Christensen et al., 2020) with zero citations. This paper was written within the scope of critical management studies and discusses a GenderLAB workshop. Within the rationale of the article, the authors draw on theories of performativity to illustrate the roles of women and men is various design settings. The idea was to accentuate bias and fixations within 'taken-for-granted' roles and actions deliberated within cultural and societal norms.
These search terms also returned single hits in the following areas: co-creation -co-creation in design thinking; computer programming -utilizing design thinking in computer programming; ethics in design thinking -gender and ethics in design thinking; philosophical view -differences between dualistic and multiplistic thinkers; and social innovation -how to use design thinking in social innovation.

'Gender' and 'design thinking' and 'ethics'
Altogether, 20 publications are reviewed from this category (see Appendix 3). The overall citation amount at the time of retrieval was 5ʹ087 from 2001 to 24 March 2021. The article with seemingly the most influence in terms of citation quantity (2924) was once again Dym et al. (2005). Dym et al.'s main tenet was that design is a complex process that should be harnessed also by engineers, particularly considering the systemic technological future that is emerging. Design thinking is treated as multi-faceted, far beyond any understanding of a five-step method. Besides posing some open questions regarding design thinking, the article does not delve deeply into the nuances of considering design thinking from a critical gender perspective. Brown and Wyatt (2010) was the most cited in this search category. It is seen in this article that issues of gender in design thinking are tightly intertwined with the ethical considerations of wellbeing and equality. The third most cited article (102 citations), 'Combining critical reflection and design thinking to develop integrative learners', by Welsh and Dehler (2013) focuses on the managerial education domain. The paper argues that combining design thinking with managerial education affords students the possibility to learn additional skills, such as critical reflection and knowledge transformation. The combination of managerial and design thinking education is beneficial for learners. Once more, gender does not arise as a key factor of concern within the study.
Once more, education and learning are in the spotlight of papers in this category. Educational fields include entrepreneurship, medical education, reforming education, and education. Two articles focus on design thinking in health care One concentrates on gender-affirming design thinking within medicine protocols and another on caring design as an application of design thinking. Two more articles focus on ethical design thinking. One centers on design thinking and diversity, and another on design thinking and racialized factors.
Additionally, these search terms returned single hits in the following areas: design thinking and organization -examining design thinking in organizational changes; design thinking and placestudying the relationship between design thinking and place; design thinking and robotics -using design thinking in robotic design; design thinking and social concept -design thinking on social innovation; design thinking and well-being -design thinking and community well-being; philosophical view -theoretical issues behind the practical approach; and design thinking in workplaceutilizing design thinking in workplaces.

'AI' and 'ethics'
Altogether, 18 publications are reviewed from this category (see Appendix 5). The increase in popularity of intelligent technologies has brought with it an ever present need to consider and address ethical issues (see, e.g., Vakkuri et al., 2021Vakkuri et al., , 2020. As seen in user-centered design and development embedded in design thinking, ethical questions are presented from the ground up in techniques such as user stories . The article with seemingly the most influence in terms of citations (414) was Jobin et al.'s (2019), 'The global landscape of AI ethics guidelines'. During the past few years various organizations, private, research, and public, have been defining ethical principles and guidelines for AI development. The debate is ongoing around what ethical AI means and what requirements, standards, and practical issues are required for AI usage. In this research, the authors mapped global agreements and understandings of the area. They discovered that five ethical principles are globally discussed: transparency, justice and fairness, non-maleficence, responsibility, and privacy. Their work revealed that the research and development of this area is very important. Hagendorff (2020, cited 163 times) focused on analyzing 22 sets of AI guidelines and principles. This article echoed the sentiments of Jobin et al.'s paper in terms of the necessity of translating the principles into practice, yet additionally suggests ways in which to apply them practically in development.
Similarly, to the other two articles, in his much earlier article McLaren (2003) described the difficulties in concretizing ethical principles into action. Yet, what is significant about this paper is the fact that they explicitly stress the necessity of gathering information about past cases and examples to isolate details about the implicated situations, their contexts, and then about the causality of ethical factors during moments of operationalization. McLaren argued that we learn from previous experience to anticipate the future. This follows in a discussion by Musschenga (2005) who discusses the relatively and context-sensitivity of ethics and moral frameworks.
Principles and guidelines dominate most of the articles (eight articles) (see Appendix 4) with the main concern of how to apply these in practice. The article covers topics concerning existing AIrelated guidelines and principles, their flaws, usage in practice, and application. One article is related to ethics in addition to where and how to use AI for doing good instead of harming anyone or anything. The second-largest group of articles (4) concentrates on ethics documentation. Articles are related to AI ethics overview, importance of ethics and international cooperation. These search terms returned also single hits in the following areas: automated AI oversight -ethical issues in AI surveillance; concerns -common concerns of AI ethics; and ethical design -the importance of ethical training of AI practitioners.

Discussion
The results of the literature search performed in this quick review were surprising not in terms of what arose, but what did not arise. The key themes of the review can be seen in Figure 2. There were strong patterns in the themes that arose particularly in terms of design thinking research being connected to educational methods and approaches (Welsh & Dehler, 2013). From the AI ethics perspective, the greatest emphasis was placed on the transition of ethics from principles to practice (Hagendorff, 2020;Halme et al., 2021;Jobin et al., 2019;McLaren, 2003). Then, there were also the papers that represented the philosophical and pragmatic traditions implicated in design thinking processes that are linked to embodied and emotional experience and empathy (Dalsgaard, 2014;Jensen & Vistisen, 2013;Kirkman et al., 2017;Spiekermann & Winkler, 2020). Dalsgaard's approach through John Dewey's pragmatic theorization connects with the rationale of design thinking as an effective learning tool. Values are also tied into the emotional weighting of ethics (Spiekermann & Winkler, 2020). Where gender did arise within the results was in relation to equality and social innovation (Christensen et al., 2020;Dym et al., 2005).
What did not arise was research into distinctions between genders in design thinking processes and attributions or critique of creativity and creative framing of practitioners who engaged in design thinking. One could stop to wonder whether or not findings such as Zwijsen et al.'s (2011) discovery of the overuse and misuse of constructs and ethical assumptions such as the need for human autonomy and self-determinism could have introduced a form of reflectionism that incorporates critical theory and gender at its core for deciphering from where these assumptions arise and what they conceptually denote. Moreover, Jobin et al.'s findings of the five pillars of AI ethics -transparency, justice and fairness, non-maleficence, responsibility, and privacy -could be challenged in relation to various readings of these factors, as well as how and by whom they were established. Does gender matter when approaching an ethical design problem through design thinking in terms of possible readings, interpretations, and actor (designer, professional) -subject (consumer, civilian) relationships? One also needs to wonder as to whether the key to effectively developing AI systems within a design thinking process is not simply a matter of concretely (literally) translating theory to action, but rather creatively (laterally) attributing qualities to specific traits and functions.
There are several limitations to this research, one such being the reliance on Google Scholar as the main search engine -in order to gain an accurate view on the types of representation, application, and framing of design thinking and ethics in research, a larger systematic study of the alternate digital libraries (EBSCO, Science direct, ACM, IEEE, etc.) would be beneficial also to avoid biases in the Google algorithms. The addition of researchers and thorough review of all results would additionally give a more rigorous view on the state of ethics in design thinking research. This would also assist in the exclusion of gray literature from the data. In future research, it would be interesting to investigate how women have experienced their level of influence in design decisions and the roles they adopt in design thinking processes (i.e., design lead, mediator, facilitator, creator, mentor, etc.). Future research will concentrate on empirical interview studies that also gauge how professionals experience potential changes in dynamics that are contingent on gender, role in relation to creativity, customer/user, and types of technology. A technological review, and review of AI systems biases may enlighten the situation on how these dynamics affect aspects such as algorithmic logic.

Conclusion
In this article, the authors described a quick review of literature under the terms of 'design thinking', 'gender', 'ethics,' and 'AI' in order to capture a glance of research both in relation to the connection between design thinking as a method, in relation to the demographics of those implementing it (designers and gender), the design and technological space that many projects currently focus on and that draw the most ethical attention (AI systems), as well as the ethics in logic behind those systems (e.g., bias caused by lack of diversity in design teams). This mode of approach connects design thinking as a method, to a more cognitive scientific understanding of design thinking as cognition through design -meaning that the thought in and behind the processes informs the design outcome, and a multidisciplinary design team represents an assemblage of collective cognition.
Based on the main research interest and professional experience of the authors, 'gender' and 'ethics' in relation to design thinking was particularly in focus. The authors were interested in understanding how the gender of designers and developers engaged in design thinking methodological processes affect the ethical outcomes of predominant contemporary design projects (these days often linked to AI system development). For this particular study, they conducted a quick review of the literature to understand the current state of these considerations in published research. While there were common themes and focuses within the areas of design thinking as method, design thinking and ethics, and even AI ethics, such as design thinking's application as a method for education and learning in various disciplines -medicine and health care, engineering, and organizations.
While the results for a straight search in gender and ethics were extremely broad and indirectly related, what was interesting to observe was that at the time of keyword searches, results with these terms (3,3 million) in Google Scholar only ranged between 1996 and 2017. Hits directly on gender per se dropped off and were overtaken by those more focused on ethics in general. This was coupled by a rise in AI ethics related literature. The authors ponder over whether considerations for gender in the power of design have decreased or whether consideration for ethics overall automatically incorporates an understanding that gender is one variable among many that need to be accounted for. A curve in popularity of the topic may be attributed to a range of factors, none-the-least current emphasis on predominant seemingly non-gendered (uni-sex) AI ethics discourse. This may represent an algorithmic bias in the field's development, or potentially embodies a non-binary approach to AI ethics discourse. On this note, 'AI' and 'ethics' received a significant amount (1,98 million) of hits. The timeframe within the first two pages is from 2018 to 2021. This reveals that indeed, research on ethical considerations and applied ethics within the scope of technology development has shifted focus toward intelligent technological development.
'Ethics' and 'design thinking' revealed a current trend in the field of AI to consider ethics as an important part of development. The search results show that relatively little scholarly attention has been placed on ethics, design thinking, and particularly the matter of creativity -as a praxis and discourse -through the lens of gender. Despite projects aiming directly at equality and increased quality of life as seen in, for instance, by Brown & Wyatt (2010) and Lopes & Vogel (2020), ethical issues pertaining to gender representation among designers and developers seem relatively unexplored. In fact, the impact of the young [white] trendy designer/entrepreneur/developer/architect stereotype, myth, or fact in predominant software practice and business still appears to be overlooked when discussing the relationship between ethics and AI logic. Some talk has occurred regarding over-representation of this category of developer among Silicon Valley businesses (see, e.g., Rangarajan, 2018;Watson, 2016). It would be highly worth applying a cognitive design perspective toward investigating the impact of the relationship between developer/designer demographics and bias within ML and AI system logic (Storey et al., 1999).
'Gender' and 'design thinking' and 'ethics' received the lowest number of hits (2280 hits). In most selected articles, gender was a background variable rather than being the focus. This in itself, the authors of this current article feel is interesting, as it underplays the significance of gender difference and gendered logic (no matter how technological or constructed, e.g., De Lauretis, 1987) within the framework of AI systems and how they operate in societal formations and socio-cultural structures. The same applies to the category 'gender' and 'design thinking'. This seems to be a comparative basis for analyzing results, yet does not actively test how the gender of, for instance, designers and/or developers affects the ways in which: a) they engage in design thinking processeswho do practitioners empathize with stakeholders, and does gender matter?; b) how do they see themselves within the process in light of socio-cultural and technological positioning (reflective/ reflexive practice and praxis); and c) how do cognitive-affective and reflective processes impact the process outcome -can normative biases be detected, analyzed, and somehow resolved? Then, to combine the strong representation of education in the design thinking literature, with AI ethics research investigating bias, one could pose the question: who are training future designers and developers and how does this reflect in decision-making and design logic? Notes 1. https://www.designkit.org/ 2. The paper documents empirical examples in which Deweyan theoretical concepts (peepholes and inquisitive use) were translated to practice.