On the usefulness of map-based dashboards for decision making

—This paper presents an investigation of the usefulness of map-based dashboards and how these can be built to support science-based spatial planning. It includes a comparison of 21 dashboards in addition to an in-depth case study evaluating a dashboard using an insight-based methodology. The results suggest that useful dashboards have pre-deﬁned clear goals and are customized to their targeted audience. Tailored summary charts and data overviews guide users through the data and encourage them to go from data to action. Furthermore, a list of guidelines, e.g. work iteratively and deﬁne audience and goals, to support analysts creating dashboards for decision-making is provided.


INTRODUCTION
P LANNING for the future can be an extraordinary challenge [1]. Spatial planning requires various factors to be taken into account such as water management, provision of food, risk reduction from natural habitat, and socioeconomic structures [1], [2]. To this end, scientific findings about the impacts on nature as well as knowledge about local regulations and culture can be useful [3]. Since most decision-makers are not experts in all relevant areas, effective communication techniques such as visual representations and graphics are crucial to bridge the gap between science and decision-making [4], [5]. To this end, mapbased dashboards have been developed to support decisionmaking. [6], [7], [8]. By combining interactive maps with key performance indicators (KPIs) it is possible to visualize research findings in user-friendly formats beneficial to spatial planners [1].
Developing and maintaining dashboards requires a lot of resources [5], therefore it is important to evaluate the efficiency and usefulness of these tools. The purpose of the present study is to investigate if interactive map-based dashboards are successful in supporting decision-makers in complex environmental reasoning leading to better, sciencebased, spatial planning.
There are three main contributions of this paper. Firstly, a competitive analysis [9] of dashboards built for communicating research for spatial planners was conducted. This created a quantitative assessment of interaction operators [10] and visual-variables [11] as well as qualitative data about how developers worked with prototyping, customiza- tion and evaluation. Secondly, how users interact with a dashboard was more in depth analysed using the insightbased methodology proposed by North [12] in a case study of a specific dashboard (see Section 3.2). This aims at filling the gap about how users gain insights from visual representations [1], [13]. Thirdly, a set of guidelines based on the knowledge and lessons learned from interviewing developers, cartographers, and visualization experts is presented. This suffices the need of supporting developers to create their own dashboards [14].

BACKGROUND
Dashboards are visual displays containing the most important results of a specific dataset [14]. The objectives are to create awareness and facilitate actionable understanding for the sake of helping users to make well-informed decisions [3], [4], [15]. Dashboards can communicate an overview of the data, present the bigger picture of a complex situation, and thus trigger a discussion about strategy and preventative action [4], [5], [16].
Understanding the usability of map-based dashboards includes both a cartographic perspective as well as evaluation methods from Human computer interaction (HCI) and the visualization field [17]. This type of research, bridging the gap between visualization research and applied cases, is important for making sure that visualization prototypes reach a state that can be used by the end users [18].
Interactive maps hold potential for improving spatial decision-making [19]. These maps allow decision-makers to generate new, previously serendipitous insight by enabling them to consider both spatial but also environmental, economic, and social aspects. The maps help transfer the knowledge from the mapmaker to the map user [20]. However, more research is needed to understand what interactions are needed for which decisions and decision-makers [19].
Furthermore, despite the increasing use of evaluations of tool functionality and usability [13], [21], extensive user evaluation of visual representations built for domainspecific work are not very common [22]. In the cases where evaluations are done, they are often stated without any explanation about how these insights were achieved and reported too informally to use for cross-comparison with other visual representations [1], [13]. Moreover, the fact that tools for decision-making are used for both various purposes and by various types of users, makes them harder to evaluate in a quantified manner [23]. Isenberg et al. [13] mentioned that they often found reviewers rejecting qualitative approaches based on the preconceptions that such studies were nonrigorous and not valid. However, a qualitative approach has proven to be critical when evaluating visual analytic tools since these methods investigate real problems of real users which is of importance when building tools for real applications [13], [23].
For evaluating higher level issues such as exploration, insights, and decision-making, a scenario of evaluating Visual Data Analysis and Reasoning (VDAR) [23] has been presented. This scenario includes the insight-based methodology used for the case study in the present study. The aim of VDAR is to assess a visualization tool's ability to support visual analysis and reasoning about data. The outcomes from such an evaluation can be quantifiable metrics regarding insight as well as subjective opinions about quality and data analysis experience. Only 2.9% of the evaluations were mapped to VDAR [13]. However, conducting more evaluations using VDAR would be beneficial, both in general, but also within the context of land-use policy visualization [1], [13]. It could investigate how the visual representations support hypothesis generation and further how it converts scientific findings to action.

METHODS
The present study has consisted of both an analysis of twenty-one dashboards as well as an in-depth evaluation of a specific dashboard using an insight-based methodology.

Comparing and analyzing twenty-one dashboards
Twenty-one interactive dashboards were analyzed and compared through interviews with developers in addition to a competitive analysis [9]. Competitive analyses are used for comparing similar systems and are commonly used in the prototype phase of a new product [24], [25]. The analysis aimed to answer the following list of questions. 1) What techniques and features are used today? 2) What are the outcomes of the tools? 3) How are projects working with testing and user evaluation?
The competitive analysis helped to answer question (1), while the qualitative interviews were more focused on questions (2) and (3).

Dashboard sample
The requirements for the selected viewers were that they had to be map-based and used for communicating environmental topics in spatial data. Additionally, dashboards were chosen to cover different scales (10 global, 7 regional, and 4 local), for various users (6 scientists, 18 policy-makers or spatial planners, and 9 interested public).

Coding features
All dashboards were evaluated based on four code groups (see Figure 1), assessing; (1) Visual variables -how the data was symbolized based on cartographic principles [25], (2) Interaction operators -how users could interact with the map [10], (3) Communication functionalities -which were features such as tutorials or summary charts aimed to facilitate decision-making and help users understand complex spatial phenomenons, and finally (4) Usabilitywhich was tested using three modified statements from the SUS questionnaire [26] and subjectively evaluated usability.
Code groupings 1-3 were all assessed based on the presence or absence of each operator, variable, or functionality. A single coder was used since the coding was a simple assessment of whether the feature was implemented or not. For code group 4, which was a subjective measurement assessed quantitatively, three coders rated the statements independently and then triangulated the results. The statements were rated on a five-point scale from 1, strongly disagree, to 5, strongly agree.

Open-ended interviews with developers
Open-ended interviews were conducted with developers or project leads to all the selected dashboards. The semistructured interviews began with questions regarding the targeted user groups and aimed goals of the viewers. The interviews continued with a conversation about the outcomes of the viewers as well as some lessons learned during development. The respondent was also asked to rate the usability statements from code group 4. This enabled a comparison between experts' and the developers' analysis of the usability.

Case study -Insight-based evaluation of the PRO Agua viewer
The PRO Agua (Proyecto Resiliencia y Ordenamiento Territorial del Agua translated Water Resilience and Land Management project) is a collaborative effort conducted by local projects in South America and the Natural Capital Project at Stanford University. The project aims to demonstrate the benefits of ecosystem services and comprehensive watershed management for the health and well-being of the growing population in the Amazon. It aspires to increase understanding of the proper use of the area and its resources to help sustainable development for a better future.
The PRO Agua viewer [27] was developed to share and communicate the project's complex data and results in a user-friendly format. Using the dashboard the project's findings were displayed through a series of interactive maps, graphs, and photographs. The viewer was mainly built to support decision-makers, both on a local-and national level.

Insight-based evaluation protocol
An insight-based methodology [12] was used for evaluating the PRO Agua viewer. This VDAR methodology is a way of observing what insights users gain on their own instead of instructing users on what insights they should get. For this context, insights can be described as observations or discoveries found by the users [28]. The insights gained from using a dashboard are therefore a measurement of how useful a dashboard is.
The insight-based methodology has been used for applications of clinical data [29], health and well being [30], and bioinformatics [28], [31]. Common for these studies is their focus on open-ended protocol, quantitative insight analysis, and the domain knowledge relevance of the chosen participants [12].
In the present study, the user tests were conducted as short (30 minutes) recorded interviews. At the beginning of each interview, the participants were asked to answer some background questions about their occupation, previous time spent with the viewer, knowledge about the PRO Agua project, and knowledge about data analysis and visual representations. The interviews were then divided into two parts where the first part consisted of an exploratory session, and the second part of the usability questions and an open discussion about the tool.
The exploratory part of the interview started with a short 2-3 minutes tutorial of the viewer to present which part of the viewer the participant could use. During 10 minutes they were then requested to act as a local decision-maker and use the viewer to understand the topics of flooding and dengue fever and how those relate to land use. The participants shared their screens during the entire session. A think-aloud protocol was used and the participant was encouraged to talk about everything. It could be both hunches or real validated reasoning. During the 10 minutes, the participant could ask questions about the functionalities but not about the data itself.
When the 10 minutes were up the exploratory session ended. The participant was then asked to rate the same usability statements from code group 4 (see Figure 1). Finally, each interview ended with an open discussion where the participant could provide feedback on the viewer and the research.

Participants
In total sixteen people were interviewed and twelve met the inclusion criteria of no previous usage of the viewer. Ten of these worked in direct domain-relevant fields in the Amazon area and two were master students in natural sciences.

Coding user insights
The user insights were coded with the characteristics presented below. This allowed further analysis and distinguishing between different types of insight. The characteristics used were based on the original work by North [12] and previous similar case studies [28], [30], [31].
• Observation/fact -The finding or observation made by the participant. It should be connected to the existing domain knowledge and go beyond simple data statements to relevant domain impact.
• Domain Value -The value of the insight. This is defined by the complexity and depth of the finding and was very context-based. The scale was coded on a five-point scale from 1, an obvious fact in the data, to 5, a deep understanding of underlying relations and which integrates prior knowledge about the area or topic.
• Hypothesis -If the insight was leading the participant to identify a new relevant hypothesis or not.
• Directed vs Unexpected -Directed insights were those expected by the researchers and developers of the tool while unexpected were those that were not considered in the design but emerged from using the tool.
• Correctness -Level of correctness. This was also coded on a five-point scale from 1, fuzzy not entirely correct insight, to 5, precise and correct insight. The insights were coded independently by two coders. For Domain value and Correctness, the mean of the two coders results were used. For Hypothesis and Directed vs Unexpected, inconsistencies between the coders were discussed until consensus was reached.

RESULTS AND ANALYSIS
The results from both expert ratings of usefulness as well as the insight-analysis suggest that map-based dashboards can be useful for aiding complex spatial reasoning about a dataset. The result presents the technical features used, the importance of having defined goals, and how user evaluation can foster the development of most effective dashboards.

Most used features
The evaluation of the twenty-one dashboards shows that dashboards can be implemented in multiple ways. Figure 2 present the most commonly used features. Fig. 2: Import, overlay, pan, and zoom are implemented in all dashboards. Hue and lightness are the most common visual encodings and scenarios and summary charts the most used communication techniques.
All dashboards had interaction operators for importing, overlaying, panning, and, zooming which could therefore be called "essential" for a map-based dashboard. Search, retrieve, and re-symbolize were also commonly used (14,13, and 11 of 21). Re-symbolization was always done by allowing for change of opacity. Hue was used as a visual variable for representing one or more data sets in all dashboards. Lightness was also commonly used (14 of 21).
Thirteen dashboards included functionalities to visualize future scenarios. However, not all of these extensively described what these scenarios were based on. The qualitative data from the PRO Agua viewer evaluation showed that many users asked for a clear definition of the given scenarios. One way of describing the scenarios could be through the use of storytelling elements which are reported helpful for interpreting the data [32].

Useful dashboards include various sets of features
Using the usability ratings (see Section 3.1.2), the four best dashboards was extracted (Resource Watch [8], Ocean Reports [7], Resilience and Preparedness [33], and the PRO Agua viewer [27]). Figure 3 presents the features used by the dashboards with the highest usability ratings.
The most useful dashboards have included various features but common interaction operators are saving and exporting. The PRO Agua viewer was given a high usability score even though it did not include any of these features, however, according to the user interviews these features were sought after.
Summary charts and tables were also implemented in all of the top 4 dashboards, some also included functionalities for cost/saving potential, adaptation strategies, and key performance indicators. This suggests that in order to make a dashboard truly useful, concrete examples of potential benefits and strategies need to be included. This finding was also supported by the qualitative data from the interviews with users and experts.

Dashboard elements should be adapted to the user
In conclusion, when choosing visual variables, interaction operators, and functionalities for a dashboard, previous studies, and the qualitative data suggest it is important to consider the targeted users early on [5], [14]. A nontechnical decision-maker with a low data literacy might not be able to understand a complex map with multiple ways of re-expressing. In this case, clear legends, some context information, and a simple interface will be needed. For users with higher data literacy, more complex features that allow for further analysis can be presented Different combinations of all features and functionalities can be used in order to create a useful dashboard (see Figure  3). That being said, there is no need in using all of them. There is such a thing as too much functionality. From the dashboards with a functionality rating of 4 or higher only 50% also had a high (4 or more) ease-of-use or confidence rating.

Customized dashboards with clearly defined goals are more likely to be useful
When asked about the outcomes of dashboards all developers stated that the dashboards were key elements in communicating their work. However, even though developers described their tools as useful, they also argued that it was difficult to distinguish the isolated viewers' outcomes from outcomes facilitated by the data itself.
Furthermore, the original goals of the dashboards were not always defined. According to the qualitative data, the goal of some of the dashboards was only broadly defined e.g. "be a data discovery tool". The results indicate that undefined goals might be problematic as this makes the discussion about the usefulness of the tool unclear. It is hard to judge the usefulness of a dashboard when it is not clear what it should be used for.

Goal definition vs usefulness rating
During the competitive analysis, the experts rated the usefulness of each dashboard. As help, they had some keywords of how the developers had described the dashboard. Ten dashboards were rated highly useful (usefulness rating 4.0-5.0), six were rated less useful (usefulness rating 3.5-1.7).
By mapping the rated usefulness with how defined the goal was, a pattern seemed to emerge (see Figure 4). Dashboards with a more defined goal were in general rated more useful (r = 0.74).

Customization vs usefulness rating
The level of customization for the dashboards was also analyzed. This investigated how the dashboards were built. Most dashboards (12 of 17) were highly customized with a lot of special features created for the specific data set while the rest were more based on ready-to-use templates, e.g. from Esri and ArcGIS. The level of customization was also mapped against the usefulness rating. Like the goal definition, the data suggest that a more customized dashboard are more likely to be useful (see Figure 4).
Noteworthy is that just because a dashboard is customized does not automatically make it useful. As shown in Figure 4 the data also presents examples when the customization is not helping the dashboard.

Evaluating if the goal is fulfilled
The goal of the PRO Agua viewer was to provide complex domain-relevant insights to decision-makers working with the topic of water resilience and land management in the Amazon. The dashboard was rated as useful by the experts (with a rating of 4 out of 5) and the results of the insightbased evaluation showed that users were able to gain valuable insights. This further makes the case that dashboards can be proven useful when they have defined goals that, based on the outcomes of evaluation, are fulfilled.
The analysis of the insights users gained suggested that the goal of the PRO Agua viewer was fulfilled. 25% of the insights had a domain value of 4 or higher which indicates that the users were able to gain complex knowledge about the data. Furthermore, the average correctness was 3.9 (see Figure 5) and only 4% of the insights were given a correctness value less than 3. These results suggest that the dashboard was not misleading in its presentation of the data. The insights gained were also grouped based on the participants' prior knowledge about data analysis and the PRO Agua Project (see Figure 5). The results suggest that there is a difference in the domain values of the insights. Respondents with more knowledge about the project or about data analysis could gain more complex insights. However, the correctness of the insights remains more or less constant. This means that the dashboard can be useful for various types of user groups. Nonetheless, people with less knowledge about the data and project might still need further explanation to understand the most complex parts of the data presented.

Outcomes of dashboards can be various
From the qualitative data, the following overall outcomes have been extracted and synthesized. These don't cover every possible outcome of the dashboards but could serve as a starting point for a discussion when creating a new dashboard.
• Monitoring and creating data transparency -Dashboards can help to monitor the status of a variety of topics such as forests, mangroves, coral reefs, etc. Reported outcomes are ranging from finding illegal logging roads in South America to being able to assess the damage of a coral reef after a ship accident. Furthermore, the dashboards can provide data transparency and were used by non-profit organizations and journalists.
• Being a key tool for discussion -Dashboards can be key pieces in facilitating discussion. Web-based dashboards built for non-scientist allow basically anyone to access them and inform themselves with the data provided. Examples are dashboards used in discussions regarding fishing areas vs turbines, agreeing on suitable logging areas as well as discussion about the relevance of the data itself.
• Creating research exposure and help to secure funding -The final group of outcomes is the exposure these dashboards give to the associated research project. Since dashboards potentially are more easily accessible than the report itself they can be a link to the interested public, other scientists, or potential funders. Dashboards thereby can help the scientist get feedback on their work as well as being used to allocate funding to new projects.

The importance of user evaluation
Evaluating dashboards can be difficult and not all developers prioritize doing it (10 out of 21).
The PRO Agua case study provided, in addition to the insight analysis, an overview of users' opinions of the tool by also asking them to rate the usability statements (see Figure 1). When comparing the results of insight analysis with the self-reported ratings the data shows that users tended to be too positive when rating the statements. They often overrated their capacity of understanding the tool. Even those respondents that could not gain more than 3 insights rated that the viewer was easy to use, had all the functionalities they wanted, and that they felt confident while using it. This demonstrates that just letting users answer statements about usability will not present the full picture of how well the tool is actually working. To understand how the users actually use the tools, a qualitative user evaluation is also important (see discussion in Section 2).

Evaluation leads to more useful tools
The data suggest that conducting some type of user evaluation, regardless of the method, creates a better understanding of how users perceive the dashboard and in extension helps developers create a more useful tool. Dashboards that were evaluated tend to both have better usability ratings, and more agreements between expert ratings and developers ratings. Developers tend to overrate the usability ratings, especially when they have not conducted any evaluation (see Table 1). Time and lack of resources were the main explanations for why developers did not conduct evaluations. The developers indicated that they would like to have done more rigorous testing. However, creating the dashboards required a lot of time leaving no resources nor funding for evaluation.

DISCUSSION
The results of the present study have highlighted some aspects relevant to the usefulness of the tools and also discovered some unmet user needs.

The value of customization
The results of the comparison of dashboards show that there are no specific visual encoding or interaction operators that automatically make a dashboard useful. What instead is found is that the level of usefulness is dependent on the combinations of these techniques and how they relate to the goal of the dashboard. In other words, how customized the dashboard was to fit the needs of its users. In the present study this is proved by comparing the definition of goals, and level of customization with the usefulness (see Figure  4), but also by studying the communication functionalities that are used by the most useful dashboards (see Figure 3). The value of customizing the dashboards to the users' needs has also been shown in previous studies [14], [22].
Summary charts and presenting adaptation plans can be seen as a way of customizing the data to be of more help to the decision-makers [5]. Instead of just placing the data directly on the map, those elements highlight what is important and thereby guide the user through the data.

Unmet user needs -accessibility
The outcome from the competitive analysis and interview found a few points for future development. The main focus has been on the accessibility of the tools. For several dashboards, more than half of the users tried to access the viewer from a mobile device. However, only three of the dashboards from the competitive analyses were adjusted for mobile devices. When more users access the dashboards from their phones appropriate adjustments have to be done to the interface [34].
Moreover, interactions like exporting, saving, and editing are features that were sought after by users but not often implemented. Exporting the maps was reported as useful for sharing the result with others. It can also be useful in cases where the internet connection is not reliable.

Considering the evaluation methods
The benefits of evaluating dashboards have been shown in previous papers about dashboards [5], [35]. Therefore, it is remarkable how little time most dashboards projects actually spend on user tests, according to the findings in the present study. Apart from dashboards developed by specific companies, the projects seem to mainly conduct peer feedback sessions and maybe a short questionnaire.
There are various ways of evaluating a visualization and a dashboard (see example in [13] and [23]). In the present study, benchmark testing in the format of a competitive analysis was conducted. This method was used to understand which features were included in the different dashboards. For an even more rigorous evaluation, the different variables could also have been rated on how well they were executed. However, that would have required experts trained in evaluating those features.
Furthermore, by using the insight-based methodology insights gained by the users could be characterized and evaluated. This provided an indication of how the users interpreted the data. However, the rating of the different characteristics was subjective, and different coders could have presented a different result. This created difficulties for cross-comparison, but the results were still considered helpful for finding improvements to the specific dashboard.

GUIDELINES FOR CREATING A DASHBOARD
Several aspects of designing, implementing, and evaluating useful dashboards has been identified (see Figure 6). Fig. 6: Guidelines for creating a map-based dashboard.

Plan and prototype before coding
Findings from the present study as well as previous studies [5], [35] have emphasized the importance of thoroughly defining goals and targeted user groups of the dashboard before any sketch work can start. Discussions with a crossinterplanetary team of developers, stakeholders, scientists, and end-users can be used to specify the users' needs which then can be transformed into technical features.
Furthermore, developing a useful dashboard takes time [36] so it is important to allocate enough time and resources for development and user evaluation. Customized dashboards are suggested to have a better chance of being useful but come with the cost of longer development time. Dashboards using templates can also be useful but it is important to always have the goal in mind and only highlight the interaction that helps the user reach it.

Guide the user through the data
The next step is to decide what parts of the data that should be displayed. Choose metrics and key performance indicators that align with the aimed outcomes of the dashboard and the users' needs [5], [14]. If the dashboard is built to support concrete decisions it could be beneficial to include some strategies and cost/saving potential.
To facilitate understanding and create engaging dashboards it is common to start with an overview to present the big picture, e.g. with a summary chart or graph. Allowing for drill-down options like zoom, filtering, or retrieving can then be added to allow for a deeper analysis of a specific area [14], [15].

Conduct evaluations throughout the process
Including user evaluations iteratively and collaborating in cross-functional teams has proven to help create useful tailored visualization [5], [15], [35]. User evaluation with end-users is also likely to motivate further studies increase the chances of user action [2], [3].

CONCLUSION
The present study aimed to investigate the usefulness of map-based dashboards for spatial planning. A competitive analysis was conducted in addition to an in-depth case study that evaluated the insights gained from using a specific tool. The findings were then synthesized to a set of guidelines and the following conclusions.
Dashboards provide insights -Map-based dashboards can be useful and help decision-makers gain complex domain-valuable insights. Summary charts and relevant concrete adaptation strategies can guide the user through the data and help them go from insights to action.
Customization and clearly defined goals are key for a useful dashboard -Defining goals and having a comprehensive understanding of the users, supports the creation of a more useful dashboard. By customizing the dashboards based on the users' needs, the key messages of the data sets are more easily understood and communicated.
User evaluation is important -The present study emphasizes the benefits of rigorous user evaluation. The results suggest that evaluated dashboards are used more used and enhance users' confidence. User testing provides knowledge and insights into how a dashboard can be used for solving real problems.
Finally, to further understand how the design of different features and interactions affect the usefulness of a dashboard, future studies implementing control vs treatment of a specific dataset would be of interest. This could provide more direct guidance on how to include the different features presented in this paper. Also, we encourage analyzing the long-term effects of using a dashboard for spatial planning, adding further insights regarding user needs and requirements. This would be even more similar to real use cases where map-based dashboards are used by decision-makers to support spatial planning.