Meeting Student Needs for Multivariate Data Analysis: A Case Study in Teaching an Undergraduate Multivariate Data Analysis Course

ABSTRACT Modern students encounter large, messy datasets long before setting foot in our classrooms. Many of these students need to develop skills in exploratory data analysis and multivariate analysis techniques for their jobs after college, but such topics are not covered in traditional introductory statistics courses. This case study describes my experience in designing and teaching an undergraduate course on multivariate data analysis with minimal prerequisites, using real data, active learning, and other interactive activities to help students tackle the material. Multivariate topics covered include clustering and classification (among others) for exploratory data analysis and an introduction to algorithmic modeling. Supplementary materials for this article are available online.


Introduction
Modern undergraduate students are encountering and using data in ways unlike their past counterparts. Their first experience with data is typically not in statistics courses and the type of data they encounter (large, messy, complex datasets) is not usually part of introductory statistics courses, which cover basic figures and descriptive statistics and work up to hypothesis testing and confidence intervals in one or two variable settings (Gould 2010). When our students graduate and enter the workforce, many of their jobs include data management (retrieving, filtering, or cleaning) and performing basic exploratory data analysis ( Nolan and Temple Lang 2010). However, they are unlikely to see these topics in a traditional introductory statistics course. We need to adjust to our students' changing needs. There is good news on this front.
Statistics courses are undergoing significant change with rapid development in areas such as data science. We are moving away from the "tyranny of the computable" (Cobb 2007) and many instructors have shifted toward using randomizationbased procedures to introduce inference in introductory courses (Tintle et al. 2011). Data science courses and programs are being introduced around the country, and coursework in statistics is becoming more data-focused, with guidance provided by the updated American Statistical Association (ASA) Curriculum Guidelines (ASA 2014). However, we still are not meeting the needs of our undergraduate students in terms of dealing with complex datasets, visualizing data, and performing exploratory data analysis.
How can we introduce our undergraduate students to more complex datasets and basic techniques for multivariate data analysis? Clearly, adding multivariate analysis topics to introductory statistics would strain an already taxed curriculum. Some statistics educators have suggested some multivariate top-CONTACT Amy Wagaman awagaman@amherst.edu Department of Statistics, Amherst College, Amherst, MA . Supplementary materials for this article are available online. Please go to www.tandfonline.com/r/TAS. ics (simple classification or decision trees, e.g.) might replace or become topics in introductory statistics (De Veaux 2015). However, instead of adding topics to introductory statistics or otherwise modifying that course, we consider a new course, that can be taught without any prerequisites.
To meet the needs of our students, the course needs to have exploratory data analysis and cover techniques such as classification and clustering methods, taught at a level that undergraduates (including students without a previous statistics course) can handle. Many classification and clustering methods fall into an "algorithmic" modeling culture, as described by Breiman (2001). These methods tend not to focus on the underlying stochastic data model, and in particular for the classification methods, such as trees and random forests, focus on prediction. If we want to serve our undergraduate students well, we should find a place for these methods in our curriculum, because these are methods they actually see employed in practice (Breiman 2001).
We designed a course to introduce students with a minimal background to multivariate analysis techniques. The overall course goals were for students to be introduced to more complex datasets than is typical for introductory statistics (say, at least six variables to start and most ending up with datasets with 15-25 variables for their projects), and gain an applied understanding of exploratory data analysis and several multivariate analysis techniques. That meant students should be able to recognize appropriate and inappropriate applications, be able to interpret results, including justifying choices they had to make during the analysis, and communicate their results. To accomplish this, it was clear that students would need to see many examples, have time to practice on their own, and receive feedback on their communication of ideas and results.
By making the course applied, instead of theoretical, we did not need to rely on student background knowledge, and initially developed the course to be taught with no prerequisites. As our statistics program evolved, we also taught this course with an introductory statistics pre-requisite. The introductory statistics prerequisite has been a boon to the course in terms of improved student computational knowledge (improved based on what they see in our introductory course). Ideally, this course should have minimal or no prerequisites to have a broad student audience (O'Shea and Pollatsek 1997). Further discussion of the issue of pre-requisites is postponed to later sections, but we emphasize that it can be taught without any.
To develop the course, we researched teaching approaches that would help the students learn the techniques. There are great examples available from statistics education research and suggestions from many educational reports to pull from. For example, we used real data (often from the web or from a repository) throughout the course, as suggested by the updated ASA Curriculum Guidelines, the Guidelines for Assessment and Instruction in Statistics Education (GAISE) college report and others (Singer and Willett 1990;Franklin and Garfield 2006;ASA 2014). Students learned to deal with some challenges with the data-working with multiple sources, creating new variables or subsetting, etc. In particular, using the framework of Grimshaw (2015) to describe an authentic data experience, the data skills required across all assignments for the class ranged from good to best for both the breadth and depth categories. More information on this framework and how it worked for one module of the course are detailed below.
We structured the course with a mixture of lectures and interactive activities ("labs"), including class discussions and group work opportunities, to provide students with a variety of learning methods to help them with each topic we were covering. As pointed out in Snee (1993) and suggested in the Curriculum Renewal Across the First Two Years (CRAFTY) statistics report (Moore, Peck, and Rossman 2000), students learn in different ways and we need to incorporate a variety of methods into how we teach. Additionally, we used assessment methods that let us give detailed feedback to students on their work. It is known that students' learning is enhanced when students receive good feedback on the expressions of their ideas . For this particular course, we felt good feedback was essential as students wrapped their minds around the idea that in data analysis, there may be several viable ways to tackle a problem, and that justification of decisions during analysis is very important. In addition, these opportunities allowed students to practice communicating their statistical results and ideas to others, culminating in a group presentation of a course project. Providing opportunities to practice communication skills (with refinement) is a key point in the updated ASA curriculum guidelines (ASA 2014).
We incorporated active learning activities in the course (in many of the lab activities). There has been general enthusiasm about these activities, where students take an active role in their learning in the classroom. Active learning encourages students to think about underlying concepts or processes while giving them a meaningful task. Numerous studies have found active learning to be effective in statistics, though there are some conflicting accounts. For details, as well as a more substantial review and discussion on active learning approaches, see Carlson and Winquist (2011).
Another decision motivated by the course goals and student needs was to spend a solid part of the course on exploratory data analysis and data visualization. Clustering (as an example of exploratory data analysis) is a useful topic that students have intuition about and is a natural topic to include so students see ways of uncovering patterns in data. Including data visualization was motivated in part because the GAISE college report states that appropriate figure use is a learning outcome in introductory statistics courses (Franklin and Garfield 2006). Learning and practicing appropriate figure use should be relevant for any statistics course, and in fact, data visualization can be taught as an entire course on its own (Nolan and Perrett 2015). For this course, we wanted to keep the focus on students learning what figures could show, and not become mired in the details about their creation. This is in line with research about how to get students to view figures as reasoning tools rather than just "pretty" pictures . Figures can also be helpful teaching tools. For example, Valero-Mora and Ledesma (2011) discussed their experiences teaching some multivariate data analysis techniques using interactive figures.
To have the students generate figures and develop some computational skills, it was clear that statistical software was needed for the course. The use of statistical software is also recommended by GAISE (Franklin and Garfield 2006), and the development of computational skills is highlighted in the updated ASA curriculum guidelines (ASA 2014). In course design, it is important to consider what level of programming the students can handle. For example, Samsa et al. (2012) discussed student attributes as part of their course development considerations and noted that their students were not skilled programmers. We used R (R Core Development Team 2009) each time we have taught the course, with different interfaces, due to changes in the computing background of the students. A variety of other software packages could have been used; see Chance et al. (2007) for other options. When working with any software, educators should note that students may be overwhelmed by software commands. Chance et al. (2007) gave guidance for how to avoid this, and we designed the student activities with this in mind.
We have taught the course three times, while adapting to changing needs in our curriculum and advances in technology. In the following sections, we present course design details, an example module (and activities), and outcomes/course evaluation for this applied undergraduate statistics course as a case study. We begin with course design in Section 2, offering further details on class structure, software, and assessment. In Section 3, we present the clustering module from the course as an example module. This was the fifth module for a multivariate analysis technique in the course, but it could have easily been an earlier module. This example module shows how the various course design components were integrated. Finally, in Section 4, we include some evaluation of the course to assess how well we met the course goals. This case study ends with considerations for the future and advice on determining pre-requisites in Section 5.

Course Design and Structure
In this section, we describe basic course design considerationsavailable classrooms, course software, course topics, general module structure, and course assessment. The primary reflection deals with teaching the class twice with no prerequisites.

Classroom Space
Class time each week was divided into four fifty minute blocks in a lecture-style classroom with large two-person desks and a computer projection system, suitable for 36 students, with plenty of power outlets. We required students to bring their laptops for computer labs (or work in pairs). Having the desks and laptops available meant it was easy to go from an example on the board to an example on the students' laptops. Over the entire course of the semester, we used computers in the class an average of three of the four class meetings each week. The classroom setup was also nice for group work-whether that involved the computers or not.
An alternative setup was used in previous course offerings. We had access to two classrooms-a lecture classroom and a computer lab classroom on fixed days. This schedule meant careful planning was necessary to optimize the time in the computer lab, where students could try techniques on their own.
We found the flexibility in the current structure to be beneficial to planning for the class. In particular, if an activity on the computer was taking longer than planned, we could continue it the next day, rather than waiting until we were back in the computer classroom.

Software and Background Knowledge
Here, we consider teaching the course with and without an introductory statistics prerequisite.

... Then: A Reflection on the Course with No Prerequisites
The first two times we taught the course, students enrolling in it were not expected to have background statistical or linear algebra knowledge, or any programming experience. In other words, they were being introduced to these concepts without prerequisites. The first two weeks of the course were devoted to covering background material to give students a common frame of reference and be introduced to the software.
With no prerequisites required, we were aware that our students were not likely skilled programmers, and most would not have seen the R software previously. This motivated us to use the graphical interface The R Commander for R (Fox 2005). The interface for The R Commander is menu-driven and can be a comfortable starting point for students, while still allowing them to learn the script language. Eventually, all students in the course had to use R script for some of the multivariate techniques. We also used the additional free software Ggobi to aid in data visualization and exploration (Swayne et al. 2003). The first two weeks in the computer lab were devoted to learning ways of visualizing multivariate data, covering basic univariate and multivariate graphical displays, and using the software.
We covered other background material in the classroom in those two weeks of class. This included some statistics (types of variables, regression basics, ideas on variable selection and transformation), and some linear algebra (matrix notation, scaling effects, eigen decomposition, and singular value decomposition). Overall, we believe the course as taught with no pre-requisites was a useful and valuable experience for these students, and the course reached a broader audience than would have been possible with pre-requisites. For example, the first time the course was taught, only half of the 19 enrolled students would have satisfied an introductory statistics prerequisite.

... Now: The Course with an Introductory Statistics
Prerequisite In Fall 2014, major changes to the course were made. It was taught with a pre-requisite of introductory statistics, and was designed to fill the role of an elective course in our new statistics major. The change to the prerequisites meant that many students came into the class familiar with R, as we use R with RStudio (2011) as the interface in our introductory courses. Students were also already comfortable with basic statistical concepts, such as types of variables and univariate figures. We were able to capitalize on the student knowledge by going further with the material in several modules. For example, we used the lattice (Sarkar 2008) and ggplot2 (Wickham 2009) packages in R for visualization in that module, rather than the Ggobi software. We spent the first two days of class doing review activities via RMarkdown (RStudio 2014) to help students remember their R (or get a gentle introduction to it, if they were coming straight from AP statistics or had some other background in statistics). We still covered some linear algebra material as background information, but overall the first few weeks of class (introduction and visualization module) were radically different. Building on student's computing knowledge and basic statistics background made the visualization module much more enjoyable, and the students were able to tackle much more in that regard than in previous semesters.
The statistics prerequisite knowledge was also useful when topics such as hypothesis testing (in factor analysis) or the multivariate normal distribution arose. We did not have to introduce students to the basic concepts of hypothesis testing, or lay out the specifics of the univariate normal distribution before building up to the multivariate case, as they were assumed to have knowledge in these areas.

Course Topics, Principles, and Module Structure
The course topics (listed in Table 1) were covered in a module structure.
The textbook for the class readings was Everitt and Hothorn's (2011) An Introduction to Applied Multivariate Analysis with R, which was sufficient for all topics except VI: Classification, which was supplemented with other notes. In previous semesters, students used Lattin, Carroll, and Green's (2003) Analyzing Multivariate Data text. The topics included are all frequently used multivariate data analysis techniques, covered in most multivariate texts. For some topics, theoretical depth was limited due to the level of linear algebra taught in the course.
Students have natural intuition about some of these techniques  (e.g., spam filters classify emails as spam or not). Each module received 2-2.5 weeks of class time, and was followed by a homework assignment. The daily outline of a typical module is provided in Table 2. We lectured to convey basics about the topic, at least one example (in R), considerations (such as tuning parameters) and relationships to other topics. Complete code for the examples from class was provided via RMarkdown files. One day in each module was an "application day" which was devoted to examining applications of the technique in published work. This day took on several forms, with all three of the following formats used in Fall 2014. In the first format, we prepared slides showing key points from articles using the techniques. By showing real published articles and having students walk through the analysis as a group, students could show what they had learned about the topic so far, help each other, and learn to critique statistical work. In another format, we randomly divided the students into groups, each of which had a task to complete relative to the topic at hand and based on a published analysis article, with group presentations. In a final format, which the students seemed to prefer, we brought about 10 articles that included relevant methods to class, from a variety of subject areas, and students read through them to report to the class (in a short informal presentation) about how the methods were applied in the article, as well as any critiques they had about the application, etc. This could either be done in one or two class periods, and the reports could be made formal as well. Appendix B in the supplemental materials contains a list of articles used for the Clustering Module.
For every topic, students explored the techniques using their laptops and examples in R/RMarkdown via RStudio. These "lab" activities were self/group-paced with instructor availability by request to help with any issues. We carefully designed the labs using RMarkdown, with some code provided and some not, and had flexibility for students to choose various options relevant to the techniques. The activities also provided a time to explore issues not covered in lecture and for students to learn about the challenges of real data analysis (Singer and Willett 1990). For example, we spent time discussing outliers and missing values, as well as how the computer software treated those values for each technique.
For each module, we wanted to keep an emphasis on data visualization, use real data for all examples, incorporate active learning activities, and use a variety of learning methods. Labs always began with data visualization and exploratory analysis, so the students were exploring the data before applying the new techniques they were learning. Active learning was incorporated in some lectures where students helped guide the discussion, as well as in most labs, where students worked through examples while addressing various questions. Finally, students had access to a variety of learning methods-readings from a text (Everitt and Hothorn 2011) and published articles, homework and a course project, class discussions, activities with exploratory components, with opportunities to share with classmates. The datasets used for class and lab examples were a mixture of classic, tidy datasets and datasets requiring some data management (e.g., requiring subsetting, having missing data coding problems, or needing to create new variables), from a variety of disciplines including (but not limited to) biology, music, restaurant management, economics, medicine, chemistry, real estate, sociology, and anthropology. To summarize the data experiences of the students in the class based on one module, we constructed Table 3 based on Grimshaw's framework (Grimshaw 2015) and our clustering module. The framework divides the skills needed for analysis into "Data from Different Sources and Formats" and "Data Management. " In each of these groups, three categories are defined to distinguish the depth of skill necessary-"good, " "better, " and "best. " (For details on the breakdown, see Grimshaw (2015)). The coursework achieved the "better" and "best" categories for dealing with data in different courses and formats mostly during course projects which involved clustering, but the "better" category for management was obtained via the creation of new variables, requiring subsetting, and working with outliers in different examples and assignments (see the example homework assignment in Appendix C). Again, this table summarizes the experiences in a single module from the course. Clearly, we can work more on incorporating data in different sources and formats into the course.
As a final note about the data in the course, only one artificial (i.e., generated with context "added") dataset was used throughout the entire course, so students were working with real data for every technique. Simulated datasets were used in some concept illustrations, but were clearly denoted and had no context added.
(A quick note about that solitary artificial data set: we designed it so that using scatterplots would result in letters being spelled out in the display, which could be properly arranged to spell a word. This dataset was designed to reinforce the power of data visualization. For details, contact the author.)

Assessment
Designing assessment for this course was a challenge. Statistical data analyses can take many correct "paths, " and we wanted that to be clear to the students. We also wanted to give the students good feedback on the expressions of their ideas and offer several ways to convey their ideas. The breakdown of assessment tools that we decided on for the course is shown in Table 4. (Previous iterations had a take home final exam and less emphasis on the project.) Each course concept had a related homework assignment with two portions, a theoretical portion on the technique and a data analysis portion, which was an RMarkdown submission of several pages of written analysis of a dataset with supporting work and figures included which addressed a few questions we asked. The students grappled with homework with substantial decisions (which variables to include, e.g.), and they had to produce and provide appropriate supporting work for their analysis and solutions. For each assignment, we determined a basic rubric to grade on the statistical content, and while we commented on their writing and made corrections, we did not include a major writing assessment on homework. We designed the rubrics so all data analysis assignments were a similar point value, giving points for preliminary analysis, appropriate technique choice and discussion of relevant option choices (e.g., variable selection, rotation choice, or number of PCs/factors/clusters), as well as addressing any specific questions asked.
Midterms and the quiz were held in class and consisted of several problems each with multiple parts (with perhaps shared datasets). We supplied all results (figures and output), so the students were interpreting the results or picking between different options with explanations for their choices, and not applying techniques themselves. (Thus, they were not tested on their ability to code quickly.) We did not write exam questions where students were given a problem and then had to suggest an appropriate analysis. Instead, midterms focused on students demonstrating understanding of each technique, reading and interpreting the output. We relied on the course project to give better insight into student abilities to apply appropriate techniques.
The course projects were group projects where students applied at least two of the main techniques from class on a dataset (based on a dataset they found or constructed) after brainstorming some questions of interest that they would try to answer. There are many websites that make data freely available and others that list collections of those sites. One of our favorite sites to find data is the University of California at Irvine's Machine Learning Repository (Frank and Asuncion 2010), and there are many other similar sites (too numerous to list here). Readers interested in additional sites used in the course should contact the author.
For their projects, students had to submit a proposed set of questions and receive approval before delving into their data analysis. They also had to introduce the class to a new statistical topic relevant to their analysis. This topic was tailored to each group with the difficulty level tuned to that group. For example, one group learned to code a bootstrap analysis in a new setting, while another group learned about support vector machines and implemented one. The groups made class presentations to share their findings with the class during the last week of classes. For their final submission, groups summarized their results in a 10-12 page paper, with a technical Appendix in RMarkdown.
This approach to the projects was a change from individual projects we had used in previous semesters. Overall, we felt the group projects were better quality than the individual projects from previous semesters. Course projects are a significant time investment on the part of the instructor-whether they are group or individual, but are very rewarding in our experience.
To demonstrate how all these course design aspects came together in the course, we present the clustering module in the next section.

An Example Module On Clustering
The clustering module was the fifth module in the course. At this point in the course, students were also brainstorming for their projects, and clustering gives them many possible techniques to explore that we do not cover in class. Briefly, we present the days in the module stating what happened in class and give some additional comments. We also describe the clustering related assessments used in the course. Many materials described here are provided in the supplementary material. Additional materials may be provided by request.

... Prep
What: Students were assigned to read the clustering chapter from the text with sections spread out over the course time on the module based on class progression.
Comments: If assigned, reading was usually 1-3 sections per day from the text. There were days with no reading assigned when the students were focused on an activity or application (where they might have to read an article).

... Introduction to Clustering: Lecture 
What: Introduction and motivation of the technique using interactive examples and data from our research, followed by simple examples for the students to see. The basics of linkage were discussed for hierarchical solutions.
Comments: Students worked in pairs to cluster 18 animals into two groups. Their results were shared with the class. This brought about a wonderful discussion of dissimilarities due to the wide variety in the solutions.

... More on Clustering: Lecture 
What: Continued developing understanding of hierarchical clustering and linkage with simple examples. Advanced linkages were discussed, as well as choice of dissimilarity. We began discussion of K-means.
Comments: The students reviewed a single linkage example and then tried a complete linkage example on their own, before the class moved on to K-means.

... Clustering in R: Lab 
What: Finish discussion on K-means and move into lab activity where students explored clustering solutions performed with Rcode.
Comments: We designed this lab to familiarize the students with performing clustering in R based on all the code and examples in their text. Much of the R code is provided, though students have to make adjustments or code up their own solutions as they go-but example code is always close by. (See Supplemental Material: Appendix A for this lab activity.)

... Clustering in R: Lab 
What: Students continued working on the introductory lab activity. When completed, they had access to a second lab activity where clustering was being used to look for groups of crabs and decide if those groups matched sex or species of the crabs.
Comments: The second lab was designed to show students that the "natural groups" clustering finds in data may not match any categorical variables in the dataset. The class had a discussion about stripe plots to be sure that everyone could understand them using those plots presented in Lab 1. Not all students made it through Lab 2, so they were encouraged to complete this on their own and check their responses versus our posted comments.

... Clustering Challenges: Lecture 
What: An RMarkdown (compiled) file was provided that showed various challenges in clustering. For example, a situation where K-means fails to find the "correct" three clusters expected based on two starting points falling in one large cluster in a simple two-dimensional setting was shown. The class walked through the different examples with guidance.
Comments: This series of examples was very useful to show students that different clustering techniques work well in different situations, but that there may be issues to be aware of. The K-means example in particular shows the importance of starting points of algorithms, and the students gained understanding about why it is necessary to randomize those starting points and consider multiple solutions.

... Application Day: Lecture 
What: Application day. Eleven (semi-short) articles were prepared for class discussion. The articles were applications of clustering in various disciplines and contexts. Students read/skimmed the articles in pairs or triples (eight groups) and presented some findings about how clustering was used in the articles to the class in informal presentations. (See Supplemental Material: Appendix B for article references and class handout.) Comments: Application days were discussed in Section 2.3 in some detail. The students seemed to prefer this setup-groups worked with an article and presented findings briefly to the class. Many variants on this activity are possible. Our idea was to bring current (year 2000+) examples of applications to the students so they could really appreciate the technique they were learning.

Clustering Assessment
As a main course topic, clustering had several related assessment materials-a homework and exam question (second midterm), and could also have been used for the course projects.
The clustering homework (as with all the homeworks) had two associated components. First, there was a theoretical component where the students demonstrated their understanding of linkage and answered some questions. Then, they had an applied data analysis component. For this component, students had to do some preliminary data exploration, perform an appropriate clustering analysis, and explain what their analysis revealed. The clustering homework and the exam question on clustering are shown in the supplemental materials-Appendices C and D.
Clustering played a major role in several of the course projects-four of the nine groups planned to use it as one of their two required techniques in their original project proposals. Two groups used different aspects of cluster validation as their expository topics to share methods/ideas with the class that were not covered in the text. One group used clustering to identify natural groups of observations based on one subset of variables and then tried to follow up with classification techniques on a different subset of variables to see if the clusters could be recovered with different data (though this did not work unfortunately). With the course projects and presentations, students got to see firsthand how different techniques worked well for some data sets and associated questions and not for others.

Outcomes and Course Evaluation
Recall that the goals we had outlined for the course were for students to be introduced to complex datasets (with more variables than usually presented in an introductory course), and gain an applied understanding of exploratory data analysis and several multivariate analysis techniques. This section provides some assessment of the course and whether the goals were achieved based on student course evaluations and short survey replies. Course evaluations were completed before the last day of classes in each semester the course was taught. The student evaluation included questions designed to generate feedback on the course and various course aspects, such as the application days, lab activities, and software. Over the three semesters the course was taught, the evaluation completion rate was around 80%. The short survey was sent to students who took the course the first time it was offered to obtain feedback nearly two years after that course offering (18 of 19 students in the first cohort were sent the survey, and six replied).

... Class and Lab Activities
The class and lab activities were a crucial course aspect where students practiced techniques, worked with other students, and learned to code in R. Over the course of the semester, the activities got the students thinking about how data are stored and shared, the importance of appropriate visuals in data analysis, and how vital it is to understand the problem at hand, as well as the techniques being used. Most activities had an active-learning style and the students used RMarkdown via RStudio (RStudio 2011) for their analyses. Student comments about the activities and software were overwhelmingly positive. Students enjoyed the activities, felt they were vital to their learning process about the techniques, and felt the software was not a deterrent to learning. (It was recommended that we create a master RMarkdown file of coded examples for reference-a good project for the future!). Prior to using R with RMarkdown, a few students had commented that they would like to see other software used, such as Excel or Stata, because they were already familiar with those programs from other courses. However, we think the benefits from R (free, very nice interface in RStudio, access to packages, and reproducibility in RMarkdown) will continue to make it a valuable software choice.

... Application Days
Another major feature of the course was the application day(s) in each module, and we asked the students what they thought of these days on their course evaluations. Each semester the course has been taught, these responses have been very positive. Recall that there were multiple formats where the students would be exposed to actual applications of the techniques in journal articles. A few students commented they did not prefer the format most often chosen by the class (majority vote), but still felt it had been useful to see the applications. The informal presentation aspect was new to the course this past fall, and several students wanted to formalize that, which likely would have meant spending at least two days on this component each module. With fewer modules, such an approach might work well, and it would give students more time to delve into the articles and practice their communication skills.

... Course Accessibility
When the course is taught without a prerequisite, perhaps the most important course aspect to consider is whether or not the material was accessible to all the students, and if they felt that they had gained good understanding of the techniques we had studied, no matter their background. To address these questions, we can examine responses from the first two cohorts of students in the class, as well as the survey results. In particular, views on the level of theory in the course were requested on course evaluations. Many students were quite content with the applied nature of the course, and felt they got a lot out of the course. They voiced this in their final course evaluations as shown in the example replies that follow. One student stated that "It's nice to have another math class that is more hands on and has more application rather than theory based courses. " Another stated: "It was good to take an applied class that I felt like had real life applications. I am very glad I took this class; it opened my eyes to ways I could use math in my future without just studying theory. " (The course was listed as a Math course for two offerings). A few students from the first two semesters commented that they would have liked to have seen more theory, and that requiring additional background knowledge might be necessary to do that. From the short survey, one student stated that he felt a deeper linear algebra background would have benefited him while learning about the techniques. Even in the third course offering, with the introductory statistics prerequisite, two of the 21 enrolled students commented that they felt students who had linear algebra as well as intro stats would likely have gotten more out of the course than those without the linear algebra. These comments suggest that there is definitely a need to balance the desire for theory with student background, and that this is trickier when the course is taught without prerequisites. For some recent discussion about "flattening" prerequisites, see Cobb (2015).

What Did Students Do with the Knowledge They
Gained in the Course?
One of our main motivations for sending the short survey out to course alumni from the first course offering was to find out how they had used the knowledge they gained in the course. We had limited feedback on this from course evaluations at the end of the semester. In this section, we discuss relevant feedback from course evaluations from all three course offerings and short survey replies from the first cohort. First, even at the end of the same semester the course was taught, several students had positive things to say about the course and how it had impacted their lives. From one student, we learned that "Seeing this course on my transcript has been a conversation starter during interviews" and this has actually happened several times since. One student listed us as a summer internship reference and we were able to state to the prospective employer that she had used R to perform a principal components analysis (PCA). Some seniors writing theses concurrently with the course used these techniques in their thesis work, and taking the course helped them to understand what they had been reading about and/or doing themselves, as well as seeing the techniques in a range of applications. From their short survey replies, two students reported greatly appreciating learning about data visualization and learning to work with large, real datasets for their current work. Another student reported seeing PCA applied often in her current field. Knowledge of R and some skill in coding has proven beneficial for several students (several have entered statistics graduate programs, and many reported this knowledge was helpful for job interviews and future work).
Finally, the course was important to several students due to its applied nature and course content. Students commented that the practical application was a nice contrast to theoretical courses (in our mathematics curriculum). Based on their evaluation comments and survey responses, students used the knowledge gained in the course in a variety of settings (other classes, projects, theses, internships, graduate school, and jobs), and appreciated their undergraduate applied introduction to the topics.

Discussion
Briefly, we discuss some considerations for the course and potential research areas for the future. First, there are some considerations related to the teaching approaches used in the course. Each time we have taught the course, the students were amenable to group work, but we did not use this to its full advantages. We hope to incorporate more of the suggestions from Garfield's work on active learning with cooperative groups in future courses (Garfield 1993). Students might also benefit from more interaction with figures, though we feel the course in its third offering did much more on visualization than previous offerings. While some data skills were covered in the course, additional data wrangling skills would also be useful to incorporate before projects begin, in particular to assist students dealing with data sets from multiple sources requiring merging.
Second, there is the major consideration of teaching the course with or without a pre-requisite. We would like to emphasize that the course can be successful with or without pre-requisites. The relevant prerequisites to consider are introductory statistics and linear algebra. The linear algebra prerequisite would lock many students out of the course, so we will focus discussion here on the introductory statistics course as a prerequisite. Our best advice is to consider the audience for the course. If the audience is statistics students (especially in a statistics program), it might make sense to offer the course as an elective after a prerequisite of introductory statistics. If the audience is broader, then, in the spirit of Cobb (2015), perhaps offering the course with no prerequisite is a better option. Another important issue to bear in mind is how much computation the students will be responsible for. The course can be taught with less emphasis on the computation, with more provided examples. Finally, if teaching the course with no prerequisites and trying to cover as many methods as described herein seems daunting, cover fewer topics! We can envision a version of the course with no prerequisites that focuses on visualization, clustering, and classification, with a computational level that can be adjusted (or uses a slow pace to build up skills) that would still be beneficial for the students. In short, the core issue of the prerequisite really depends on the student audience of the course.
A few areas for future pedagogical research are suggested by this case study. The use of real analyses from published articles as examples to help students learn techniques bears further investigation. Additionally, investigating active learning methods in statistics education in courses beyond introductory statistics will likely be enlightening, as we strive to help modern students learn the statistical concepts that they need to know in the real world.