Program Assessment for an Undergraduate Statistics Major

Program assessment is used by institutions and/or departments to prompt conversations about the status of student learning and make informed decisions about educational programs. It is also typically required by accreditation agencies, such as the Southern Association of Colleges and Schools (SACS) or the Western Association of Schools & Colleges (WASC). The cyclic assessment process includes four steps: establishing student learning outcomes, deciding on assessment methods, collecting and analyzing data, and reflecting on the results. The theory behind the choice of assessment methods and the use of rubrics in assessment is discussed. A description of the experiences of a Department of Statistics at a large research university during their process of developing an assessment plan for the undergraduate statistics major is provided. The article concludes with the lessons learned by the department as they completed the assessment development process. Supplementary materials for this article are available online. [Received December 2014. Revised July 2015]


INTRODUCTION
Program assessment is the process of collecting data to make informed decisions and changes to education programs (Peck and Chance 2007). Unlike classroom assessments, which seek to evaluate students, courses, or professors, program assessment is the systematic process that provides insight into the overall student learning experience. Through the process departments articulate the student learning that is valued and consider the match between what is valued and what the program actually delivers (Peck and Chance 2007). Accreditation agencies, such as the Southern Association of Colleges and Schools (SACS) or the Western Association of Schools & Colleges (WASC), expect all academic departments at accredited institutions to establish "student learning outcomes, develop plans for assessing their learning outcomes, and use the results for the improvement of student learning" (Lindholm 2009, p. 9) as part of the accreditation process. While program assessment is often done because it is required by accreditation agencies, it also provides useful insight into the program as a whole (Hatfield 2009). One specific benefit is helping faculty members design courses and focus instruction on the core competencies that are expected of students (Stassen et al. 2001;Jonson 2006). Program assessment opens a crucial dialogue between educators about potential improvements to their educational programs. The results of program assessment also provide data regarding the strengths and weaknesses of graduating students.
In general, the term assessment means any process of gathering and analyzing information on topics, courses, institutions, or departments (Schuh and Upcraft 2001;Stassen, Doherty, and Poe 2001). The cyclic process of program assessment has four essential steps (Peck and Chance 2007;Gordon 2013b; University Office of Academic Planning (OAP) n.d.) beginning with the first step: establishment of student learning outcomes (see Figure 1).
Step 2 includes the development of an assessment plan, in which programs determine where students can demonstrate their mastery of the learning outcomes. The third step is the process of collecting and analyzing data from the program assessment methods. The fourth and final step is reserved for reflecting on and discussing the results (Peck and Chance 2007). The most important process occurs in Step 4, where "closing the loop" takes place (OAP n.d.). In this step, faculty use the results obtained from assessment to implement changes within their departments, adapt curricula, and edit previous learning outcomes.
The article by Chance and Peck (2015) describes the theory behind Step 1 of the assessment process, presents sample learning outcomes developed by several departments, and provides an example of how their department is using program assessment to improve their undergraduate program. This article will focus on Step 2 (choosing assessment methods) by providing background on assessment methods and a description of the process our department used to complete Steps 1 and 2 of the program assessment cycle for our B.S. degree in statistics. The material in the next section is designed to help faculty understand two issues that were challenging for our faculty as we completed our process: (i) the difference between classroom and instructor assessment and program assessment and (ii) the use of rubrics in program assessment. This article concludes with a description of the lessons we learned as we developed our assessment plan.

Assessment Types and Methods
There are eight key assessment terms used to describe assessment types, described briefly in Table 1: direct versus indirect, formative versus summative, add-on versus embedded, and qualitative versus quantitative. A good assessment plan includes a variety of assessment types. The biggest distinction in types of assessment methods is between direct and indirect methods. Direct evidence tells explicitly what students have learned, for example, using a graded course assignment. In comparison, indirect evidence relates to the process of learning and can provide more insight into why they are successful or unsuccessful (MSCHE 2007), for example, student surveys that provide self-reported ratings of student knowledge.
Another distinction between types of assessment is whether it is summative, measuring outcomes of a program or formative, measured while students are in the process of developing mastery of the outcome (Suskie 2009;Gordon 2013b). Embedded assessments, integrated into courses and serving double duty as program assessment items and course assignments, help in getting responses from students that are reflective of their actual knowledge (Peck and Chance 2007). Add-on assessments are not part of course syllabi and do not affect a student's grade in any specific course so students may not provide their best possible responses (Suskie 2009). Finally, quantitative assessment methods produce responses and data that can be analyzed with statistical methods (Suskie 2009). Qualitative assessment meth- Flexible responses and data that are analyzed based on patterns Quantitative Specific responses and data that can be analyzed with statistics ods are used to find recurring patterns in student work and tend to be a bit more flexible in their administration (Suskie 2009). Both qualitative and quantitative methods are highly valued and should be combined to create stronger assessment plans (Schuh and Upcraft 2001).
There are a number of add-on, summative, indirect measures of student learning that are typically used by departments in program assessment. One example is records of student data, which could include number of graduates, majors, and/or minors, grade records from courses, performances on actuarial, certification, or graduate school entrance exams, lists of student honors and awards, and/or internship or employment information (Groeneveld and Stephenson 1999). These records may be relatively easy for departments to record and track over time. Exit interviews of graduating students are also favored because they are relatively inexpensive, easy to administer, and can be used to track changes in student opinions over time. The success of exit interviews, however, relies on the expertise of the interviewer and the quality of the prompts (Stassen et al. 2001). With a trained interviewer, exit interviews can provide a realistic and in-depth perspective of student experience (Jonson 2006). Focus groups are an extension of an exit interview: carefully structured discussions that allow groups of 6-10 students to discuss their knowledge and experiences. The open-ended nature of focus groups may bring up areas of concern that were not originally foreseen by faculty members or other assessment methods (Jonson 2006). Like exit interviews, the success of focus groups is highly dependent on the moderator and the department's ability to gather enough participants (Stassen et al. 2001;Suskie 2009).
Alumni surveys ask the program's graduates about their current positions and any skills they use in their jobs on a regular basis. The patterns of commonly used skills not addressed by the program suggest ways in which a curriculum should be changed (Stassen et al. 2001). Alumni surveys can be broad in scope and are inexpensive measures of student experiences (Stassen et al. 2001). A similar type of survey can be sent to employers or company recruiters. This survey can be used to ask about desired job skills for students who will graduate from the program. Current employers or recruiters provide unique insights (Palomba and Banta 1999) into valued skills that cannot be replicated with any on campus assessment method. Surveys can be either quantitative or qualitative depending on the nature of the questions that are asked (Stassen et al. 2001).
A less well-known indirect assessment of student learning is syllabus analysis. In syllabus analysis faculty members review syllabi to look for opportunities for students to practice the departmental learning outcomes (Suskie 2009). Syllabus analysis is a useful process and is very similar to curriculum mapping (see Chance and Peck 2015) because it promotes coherence across the department and helps to identify possible areas for improvement. It provides a formative, add-on assessment that is typically qualitative in nature.
It is important to include direct measures of student learning in a complete assessment plan (Gordon 2013b). Objective tests in any required course provide embedded direct assessment of students' basic conceptual knowledge and procedural and abilities (Maki 2010). In addition, capstone courses provide students opportunities to demonstrate synthesis and evaluation of cognitive abilities (Bloom and Krathwohl 1956;Maki 2010) through embedded assignments such as posters, presentations, and/or articles (Palomba and Banta 1999). Rubrics can be used with these assignments to provide quantitative measures of student learning from otherwise qualitative activities.
Portfolios are collections of student work that can include articles, reports, exams, or analyses (Stassen et al. 2001) and are direct add-on assessment methods of student learning. Portfolios allow programs to see trends in student growth over time as well as students' overall strengths and weaknesses (Jonson 2006). Nationally standardized and locally developed classroom tests measure basic competencies and allow programs to compare student achievement after taking the same test (Maki 2010). National tests are advantageous because they allow the examination of a large group of students in a short time frame (Palomba and Banta 1999) and have been analyzed for reliability, validity, and other psychometric properties (Jonson 2006;Maki 2010). National standardized tests, however, are often not directly aligned with program learning outcomes and typically fail to provide realistic insight into problem-solving processes (Maki 2010).

Using Rubrics in Assessment
Many programs already include projects, presentations, or articles in their courses. These assessments demonstrate individual student learning, but are difficult to use to understand the collective knowledge of the group and whether the program's learning outcomes are being mastered. Rubrics bridge the gap between course assignments and program assessment, creating direct and quantitative measurements from qualitative data (the articles) and indirect measures (grades on the article). There are two main types of rubric scoring systems: holistic and analytic. Holistic scoring focuses on an overall impression where elements are combined (Quick 2014). This type of rubric is especially useful to programs that wish to categorize student performances into levels (Maki 2010) and an example is provided in the supplementary materials.
Analytic rubrics provide detailed insight into student performance by examining elements separately at each level (Suskie 2009;Maki 2010;Quick 2014). Analytic rubrics typically have four parts: task, scale, elements, and description (Quick 2014). The task is the complete description of the assignment. The scale, which defines the levels of student achievement from lowest to highest, is placed horizontally. The example given in Table 2 contains three levels: novice, intermediate, and advanced. The descriptions in an analytic rubric should contain positive terms with a clear distinction between levels (Hatfield 2009;Quick 2014).
The performance elements or dimensions are included on the left side of a rubric and break down the attributes of students' work on the core learning areas. The four performance elements included in the example in Table 2 are content, organization, graphics, and eye contact. Finally, the descriptions are the interior cell components that detail the specific attributes for the intersection of the scales and elements.

Choosing Assessment Methods
Departments should select the assessment methods that will be the most useful for gathering information on student learning outcomes. The best combination of methods answers important questions, is manageable based on current resources, and allows faculty to view the strengths and weaknesses of the program (Stassen et al. 2001). The process of selecting multiple assessment types is called triangulating assessments (Washington State University (WSU) 2009) and helps to assess a complex task and to tell faculty about student experiences throughout the curriculum (Palomba and Banta 1999). Assessment plans should include both direct and indirect methods, as well as qualitative and quantitative methods. An objectives-bymeasures matrix linking existing or future assessment measures with student learning outcomes can be used to choose appropriate assessment methods. The matrix, similar to the curriculum map discussed by Chance and Peck (2015), can help programs to select assessment measures based on their program goals, rather than forcing outcomes to existing assessment methods. It allows programs to see where certain questions fail to be answered or where learning outcomes are not triangulated with multiple measures.

Closing the Loop
Once the data have been collected, the important work of understanding how well the curricula match with educational objectives begins. This final step in the assessment cycle is often the biggest challenge (Hatfield 2009). Analysis of program assessment data over time reveals three common patterns: (i) patterns of consensus, when disaggregating the data exposes agreement between items, (ii) patterns of consistency, Full understanding of material and complete answers to questions. Organization Presentation jumps around and is not easy to follow.
Presentation can be followed, but dull for audience.
Presentation is logical and interesting for audience. Graphics Images are not used or are not useful to the presentation.
Images are useful to the presentation. Images are explained and useful to the presentation.

Eye contact
Maintains little to no eye contact and reads off slides.
Maintains partial eye contact and uses slides too frequently.
Maintains constant eye contact and uses slides sparingly.
revealed by a longitudinal analysis of the same type of data, and (iii) patterns of distinctiveness, in which data from different assessments are evaluated to see which outcomes have the best performance (Hatfield 2009). Upon identifying patterns, programs can classify the trend, determine whether the achievements are acceptable, and discuss any surprises (Hatfield 2009). The term "closing the loop" (Hatfield 2009;Gordon 2013b) is used to describe the changes that are suggested by the patterns in the data. Programs should use data to influence curricular reform when the true and desired performances are not in concordance. Sometimes data suggest that departmental policies should be reviewed. This could include advising methods, admissions criteria, or methods for managing students through the curriculum. Most importantly, assessment data should encourage faculty discussion about student learning and programs as a whole. Some concrete examples of changes made within this step of the assessment process for undergraduate programs in statistics are given by Chance and Peck (2015).

The Undergraduate Program in Statistics at the University of Georgia
The primary purpose of our undergraduate program is to provide students with a strong statistical background that will lead to a Bachelor of Science (B.S.) degree and prepare students for a career after graduation. Between 1964 and 2014, the undergraduate program in our department awarded 554 bachelor's degrees. In addition to their statistical studies, many of the graduates of the program have double majors and/or minors. Enrollment in our Bachelor's program has more than doubled since 2008, when there were 50 declared majors and 10 graduates. In 2014, we had 106 declared majors and conferred 20 B.S. degrees.
The B.S. in Statistics requires a minimum of 50 hr of coursework. The prerequisites for statistics majors include an introductory statistics course, a computer science course, and a calculus sequence. The curriculum is designed to be flexible enough to allow students to choose courses based on their future aspirations. For example, students interested in pursuing graduate school in statistics are encouraged to take an additional course in linear algebra (Department of Statistics, n.d.). Requirements for the major include a six-credit capstone sequence (Lazar, Reeves, and Franklin 2011). The two courses are taken during students' senior year and expose students to real data problems. Students work with clients to investigate research questions, while gaining experience that could not be acquired in a traditional classroom setting.

University Assessment Process
Our University has had a program assessment requirement in place since 2003. The main components of the previous plan are shown in Table 3. Our University requires a 7-year review cycle in which each department submits brief reports, consisting of a 1-2 page summary of the assessment data collected in the previous year, in Years 1, 2, 4, and 5. In Years 3 and 6, full reports containing information about all student learning outcomes and assessment data collected from the past year as well as from previous brief reports are submitted. The work reported here was undertaken to meet the request of a university program review committee to revise the assessment plans for all of the degrees offered by our department. Information provided at workshops sponsored by our University as well as informal discussions with colleagues at other institutions indicate that the trend to provide more detailed and extensive assessment plans is national in scale. Departments that have not collected assessment data previously or that have fewer resources may find the assessment plan we devised overwhelming in scope. Such a department is encouraged to start with a less ambitious plan.

Our Process
Assessment needs to be a collective effort on the part of all faculty members in a department. In response to the request for revision to our assessment plan, the department head convened a Learning Outcome Assessment (LOA) Committee to discuss changes to current outcomes and program assessment at the three program levels (Bachelors, Masters, and Doctoral). The establishment of an assessment committee provided direction and leadership for the assessment process, began discussions, supported department buy-in, and encouraged faculty members to see the value of assessment to student learning as is supported  Students should be able to obtain and use real data to answer a question requiring data; describe the different types of study designs (survey, experimental, observational), the role of randomization, and common sources of bias; interpret graphical displays and numerical summaries of data; explain that variability in data is natural, predictable and quantifiable; and make use of statistical inference. Students should be able to distinguish distribution types (population, sample or data, and sampling) and explain conceptually the role of a distribution and, statistical significance (including p-value and significance level), and margin of error and confidence level with intervals. Students should be able to interpret statistical results in the context of the original question. Students are expected to critique statistical information and develop a healthy skepticism about how statistics is used in practice.

Theory
Students should be able to explain the basic statistical theory underlying much of formal statistical estimation and inference. They should be able to justify theoretically the properties of the most commonly used frameworks for statistical estimation, inference, and modeling. They should be able to apply the properties and criteria that can be used, in general, to derive, evaluate, and compare statistical methods on theoretical grounds.

Computing
Students should be able to use technology to analyze data, with students focusing on the interpretation of results and the checking of conditions for inference. Students should also use technology to visualize concepts and illustrate abstract ideas through simulations. Students should become proficient using statistical packages such as Minitab, SAS, and R and working with spreadsheets and using web-based resources. Students should use technology to work with large, real and sometimes messy datasets.

Communication
Students should be able to use the language of statistics effectively to communicate ideas verbally and in writing. Students working in groups should be able to successfully collaborate with researchers when a more senior statistician is present. Students should be able to explain basic statistical concepts, methods, and results in language that is clear to nonstatisticians and relevant to applied problems.
by the literature (Peck and Chance 2007;Hatfield 2009;Maki 2010). The committee consisted of five faculty members (the associate department head, the director of graduate admissions, the graduate advisor, the undergraduate advisor, and a tenuretrack faculty in statistics education) and a graduate student. The work of the committee was done prior to the release of the 2014 Curriculum Guidelines for Undergraduate Programs in Statistical Science (American Statistical Association 2014). Therefore, the committee used the 2000 ASA guidelines (American Statistical Association 2000) and learning outcomes from similar statistics programs as inspiration when writing the student learning outcomes (SLOs). The 2000 guidelines contained five subcategories of skills needed by a graduate of an undergraduate statistics program: statistical, mathematical, computational, nonmathematical, and substantive area (ASA 2000). As is recommended in the literature, the committee limited the SLOs to four core areas derived from the five subcategories: theory, data analysis, computing, and communication. While only the SLOs for the B.S. program are given in this document (see Table 4), the complete set of SLOs, which build the learning objectives in cumulative levels throughout the B.S., M.S., and Ph.D. programs, are provided in the supplemental materials.
The LOA committee next developed a curriculum map based on the four core SLO areas to ensure the alignment of the curriculum and undergraduate student learning outcomes. In our mapping process, we used the letter I to denote classes in which the outcome is Introduced, D for courses that give the opportunity to Develop the outcome, and A for the courses in which the outcome will be Assessed. The curriculum map (Table 5) shows that our undergraduate program requirements address each of the student learning outcomes at some point in the curriculum. The program could, however, include more courses to introduce communication skills prior to their assessment during the capstone sequence. The student learning outcomes associated with communications may need to be investigated further during the later stages of the assessment process. While the calculus and linear algebra requirements are perceived to develop the core area of Theory, it is possible that these courses also introduce and/or develop SLOs in computing or communication. The SLO committee needs to communicate with the Department of Mathematics about the SLOs associated with these courses. Overall, the curriculum map suggests that the capstone course will be the primary focus during assessment of the SLOs. The next step in developing the new assessment plan was to determine when and how to assess outcomes within the program. The previous assessment plan, the new curriculum map, and an empty objectives-by-measures matrix were used to choose assessment methods. The LOA committee created a plan that was feasible and would provide the most valuable assessment data with minimal time and effort, while providing opportunities to assess each SLO and triangulate the data by using multiple assessment types. The completed objectives-by-measures matrix  for the undergraduate program (see Table 6) incorporates an alumni survey, exit interview, faculty discussion, records of student data, and a student focus group as the indirect assessment measures. The check marks in the Objectives-by-Measures Matrix indicate that the assessment method can be used to evaluate student learning for a particular core outcome. The online supplementary materials include an example alumni survey, exit interview, and record of student data, all indirect measures of student learning. The alumni survey includes applications of the four core areas in the workforce, a comment section for program improvements, and questions on current employment status. The exit interview questions are very similar to the version used in the 2003 assessment plan. The working knowledge questions (#8) are grouped based on the four core learning areas. Historically we have found that using the undergraduate advisor as the moderator of the exit interviews provides relatively unbiased data and that the students are comfortable sharing their thoughts with her. Records of student data are updated during each academic year and include graduation rates, course enrollment, course grade averages, and student honors. A faculty discussion is used to synthesize the data collected and close the assessment loop. This discussion is like a focus group of the faculty members involved in the program. Finally, focus groups of students are used to gather qualitative and indirect data. The choice of the faculty moderator and guiding questions was done to maximize the quality of data collected from the focus groups.
Direct assessment methods include objective tests, for example, midterms or final exams, in upper division courses, such as Mathematical Statistics I and II, and assignments graded with rubrics. This helps the department to gather additional direct evidence in courses that were marked with an "A" in the undergraduate curriculum map (Table 5). Instructors of these courses are responsible for reporting student's working knowledge in the related core learning area to the LOA Committee. Rubrics have been added to the capstone poster session and are being considered for the presentation. Two example rubrics for the capstone presentation and poster session are included in the supplemental materials. The capstone presentation rubric is an analytic rubric based around the four core learning areas. Faculty members or other audience members who watch the presentations would complete this rubric for each group. The capstone poster rubric is also focused around the core areas, but functions more like a checklist. This rubric is distributed to and completed by the departmental poster session attendees. Both rubrics aid in grading the capstone assignments and understanding student learning across the program.
The proposed timetable, which is based on the selected measures and the university 7-year cycle and included in the supplemental materials, suggests scheduling a dedicated faculty discussion of assessment results during a fall faculty meeting. This will help the assessment committee to close the loop and write the required annual reports. Objective tests in upper division statistics courses will also be used during the fall semester. During the spring semester the department will use capstone presentation rubrics and capstone poster rubrics. The exit interview will be conducted at the end of each academic year to the graduating seniors. A data review is also included on the timetable for each summer in the assessment cycle. The data review will include completing the record of student data as well as combining any assessment data that were collected during the previous academic year. This step, completed in the summer semester, will ensure that data are prepared for the fall semester's faculty discussion and report writing. Additionally, focus interviews will be scheduled twice in the 7-year cycle. If the focus interviews are scheduled right before the writing of the two full reports, faculty members could gain additional insight from students before reporting to the university. Lastly, scheduling an alumni survey once every assessment cycle (i.e., every seven years) will help the department to keep up with their recent graduates. The committee must still determine who is responsible for each item contained in the table. For example, the capstone instructors may be responsible for collecting rubric data on presentations, but a different person might be selected to summarize these data. There should also be a point person who is responsible for submitting the required reports to the university, but the timetable will be very helpful in keeping the assessment plan and reporting cycle on track during each academic year.

LESSONS LEARNED
Assessment can only be effective if it is endorsed by faculty of the program (Lindholm 2009). Faculty members tend to be skeptical of the process initially and believe that the assessment trend will "blow over" (Peck and Chance 2007;Hatfield 2009, p. 2). Often faculty members have so many other things to do that assessment is not high on their priority list (Hatfield 2009). As suggested by the literature, there was initial skepticism by the faculty in our department about the LOA process. The main issues for faculty stemmed from the confusion between course assessment that is commonly used by faculty, and program assessment. After the members of the LOA committee explained that program assessment is meant to evaluate programs as a whole and not individual instructors, classes, or students, the skepticism was greatly diminished.
A key recommendation for departments developing program assessments is to ensure that at least one member of the LOA committee is knowledgeable about program assessment. This person should also understand the process and requirements of the university with regard to assessment. The first author of this article was that person for our department. As an M.S. student in our department, she read the literature about program assessment and attended two university sponsored workshops on writing outcomes and deciding on assessment procedures (Gordon 2013a(Gordon , 2013b. We recommend Stassen, Doherty, and Poe (2001) and Hatfield (2009) as starting points for reading about program assessment. Larger volumes, Suskie (2009) andMaki (2010), contain everything a department will need to complete the program assessment cycle, but may be overwhelming as a point of entry. The first author used the information obtained from her reading and the workshops to guide the assessment process within the statistics department and her thesis was a record of the theory and process that underlies program assessment. Departments may need to be creative in identifying and supporting someone in this role during the process of developing an assessment plan. Some departments may have the resources to hire a student or give a course release to a faculty member, other departments may give credit for independent study course or use the time allocated to the undergraduate advisor as support.
The use of the 2000 ASA curriculum guidelines as the foundation of our SLOs was invaluable to our process; we used the narrative describing the major skills needed by graduates to write our own learning outcomes. We, therefore, suggest that any department undergoing a similar process use the 2014 ASA curriculum guidelines as the basis for writing or revising departmental SLOs. In particular, SLO committees should attend to the descriptions of the five major areas found on pp. 9-10 of the 2014 curriculum guidelines (ASA 2014), statistical methods and theory, data manipulation and computation, mathematical foundations, statistical practice, and discipline-specific knowledge, and follow the advice given by Peck (2000, 2015) to create SLOs from the ASA provided narrative description of skills.
The first step in our process, writing SLOs, was an iterative process with a series of critiques and rewrites before they were considered by the full faculty. While faculty members could easily identify the knowledge students graduating from our program should possess, they had difficulty phrasing the knowledge components using action verbs, instead relying heavily on the terms "understand" and "know." The outcomes were repeatedly edited to change "understand" and "know" to action verbs (see Chance and Peck 2015). For example, in the computing core area a statement originally read, "Students should also use technology to visualize concepts and understand abstract ideas through simulations." The word, understand, was changed to the action verb, illustrate, to create a measurable student learning outcome.
The learning outcomes were well received by the faculty in our department and started a productive conversation. Faculty members made interesting and useful observations on the outcomes and curriculum maps. In reaction to the lack of opportunity for students to develop communication skills prior to the Capstone course, one faculty member commented that in previous teaching of courses for our majors he had focused on teaching a lot of technical material rather than working on communication skills through articles, group work, or presentations. Other faculty agreed, saying that they could do a better job of incorporating communication into their courses. This is one example of how our department will use the program assessment to close the loop and make positive changes in our program.
When the discussion moved to the future of program assessment in the department, members were concerned about the university guidelines and restrictions with respect to assessment. In designing the assessment plan, the LOA committee took care to create a plan that is manageable and informative for the department; the university criteria were used as guidelines rather than strict rules. We recognize that our plan may seem daunting to a department with no previous experience with program assessment and we encourage such departments to start small, but include at least one direct measure of student learning in the plan. As statisticians, we have the appropriate background and training to design, conduct, and analyze an alumni or employer survey as an indirect measure of student learning, but departments are encouraged to collect direct measures of student learning through embedded assessments, either from an analysis of the final exams in upper division courses or through the use of rubrics in conjunction with a poster fair or capstone project.
When the use of rubrics was raised as a potential addition to our assessment plan, the reaction to the term rubric by faculty was quite negative. While we cannot explain this negative reaction, we found that replacing rubric with the word checklist reduced the issue faculty appeared to have with the concept of rubric or checklist. We, therefore, caution others embarking on this procedure to be on the lookout for other assessment terms to which faculty may have an unexpected negative response. On the positive side, one professor did point out that within our courses, topics covered and teaching methods should evolve based on the outcomes and new data received from assessment. This is a very key item and it emphasizes that program assessment is an iterative process meant to improve student learning.
In summary, the literature indicates that departmental assessment committees should start with advice from other programs' assessment plans and university resources (Palomba and Banta 1999). The plans they create should be dynamic, ongoing, systematic, manageable, and generate meaningful data (Palomba and Banta 1999;Peck and Chance 2007;Hatfield 2009;Lindholm 2009). An effective plan is designed and implemented by the faculty and is aligned with the curriculum. Successful plans use multiple methods and help to distinguish between levels of achievement (Stassen, Doherty, and Poe 2001;Hatfield 2009). Larger programs should also use sampling to reduce the assessment burden (Peck and Chance 2007). Similarly, programs might focus on assessing only 1 or 2 student learning outcomes per academic year (Gordon 2013a). Programs deciding on assessment methods need to include a detailed timeline with their assessment plan. This timeline contains the faculty in charge, time frame, resources, and how feedback is used to make changes to the program (Lindholm 2009). The time frame should allow time for mistakes as well as ongoing reflection, input, and improvement (Palomba and Banta 1999;Stassen, Doherty, and Poe 2001). Our department has tried to follow this advice and this document should provide an example for other departments to follow.
Program assessment is especially important for statistics departments because of the relatively rapid increase in enrollment at the undergraduate level and the increased demand in industry for employees trained in statistics due to the availability of large datasets and the need for analysts of such data. These changes are forcing departments to adapt undergraduate programs to meet industry and university demands. Historically, there has been "a wide disparity in the curriculum at various institutions and confusion on the part of employers as to the skills and abilities that can be expected of a bachelor's level statistician" (Bryce et al. 2000, p. 6). Program assessment can help to reconcile these expectations and curriculum realities within the department. By analyzing student learning through program assessment, departments can determine ways to improve their program and increase the value of their graduates in the workforce.

SUPPLEMENTARY MATERIALS
The supplemental materials include examples of direct and indirect assessment measures, a holistic rubric, analytic rubrics for capstone presentations and posters, alumni survey, exit interview, and record of student data. In addition, the supplemental materials include the complete list of student learning outcomes for our department for the B.S., M.S., and Ph.D. programs and the proposed timetable for assessment, which is based on the university 7-year cycle.