Improving Student Learning of Calculus Topics via Modified Just-in-Time Teaching Methods

Abstract Although the use of traditional just-in-time teaching techniques has long been viewed positively by students and instructors in undergraduate calculus courses, past studies in this area have not addressed gains in student achievement with respect to specific calculus topics. This paper investigates the latter by administering modified just-in-time review modules over college algebra topics in a first-semester calculus course at Kansas State University during Spring 2012. Analysis of student performance revealed significant gains in learning on the topics of limits and the chain rule for derivatives. Issues pertaining to the timing of the reviews, along with descriptions of the review assignments, analysis of student performance, and implications for classroom instruction are discussed.


INTRODUCTION
Just-in-time teaching (JiTT) and similar techniques have been widely used to address issues of remediation/concurrent review in undergraduate math courses across the continent (see [1][2][3]5], for instance). Although these studies either report positive feedback or improvement in terms of final grades, none of the literature addresses actual improvement in learning on particular calculus topics after implementation of JiTT in the classroom. One of the main purposes of the performed teaching experiment was to determine if such JiTT tools do lead to specific gains in learning, as well as whether or not the just-in-time feature makes a difference in student achievement on various calculus topics.
In light of continuing troubles surrounding students' lack of fluency in pre requisite mathematical skills/concepts, assisting calculus students and helping them succeed and learn mathematics is of prime importance. In particular, helping first-semester calculus students mend gaps in their understanding of college algebra and trigonometry is critical. With this goal in mind, the performed research investigated the following ideas.
1. Do modified just-in-time online teaching tools covering review topics in algebra lead to gains in student learning on specific calculus topics? 2. Does timing of these review assignments make a difference in terms of student achievement on calculus online homework assignments?

THE TEACHING EXPERIMENT
We compared two batches of Calculus-I students from the Spring semesters of 2011 and 2012, that had initial enrollments of 285 and 341 students, respectively. To verify that both populations of students were similar in terms of mathematical ability upon entering the course, average math ACT scores of both populations were considered. These scores were 24.9 and 24.6 in Spring 2011 and Spring 2012 respectively, and a two-sample t-test assuming equal variances revealed that the mean ability of both groups of students as represented by their ACT math scores was the same with 99.95% confidence. The Calculus-I ("Analytic Geometry and Calculus I") course at Kansas State University follows a traditional lecture-recitation format where students meet in a large lecture setting twice a week and smaller breakout recitation sections twice a week. The topics covered in this course include limits, differentiation, applications of differentiation, integration, and a few applications of integration including computing the area between curves and volume of solids of revolution during the Spring 2011 semester. The syllabus in this course aligns well with other traditional Calculus-I courses at institutions with accredited engineering and reputable undergraduate mathematics programs.
During both semesters, the lecturers were different; however, they covered the same topics and assigned written homework from the same text. Online homework was a major component of the coursework as well and both groups of students used the same online homework system.

THE ONLINE HOMEWORK ENVIRONMENT
The review assignments that were developed for this teaching experiment followed the same format as the online homework sets that were already in place in the Calculus-I course. Both review and regular online homework assignments contain three or four problems. After logging in, the students may enter their answers and save their work, and they may come back and work on the problem set at any time before the deadline for the assignment. When students initially submit their answers for grading on the first problem set, see Figure 1, the system instantly marks which responses are correct and incorrect, and the students have one free chance to correct their initial mistakes. After the second round of grading, see Figure 2, students may choose to keep the score earned, or, they may log in again and try a different problem set. Students may do as many different problem sets as they wish until they are happy with their score. The system records the student's highest score attained up until the deadline for each individual assignment. All of the review assignments were written in PHP, with automatic, instantaneous grading routines written in PHP/Javascript/Java.
Numerous aspects related to individual student performance can be tracked, such as average time until one earns a perfect score, mean scores on an assignment, and how often the student accesses optional additional help. The system presents students with worked out solutions if they haven't scored 100% on a given problem set. The online homework system randomly generates different problems for every student; the system supports a variety of question formats, including graphical, tabular, fill in the blank, and whichof-the-following apply types of questions. Students truly complete different assignments so that copying solutions from each other is not an option.

REVIEW TOPICS
Traditional JiTT assignments are quite different from those designed for this teaching experiment [4]. In particular, the preparatory assignments designed in this study are not strictly administered right before a lecture, nor are they based on out-of-class reading. Instead, the preparatory modules created here are solely administered online at some time during the semester, simply at some point before students need to apply the ideas when they arise in calculus. The reviews are designed in such a way that lecturers have the option of using them as review tools outside of class, or, they may use them in a pure JiTT manner by adjusting their lectures based on student responses.

Rational Expression Review
In this particular assignment, the student must submit solutions to three questions. The first problem requires adding two rational expressions, possibly needing to find a least common denominator in the process. Answers must be simplified as far as possible. The second question covers compound fractions, and involves manipulating fractions using basic rules of algebra. The last exercise is a lengthier computation that one would see when applying the definition of derivative to a quadratic or rational function. The three problems have optional help links attached to them. The help links contain step-by-step explanations to the student's specific problem which may be viewed after having submitted answers for the second (and final) round of grading (see Figure 3). This assignment was administered the week before the online assignment over limits.

Composition of Functions
This assignment also takes students through three tasks. The first question is a straightforward computation of f (g(x)) given two functions f (x) and g(x). The student must simplify the composition as far as possible, according to the assignment directions. The next problem asks students to find an outside and inside function which upon composition yields a given function. The third question is purely conceptual, asking students to find function composition values from a table of values for two functions f (x) and g(x). All of these questions also have optional help links attached to them and a different added feature in the assignment involves a video explanation of how to do the problems as well. The video clip was created using both Camtasia recording software and Windows Journal. Camtasia is a software that captures all activity taking place on a computer screen; Windows Journal allows one to write on notebook or graph paper just as though one is taking notes in class. This assignment is administered roughly 5 weeks before the lecture on chain rule takes place.

Order of Operations
To assist students with interpreting computer syntax and correct usage of parentheses, this review assignment focuses on basic algebra. The first question involves using a calculator to simplify a fraction in which the numerator contains a natural logarithm. The next exercise presents a fraction with various parameters and asks the student to select choices that are equivalent to the given fraction. The last problem displays an expression that one might run into when applying the quotient rule to a function involving exponentials; from a given list, students must select appropriate choices that are equivalent to the given expression. Like the other assignments, this assignment contains optional additional help. The review is given roughly 2 weeks prior to the lecture during which the quotient rule is covered.

RESULTS OF TEACHING EXPERIMENT: ACHIEVING PERFECTION
The results from individual chi-square tests for independence revealed that there is a statistically significant relationship between student performance on the limits assignment and the chain rule assignment and the semester in which these assignments were administered (chi-square with one degree of freedom, p = 0.03 and p = 0.02, respectively).
Implications of the chi-square test for independence are certainly favorable. Higher percentages of perfect scores on the limits and chain rule assignments (summarized again in Figure 4) were attained during the Spring 2012 semester, the semester during which the just-in-time modules were the only newly added content-specific components in the Calculus-I syllabus. All other features related to course content during the Spring 2011 and Spring 2012 semesters, including the textbook, textbook assignments, online calculus assignments, syllabi, and number of exams were either identical or very comparable. It is safe to conclude that the just-in-time tools certainly   contributed positively towards significantly raising the number of students that reach perfection on the limits and chain rule assignments. The differentiation assignment was a regular calculus online homework assignment that did not have a modified JiTT assignment attached to it. The differentiation online homework assignment contains problems that utilize the power rule, and does not include problems involving the chain or quotient rule. The chi-square test for independence for the differentiation assignment gave a p-value of virtually one (chi-square with one degree of freedom, p = 0.98 -see Table 1). Being an untargeted calculus assignment, this result provides additional support that the control and experimental groups of students are similar in terms of mathematical ability.
Ideally, we would have liked to run a similar experiment that compares two batches of Fall semester students in Calculus-I. Due to certain limitations, this was not possible. In particular, the review modules were in the piloting phase in the Calculus-I courses during Fall 2011 and have been part of the syllabus ever since then. For Fall semesters prior to 2011, online homework (the source of our control data) was not a course component.

DOES TIMING MATTER?
Ideally, reviews created in accordance with JiTT would have been administered no more than a few days prior to when the associated calculus topics were covered in a lecture. In this study, the reviews were not implemented in a pure JiTT manner. In fact, one of the review assignments, namely the composition of functions review, was given 5 weeks prior to the chain rule online assignment. The other review assignments were given roughly 1 to 2 weeks before the associated online calculus homework was assigned (consult Table 1 for the exact timeline).
Several of our results affirm that timing of the JiTT reviews is not a crucial factor when it comes to improving student performance. Figure 5, for instance, re-caps average scores on select Calculus-I assignments, which were higher on the target calculus assignments (these were the online assignments with the JiTT intervention during the Spring 2012 semester) and about the same on an assignment that had no intervention (the differentiation assignment).

IMPLICATIONS FOR INSTRUCTION
As long as students work through the review material at some point during the course prior to their application in calculus, the reviews seem to have a positive effect. In fact, the review assignment that was administered most ahead of time (5 weeks ahead of time) was the composition of functions review, which corresponds to the chain rule assignment, and the most improvement resulted on this particular calculus assignment. Rather than having completed the reviews just-in-time, it seems that having seen the prerequisite material at some point during the course of the semester did have a positive impact.
Another possible advantage of completing the review modules ahead of time rather than just-in-time is that if the review covered a topic that a student truly did not know/remember, the student still has some level of flexibility in terms of brushing up and reviewing these specific skills on their own time before seeing it applied to new content in calculus. Had they seen these review topics in a pure just-in-time setting, there would not have been a long enough break in time to sit down and learn/relearn/revisit the review topic on their own. Overall, the review assignments created here serve as a catalyst for triggering connections between prior knowledge and new calculus topics, even when the just-in-time aspect is somewhat relaxed.
Since our results reveal that gains in learning occur regardless of the timing of the assignments, we have a flexible tool that instructors can easily adapt without having to follow a pure just-in-time approach. Although instructors have the option of using these reviews as a traditional just-in-time tool (adjusting their lectures according to student responses), this tool lends itself well to instructors who choose utilize the assignments purely for review purposes early on in the semester.
Although the review tool used in this teaching experiment was built inhouse and solely administered online, the review exercises presented here can be easily adapted in other online homework/course management systems (Blackboard/ALEKS for example) as well as traditional paper/pencil-based reviews. One key observation worth noting is that concept-based reviews seem to have the highest impact in comparison with skill-based reviews (composition of functions helping most with chain rule, versus order of operations and the quotient rule assignment, for instance).

FUTURE WORK
A fourth review module over trigonometry was also created based on students encountering trigonometric relationships in many calculus applications. In particular, the module was intended to prepare students for related rates problems. The effectiveness of the trigonometry module was not directly analyzed in this study since we did not assign an online homework assignment in calculus related to such problems. A future study might investigate whether or not the trigonometry module helps with such applications in calculus.
Other areas of review are also worth investigating. In particular, an assignment that tests students' knowledge on graphing common functions might be useful. The motivation for such a review lies behind the concept of graphing the derivative function; being able to quickly recall graphs of known functions is a necessary skill. Finding equations of lines in point slope form might also be a relevant review module when students embark on computing tangent line equations; likewise, reviewing concepts from physics, such as displacement, velocity, and acceleration might also be another area to look into when students begin the differentiation unit in Caclulus-I.
Assessing the effectiveness of a modified just-in-time teaching approach in subsequent calculus courses might also yield useful information pertaining to retention of students at the university level. In particular, since we have demonstrated that such tools do lead to gains in student achievement, adapting them might lead to more success in mathematics courses that traditionally struggle with producing higher pass rates. Perhaps a longitudinal study across the whole calculus sequence and differential equations in which modified JiTT tools are utilized in each course along the way might better address questions surrounding retention of students from one course to the next.

CONCLUSIONS
The research study presented here investigated the effectiveness of a modified just-in-time set of review modules covering algebra topics in the context of relevant calculus material. By administering these review modules via the online homework system traditionally used in this calculus course, we have discovered a flexible, time-independent method for addressing gaps in background knowledge that also leads to a statistically significant increase in student achievement on the topics of limits and the chain rule. With limits being an inherently difficult topic for first-semester calculus students to master, the success of this tool in context of this topic is encouraging. With added flexibility of dropping the just-in-time feature, easy implementation is another positive aspect of these review modules. We hope that this study leads into future research that offers new perspectives and insight regarding the connections that students make between background mathematical knowledge and new content.