Meaningful Learning and Transfer of Learning in Games Played Repeatedly Without Feedback

Psychologists have long recognized two kinds of learning: one that is relatively shallow and domain-specific; and another that is deeper, producing generalizable insights that transfer across domains. The game theory literature has only recently considered this distinction, and the conditions that stimulate the latter kind of “meaningful�? learning in games are still unclear. Three experiments demonstrate that meaningful learning – the acquisition of the principle of iterated dominance – occurs in the absence of any feedback. We also demonstrate that meaningful learning transfers to new but strategically similar games, and that such transfer does not occur when prior games are played with feedback. The effects of withholding feedback are similar to, and substitutable with, those produced by requiring players to “self-explain�? (provide written explanations for behavior), a method commonly employed in psychology to increase deliberation. This similarity suggests that withholding feedback encourages players to think more deeply about the game.

1 Considerable research in economics attempts to understand how people learn in strategic environments. Many experimental studies on games demonstrate that players do not initially play equilibrium strategies, but that with repetition their behavior converges towards equilibrium. Several models attempt to provide a theoretical basis for this regularity (e.g., Cheung and Friedman, 1998;Fudenberg and Levine, 1998;Erev and Roth, 1998; see Chapter 6 of Camerer (2003) for a review). While these models vary in the details of how they assume learning occurs, most share the assumption that learning operates by players observing how well different strategies perform -either by playing those strategies, observing others playing them, or observing (foregone) outcomes produced by unselected strategies -and then adjusting their subsequent behavior in the direction of better-performing strategies. Thus, most learning models in economics focus on understanding how players gradually figure out what strategy produces the highest payoffs in a specific game, a process best described as strategy learning.
Economists have devoted considerably less attention to understanding a distinct process that might be called meaningful learning, whereby individuals come to obtain meaningful cognitive representations of higher-order concepts, rules, and relationships that can be transferred to novel domains. While a small amount of empirical and theoretical work in economics has uncovered some conditions under which higher-order concepts learned in one context transfer to new contexts (Rankin et al., 2000;Stahl 2000,a,b;Haruvy and Stahl 2008;Cooper and Kagel 2003, 2005, the factors that stimulate meaningful learning in games are still not well understood within economics.
In this paper, we attempt to advance economists' understanding of meaningful learning by importing relevant insights from psychology. Although economists have only recently begun to study more than one type of learning, psychologists have for decades recognized a distinction between two kinds of learning, based on the way in which people learn, the kind of knowledge produced by learning, and the ability of individuals to transfer what they learn to new domains. This distinction is important because it highlights significant differences in the depth of what is learned and the ability of individuals to generalize their learning to new contexts.
One type of learning studied within psychology, variously referred to as "implicit," "procedural," or "unconscious" learning (Holyoak and Spellman, 1993), is an unconscious process that yields knowledge that is usually neither accessible to cognition nor verbalizable (Reber, 1967(Reber, , 1989Mandler, 2004). It is demonstrated, for instance, by showing that subjects exposed to massive amounts of information demonstrate improved performance in pattern matching, but that such improvement exceeds their ability to articulate or generalize their knowledge (Berry and Broadbent, 1984;Nissen and Bullemer, 1987;Hayes and Broadbent, 1988). A key property of this kind of learning is that it operates through perceptual and associative processes, rather than through cognition, and therefore fails to produce cognitive or conceptual representations of what is learned (Mandler, 2004). An important consequence of the absence of such meaningful representation is that what is learned through implicit learning cannot be consciously manipulated or transferred to new domains (Holyoak and Spellman, 1993).
The other type of learning, commonly referred to as "explicit," "declarative," or "conscious" learning (Holyoak and Spellman, 1993), is a process through which individuals come to obtain meaningful cognitive representations of underlying concepts, rules, and relationships. Unlike the knowledge acquired via implicit learning, the knowledge acquired via explicit learning is consciously accessible, generalizable, and verbalizable. Moreover, explicit learning involves cognition, the evaluation of hypotheses, and often results in the development of improved general problem-solving ability (Hayes and Broadbent, 1988;Mandler, 2004). Thus, a key property that distinguishes explicit from implicit learning is that the former is less context-dependent and generates knowledge that can transfer to novel situations.
We propose that the strategy learning commonly observed in games more closely resembles the former than the latter, more meaningful, kind of learning. This correspondence is perhaps best illustrated by the lack of transfer of learning to new games. Despite many experiments on learning in games -in which subjects converge towards equilibrium when playing a game repeatedly with prompt outcome feedbackthere is little evidence that what is learned transfers to new strategically similar games (i.e., games in which a meaningful principle, such as dominance or backward-induction, applies non-trivially to both games). For instance, Ho et al. (1998) explicitly test for transfer in two closely-related dominance-solvable games and find no transfer from the first game to the second. Similarly, in a series of papers, Cooper and Kagel (2003, 2005 find that transfer does not occur when subjects play two signaling games sequentially under typical feedback conditions, unless experimental treatments (such as meaningful context or team play) facilitate subjects thinking more deeply about the game.
Given the correspondence between strategy and implicit learning, manipulations commonly employed in psychology experiments to inhibit implicit learning and stimulate explicit learning could serve an analogous function in game theory experiments and could facilitate meaningful learning. One such manipulation involves the amount of feedback participants receive about task performance. Counterintuitively, psychologists have often found that deeper and more meaningful learning occurs more often when people receive minimal or delayed feedback than under full and immediate feedback (e.g., Salmoni et al., 1984;Winstein and Schmidt, 1990;Goodman, 1998;Lurie and Swaminathan, 2009).
Indeed, there is also some evidence in the economics literature that people can engage in a type of learning inconsistent with strategy learning when playing games repeatedly without any feedback. Weber (2003) conducted an experiment in which subjects played a dominance-solvable game 10 times without any outcome information between plays of the game. Across several treatments, significant learning occurredbehavior converged towards equilibrium. 1 Such learning cannot be considered strategy learning, which requires regular feedback. Although it is impossible to conclude that the feedback-free learning observed in Weber (2003) was meaningful without examining whether such improvements in performance transfer to a new game, these findings combined with the relevant psychological research suggest that withholding feedback in games may stimulate meaningful learning.
The main hypothesis for our research is that withholding feedback in a game played repeatedly will produce meaningful learning that will transfer to the first period of a new but strategically similar game. 2 We focus on iterated dominance as the principle that is learned and transferred. We conduct three experiments, all of which demonstrate that the feedback-free learning that occurs in earlier dominance-solvable games transfers 1 A handful of other papers also provide support for the notion that people can learn in environments where they make repeated choices without feedback (Grether, 1980;Cason and Mui, 1998;Rapoport et al., 2002). However, none of these studies directly explores this phenomenon or focuses on learning. 2 We focus on "immediate transfer" (to the first period of the new game) because such one-shot behavior is the focus of much of the research on iterated rationality (e.g., Costa-Gomes, Crawford and Broseta 2001) and also because it is a good way to measure prior learning in games (Merlo and Schotter 1999). to later dominance-solvable games. Moreover, our second and third experiments demonstrate that the kind of (strategy) learning that typically occurs with regular outcome and payoff feedback fails to yield similar transfer. We interpret this difference in transferability as evidence that the kind of learning produced by feedback-free repetition is more meaningful than the learning produced with immediate feedback.
We also explore, in our third experiment, why withholding feedback might stimulate meaningful learning (or, equivalently, why providing feedback impairs the development of meaningful learning). Goodman (1998) proposed that "external feedback," or learning the correct answer to a problem one just attempted to solve, can impair meaningful learning by diverting attention from "response-produced feedback," or what is learned by simply performing a task (see also Anzai and Simon, 1979). Similarly, in a meta-analysis of studies that examined the effectiveness of external feedback, Kluger and DeNisi (1996) found that the ability of external feedback to induce meaningful learning decreases as it moves attention away from important features of the task being performed. 3 If feedback impairs meaningful learning by reducing the ability and motivation to think carefully about the task (or game), then interventions that force people to think more deeply about their decisions should reduce the negative influence of feedback on the development meaningful learning. Requiring people to generate "self-explanations," or engage in verbalization during problem-solving, is one commonly employed method for inducing deeper thought (Gagne and Smith, 1962;Chi et al., 1994). Therefore, in our third experiment we vary in a first game both whether feedback is withheld and whether players are required to engage in self-explanation as they play the game (in the form of written explanations for behavior). If withholding feedback stimulates the type of deeper thinking that occurs when prompted to self-explain, both interventions should have similar and substitutable effects on meaningful learning. This is what we find.

Experiment 1
Experiment 1 examined whether it is possible to produce meaningful learning, measured by transfer across games, when playing games repeatedly without feedback.
Subjects played four normal-form games repeatedly without any feedback until the end of the experiment. They played each game 20 times before proceeding to the next game, and the order of games varied across sessions. Table 1 presents the four games used in the experiment. The payoffs represent points, with 200 points equal to $1.
Game A is a symmetric stag-hunt game with three Nash equilibria: the Paretodominant equilibrium (Top/Left), the risk-dominant equilibrium (Bottom/Right) (Harsanyi and Selten, 1988), and a mixed-strategy equilibrium in which subjects play Top/Left with probability 0.56. Games B, C, and D, on which we will focus, have unique equilibria resulting from iterated deletion of dominated strategies. 4 In Game B, Bottom is dominated (step 1) and then Left is dominated once Bottom is removed (step 2), producing the equilibrium prediction of Top/Right. For Game C, which is symmetric, iterated dominance eliminates Bottom/Right (step 1) and Top/Left (step 2), resulting in the equilibrium of Middle/Middle. Finally, in Game D iterated dominance eliminates Left 4 We included Game A to test one possible interpretation of feedback-free learning -that a player's change in behavior results from best responding to her own prior choices. Since we find little evidence of this phenomenon (players in fact regularly move away from a best-response to their own prior strategies), we focus our attention primarily on the three dominance-solvable games.

A. Experimental Design
There were four sessions. In each session 18 to 20 subjects played the above four games in 20-period blocks. The sequence of games is presented in Table 2.
Each subject sat at a computer monitor. Subjects' roles (Row or Column) were fixed and each subject was anonymously paired with someone of the opposite role for the duration of the experiment. Matching and role assignment were done by random assignment of participant numbers. At the beginning of a session, subjects received extensive instruction in how to interpret game matrices with generic payoffs. 5 At the beginning of each 20-period block, the computer displayed the game matrix for those periods. The experimenter read all of the payoffs in the matrix aloud.
Subjects then proceeded through each of the 20 periods by clicking on a choice. After each choice, the computer screen froze, displaying the subject's choice and the matrix for 20 seconds. Subjects received no payoff information until the end of the experiment.
Subjects were recruited from an e-mail list of graduate and undergraduate students at the University of Pittsburgh. The experiment lasted approximately 1.5 hours.
Subjects were informed at the beginning of the experiment that their earnings would be determined exclusively by the points they accumulated during the 80 rounds of play. At the end of the experiment, subjects were privately paid one at a time.

B. Results
We first focus on learning within games, pooling across sessions, to examine whether we also find evidence of the feedback-free learning observed in Weber (2003).
We then turn our attention to whether there is any transfer of learning across the dominance-solvable games.

Learning within games
The aggregate choice frequencies -by 5-period blocks -are presented in Table 3. Table 4 presents logistic regressions that systematically examine changes in behavior across periods. The first four regressions explore the extent to which subjects play Nash equilibrium in the three dominance solvable games and the risk-dominant equilibrium in Game A. The change in behavior across periods is significant in all four games.
Subjects clearly learn in our experiment -their behavior shows systematic change in the direction of the predictions of rationality. However, it is also important to understand whether what they are learning corresponds to meaningful principles. The dominance-solvable games allow us to explore this issue.
To explore whether the above learning involves the acquisition of iterated dominance, we consider behavior consistent with the first two steps of this principle in the three dominance-solvable games. If subjects learn to avoid dominated strategies ( Table 4 examine D1 and D2 violations across periods. The frequency of D1 violations decreases significantly for all three games, while for D2 the decrease is significant in Games C and D.
We find that subjects learn when they play games repeatedly without feedback, and that this learning appears to involve the acquisition of iterated dominance. But to more definitively determine whether subjects are acquiring a meaningful understanding of iterated dominance, we next examine whether this acquisition transfers across games.

Transfer across games
Games B, C, and D all have in common the applicability of two steps of iterated dominance. To test whether learning transfers across games, we compare the frequency with which subjects violate these principles by games' positions within sessions. Table 5 presents, by game position, the frequency with which subjects violated the first two steps of iterated dominance. The table compares behavior in exactly the same three games. 7 The frequencies of both kinds of violations decrease with game position. For both D1 and D2, roughly 10 percent fewer choices violated the principle in the last game than in the first. 8 This can also be seen in Table 6, which presents logistic regressions of the frequency with which subjects violate each of the two principles by game position within a session (i.e., the "Game Position" variable takes on values from 1 to 4). The coefficients on Game Position are negative and significant; thus, subjects are less likely 6 In Games B and D, D2 only applies to players in one of the two roles (column in B, row in D), so we can look directly at behavior consistent with D2. In Game C, which is symmetric, the principle implies that subjects should neither play dominated strategies (Bottom/Right) nor strategies dominated in the second step (Top/Left). An alternative approach for Game C would be to label a subject as "not violating D2" if she chose either Bottom or Right, but this would allow a violation of D1 to count as not violating D2. 7 The number of subjects varies across cells because session 4 had a different number of participants. 8 The magnitude of the transfer between the first and last game in a session is roughly 1.5 times greater than the within-game learning we found between the first and last five blocks in a game. to violate the two principles when playing games later in a session. 9 Another consequence of meaningful learning is that, once a subject learns to identify dominated or iteratively dominated strategies, (s)he should cease to play such strategies for the remainder of the experiment. Therefore, we also consider the number of subjects in the first and last games who never violated each of the first two steps of iterated dominance. For D1, 12 of 37 subjects (who had an opportunity to do so) never played a dominated strategy in their first game, but in the fourth game this proportion goes up to 21 of 36 (χ 2 (1) = 4.94, p = 0.03). Similarly, every subject violated D2 at least once in their first game (0 subjects with no violations). However, in the last game, 9 of 36 subjects never played such strategies (χ 2 (1) = 10.55, p = 0.001). It appears that many subjects -at some point in the experiment -learned to apply iterated dominance, and continued to apply this principle throughout the remainder of the experiment.
We also find evidence of "complementary acquisition" of the two steps of iterated dominance (i.e., subjects need to acquire D1 before they can acquire D2). We classify subjects according to whether they "never violated," "stopped violating," or "never acquired" a principle throughout the experiment (with the last classification for subjects who violated the principle in at least one of the last 5 choices in which they could do so).
The number of subjects who never violated D1 but violated D2 at least once (19) is considerably higher than the number who never violated D2 but violated D1 at least once (1). Similarly, of the 19 subjects who never acquired D1, only 1 never violated or stopped violating D2, but of the 44 subjects who never acquired D2, 26 never violated or stopped violating D1. Both of these comparisons are consistent with the fact that acquisition of D2 requires prior understanding of D1.

C. Discussion
When playing different games repeatedly without feedback, subjects learn to stop violating the first two steps of iterated dominance and transfer such learning to later games. This transfer of learning stands in contrast to the limited evidence of cross-game transfer in previous experiments that provide regular outcome and payoff feedback. The results thus suggest that withholding feedback might stimulate the kind of meaningful learning that transfers to new games.
However, an obvious limitation is that Experiment 1 did not include a comparison of the kinds of learning that occur with and without feedback. Such a comparison can determine whether withholding feedback is in fact more likely to stimulate meaningful learning than providing feedback. In Experiment 2, we perform this comparison.

Experiment 2
Experiment 2 examines whether meaningful learning occurs to a greater extent when a first game is played with no feedback than when it is played with payoff and outcome feedback at the end of every period. We measure meaningful learning by the extent to which learning transfers to the first period of a new game and by the extent to which it is consistent with the acquisition of iterated dominance.
The experiment uses a procedure similar to that of Ho et al. (1998). We use two versions of Nagel's (1995) competitive guessing ("p-beauty contest") game. In the game, N players each choose a number in a given range ( ] , [ s s s i  ). The average of the N numbers is then multiplied by a constant (p) to obtain a target number. The player whose choice is the smallest absolute distance from the target number wins a fixed prize.
We use one version of the game with p < 1 (p = 0.  Table 7). We varied the feedback provided between plays of the first game. In the Feedback treatment, subjects received outcome and payoff information at the end of each period, as in Ho et al. In the No Feedback treatment, subjects did not receive any feedback between periods of the first game -they found out first-game outcomes only after completing the second game.
Following 10 periods of the first game, either with or without feedback, subjects played the second game with feedback. Since iterated dominance applies in both games, we are interested in whether what is learned in the first game produces immediate transfer to the second game. We predict that initial (Period 11) choices in the second game will deviate less from Nash equilibrium than the choices of inexperienced subjects (in Period 1), thus exhibiting positive transfer, but only in the No Feedback treatment.

A. Experimental Design
Participants were Carnegie Mellon and University of Pittsburgh students. There were 16 sessions. In each session 8 to 10 subjects played two versions of the competitive guessing game -IT and FT -for 10 periods each. The winner in each period received $3, and in case of a tie this amount was equally divided among the winners. Subjects played one game first and then received instructions for the other, and the order of games was counterbalanced. Aside from the game order, sessions also varied by whether the first game was played with or without feedback. Table 7 presents the different kinds of sessions. The second game was always played with feedback.
At the beginning of the experiment, subjects received instructions describing general procedures and the first version of the game. After asking any questions, subjects proceeded to the first period. In each period, subjects recorded their choices on a table at the bottom of their instruction sheet. The experimenter recorded these choices and entered them in a laptop, which computed the outcome and payoffs for that period.
In the Feedback treatment, subjects received feedback after every period -the average, target number, and participant number(s) of the winner(s) were written on the board and read aloud, and subjects recorded this information. In the No Feedback treatment, the experimenter determined the outcome, but this information was not revealed to subjects. Instead, the experiment proceeded to the next period.
In both the Feedback and No Feedback treatments, subjects played the first game for 10 periods. After the 10th period, subjects received a new instruction sheet that described the other version of the game. They then played 10 periods of the second game, with feedback after each period. Following the second game, subjects in the No Feedback treatment received full outcome information for the first game. Subjects were then paid privately.

B. Results
In presenting the results, we first briefly examine whether learning occurred in the first game. We then explore whether there is transfer of learning to the first period of the second game and whether such learning differs by feedback treatment.   10 Thus, we compare first-period choices in the second game from one sequence (when subjects have experience with the other game), to first-period choices in the first game from the other sequence (in which subjects play the same game, but without any prior experience). We also pool first-game choices from the two feedback treatments (i.e., pooling the first row of Table 8 by game), which proceeded identically up to the end of the first period (when one treatment received feedback and the other did not). If we compare first-period choices by feedback treatment, we find no significant differences between the Feedback and No Feedback treatments (IT: t 74 = 0.79; FT: t 74 = 1.53). 11 In later periods of the FTIT treatment, choices move away from Nash equilibrium. In a few sessions, subjects began experimenting with choices of 100 (perhaps out of boredom or to try to gain a strategic advantage -cf. Weber, 2003). In some cases this occurred in consecutive periods, raising the average substantially and persisting into subsequent periods. In the No Feedback treatment, these choices of 100 started to occur somewhat earlier in two sessions (contributing to the slower convergence towards zero).  Table 8), which are both significantly closer to Nash equilibrium than the corresponding choices of inexperienced subjects (IT: t 111 = 2.63, p = 0.005, one-tailed; FT: t 112 = 1.86, p = 0.03, one-tailed).
We also observe evidence of transfer in the No Feedback treatment in Table 9 (right column). In Periods 1-3 of the first game, 19 percent of subjects violated the first step of iterated dominance at least once and 56 percent did so for the second step. In Periods 11-13, however, these proportions decrease to 12 percent and 31 percent, respectively. While the decrease for the first step is not significant (χ 2 (1) = 1.28), the decrease in violations for the second step is (χ 2 (1) = 9.80, p = 0.002). 12 Thus, we find immediate transfer to the second game when the first game is played without feedback, even though we observe no such transfer when the first game is played with feedback. One caveat worth highlighting is that, as Figure 1 13 An anonymous reviewer suggested an alternative interpretation of our main results in Experiment 2, which Table 10 helps rule out. This interpretation, based on "anchoring and adjustment" (Tversky and Kahneman, 1974), posits that subjects simply anchor on, and adjust from, their Period 10 choice when making their Period 11 choice; the adjustments are approximately equal across treatments, and No strategies close to equilibrium at the end of the first game also do so at the beginning of the second game. This is robust to controlling for subjects' Period 1 choices (how much knowledge they had at the beginning of the experiment). Thus, subjects who appear to "learn" a principle by the end of the first game are more likely to behave consistently with that principle in the second game in the No Feedback treatment, than in the Feedback treatment. 14 Thus, when we move beyond a simple comparison of means in the second game, we find little evidence that subjects in the Feedback treatment are applying something they learned in the first game to the beginning of the second game, while for No Feedback subjects we do find such a relationship.

C. Discussion
Consistent with much previous research on strategy learning, the kind of learning obtained with feedback does not immediately transfer across strategically similar games.
But the kind of learning produced by repeated play without any feedback transfers to the first period of the second game. Thus, we find that meaningful learning is more likely to occur when feedback is withheld than when feedback is given.
Feedback subjects are closer to equilibrium in Period 11 simply because they were farther from equilibrium in Period 10. Note that this account would predict a negative correlation between distance from equilibrium in Period 10 and Period 11 for both treatments. But as Table 10 reveals, the relationship is positive for both treatments, and significantly so for the No Feedback treatments. 14 We observe a similar pattern if we examine the relationship between violations of iterated dominance in the last three periods of the first game and the first three periods of the second game. In the Feedback treatment, 74 subjects never violated dominance in the final three periods of the first game (see Table 9). Of these, 17 (23 percent) violated this principle at least once in the first three periods of the second game. However, in the No Feedback treatment this proportion is lower (9 of 72, or 13 percent; χ 2 (1) = 2.73, p = 0.10). For the second step of iterated dominance, the proportion of Period 8-10 non-violators who violated this principle in Periods 11-13 is also higher in the Feedback treatment (32 of 74; 58 percent) than in the No Feedback treatment (14 of 61; 23 percent; χ 2 (1) = 6.13, p = 0.01).

Experiment 3
The results obtained thus far suggest that feedback-free repetition results in meaningful learning that transfers across games (Experiments 1 and 2), even when feedback-based learning does not (Experiment 2). Experiment 3 examines why meaningful learning is more likely to occur when feedback is withheld than when feedback is given.
Prior psychological research has proposed that feedback can reduce people's ability and motivation to think carefully about a task (Einhorn, 1980;Goodman, 1998). If so, then interventions that force people to think carefully about the task at hand should moderate the influence of feedback on meaningful learning. "Self-explanation" is one such intervention: people who explain to themselves why they are doing what they are doing tend to think more deeply about the task at hand than people who do not generate such explanations (e.g., Chi et al., 1994). Such explanations also tend to facilitate the development of knowledge that transfers across related tasks (Gagne and Smith, 1962 In the experiment, subjects initially played 10 periods of the Finite Threshold (FT) competitive guessing game from Experiment 2. We varied whether or not subjects received feedback at the conclusion of each round, as well as whether or not subjects were asked to explain their decisions and the (revealed or predicted) decisions of others during each period. At the conclusion of these 10 periods, subjects played a single period of the Infinite Threshold (IT) game. As in Experiment 2, we measure meaningful learning by the extent to which choices in the second game deviate from Nash equilibrium. If feedback-free repetition stimulates people's ability and motivation to think carefully about a game, then its effects should be similar to those produced through selfexplanation. Therefore, we predict that the presence of either feedback-free repetition or self-explanation should facilitate meaningful learning, but that their combination should produce no additional benefits.

A. Experimental Design
Participants were University of Pennsylvania students. In each session 7 to 10 subjects played 10 periods of the FT competitive guessing game, followed by one period of the IT competitive guessing game. The winner in each period received $2, and in case of a tie this amount was equally divided among the winners.
At the beginning of the experiment, subjects received instructions describing general procedures and the FT game. After asking any questions about the game, subjects proceeded to the first period. The experiment was computerized, using the z-Tree software (Fischbacher, 2007), and in each period subjects typed their choices and recorded them on a table at the bottom of their instruction sheet.
In the first game, we varied whether or not subjects received feedback, as well as whether or not subjects were asked to explain their own decisions and the (revealed or predicted) decisions of others. In the Feedback (F) treatments, the computer revealed the average, target number, and the amount of money earned at the end of each round.
Subjects recorded this information on a record sheet before proceeding to the next round.
In the No Feedback (NF) treatments, the computer asked subjects what they thought the average choice was at the end of each round. Subjects typed this number and recorded it on a record sheet before proceeding to the next round. 15 In the Explanation (E) treatments, subjects were asked to enter a choice and then, on the same screen, to "type a sentence or two indicating why you made the particular treatments, subjects made their decisions and either received feedback or guessed the average, but were not prompted to explain their own decisions or the decisions of others.
Following the 10th period, subjects received new instructions that described the IT game. After asking any questions, subjects played one period of the game. Subjects did not explain their decisions or the decisions of others, and all subjects received feedback at the end of the period.
After the second game, we administered the short form of the Need for Cognition scale (Cacioppo, Petty, and Kao, 1984), which measures individual differences in the tendency to engage in and enjoy thinking. 16 Next, No Feedback subjects received all of the outcome information for the FT game. Subjects were then paid privately. Table 11 presents the number of sessions and subjects per treatment. 17 We first briefly examine whether learning occurred in the first game. We then explore whether there is transfer of learning to the second game and, more importantly, whether selfexplanation moderates the influence of feedback on transfer.

B. Results
1. Learning in the first competitive guessing game Figure 2 presents the mean choices across periods, by treatment. As the figure reveals, there is convergence towards equilibrium in all treatments. Average choices move significantly in the direction of equilibrium between the first and tenth periods in all treatments (NF+NE: 11.0, t 49 = 2.17, p < 0.04; F+NE: 33.9, t 56 = 9.60, p < 0.0001; 16 Since individuals high in Need for Cognition may be more likely to spontaneously engage in selfexplanation, we included the scale to control for this potentially important individual difference. 17 We excluded one session (N = 10) from the analysis. Partway through this session, the experimenter had to leave abruptly, and the remainder of the session was administered by a lab assistant. One subject in this session, either due to boredom, misunderstanding, maliciousness, or some combination of these factors, consistently chose numbers whose individual digits summed to or included seven (e.g., round 6: 131.11, reason given: "seven. I was trying to reach seven"; round 8: 124, "1+2=3 3+4=7"; round 11: 77.77). Because this was a Feedback session, this subject's erratic behavior coupled with the experimenter leaving raised suspicion among other subjects about whether the game was real (e.g., subject 5 in round 9: "Perhaps there is a computer player choosing random numbers"; subject 1 in round 10: "This is real, right?"). An analysis of data from this session clearly reveals it to be an outlier relative to other sessions.

C. Discussion
Experiment 3 sheds light on the process by which withholding feedback stimulates meaningful learning. Self-explanation significantly enhances meaningful learning and transfer, but only in the Feedback treatment. Under No Feedback, selfexplanation has no effect. Given that self-explanation has previously been demonstrated to deepen thinking, this interaction suggests that the enhancement of meaningful learning produced in the No Feedback treatments was the result of a process similar to that which 18 Similar results are obtained when examining violations of iterated dominance. For example, the proportion of F+NE subjects violating the second step of iterated dominance in period 11 is only insignificantly smaller than the proportion violating the second step of dominance at least once in periods 1-3 (42% vs. 51%; χ 2 (1) = 0.88, p = .35). However, this difference is significant in NF+NE (20% vs. 54%; χ 2 (1) = 12.40, p < .001), NF+E (21% vs. 52%; χ 2 (1) = 8.64, p = .003), and F+E (19% vs. 61%; χ 2 (1) = 12.99, p < .001).
was produced by self-explanation. Put differently, our No Feedback treatments appear to facilitate the same kind of deeper thinking that is produced through self-explanation.

Conclusion
Game theorists are beginning to devote significant attention to the process by which players acquire generalizable knowledge that transfers to strategically similar settings (e.g., Stahl, 2000a,b;Rankin et al., 2000;Cooper and Kagel, 2003, 2005. We contribute to this literature by demonstrating that withholding feedback in repeated play stimulates meaningful learning and cross-game learning transfer in dominancesolvable games, including normal-form games (Experiment 1) and competitive guessing games (Experiment 2). We also demonstrate that withholding feedback is more likely to stimulate meaningful learning than providing regular outcome feedback (Experiment 2), and that the effects of withholding feedback are very similar to those produced by selfexplanation, another intervention known to facilitate deeper thinking (Experiment 3). 19 Our work demonstrates that withholding feedback, a manipulation previously employed in game theory experiments to eliminate learning (e.g., Costa-Gomes and Crawford, 2006), can have counterintuitive effects, namely the facilitation of meaningful learning. While games played only once or twice each (as in Costa-Gomes and Crawford) might not yield meaningful learning when played without feedback, we demonstrate that such learning can occur with 10 or 20 repetitions of a game. 20 Since our work is based on a large body of research in psychology, we present an opportunity for game theory to further integrate knowledge from other disciplines. For instance, one goal of subsequent research should be to further develop theoretical models that account for the two kinds of learning. Turning to the existing theoretical literature in psychology may be helpful in this regard. Psychological models that allow for two kinds of learning (e.g., ACT-R; Anderson and Lebiere, 1998) present a potential starting point.
It is also worth highlighting that our work does more than just replicate what psychologists already know about learning. To the best of our knowledge, none of the related prior psychological work paid subjects based on their performance, which is an important factor for facilitating learning, presumably by increasing effortful thinking (cf. Camerer and Hogarth, 1999), and which could have therefore mitigated the feedbackbased effects on learning typically observed in psychology experiments. We find that the benefits of withholding feedback for developing meaningful transfer exist even when subjects are motivated by monetary incentives. Additionally, none of this prior research explored learning in strategic contexts, such as games. Finally, our third experiment sheds new light on why withholding feedback stimulates meaningful learning, a contribution to both economics and psychology.
Of course, the procedure we use -feedback-free repetition -is only one way to develop meaningful learning, and there may be other, potentially better, methods. For example, would providing time for introspection function as well as feedback-free repetition? Some models in economics propose that players can develop improved reasoning ability by introspecting prior to playing a game (Goeree and Holt, 2002; iterated dominance in the first three periods of the second game in Experiment 2). But our results do demonstrate that, in certain situations, meaningful learning is more likely to occur when feedback is withheld than when it is provided. MacLeod, 2002;Capra, 2003). However, it is also possible that time alone might not produce meaningful learning. For example, empirical research in psychology suggests that people tend to postpone thinking concretely about specific aspects of a situation until it is imminent (Trope and Liberman, 2003). This work suggests that, no matter how long people have to think about a game they are about to play, they may not think about it carefully until they actually begin to play.
Moreover, there may be conditions under which meaningful learning develops even when feedback is provided. Experiment 3 demonstrated one such condition (selfexplanation), but there may be others. In Experiments 2 and 3, subjects played one game and were then "surprised" by a second, strategically similar game (cf. Merlo and Schotter, 1999). It would be interesting to examine whether warning subjects in advance that they will play a different but related game in the future allows meaningful learning even with feedback. Moreover, previous research in psychology demonstrates that delayed feedback is useful for producing the kind of learning that transfers, and a considerable literature explores the optimal timing of feedback (e.g., Schmidt et al., 1989;Erev et al., 2006). Therefore, subjects in our experiments might demonstrate even greater meaningful learning if presented with feedback after some such "optimal" delay. 21 Future work should also examine the extent to which the benefits of feedback-free repetition generalize across contexts. Games like the competitive guessing game are contexts in which players do not need meaningful learning to perform well when feedback is provided. These are precisely the kinds of contexts in which providing feedback should inhibit transfer, and therefore our feedback-free environments are useful for facilitating meaningful learning. By contrast, we should observe less of an advantage for feedback-free learning in games and environments where meaningful learning is required to perform well even when feedback is provided. Future work should also examine whether feedback-free repetition facilitates meaningful insight into gametheoretic principles other than dominance (e.g., backward-induction).
Our experiments provide a useful starting point -along with other recent workfor further exploring distinctions among different kinds of learning in games, and the source and consequences of such a distinction. But much remains to be done before this distinction can be fully incorporated into economics and game theory.     Standard errors in parentheses * -p < 0.1; ** -p < 0.05; *** -p < 0.01; a -one-tailed Standard errors in parentheses * -p < 0.1; ** -p < 0.05; *** -p < 0.01; all one-tailed