Studying technology-based strategies for enhancing motivation in mathematics

Abstract

Background

During the middle school years, students frequently show significant declines in motivation toward school in general and mathematics in particular. One way in which researchers have sought to spark students’ interests and build their sense of competence in mathematics and in STEM more generally is through the use of technology. Yet evidence regarding the motivational effectiveness of this approach is mixed. Here we evaluate the impact of three brief technology-based activities on students’ short-term motivation in math. 16,789 5th to 8th grade students and their teachers in one large school district were randomly assigned to three different technology-based activities, each representing a different framework for motivation and engagement and all designed around an exemplary lesson related to algebraic reasoning. We investigated the relationship between specific technology-based activities that embody various motivational constructs and students’ engagement in mathematics and perceived competence in pursuing STEM careers.

Results

Results indicate that the effect of each technology activity on students’ motivation was quite modest. No gains were found in self-efficacy; for implicit theory of ability, a lower incremental view of ability was found; we found modest declines in value beliefs. With respect to math learning, students in all three inductions had modest improvements in their scores on the math learning measure. However, these effects were modified by students’ grade level and not by their demographic variables. In addition, teacher-level variables did not have an effect on student outcomes.

Conclusions

The present findings highlight the importance of tailoring motivational experiences to students’ developmental level. Our results are also encouraging about developers’ ability to create instructional interventions and professional development that can be effective when experienced by a wide range of students and teachers. Further research is needed to determine the degree, duration of, and type of instructional intervention necessary to substantially impact multi-dimensional, deep-rooted motivational constructs, such as self-efficacy.

Success in algebra during the middle grades is widely recognized to be a critical gatekeeper that constrains students’ decisions about whether to pursue further educational opportunities in Science, Technology, Engineering, and Mathematics (STEM) fields (Adelman 2006). Unfortunately, during this developmental period many students show significant declines in motivation toward school in general and mathematics in particular (e.g., Archambault et al. 2010; Blackwell et al. 2007). One way that researchers have sought to spark students’ interests and build their sense of competence in mathematics is through the use of various technological media. These technologies have ranged in complexity and cost from the simple and inexpensive, such as repurposing television programs, to the more complicated and expensive, such as specially designed mathematical experiences based on immersive virtual environments and computer games.

Despite the widely accepted notion that all technology-based activities are inherently engaging, the evidence regarding their motivational effectiveness is mixed (Moos and Marroquin 2010). Part of the reason may be that many different types of technologies are available, and each can be designed well or poorly to leverage various aspects of motivation (e.g., engagement, self-efficacy, tenacity) in different ways. Another explanation for these mixed findings is that much of the research on technology-based activities considers motivation as a unidimensional construct intrinsically generated by technology usage rather than as a construct with multiple dimensions that may be impacted via various affordances of technology. This latter reason may be due to many developers lacking strong theoretical grounding in well-studied motivation constructs (Chen et al. 2013; Moos and Marroquin 2010).

As a step toward improving our understanding of the potential impact of technology-based activities on students’ motivation in mathematics, the goal of this project was to investigate the relationship between (a) specific technology-based activities that exemplify various motivational constructs, (b) students’ engagement in mathematics and perceived competence in pursuing STEM careers, and (c) students’ mathematics learning from a short algebra lesson. As part of a four-day school-based intervention, students in grades 5 to 8 in a large school district were randomly assigned to three different technology-based activities, each representing a different framework for motivation and engagement designed around an exemplary lesson related to the learning of algebra.

Our research questions were as follows. First, what is the impact of the four-day intervention on students’ motivation in mathematics, including interest in pursuing STEM careers? Second, to what extent is this impact influenced by factors such as the type of technological induction the students received and/or students’ demographic and academic characteristics (e.g., gender, race/ethnicity, prior achievement)? Third, to what extent is this impact influenced by teacher-level factors such as credentialing in mathematics education, undergraduate major, years of experience, and teachers’ beliefs (e.g., teaching self-efficacy)?

We begin by reviewing evidence on how and why technology-based activities might impact students’ motivation in STEM fields.

Background

As the National Academy of Sciences ( 2011) indicated, certain key ingredients are relevant for students who want to pursue STEM careers. These ingredients include a robust confidence in math and science capability, the ability to see one’s abilities in STEM as able to improve over time, and the ability to develop a passion or sustained interest in becoming a scientist or engineer. Within the educational psychology literature, these key ingredients translate into three constructs, each of which has received substantial attention in the field of motivation: self-efficacy, implicit theories of ability, and value beliefs. We discuss each in turn.

Capable students plagued by a loss of confidence about their capacity to succeed in math and science typically avoid careers that require a strong background in those subjects (Lent et al. 2005). Decades of research have shown that students’ self-efficacy, defined by Bandura ( 1997) as “the belief in one’s capabilities to organize and execute courses of action required to produce given attainments” (p. 3), is a powerful influence on motivation and achievement. Bandura ( 1997) hypothesized several sources of self-efficacy, including mastery experience (the interpreted results of one’s past performance), vicarious experience (observations of others’ activities, particularly individuals perceived as similar to oneself), and physiological and affective states (anxiety, stress, and fatigue) – each of which has been linked to performance in math and science, including students’ persistence in STEM fields and choice of STEM majors (Andrew 1998; Beghetto 2007; Britner and Pajares 2001; Chen and Usher 2013; Gwilliam and Betz 2001; Lau and Roeser 2002; Lent et al. 1984; Luzzo et al. 1999).

Accumulating evidence demonstrates that underrepresentation of women and racial/ethnic minorities may be substantially explained by considering the sources of self-efficacy. For example, Lent et al. ( 1991) found that gender differences in math self-efficacy could be accounted for by students’ mastery experiences, suggesting that women viewed their past experiences with math and science in a more negative light than did their male counterparts. Zeldin and Pajares ( 2000) found that women’s decision to stay in the STEM pipeline could be attributed to the (vicarious) role models with whom they strongly identified, as well as the powerful social persuasions that came from women’s most trusted sources (e.g., a mentor). Men, however, drew mostly from their mastery experiences—discussing their past successes and accolades as reasons for staying in the STEM pipeline. Therefore, in influencing students’ participation in STEM fields, educators would be wise to look toward the sources that feed each individual student’s self-efficacy to pursue such careers.

Like self-efficacy, implicit theory of ability (defined as a belief about the nature of intellectual ability (Dweck and Leggett 1988)) plays an important role in motivation. Some individuals believe that their abilities are a fixed characteristic, and that nothing can be done to change that (i.e., “I’m not smart in math, and there isn’t anything I can do about it”). This is referred to as a fixed theory of ability. On the other hand, other individuals believe that, with sufficient effort and the proper strategies, one can become more able (i.e., “If I work hard in my math class, I can get smarter in math”). This is known as a incremental theory of ability. A large body of research has shown that implicit theory of ability plays a key role in students’ academic motivation, achievement, and career choices (Blackwell et al. 2007; Chen and Pajares 2010; Chen 2012; Cury et al. 2006; Good et al. 2012; Grant and Dweck 2003; Hong et al. 1999; Stipek and Gralinski 1996). For example, Blackwell et al. ( 2007) found that, although Grade 7 math students’ who held a fixed theory of ability and those who held an incremental theory of ability both started at the same level of math achievement, by the end of the two years students who held an incremental view of ability achieved higher grades in math than did their fixed theory peers. Related, other work has suggested that teachers’ beliefs about the nature of intelligence may promote students’ conceptions of ability (Good et al. 2012; Rattan et al. 2012) and that gender and ethnicity may influence students’ conceptions of ability (Good et al. 2003).

If, as Dweck and her colleagues have suggested, an incremental theory of ability can serve a protective function for students’ motivation and achievement, it would benefit researchers and educators to know what the sources of such a belief are. Little research has investigated this topic, however. Some studies suggest that process feedback highlighting the strategies and effort that lead to success can promote the view that ability is augmentable, whereas product feedback highlighting the accomplishments, but leaving out the perseverance required to get there, promotes a fixed view of ability (see Dweck and Master 2009 for a review).

In addition to the self-efficacy and implicit theories of ability, value beliefs are also a significant determinant in students’ motivation and achievement (Eccles Parsons et al. 1983). Value beliefs in mathematics and science deal with the question, “Do I want to pursue more opportunities in mathematics and science?” Eccles et al. defined values as being composed of several distinct constructs. First, students’ interest or intrinsic value can affect the activities they pursue—activities that are more enjoyable are more likely to be pursued than are activities that are perceived to be lackluster. Second, students’ perceptions of the utility of an activity refer to how valuable students perceive an activity to be. If an activity is perceived to be a steppingstone toward students’ desired future endeavors, then students are more likely to pursue it. Finally, doing well in mathematics and science may influence students’ identity or feelings of self-worth. This attainment value describes how important doing well in mathematics and science is to students’ identity or feelings of self-worth.

Numerous studies have found that interest value predicts STEM career choice (Lent et al. 2008; Lent et al. 2010), as well as choice in taking STEM courses (Eccles et al. 1984; Watt et al. 2006). Attainment value in mathematics and science is closely aligned with students’ mathematics and science identity. The empirical literature supports that persistence and success in STEM careers may be rooted in students’ identification with the roles and work of STEM professionals (Bonous-Hammarth 2000; Estrada et al. 2011; Hernandez et al. 2013). As such, attainment value predicts students’ persistence in STEM careers (Carlone and Johnson 2007; Oyserman and Destin 2010).

Empirical literature also supports the notion that students’ utility value predicts STEM success and choices. For example, Maltese and Tai ( 2011) found that students who perceived science to be useful were more likely to major in STEM subjects in college. Some have found convincing students that mathematics is useful for their future endeavors increased the interest of students only if they had high expectancies for success; those who expected to do poorly lost interest. However, Hulleman et al. ( 2010) found that, instead of telling students about the importance of an activity, if students discovered the usefulness of an activity on their own, the interest of those who had low expectations for success increased. For those whose expectancies for success were already high, no changes in interests were observed. Therefore, utility value can be influenced if students discover the utility of a subject on their own, with positive consequences for motivation and achievement.

Motivation and technology

How can the constructs described above be targeted through technology-based educational experiences to support the motivation of students in mathematics and science? Although the literature on technology and motivation is quite large, relatively few of these studies employ frameworks that are grounded in well-studied psychological theories of motivation (Moos and Marroquin 2010). Moos and Marroquin noted that the results about the effectiveness of technology as a motivational tool are mixed. One might expect lackluster outcomes if technology is applied as a “secret sauce” to automatically enhance students’ engagement, rather than utilized in a principled manner to help an individual to find a robust sense of confidence in math and science capability, see his or her abilities in STEM as able to improve over time, and develop an interest for becoming a scientist or engineer.

With regard to self-efficacy, there is some evidence that engagement with innovative technology in academic settings can positively impact self-efficacy toward STEM. For example, Ketelhut and colleagues (Ketelhut 2007; Ketelhut et al. 2010) found that students’ self-efficacy for scientific inquiry before using a Multi-User Virtual Environment (MUVE) called River City was related to their behaviors within the virtual world. In particular, less self-efficacious students manifested a self-efficacy boost through mastery experiences gained through engagement in the activities of the MUVE. Similarly, Liu et al. ( 2006) explored middle school students’ science learning within a computer-enhanced Problem Based Learning (PBL) environment called Alien Rescue and found that students’ achievement and self-efficacy increased after participating in Alien Rescue.

Building on studies such as these, one additional promising avenue in exploring how innovative technologies can be used to tap the sources of self-efficacy deals with the capability to use virtual representations of the self (avatars) in creative ways. For example, Fox and Bailenson ( 2009) reported that, when individuals watched a virtual representation of themselves experiencing the benefits of exercising, these individuals were significantly more likely to engage in exercise after the intervention was done. In contrast, individuals who watched virtual representations of themselves loitering did not engage in exercise after the intervention nor did individuals who watched a virtual representation of others. As another example, Rosenberg-Kima et al. ( 2008) reported greater gains in self-efficacy for pursuing engineering careers when participants saw virtual avatars on a computer interface who looked similar to themselves. These results suggest that virtual models of a person successfully attempting a task can be effective in shaping a person’s self-efficacy and behavior.

Technology also seems to be a promising avenue for impacting implicit theory of ability. In particular, Dweck and her colleagues have developed a web-enabled intervention, Brainology®, based on the paper and pencil version of their curriculum materials designed to enhance implicit theory of ability. Students are introduced to two cartoon characters who guide them through the web-based environment, where they learn about the functions of the brain, including that the brain is like a muscle—with conditioning, it can get stronger – an attitude which is linked to an incremental view. Donohoe et al. ( 2012) conducted a quasi-experimental study on 33 adolescents (ages 13–14) and found that Brainology® led to a significant increase in students’ incremental view of ability. More generally, although a substantial literature base has shown that manipulating students’ beliefs about the plasticity of ability leads to positive motivational and achievement gains, the research base concerning how technologies can be used to tap this construct is quite limited.

With respect to value beliefs, the research base about technology is similarly small. However, researchers have argued that well-designed technology-based activities can be used to target students’ interest value beliefs by making learning goals relevant and meaningful, and by allowing students to identify with characters within the technology environment (Gee 2003; Squire 2003). For example, Moos and Azevedo ( 2008) found that a hypermedia environment enhanced the development of students’ interest but not their utility value beliefs. Similarly, Hickey et al. ( 2001) showed that the use of The Adventures of Jasper Woodbury videodisc activity led to gains in students’ mathematics interest, although these gains appeared to result both from the technology as well as from teachers’ beliefs and instructional practices.

Context of the present study

To investigate the potential impact of technology-based activities on students’ mathematics motivation, we designed three different types of technology activities (or ‘inductions’). (We use the term ‘induction’ to refer to the technology activities, to avoid possible confusion between the technology activities and math lesson activities (described below)). The inductions differed along two main dimensions. First, the design of each induction was based on a different motivational construct; in other words, the theory of change underlying each induction differed (as we elaborate below). Second, the inductions differed in the expense and technical sophistication that were required for their creation and implementation, ranging from the very expensive-to-produce and technically advanced to the modest and inexpensive. Below we describe each induction in more depth.

Induction 1: virtual environment

At the core of Induction 1 was an Immersive Virtual Environment (IVE) - a game-like activity we designed to introduce students to the mathematical concepts that were to follow in a subsequent lesson. The IVE was professionally produced such that it was similar in look and feel to video games that students may have had experience playing.

For the storyline of the IVE, students were provided with the opportunity to explore an outer space environment in the context of a space rescue mission. Various mathematical puzzles were encountered as students moved around the planet; all puzzles related to the generation of and identification of mathematical patterns, similar to what would subsequently be discussed in a mathematics lesson. The initial puzzle was designed to be relatively easy; in later stages of the experience, mathematically related, more complex puzzles were broken down into many smaller steps to scaffold students’ progress and to reduce the likelihood that students would be overly frustrated. Similarly, hints were also provided by the IVE for students who requested help in completing any of the puzzles.

Prior to beginning the IVE, each student viewed a short (5-minute) video clip of a young STEM professional who talked about the nature of the work they do (e.g., designing astronaut space suits), the difficulties they had encountered in their K-12 math and science classes, and how they were able to overcome these difficulties. Students were provided with a selection of several of these videos, which varied according to the demographic attributes of the STEM professionals (e.g., gender, ethnicity); students were allowed to select whichever single video they wanted to view before beginning the IVE.

Motivationally, Induction 1 was designed to primarily impact students’ self-efficacy. In particular, we attended to the sources of self-efficacy beliefs theorized by Bandura ( 1986, 1997) and described above. The IVE experience supported mastery experiences by allowing students to experience incrementally more difficult mathematical challenges, and by providing the scaffolds necessary for students to succeed when they were met with obstacles. Vicarious experiences were included in Induction 1 by including real-life, young, STEM professionals who discussed their jobs and the types of obstacles that they faced (and overcame) as they pursued a STEM career. Finally, emotional and physiological states were addressed by ensuring that students felt comfortable and relaxed about solving the mathematical challenges in the IVE. For example, we made the design decision not to include a timer that gently reminded students to work more quickly if they were taking too long, because such a timer would likely cause a good deal of anxiety—a common experience for many students in mathematics.

Induction 2: Brainology® web-based activity

For the second induction, we used a commercially available series of web-based modules designed to teach students about an incremental view of ability. These modules are based on the work of Dweck and colleagues and have been shown to be successful at influencing students’ motivation and achievement (e.g., Blackwell et al. 2007). Students assigned to Induction 2 were given access to an abridged version of the Mindset Works® StudentKit - Brainology® program (http://www.mindsetworks.com). (This abridged version was created by Dweck and colleagues specifically for the present study). In a series of interactive modules, animated characters taught students how the brain works and how the brain grows stronger with effort. Students progressed through the modules at their own pace. The intervention that students experienced was relatively short compared to the entire Brainology® program, which contains over two hours of online instruction and up to 10 hours of additional activities to do over a recommended period of 5 to 16 weeks. Brainology is specifically designed for 5th to 9th grade classrooms. Note that the Brainology® modules do not have a specific focus on mathematics, nor do they incorporate any mathematical problem solving or algebraic reasoning.

With respect to motivation, the Brainology® program is explicitly designed to impact students’ implicit theory of ability. As noted above, Dweck and her colleagues (Dweck and Leggett 1988; Blackwell et al. 2007) have shown students possess particular ‘mindsets’ that can influence their motivational and developmental trajectories through the course of school (e.g., fixed theory of ability vs. incremental theory of ability). The Brainology® program activities have been found to encourage students toward a incremental view of ability.

Induction 3: video on mathematical patterns

Induction 3 was intended to provide an off-the-shelf experience for students related to some of the mathematical ideas that were to come in the mathematics lesson. We selected a commercially available PBS NOVA video on fractals because of its engaging storyline and graphics, its focus on mathematical patterns, and the accessibility of the content to our target population of students in grades 5–8. The 2009 video, Fractals: Hunting the Hidden Dimension, is 56 minutes long and includes visually appealing animations, interviews with mathematicians, and accessible explanations of the mathematics of fractals and their applications to everyday life, such as building smartphone antennas and generating visual effects in movies.

In terms of motivation, movies have long been used by educators to motivate and engage students in the classroom. Although this movie did not specifically target a particular motivation construct, movies are often used in educational settings as an inexpensive, simple means that teachers can employ to help students see connections between what they are learning and real-world applications.

Mathematics content focus

Within the general landscape of STEM, we chose to situate the present study in the content area of algebra. Algebra is widely recognized as a crucial peg in the trajectory of mathematical learning, because of the conceptual and procedural groundwork it lays for accessing higher mathematics and because it presents a shift in how students are expected to think mathematically (Kieran 1992). Algebra is often the first time students are introduced to some of the most important and useful ideas in the field of mathematics, such as the concept of a “variable” or the generalization of patterns in generated data (Star & Rittle-Johnson 2009). However, students’ difficulties in algebra are well documented on both national and international assessments (e.g., Beaton et al. 1996; Blume and Heckman 1997; Lindquist 1989; Schmidt et al. 1999). For example, in the eighth-grade data from the US National Assessment of Educational Progress [NAEP] show that students continue to struggle on very straightforward algebra problems: Only 59% of 8th graders were able to find an equation that is equivalent to n + 18 = 23, and only 31% of 8th graders were able to find an equation of a line that passes through a given point and with a negative slope (National Assessment of Educational Progress, Question Tool, 2011).

Within the larger landscape of algebra, we focus here on an aspect of algebra that many mathematics educators refer to as algebraic reasoning (e.g., Kaput 1999), which includes using arithmetic for generalizing, working with patterns to describe functional relationships, and modeling as a way to formalizing generalizations. Algebraic reasoning has begun to play an increasingly important role in US mathematics instruction, as evidenced by its emphasis in several grade levels of the Common Core Standards (Common Core State Standards Initiative 2010). Furthermore, the exploration and modeling of data that lie at the core of algebraic reasoning are central to the work of scientists, engineers, and other STEM professionals (Hoyles et al. 2010). In many middle grades mathematics classrooms, algebraic reasoning is instantiated through the identification, justification, and generalization of numerical patterns in given or generated data.

At the core of the present study is a two-lesson mathematics activity in which students engaged in an exploration of mathematical patterns. We designed the activity around a combinatorics task often referred to as a “trains” problem, because it involves the creation of integer-length “trains” using different numbers and lengths of integer-length “cars.” For example, students may be asked to determine the number of possible trains of a certain length n that can be created. If the task is to create a train of length 4, there are 8 ways to do so (using only integer-length cars, where the order of the cars matters): 1-1-1-1, 1-1-2, 1-2-1, 2-1-1, 1–3, 3–1, 2–2, and 4. Similarly, to make a train of length 5, there are 16 ways to do so. There are a large number of interesting variations and extensions of the trains problem, such as: How many trains of length n can be made using only cars of length 1 and 2, or only with cars of length 2 and 3? Or how many trains of length n can be made that begin with a car of a given length?

The trains problem was a useful context in which to ground our study for the following reasons. First, the mathematical content of the trains problem, which includes identifying, justifying, and generalizing numerical patterns, is well aligned with current state and national content standards for algebra. Second and similarly, the instructional practices involved in optimally implementing the trains problem (including use of mathematical manipulatives or representations to depict the trains, small group work leading to whole class discussions, and the sharing and comparing of students’ problem solving strategies) are consistent with current “best practices” in mathematics education (e.g., Common Core State Standards Initiative 2010, National Council of Teachers of Mathematics 2006). Third, as noted above, the intellectual activities of the trains problem, including exploring and modeling data, are central to the work of many STEM professionals. And finally, the trains problem is approachable to students from a variety of grade levels.An overview of the math activity is as follows (see Figure  1). The lesson was designed to occur on two consecutive days; teachers were given latitude to decide where the break between the first and second days of the lesson would occur. Teachers were provided with a variety of materials to aid in their implementation of the lesson, including detailed and condensed lesson plans, poster-sized visual aids, and concrete and virtual manipulatives.

Figure 1

Condensed two-day mathematics lesson plan.

The current study

The goal of the present study was to investigate the relationship between specific technology-based activities and students’ motivation in math. Students in grades 5 to 8 participated in a four-day classroom-based experience, beginning with a one-day technology activity, followed by a two-day mathematics lesson on algebraic reasoning, and concluding with revisiting the same technology induction on the final day. Students were assigned to one of three different types of technology inductions (as described above), each representing a different motivational framework. An assessment that targeted motivation was administered before, immediately after, and two months after the intervention.

We hypothesized that Inductions 1 and 2 would have the strongest effect on the motivational constructs that they were designed to influence. In particular, we hypothesized that Induction 1 would have the strongest impact on students’ self-efficacy and that Induction 2 would have the strongest impact on students’ implicit theory of math ability. Because Induction 3 was not designed with a particular theory of motivation in mind, it did not intentionally target any particular motivation variable. However, because of the content in the movie, we hypothesized that this third induction would have an impact on students’ value beliefs, especially their utility and interest value. Finally, with respect to developmental issues in motivation, the literature is clear that there is a general decline in motivation as students progress through school (Archambault et al. 2010; Eccles et al. 1984). Because the structure of schooling for students in middle school (Grades 6–8) is different from that of elementary school students (Grade 5), and because students conceive of competence differently based on age (Dweck 1986), we expected the first two inductions to have differential impacts on students depending on their age.

Method

Sample

Data come from all 5th, 6th, 7th, and 8th grade students and their teachers in the Chesterfield County Public School district in Virginia. A total of 18,628 students participated in the study, along with their 476 teachers, from 38 elementary and 12 middle schools.

A number of teachers in our original teacher pool were assistant, ESL, or special education teachers who did not have their own classroom. We removed these teachers from our sample, ending up with 339 teachers in our active teacher sample who participated in random assignment. In the elementary schools, the 163 5th grade teachers, who taught all subjects to the same group of students each day, implemented the intervention with their homeroom students. In the middle schools, the 60 6th, 57 7th, and 59 8th grade teachers were all math specialists and implemented the intervention in each mathematics classes that they taught. In total, the intervention was implemented in a total of 545 distinct mathematics classes.

We removed students who did not have parental consent to be a part of the study, which left us with 16,879 students. In addition, we had to exclude the 8,979 students (and their 113 teachers from 5 schools) who were missing pretest or posttest data used in our analyses. Most of this missing data was due to a miscommunication between the research team and the district relating to the student identification numbers that students were instructed to use at pre-test. Almost 5,000 students used an incorrect identification number, making it impossible to match students’ pre- and posttest scores. Little’s ( 1988) Missing Completely at Random (MCAR) test confirmed that these data were not missing completely at random (χ² (1576) = 7162.88, p < .001). In particular, students with missing data were more likely to be male, African-American or Hispanic/Latino, with ELL status, and from schools with a high percentage of free or reduced lunch. After removing those students with missing data, we report on the 7,900 students and 226 teachers from 44 schools who remained in our analyses.

Due to the large amount of missing data (about 53% of students were missing demographic, pre- or posttest data), it was not advisable to use multiple imputations to include more of these students and teachers in our analyses. As a result, we compared those participants included to those excluded using χ² tests and t-tests to examine differences in our demographic and pretest variables. We found several differences (see Additional file 1: Table S1). For instance, excluded participants were more likely to be male, African-American or Hispanic/Latino, and to have ELL status. They were also more likely to come from schools with a higher percentage of students receiving free or reduced lunch. There were few significant differences between the groups on student pretest variables, with the one exception being that excluded students had lower self-efficacy than included students (p = .037). There were significant differences between the groups on several teacher pretest variables. The excluded participants had teachers with lower self-efficacy for student engagement and instruction (p = .002) and self-efficacy for technology use (p < .001) than included participants. However, excluded participants had teachers with higher mathematics self-efficacy (p < .001) than included participants. The implications of these differences are examined in the discussion section.

The included 7,900 students were approximately equally divided across grade levels (see Table  1). The majority of students (60%) were White, with 23% African-American, 8% Hispanic, and 3% Asian. Four percent of students were identified as English-language learners [ELLs]. School level information was collected about students’ eligibility for free or reduced lunch; participating schools had an average of 34% of students who were eligible for free or reduced lunch, with eligibility at the school level ranging from 2% to 85%. We also collected students’ most recent scores on the state standardized test in mathematics, the Virginia Standards of Learning (VA-SOL) test; this test is given annually to students in grades 3–8.

Table 1 Student demographic information by condition

Design and procedure

We used a pre-test/post-test^a experimental design. Prior to the start of the intervention, students and teachers were administered a pretest. After pre-test administration, teachers were randomly assigned to one of three inductions described above – see Table  1 for student demographics for each induction. Participation in the main part of the intervention occurred over a period of four consecutive days. On Day 1, students worked on the induction to which they were assigned. On Days 2 and 3, teachers taught the two-day mathematics lesson. On Day 4, students again worked on the induction to which they were assigned.

For students in Induction 1, Day 1 of the intervention was spent in the school’s computer lab. Each student sat at his/her own computer, with headphones, and watched the short interview of a STEM professional and then played the IVE game for approximately 30 minutes. On Day 4, students returned to the computer lab and restarted the technology-based activity, including watching a video of a STEM professional and restarting the IVE game from the beginning – again playing for about 30 minutes. Similarly, for students in Induction 2, Days 1 and 4 were spent in the school’s computer lab, with one student at each computer with headphones, playing the Brainology® program. Finally, Induction 3 students watched the first half of the Fractals: Hunting the Hidden Dimension video (about 28 minutes) on Day 1; on Day 4, these students watched the second half of the video.

Professional development

All teachers were provided with a one-day (6.5 hours) professional development (PD) workshop, administered within one week of the start of the intervention. The PD workshop was designed and implemented by project staff. An identical PD was repeated for five consecutive days; district administration determined which teachers would attend on each day, with the attendance ranging from 56 teachers to 123 teachers. Each PD workshop included teachers from all three inductions and all four grade levels.

Most of the PD (approximately 4 hours) was devoted to introducing teachers to the two-day mathematics lesson. Teachers were provided with detailed lesson plans as well as visual aids, handouts, and manipulatives that accompanied the lesson. Under the facilitation of the first author, an experienced mathematics teacher educator, the PD workshop provided teachers with the opportunity to engage with the mathematics of the lesson and to plan for the enactment of the lesson. Approximately one hour of the PD was spent providing teachers with an overview of the project procedures, measures, and logistics. For the remainder of the PD, we provided teachers with induction-specific training. Teachers were divided into groups based on which induction they were assigned to. Induction 1 teachers played the IVE in a computer lab, Induction 2 teachers explored the Brainology® program in a different computer lab, and Induction 3 teachers watched the Fractals: Hunting the Hidden Dimension movie in a seminar room.

Measures

All assessments were administered to teachers and students online, via a password-protected website.

Student motivational measures

All students were administered a pre- and post- assessment, in a proctored computer lab in each school, during the regular school day. The pre-test, taken between one and three weeks prior to the start of the intervention, targeted students’ motivation, with measures corresponding to the three motivational constructs that were related to the inductions – self-efficacy, implicit theories of ability, and value (see Table  2 for descriptive information on student variables; see Table  3 for sample items and alphas).^b The post-test was administered on Day 4, after the implementation was completed. The motivational items on the post-test were identical to the pre-test. As described below, we used validated scales that have been commonly used in other motivation studies. Also, an exploratory factor analysis and scree plot indicated that our items mapped well onto three factors, with all self-efficacy items loading best onto one factor (factor loadings from 0.59 to 0.72), all value items loading best onto the second factor (factor loadings from 0.41 to 0.71), and all implicit theories of abilities items loading best onto the third factor (factor loadings from 0.53 to 0.61).

Table 2 Descriptive statistics on student motivation and learning variables

Table 3 Motivational measures

We assessed self-efficacy students with a 13-item measure that was drawn from Bandura’s ( 2006). The degree to which students endorsed an incremental view of ability (as opposed to a fixed view of ability) was assessed using a 6-item instrument that was adapted from Dweck ( 1999). Note that for the analysis of implicit theory, we reverse-scored the fixed theory of ability items and calculated a mean theory of ability score with the incremental items – thus higher scores represented stronger agreement with incremental theory of ability. Finally, interest, attainment, and utility value beliefs concerning their mathematics class were assessed using scales taken from the Michigan Study on Adolescent Life Transitions (MSALT), which has been used extensively in the past (e.g., Eccles et al. 2003).

Student mathematics learning measure

Assessing students’ mathematics learning was not a major focus of the present study, mainly because of the absence of a priori hypotheses related to the differential impact of the three technology inductions on student learning and also the short duration of the math lesson. However, as a manipulative check, we included a short five-item assessment on mathematics learning on both the pre- and post-tests. These five items were on algebraic reasoning as related to the two-day mathematics lesson, specifically data organization, pattern identification, and the ability to make generalizations. For example, an item on pattern identification asked students to identify the number that is most likely to come next in the number pattern: 3, 7, 11, 15, ?. As another example, an item asked students to determine how many different lunch plates could be made by choosing one main course (from two choices), one side (from four choices) and one drink (from two choices). Items on the pre- and post-tests were non-identical but isomorphic (same problem structure but with different contexts and numbers). The reliability of the math learning measure was low (α = 0.30 and 0.40 for the pre- and post-test); as a consequence, the results from this measure must be interpreted with caution.

Teacher measures

All teachers were administered three assessments. Teachers completed the surveys at a time (within a given survey administration window) and place of their choosing.

First, teachers were given a pre-test immediately prior to the start of the professional development workshop. The pre-test collected background and demographic information about teachers, such as number of years teaching, undergraduate major, advanced degrees held, and national board certification status. In addition, the teacher pre-test included items that tapped teachers’ own teaching self-efficacy for instruction and student engagement (22 items), technology use (7 items), and mathematics (12 items). Items were drawn or adapted from Bandura ( 2006). To confirm the validity of the self-efficacy items, we first conducted an exploratory factor analysis. This analysis indicated that our self-efficacy items mapped well onto three factors, with all self-efficacy items related to student engagement and instruction loading best onto one factor (factor loadings from 0.51 to 0.76), all self-efficacy items related to technology use loading best onto the second factor (factor loadings from 0.45 to 0.82), and all self-efficacy items related to mathematics loading best onto the third factor (factor loadings from 0.45 to 0.80). Teachers were also administered a 6-item measure of implicit theory of ability that was adapted from Dweck ( 1999). See Table  3 for sample items and alphas.

Second, teachers completed a 6-item post-professional development survey immediately after the one-day professional development workshop (see Additional file 1: Table S2). This survey assessed teachers’ views on the overall quality of the professional development workshop, how prepared and confident teachers felt in implementing the intervention, and teachers’ predictions about how students would react to this intervention. Finally, immediately after they had finished teaching the two-day math lesson, teachers were administered a six-item self-assessment of implementation fidelity asking about their adherence of this lesson plan.

Data analysis

Given that many students had the same teacher and many teachers were in the same school, we used multilevel modeling (Raudenbush and Bryk 2002) to account for this nesting of students within teachers and teachers within schools. The first level of the model, the student level, included students’ prior knowledge (VA-SOL) scores, pretest math learning scores, pretest self-efficacy scores, pretest implicit theory of ability scores, pretest value scores, and demographic information, including ELL status, grade, gender (male coded as 1 and female coded as 0), and ethnicity.

The second level of the model, the teacher level, measured the effect of experimental condition, teachers’ self-efficacy for student engagement and instruction, teachers’ self-efficacy for technology use, teachers’ mathematics self-efficacy, and teachers’ implicit theory of math ability. We specified Induction 1 (the immersive virtual environment) as the referent condition to compare it to the other two inductions. This resulted in the effect of condition being captured by two variables. One variable indicated the difference between Induction 1 and Induction 2, and the other variable indicated the difference between Induction 1 and Induction 3. To test the difference between Inductions 2 and 3, a Wald test (similar to an incremental F test) was used to examine whether the parameter estimates for these conditions were significantly different from one another.

The third level of the model, the school level, measured the percentage of students receiving free or reduced lunch in each school. Finally, we also included two cross-level interactions to test for possible interactions between induction and grade, as well as two cross-level interactions to test for possible interactions between induction and prior math knowledge (VA-SOL). All continuous independent variables in the model were grand mean centered. We ran these models to evaluate our four posttest student outcomes: math learning, self efficacy, implicit theory of ability, and value.

The intraclass correlations for the teacher and school levels ranged from 0.001 to 0.052, which were fairly small. However, we still used multilevel models because they account for dependency between observations, and produce unbiased standard errors and more stable intercept and slope estimates (Myers 2011). Similar results were obtained when using Ordinary Least Scales [OLS] regression instead of multilevel models.

Results

We begin by providing descriptive information on the quality of the implementation of the professional development workshop and the intervention, as well as by describing teachers’ view of the quality of professional development, teachers’ assessment of students’ interest and engagement with the intervention, and teachers’ self-reports of their fidelity of implementation. We then turn to our research questions by overviewing students’ scores on the motivational variables at pretest and posttest and then reporting the effects of condition at posttest.

Fidelity of implementation

Quality of the professional development

Judging from teachers’ self-reported responses to the survey administered immediately after the PD (see Additional file 1: Table S2), teachers were not especially enthusiastic about the quality of the PD, with only 29% rating the experience as very good or excellent as compared to other PD experienced in the past five years. Nevertheless, a plurality of teachers left the PD feeling prepared to implement the math lessons (45% felt prepared or very prepared), and most felt confident that they could successfully do so (58% felt confident or very confident), despite the fact that very few teachers felt that the lesson was similar or very similar to the ways that they typically taught. Most teachers (53%) felt that students would be very challenged by the content of the math lessons, and many teachers (47%) felt that students would react positively.

Implementation of math lessons

Recall that data on fidelity of implementation were obtained from self-reports of teachers on the survey administered immediately after the end of the two-day math lesson. Teachers’ responses indicated that they believed that they had very closely followed the lesson plan, with 75% indicating that they very closely or exactly followed the list of activities and 60% answering that they asked the questions very closely or exactly as suggested (see Additional file 1: Table S2).

Student and teacher pretest scores

To begin, we measured whether there were any differences between the inductions on our outcome measures at pretest and on demographic variables (see Table  2). When controlling for other independent variables in the model, there were no significant differences (p > .05) between inductions on any of the pretest or demographic variables, with the exception of prior knowledge (VA-SOL). Students in Induction 2 had lower prior knowledge than students in Induction 1, β = -15.76, p = .003, and Induction 3, χ²(2) = 13.63, p = .001. Students in Induction 3 also had slightly lower prior knowledge than students in Induction 1, β = -15.69, p = .001. Prior knowledge was included in all subsequent models, so we controlled for these differences between conditions.

Pre/Post gains

Before examining the effects of condition, we first consider whether the intervention generally led to gains in students’ motivation (see Table  2). Overall, students did not have statistically significant gains on our measure of self-efficacy (M _pre  = 4.54, M _post  = 4.55, t = -1.16, p = .246, d = -0.01). For implicit theory of ability, students’ incremental view of math ability decreased after the intervention, although this was a small effect (M _pre  = 4.22, M _post  = 4.16, t = -6.93, p < .001, d = -0.07). For value, students’ scores generally decreased after the intervention as well, although the effect was again small (M _pre  = 4.24, M _post  = 4.19, t = -8.71, p < .001, d = -0.06). For math learning, the intervention led to an average gain on students’ scores on the five-item mathematics learning assessment of ten percentage points, and this was a moderate effect (M _pre  = 0.60, M _post  = 0.70, t = 28.60, p < .001, d = 0.40).

Effects of condition at posttest

We now move to examining the effects of condition at posttest. At posttest, there were significant effects of condition on several of our outcome variables (see Table  4). As we describe below and return to in the discussion, note that most of the independent variables that significantly predicted our outcomes were at the student-level, rather than at the teacher-level.

Table 4 Parameter estimates for student outcomes

For each analysis of the effect of condition, we report three interrelated analyses, in the following order. First, we report whether Induction 2 differed from Induction 1 (main effects and interactions), and we then report whether Induction 3 differed from Induction 1 (main effects and interactions). Finally, we report results from a Wald test to investigate whether Inductions 2 and 3 differed (main effects and interactions).

Math learning

Comparing Inductions 1 and 2, students in Induction 2 earned similar math learning scores to students in Induction 1, β = 0.003, p = .872. There was also no significant interaction between Induction 2 and grade, β = 0.01, p = .129. Comparing Inductions 1 and 3, students in Induction 3 had similar math learning scores to students in Induction 1, β = -0.01, p = .409. However, there was a significant interaction between Induction 3 and grade. In particular, students in lower grades benefited more from Induction 1 than from Induction 3. Then as grade increased, Induction 3 became more effective, β = 0.02, p = .013. Thus, for students in grade 5, being in Induction 1 led to higher scores on average. For students in grades 6, 7, and 8, being in Induction 3 led to higher scores on average. Finally, post-hoc Wald tests comparing Inductions 2 and 3 suggested that there were no significant differences between Inductions 2 and 3 (χ²(2) = 1.06, p = .589); however, there was a significant interaction when considering grade (χ²(2) = 6.22, p = .045). Essentially, Induction 2 was more effective for lower grades, and as grade increased, Induction 3 became more effective. There were no significant interactions between induction and prior knowledge (VA-SOL) (p’s > .532).

Self-efficacy

There were no significant differences between any of the inductions on the student self-efficacy variable, nor were there any significant interactions between inductions and grade or inductions and prior knowledge (p’s > .128).

Implicit theory of ability

Comparing Inductions 1 and 2, students in Induction 2 had higher implicit view of math ability scores than students in Induction 1, β = 0.09, p = .039, meaning that being in Induction 2 led to an implicit theory of math ability score that was 0.09 standard deviations higher than being in Induction 1. There was also a significant interaction between Induction 2 and grade. In particular, students in lower grades had similar implicit view of math ability scores in Induction 2 and Induction 1. Then as grade increased, Induction 2 led to higher implicit view of math ability scores than Induction 1, β = 0.12, p < .001. In addition, there was a significant interaction between Induction 2 and prior knowledge (VA-SOL), β = 0.001, p = .018; however, as the coefficient indicates, this was a very small interaction. Students with lower prior knowledge had slightly higher implicit view of math ability scores in Induction 1 than Induction 2. Comparing Inductions 1 and 3, students in Induction 3 had similar scores to students in Induction 1, β = 0.05, p = .243. There was also not a significant interaction between Induction 3 and grade, β = 0.03, p = .271, nor between Induction 3 and prior knowledge (VA-SOL), β < 0.001, p = .371. A post-hoc Wald test indicated that overall students in Induction 3 had similar implicit theory of ability scores to those in Induction 2 (χ²(2) = 4.34, p = .114. However, there was a significant interaction when considering grade (χ²(2) = 23.62, p < .001). In lower grades, students in Induction 3 had similar implicit view of math ability scores as students in Induction 2, but as grade increased, students in Induction 2 tended to have higher scores than students in Induction 3. When comparing Inductions 2 and 3, there was also a marginally significant interaction between Induction and prior knowledge (VA-SOL) (χ²(2) = 5.75, p = .057).

Value

For value, in comparing Inductions 1 and 2, overall students in Induction 2 had similar value scores to students in Induction 1, β = 0.02, p = .668. There was also no significant interaction between Induction 2 and grade, β = -0.01, p = .520. When comparing Inductions 1 and 3, students in Induction 3 had similar value scores to students in Induction 1, β = 0.01, p = .795. There was a significant interaction between Induction 3 and grade. In particular, students in lower grades had similar value scores in Induction 3 and Induction 1. Then as grade increased, Induction 1 led to higher value scores, β = -0.04, p = .036. Post-hoc Wald tests suggested that there was no significant difference between Inductions 2 and 3 (χ²(2) = 0.19, p = .910). There was also no significant interaction when considering grade (χ²(2) = 4.76, p = .093). Finally, there were no significant interactions between condition and prior knowledge (VA-SOL) (p’s > .069).

Discussion

Perhaps not surprisingly given the size and complexity of the present study, our results are informative, modest, and not definitive. We begin by summarizing the results that pertain to our three research questions in turn, with particular attention to the contributions of these results to the field.

RQ1: impact on students’ motivation

Our first research question concerned the general impact of the four-day intervention on students’ motivation in mathematics, particularly self-efficacy, implicit theory of ability, and value. Overall, results from the four-day intervention were mixed. No gains were found in self-efficacy; for implicit theory of ability, a lower incremental view of ability was found; we found modest declines in value beliefs. With respect to math learning, students in all three inductions had modest improvements in their scores on the math learning measure.

RQ2: influences of induction type and student characteristics

Second, we were interested in whether the impact of the intervention was influenced by the type of induction that student received and other student-level demographic or academic characteristics. We found that induction type and student-level factors had a moderate influence on the motivational impact of the intervention. No effects related to self-efficacy were found, and effects related to value were very minor. For implicit theory of ability, there were indications that Induction 2 was more successful than Inductions 1 and 3 in impacting students’ views, especially for older students. Induction 2 led to higher incremental views of math ability for students, particularly for students in grades 7 and 8. Induction type also appeared to have a small impact on value, with some evidence that Induction 3 had the strongest impact on utility and attainment value for the younger students, as compared to the other two inductions.

Despite the complexity of these results for our second research question, three clear patterns did emerge.

Absence of effects on self-efficacy

First, Induction 1 did not have the hypothesized impact on students’ self-efficacy. Despite the fact that the IVE was designed specifically to foster changes in self-efficacy, there is no evidence that Induction 1 improved self-efficacy any more than the other inductions. There are several possible explanations for this finding. First, given the relatively short intervention, the fact that students in any induction did not experience dramatic gains in a construct as fundamental and multi-dimensional as self-efficacy is not surprising. Second, Induction 1 was the most complex in terms of cognitive and temporal “overhead” required for students to enact the experience; navigating and overcoming obstacles in a virtual world are more challenging tasks than the other inductions presented. We hypothesize that, had a longer time period been available for students to shift their focus from learning to enact Induction 1 to reflecting on the content of the experience, effects on self-efficacy would have been greater.

Finally, a third possible explanation for this finding is that, although all three inductions did target different aspects of motivation, these inductions were not the only component of the overall four-day intervention that was designed to influence students’ motivation. In fact, the two-day mathematics lesson was also designed with best practices (including motivation) in mind. Given that the two-day math lesson was implemented with reasonably high fidelity, it may be that the mastery experiences afforded by the classroom lessons washed out any self-efficacy effects that the technologies provided. And when students thought about their confidence to do these types of problems in completing the self-efficacy items on the survey, they may have reflected more on their experiences in the classroom than in their respective technology experiences.

Related, recall that the three inductions also differed on the expense and technical sophistication required to create and implement them. Does the present finding about Induction 1 and self-efficacy suggest that use of virtual worlds is not worth the trouble and expense? Particularly when inculcating sophisticated knowledge and skills, a substantial body of research suggests that this is not the case (Chen et al. 2014; U.S. Department of Education 2010; National Research Council 2011). We interpret our results as indicating that this type of complex intervention with high cognitive overhead may require more instructional “dosage” than short duration provided in the present intervention. Thus, well-designed virtual worlds, which are expensive and technically demanding, can realize their power for engagement and learning only when a sufficient investment of classroom time is made.

Effects linked to students’ age

A second pattern that emerges from the complex results of our second research question is that the effects of each induction on students’ motivation were influenced by students’ age, as evidenced by the frequency of significant induction type by grade interactions. These grade-level interactions held while controlling for prior mathematics knowledge (VA-SOL scores), indicating that the differential impact of the inductions was developmental and not merely the result of differing mathematics ability. Because the structure of schooling for students in middle school (Grades 6–8) is different from that of elementary school students (Grade 5), and because students conceive of competence differently based on age (Dweck 1986), these findings indicating differential impacts on students depending on their age are confirmatory of prior work and reinforce the importance for practitioners and policy makers of tailoring such interventions to students’ developmental level.

In addition, our results suggest that the impact of the abridged version of Brainology® on students’ implicit theory was greater for older students than it was for younger students. One possibility for this finding is that older students may be more attuned to the incremental message than younger students. Dweck ( 2002) argued that students’ conceptions of ability may not have an effect on their motivation and performance until 10–12 years old. Therefore, the incremental theory of ability message may have been more salient for these older students than it was for younger ones.

Absence of effects for student demographics

Finally, we did not find interactions between induction type and other student demographic variables such as free and reduced lunch, ethnicity, and gender. From a curricular perspective, this is a positive outcome indicating that, in contrast to many educational experiences, these types of intervention may narrow—not widen—troubling achievement gaps. That good design can produce motivational learning experiences effective across the full spectrum of students is very encouraging.

RQ3: influences of teacher-level factors

Our third research question asked about impact of teacher-level factors on students’ motivation, including credentialing in mathematics education, undergraduate major, years of experience, and teachers’ beliefs. Based on the extant literature, we had hypothesized that these factors might influence students’ motivation. However, teacher-level factors were not significant predictors of student outcomes. Viewing the intervention from a curricular perspective, this is a positive finding suggesting that our design and implementation ensured that all students received a roughly equivalent instructional experience.

With respect to the absence of a relationship between teachers’ beliefs and student motivation, although there is good theoretical and empirical evidence to suggest that these variables could predict student outcomes, it is also true that linking teacher-level beliefs to student outcomes is not a clear and straight path (Holzberger et al. in press; Klassen et al. 2011). In fact, Klassen et al. ( 2011) noted that there is a lack of evidence that links teachers’ self-efficacy to student outcomes, despite the commonly held belief by researchers that this relationship exists. Their review of the literature noted that correlations between teachers’ self-efficacy and student achievement were low to modest. Our findings confirm this perspective.

One explanation for the absence of these effects may relate to a social desirability bias influencing teachers. We note that teachers’ responses were generally quite positive on their motivation surveys, with relatively small variance. It may have been the case that some teachers were reluctant to admit they were not confident in being able to teach or manage a class effectively; similarly, some teachers might have been unwilling to admit that they saw little value in the goals of the present study. Artificially inflated teacher responses to the teacher motivational surveys may explain the lack of relationship between teacher and student motivation.

Another possibility is that the professional development that we created and implemented had the effect of eliminating much of the teacher-level variance and its effects on student outcomes. We specifically designed the professional development such that teachers emerged confident in their ability to successfully implement the two-day math lesson. We also communicated to teachers that there was considerably flexibility in their implementation of the math lesson, as long as a few basic implementation guidelines were followed. We hoped that such an empowerment-supportive way of training teachers would allow teachers to feel more autonomous and less controlled, thereby translating to better implemented curricula. It is possible that this approach (which did enable teachers to implement the two-day lesson with fidelity) also helps explain the absence of teacher-level effects on student motivation.

Limitations

There were several limitations to the present study that suggest caution in the interpretation of our results. First and foremost, as noted above, there was a very large amount of missing data – 53% of students were missing demographic, pre-, and/or posttest data – most of which occurred due to a miscommunication between the research team and the district relating to the student identification numbers that students were instructed to use at pre-test. Second, it is important to note that the length of the intervention was relatively short, both in terms of the technology-based motivational activities, the professional development, and the mathematics lesson. Although we were able to find some influence of the intervention on students’ motivation, these effects were quite modest. Further, although a delayed posttest was administered, results were not interpretable; thus, we are not able to report whether or not the effects at posttest were sustained after the end of the intervention. Third, recall that the five-item math assessment had low reliability. Taken together, all of these results raise questions about any attempt to generalize our findings. Future studies – both additional large-scale studies of longer duration, as well as shorter-term studies that afford opportunities for more qualitative exploration - can attempt to address these limitations and continuing moving toward improving our understanding of the relationship between technology, motivation, and STEM learning.

Conclusion

Investigating along a developmental span the relationship between specific technology-based motivational activities and student interest in STEM careers is important, because much potential talent in science, technology, engineering, and mathematics is now lost. Our research interweaved alternative motivational activities with effective and authentic mathematics learning, in order to take initial steps toward developing insights about the added value of technology for building confidence in math and science capability, seeing one’s abilities in STEM as able to improve over time, and developing a passion or sustained interest in becoming a scientist or engineer. Further, we studied the impacts of media with substantially different production costs, providing the basis for a cost-benefit analysis and for articulating contrasting conditions for success.

Our findings highlight the importance of tailoring motivational experiences to students’ developmental level. Our results are also encouraging about developers’ ability to create instructional interventions and professional development that can be effective when experienced by a wide range of students and teachers. Further research is needed to determine the degree, duration of, and type of instructional intervention necessary to substantially impact multi-dimensional, deep-rooted motivational constructs, such as self-efficacy.

Endnotes

^aA delayed post-test was also administered, two months after the end of the intervention. However, due to large amounts of missing data, delayed post-test results were not easily interpretable and thus are not included in the present analysis.

^bStudent and teacher assessments also included additional items assessing several other motivational constructs. The inclusion of these extra items was exploratory, in that none of the technology-based activities were designed with these constructs in mind. In particular, students were administered a short assessment immediately after the conclusion of the Day 1 technology-based motivational activities that focused on several of these additional motivational constructs. In the present analysis, we report only on those student and teacher variables that were explicitly considered in the design of the inductions and that were specifically hypothesized to be related to the effectiveness of the intervention.