At the end of the NISE Network’s 10th year, the Evaluation workgroup created a reflection document to describe how we managed the evaluation of the NISE Network, one of the largest informal education networks ever formed. This document goes into specifics of how we set up our team, what impacts we decided to measure, and what methods we used to collect data on a national scale. It also talks about ethical considerations we took into account and how we shared our work with multiple audiences. This reflection document is not meant to be a prescriptive "how-to" manual, but an example of one
The project team is developing a prototype of a web-based game utilizing the illustrations of chemical elements and science terms created by Simon Basher in his three books, The Periodic Table: Elements with Style!, Chemistry: Getting a Big Reaction!, and Physics: Why Matter Matters! The game will incorporate augmented reality (person-to-person gameplay with the support of the software) to teach grade 4 to 6 students science concepts, including an introduction to chemistry. The game will include curriculum support materials. Pilot research in Phase I will seek to demonstrate that the software prototype functions as planned, teachers are able to integrate it within the classroom environment, and students are engaged with the prototype.
Purpose: The United States (U.S.) has traditionally produced the world’s top research scientists and engineers, leading to breakthrough advances in science and technology. Despite the importance of STEM careers, many U.S. students are not graduating with strong STEM knowledge, skills or interests, and the percentage of students prepared for or pursuing STEM degrees or careers is declining. Research shows that the decreased interest in STEM typically begins in the middle school years, pose significant academic and social challenges for students. This project will develop a web-based game teach 6th to 8th students key scientific inquiry skills, along with the academic mindsets and learning strategies to facilitate engagement and effective science learning.
Project Activities: The researchers will create a prototype by mapping key Next Generation Science Standards and learning goals with concepts and content, and producing a game design document. Following completion of the prototype, the researchers will finalize the server architecture, create the core code systems, concept art, and develop a prototype in order to simulate the final user experience. Iterative refinements will be conducted as needed at major production milestones until the game is fully functional. Once development is complete, the research team will assess the usability and feasibility, fidelity of implementation, and the promise of the game to improve outcomes in a pilot study. In this study, 200 students in 10 classes will participate, with 5 of the classrooms randomly assigned to use the game and 5 who will proceed as normal. All students will complete pre- and post- program surveys assessing their academic mindsets, learning strategies, and science skills.
Product: This project will develop SciSkillQuest, a web-based multiplayer game intended to teach middle school students scientific inquiry skills and to foster academic growth mindsets in science. Students will pursue quests, employing inquiry skills to navigate and succeed in the game, including Questioning, Modeling, Investigating, Analyzing, Computing, Explaining, Arguing, and Informing. The game will include different paths to a solution, role playing elements, immersive narratives, challenge-based progressions, and peer collaboration to engage players. The growth mindset message — that ability and skill are developed through effort and learning — will be introduced and reinforced through feedback by embedded in-game characters. The games will be supplemental to the curriculum but will also be designed to be integrated within instructional practice. The game will be available for mobile devices as well as web browsers.
Purpose: In the most recent National Assessment of Educational Progress only 17% of 8th grade students performed at or above the proficient level in U.S. history. One way to engage students in learning history is to create history learning resources that are designed to be relevant and appealing to young people's interests and regular activities. Surveys find that almost all teenage boys and girls play digital games, and the majority of teens play daily. This project will leverage the potential of games and technology to engage students and increase history skills and content knowledge.
Project Activities: The team, consisting of graphic artists, content specialists, computer scientists, and programmers, will initially create wireframes and a functional game prototype. Following feedback from a group of students and teachers on the user-interface, the team will produce an online tablet app. Iterative refinements will be conducted at major production milestones until the intervention is fully functional. Once development is complete, the researchers will assess the usability and feasibility, fidelity of implementation, and the promise of the product to improve outcomes in a pilot study. The study will include 200 8th grade students in eight classrooms. Four classrooms will be assigned to play to game as part of the curriculum over three to five class periods, and four classrooms will be taught the same historical content using the business as usual curriculum without the game. Each group will complete pre- and post- assessments to assess differences in history knowledge and skills.
Product: This project team will develop a tablet-based interactive role-playing game that immerses 5th through 9th grade students in the history of the Great Depression. The game will provide players an experiential understanding of the hardships that beset Americans in the 1930s and their strategies for survival, as individuals and as a nation. Features of the game will include story-based immersive narrative missions where student's decisions continually drive the action, tips and hints for students who are struggling in the game, writing tools, and interactive maps. The game will can be integrated within a course or used as a supplement. A teacher dashboard will be developed to facilitate the use of the game within classroom settings. Finally, the final product will include upgrades to existing games, including City of Immigrants and the The Hardest Times. The upgrades will publish these games to tablets and will include deeper in-game assessment opportunities.
Purpose: An estimated 5 to 8% of elementary school students have some form of memory or cognitive deficit that inhibits learning basic math. Researchers have identified several areas where children with math learning difficulties struggle. These include a strong sense of number facts to quickly and accurately perform operations on single digit numbers, the use of strategies to solve problems which have not yet been memorized, a sense to figure out whether or not an answer is reasonable, and self-monitoring to assess one's own efficacy and understanding. To support students with math learning difficulties in grades 1 to 4, this project team will develop a series of apps for touch-screen tablets that encourage single digit operational fluency, conceptual understanding, strategy awareness, and self-understanding.
Project Activities: During Phase I project in 2012, the research team developed a prototype of the single digit addition game, following an iterative process incorporating feedback from teachers and students having difficulty with math. Nineteen students participated in a pilot study, and the researchers found that the prototype functioned well and that users were engaged by the game. In Phase II, the team will build and refine the back end system, design and develop the teacher website, and create content for games in subtraction, multiplication, and division. Researchers will carry out a pilot test of the usability and feasibility, fidelity of implementation, and promise of the game to improve learning. Students in first to fourth grade identified by teachers as having the greatest difficulty with math will participate in the pilot study. Half of the 120 students participating in the pilot study will be randomly selected to play the game as a supplement to classroom learning whereas the other half will not have access. Students in the control group will be provided the games at the end of the study. Analyses will compare pre- and post-test math scores.
Product: The web-based game, MathFacts, will include a series of apps for touch-screen tablet computers to support math learning for 1st to 4th grade students with major or sometimes intractable learning difficulties. In the game, students will learn content through mini-lessons, engage with problems in practice and speed rounds, and then receive formative feedback on their performance. Students will use and manipulate blocks, linker tubes, number lines, and interact with engaging pedagogical agents such as parrots and sloths. Students will set goals, advance to more challenging levels, and engage in competition. The game will be self-paced and will provide individualized formative assessment scaffolding when students do not know the answer to a question. A teacher management system will support professional development and will produce reports to guide instruction. The intended outcomes from gameplay will include increased fluency, conceptual understanding, strategy awareness, self-assessment, and motivation of basic math.
Purpose: There is concern about a decline in mathematics achievement scores among U.S. students during the middle school years. For example, while 4th grade U.S. students rank 8th overall on an international mathematics comparison, by 10th grade U.S. student's drop significantly to 25th in the same comparison. Some researchers posit that much of this decline relates to how math is taught in the U.S. and with how students become less engaged as learners in middle school. The purpose of this project is to develop a web-based game to engage 7h grade students in a narrative-based story which will apply learning of content and skills aligned to the Common Core State Standards (CCSS) in mathematics.
Project Activities: During Phase I in 2012, the team developed a functioning prototype and conducted usability and feasibility research with fourteen 7th grade students. Researchers found that the prototype functioned as intended and that students were highly engaged while playing the game. In Phase II, the team will develop a fully-functional user interface with animated characters, interactivity across student users, narrative scripts and accompanying art assets, 36 problem sets, and student and teacher dashboards and databases. After development is complete, a pilot study will examine the usability and feasibility, fidelity of implementation, and the promise of the game to improve math learning. The study will include 120 students in 6 classrooms in three schools, with one classroom per school randomly assigned to use the game and the other half assigned to a business-as-usual control. Analyses will compare student scores on pre and post mathematics measures.
Product: Empires is a web-based game that addresses 36 pre-algebra Common Core State Standards in mathematics for 7th and 8th grades. The game follows a storyline in a recreation of an ancient empire which is at the brink of agricultural revolution and of becoming a trade economy. As students play the game, they engage in math-focused activities to drive the action, such as taxing citizens to learn ratios and proportions, allocating resources to learn percentages, and measuring the distance and time between a neighboring empire by applying the principles of the Pythagorean Theorem. As a socially networked game, students will interact with other students in the class to complete trades that lead to encounters with different math problems. The game will include two helpful, funny, advisors who will scaffold learning through mathematical discourse, arguing over the next most important thing to do. The game design architecture will work on a wide range of computers, including desktops and iPads. A teacher's guide and companion website will provide guidance to classroom activities that complement the game.
The project team is developing a prototype of Thinking Time, a tablet-based app and game for early learners (ages 3- to 6-years-old) that provides cognitive training games based on neuropsychological research. Game play will be self-guided and adaptive, and will support the development of working memory, attention and impulse control, and flexibility. The goal is to promote academic readiness by scaffolding cognitive skills during the early years of heightened brain plasticity. In the Phase I pilot research, the project team will examine whether the software prototype functions as planned, if teachers are able to integrate it within the classroom environment, and whether children are engaged with the prototype.
The project team is developing a prototype of a mobile platform, Zaption, to support teachers in using video clips to enrich learning. The product’s user-interface will allow teachers to easily add annotations to videos, make short video clips that align to topics, and enhance videos with time-linked elements and assessments that appear at the top of each video. In Phase I pilot research, the team will examine whether the prototype functions as planned, if teachers are able to use the prototype for different purposes, and whether students are engaged by the prototype.
The project team is developing a prototype of Eco, a multi-player game to prepare high school students to be environmentally literate citizens with 21st century skills. To play the game, students will enter a shared online world featuring a simulated ecosystem of plants and animals. Students will co-create the civilization by measuring, modeling, and analyzing the underlying ecosystem. Students advocate proposed plans to classmates and make decisions as a group. Cooperation and science-based decision making activities are necessary for success in preventing the destruction of their environment. The prototype will include teacher resources to support the alignment of game play to learning goals, and implementation. In the Phase I pilot research, the project team will examine whether the game prototypes function as planned, if teachers are able to integrate it within the classroom environment, and whether students are engaged with the prototype.
The project team is developing a prototype of Happy Atoms, a game to support middle school students in learning about the composition of molecules. Happy Atoms will include physical manipulative balls with embedded magnets wirelessly connected to a tablet application (app) to recognize whether or not the created molecule exists and explain why or why not. The app will also include teacher resources including instructional videos and curriculum suggestions in order to better integrate the use of the product into classrooms. In the Phase I pilot research, the project team will examine whether the hardware and software prototypes function as planned, teachers are able to integrate it within the classroom environment, and students are engaged with the prototype.
Purpose: This project team will develop and test Zaption, a mobile and desktop platform designed to support educators in effectively and efficiently utilizing video (e.g., from YouTube, Vimeo, or their own desktop) as an interactive teaching and learning object. Personalized learning devices (e.g., smartphones, tablets) populated with video content provide opportunities for students to access educationally-meaningful content anywhere and anytime. Yet, video has yet to realize its potential as a learning tool in or out of the classroom. One reason for this is that watching video can be a passive experience for students, whereas learning requires active engagement. A second reason is that even if students are actively engaged while watching a video, there is no easy way to elicit student responses to a video. And finally, there is no easy way to feed student responses to teachers as formative assessment data to guide subsequent instruction.
Project Activities: During Phase I, (completed in 2014), the team expanded a pre-existing prototype by building a mobile app to enable anytime use and increase its functionality for teachers. At the end of Phase I, pilot research with 150 students in 7 classrooms demonstrated that the prototype operated as intended, teachers were able to integrate the videos within instructional practice, and students found the mobile app helpful and engaging. In Phase II, the team will add additional components to the prototype and will develop content-specific modules for use in high school physics classes. After development is complete, the research team will conduct a larger pilot study to assess the feasibility and usability, fidelity of implementation, and the promise of the Zaption for supporting student's physics learning. The study will include 32 Grade 10 physics classrooms, half of whom will be randomly assigned to use Zaption and half of whom will follow business as usual procedures. Analyses will compare pre-and-post scores of student's physics learning.
Product: Zaption will be a mobile and web-based platform to support the use of any video (e.g., from YouTube, Vimeo, or their own desktop) as a teaching and learning tool. Zaption will include an authoring engine where users can find and select video clips and easily insert interactive elements such as questions, discussions, and annotations into the videos. Users will then publish videos directly on Zaption's website, or on any learning management system or classroom website. Students will be able to view videos as homework or in class, respond individually to the questions and prompts, and get feedback on their responses. Teachers will use Zaption Analytics to receive immediate and actionable data showing whether students actually watched and engaged with a video, and how students responded to the questions and prompts.