Design Squad Nation: Evaluation Report

Premiering on PBS in January 2011, Design Squad Nation (DSN) is a 10-part series of TV shows and video blogs that build on the success of the award-winning PBS reality competition series Design Squad to get kids excited about engineering. Concord Evaluation Group (CEG) was hired to conduct an evaluation of DSN in 2011. The evaluation was designed to measure WGBH's success in achieving the following impacts on students and teachers: After using the DSN resources Middle school students will increase their knowledge of science and engineering concepts and the design process. Middle school students' attitudes towards engineering will improve. Middle school students' interest in engineering will increase. Middle school students' awareness of engineering work will increase. Middle school students, especially girls and minorities, will see engineering as creative, rewarding, and socially relevant. After using the DSN resources Teachers' awareness of the design process will increase. Teachers' knowledge of engineering careers will increase. Teachers' interest in using hands-on engineering activities in their classrooms will increase. Teachers' comfort level with providing engineering activities to their students will increase. In addition to assessing these impacts, another major goal of the evaluation was to collect formative data that could be used to redesign the DSN web resources for a future study. In a separate study we are planning to explore the engineering pathways that middle schoolers follow. The cornerstone of the proposed study is an enhanced DSN website with more activities, more challenges, and more social interaction designed to support students who are learning about engineering at home or at school. For the current evaluation, we performed three separate data collection activities. First, we assessed the impact of the TV episodes through a home viewing study. Second, we assessed the impact of the full range of DSN resources on students and teachers in formal settings. Finally, we also conducted a second annual survey on the home page of the DSN website for the purpose of collecting data about the demographic and background characteristics of DSN's website visitors. These studies found evidence that DSN had a positive impact on students and teachers. For instance, after using DSN resources, middle school kids were significantly MORE likely to Demonstrate an understanding of key science and engineering concepts and demonstrate an ability to identify and discuss the science concepts they applied. Demonstrate an understanding of the engineering design process and demonstrate an ability to explain and use the engineering design process. Understand the type of work that engineers do. Believe that engineers have cool jobs. Believe that engineers come up with new ideas and inventions. Understand that engineers design and build things to meet people's needs. Believe that engineers help make people's lives better. Report an interest in designing and buildings things in their classrooms and during the summer. After using DSN, the kids were significantly LESS likely to View engineering as boring. Believe that men are better at engineering than women. Believe that only super smart people can be good at engineering. In addition, roughly half the kids reported that they Were more interested in engineering after completing the DSN challenges. Thought it would be cool or very cool to be an engineer someday. These studies also found evidence that DSN had a positive impact on teachers. Four of the six teachers reported that s/he were more interested now in integrating hands-on design challenges into their classrooms after using the DSN challenges. These same four teachers also told us that the DSN challenges helped them to feel more comfortable providing engineering activities to their students. All the teachers reported that they understood the engineering design process and two of the six reported that their understanding was much better now as a result of using the DSN challenges. Four teachers told us they know more about engineering careers after using DSN. All six teachers reported that they were very likely to use the DSN challenges again. Five of the six teachers reported that they would recommend the DSN challenges to other teachers. Regarding implications for future study, in addition to assessing the impacts of DSN on students and teachers, another major goal of the evaluation was to collect formative data that could be used to redesign the DSN web resources for a future study. In a separate study, we are planning to explore the engineering pathways that middle schoolers at home follow in pursuing their engineering interests. The cornerstone of the proposed study is an enhanced DSN website with more activities, more challenges, and more social interaction designed to support students who are engaging in engineering The evaluation results support the concept that kids can enjoy learning about STEM at home and that the DSN activities can be used successfully in a home setting. For instance, kids in the evaluation reported that they enjoyed the at-home activities. Even though the DSN activities were technically homework assignments, the kids reported that they viewed them as different from their typical homework because the DSN activities were more 'hands-on' and 'fun' The DSN website is already being used by a middle school audience, including kids and families who are integrating STEM learning into their home environments. It will be important to explore how the DSN resources that are available on the Web help to nurture or guide kids along engineering pathways. The results of the evaluation studies described in the full evaluation report have several implications for the proposed study: The DSN website should offer more challenges, including more complicated challenges: Students in the school/home study reported that they would like to see even more challenges added to the website, in addition to some more difficult challenges that older kids might enjoy. The DSN website should be enhanced to make pictures and sketches easier to post and find: Of the classrooms that did post to the website, teachers reported that website limitations on file sizes made the experience of posting less interactive and less useful than intended. The appendix of this report includes the survey instruments used in the study.

If you would like to edit a resource, please email us to submit your request.