1. Introduction

This paper is an extension of work originally presented at the 2021 Integrated STEM Education conference [1].  It utilizes state of the art research methods in the employment of an exploratory research study to understand best practices in transitioning high quality STEM learning environments from in-person to virtual.

Due to the COVID-19 outbreak, and the World Health Organization (WHO) officially labeling COVID-19 as a pandemic [2], K-12 education in the United States would change dramatically.  When stay-at-home orders started going into effect in many states, many public schools were forced to close their doors and move from in-person instruction to online teaching.  Soon after, many STEM educators and researchers, including the National Science Foundation-funded STEM Sea, Air, and Land (SEALs) team from North Florida College (NFC) in Madison, Florida, began to realize the pandemic would not be easily or quickly be curbed.

In early 2020, the STEM SEALs team was planning for some highly engaging educational activities for both educators and students in their six-county service region.  The STEM SEALs team was led by STEM experts from NFC, a rural community college.  The team included educational researchers from Cynosure Consulting, LLC (Cynosure).  These activities included opportunities for participants to build rovers, boats, and drones, which they would learn to code and then maneuver to complete fun, yet rigorous engineering design challenges as part of a weeklong STEM camp hosted on the NFC campus.  This inaugural camp would expose rural middle school students to hands-on engineering and computer science experiences.  Earlier in the fall of 2019, the STEM SEALs team recruited nine middle school educators from the surrounding counties.  This group formed the design team and spent one Saturday a month together, where they tested out the curriculum and shared input on the structure and design of the student experience.

As the pandemic began to unfold, it interrupted the project’s spring plans in which the STEM SEALs team was in the middle of organizing.  In March of 2020, the STEM SEALs team was scheduled to host a large group of educators on the NFC campus to participate as a review team.  As review team members, local educators would learn about the STEM SEALs experience content, give ratings as part of the feasibility testing, and offer critiques that would continue to inform curriculum development and revision.  With the date of the on campus large review team meeting approaching, it was clear the impacts of COVID-19 were only increasing.  Each day was met with considerable conversations and discussions, which started as predictions about when in the spring the review team would meet as planned to whether it would meet virtually or not at all.  Sadly, the STEM SEALs team had to accept that holding a virtual review team was not a viable alternative.  The decision was made to postpone the spring review team meeting altogether.  As the COVID-19 numbers surged in the US and Florida’s own cases began to spike, it became evident that the inaugural STEM SEALs summer camp was in jeopardy of cancellation.

Finally, on April 24^th, the STEM SEALs team was compelled to make a decision about the fate of the inaugural STEM SEALs enrichment camp that was planned for June.  From a national perspective, it became clear that the COVID-19 pandemic was affecting many informal STEM experiences scheduled for the summer.  Despite months of developing curriculum materials, testing out prototypes, and carefully arranging an in-person STEM SEALs camp with safety as a priority, it became evident STEM SEALs could not overcome the effects of the pandemic.  Knowing the importance of the project goals to learn about best practices in offering high quality STEM exposure and interest for middle grades students in rural areas, the STEM SEALs team reluctantly made the decision to go virtual.

1.1.  The Need to Push Forward

Underrepresentation of Rural Students in STEM

The underrepresentation of rural students in STEM is not a recent phenomenon, but in the past two decades, the issue has been receiving greater attention.  Studies and literature reviews have advanced the field’s understanding of the barriers that rural students face.  Rural students struggle with issues of geographic separation and insufficient internet bandwidth to support online access and complete access to many technological advances [3].  They face limited opportunities to engage in advanced coursework in mathematics and science [4], and financial obstacles that limit future employment and educational prospects [5].  School administrative data have shown low participation in advanced coursework among low income, rural students in comparison to students from high-income families.  The result is an excellence gap that is evidenced as early as elementary school and persists through high school [6].  Education researchers assert that this excellence gap “represents a growing crisis requiring programmatic intervention” [7].  Students in rural schools, particularly those that are under-resourced, are less likely to reach advanced levels of academic achievement compared with their urban peers, even when they demonstrate high potential [8].

Challenges and Strategies for Rural Students

Rural students that show high potential for academic success confront barriers that limit options for academic acceleration, putting them at risk of becoming part of the “persistent talent underclass” [9].  Researchers studying this excellence gap have identified innovations that can mitigate access to advanced coursework in high school.  They point to programming outside of school time designed for middle school students as a potential stop gap measure [7].  These kinds of programs are advantageous for multiple reasons [6].  Spending time socializing outside of school increases positive peer interactions and stimulates social development, in addition to academic learning for middle school students [10, 11].  These benefits are larger for at-risk students, for whom researchers have documented a link between extracurricular programming and educational success [12].

Not only do informal educational experiences serve as mitigating factors for poor academic outcomes, but they can also serve as a catalyst for the decision to seek advanced coursework in high school [6].  Researchers voice that for the strategy of improving high-potential rural students’ STEM achievement through extracurricular programming to work, implementers “must also consider the inclusivity of identification models for such programming” [7].  This STEM SEALs project set out to model the potential efficacy of a widely inclusive outreach strategy with the purpose of reaching a broad pool of rural, high-potential students who are ready for STEM development opportunities.

1.2.  Guiding Framework

The STEM SEALs project was always designed to be more than the creation of a high-quality STEM camp.  The work of STEM SEALs was nested within a larger research design focused on efforts to develop rural STEM education pathways and building an evidence base for the emerging strategies and materials with the larger vision of creating broader access to high quality STEM experiences for students in rural parts of the country.  The STEM SEALs project was not simply an outreach or STEM enrichment project, but instead, STEM SEALs was from the outset framed as Design-based Research [13, 14].  Design-based Research has been widely used in education, and curriculum development, in particular, where research and design activities are often inseparable parts of improving current practices and refining design theories and principles [15].  It has also been used extensively for researching and improving professional development [16].  This systematic methodology aims to improve educational practices through a cyclical process that involves iterative periods of design, testing, evaluation, and reflection between researchers and practitioners conducted in real-world environments [17].

Design-based Research has its roots in a larger movement near the beginning of the 21^st century that looked to more effectively bridge the gap between research and practice.  It acts as a practical methodology which serves dual roles for both developing and informing learning theory and the means designed to support that learning [13, 14, 18, 19, 20, 21].  In fact, it has been used to improve STEM education in a variety of ways, including developing new curricula and instructional materials for teaching science online to middle students [22].  In this example, the new curriculum was designed to engage students, including English-language learners and students with a disability.  The Design-based Research process utilized data from multiple sources, including teacher logs, student and teacher surveys, and focus groups.  Results showed the developed curriculum to be feasible, useful, and effective with a diverse student population.  It also demonstrated that Design-based Research is a practical framework in such settings.  The Design-based Research method has also been used in designing forensic science games for middle school students [23] and developing assessment tools for measuring students’ science critical thinking skills [24].  Supporting teachers in implementing new STEM curricula and instructional materials is another context that has utilized Design-based Research methods.  In a recent example, a study by the authors in [25] used Design-based Research to develop a professional development program for supporting teachers in developing children’s spatial reasoning.  The professional development program was designed to help teachers understand the principles of the curriculum and to develop the skills they needed to implement it effectively.  Researchers found Design-based Research to be a catalyst for epistemic change.  Overall, Design-based Research is a promising methodology for improving STEM education.  It is a flexible and iterative approach that allows for the development and refinement of interventions in real-world settings.

The structure of this Design-based Research study was bolstered through the use of a modified version of the Successive Approximation Model (SAM) [26] to ensure the iterative development process (a) occurs in small steps in association with ongoing evaluation that informs iterative changes, (b) supports productive collaboration among project team members, (c) directs energy and resources effectively in order to move efficiently with intervention development, and (d) allows for manageable completion of high quality projects both on time and on budget.  Developers cycle through phases of analysis, design, and development supported by embedded research that routinely provides formative assessment and input to inform the ongoing development efforts.  Central to this approach is the use of an iterative “development-revision-testing” process with teachers and students to ensure materials and activities are understandable, appropriate, and engaging.

Design-based Research has been heralded as a research approach that could help those looking to fill the research-practice gap by bringing educational research closer to the needs of educators in the field.  The use of this approach by the current project provided clear evidence that supports this contention, with the STEM SEALs use of the Design-based Research approach emerging as a highly effective model for promoting the rapid innovation and adaptation needed to develop, implement, and build momentum under the spectra of the COVID-19 pandemic.  Ultimately, the STEM SEALs research project outcomes and findings provide significant evidence in support of the effectiveness of Design-based Research framework for bolstering innovation.  This article will provide an illustration of emergent innovations that resulted from its use by STEM SEALs to foster innovation within STEM enrichment in rural counties and a discussion of the key mechanisms of the approach associated with the bolstering of project success.

1.3.  The Phases of Design-Based Research

The Design-based Research process “consists of four phases: (1) analysis of practical problems by researchers and practitioners working in collaboration; (2) development of new solutions informed by existing design principles; (3) iterative cycles of testing and refinement of solutions in practice; and (4) reflection to produce design principles and enhance solution implementation [27].  Each of these phases was operationalized by the project. See overview in Figure 1.

Phase 1: Analysis of practical problems by researchers and practitioners working in collaboration

Assemble Diverse Team of Researchers and Practitioners.  To address the practical problems that emerged, a project leadership team was assembled with individuals bringing different expertise and skillsets.  The STEM SEALs team included members with first-hand experience as STEM teachers in the rural area, education researchers, and content experts from a rural community college representing the fields of Physics, Engineering, Biology, and Advanced Manufacturing.

Figure 1:  Design-based Research Process Operationalized

Routine and Ongoing Communication with Diverse Team around Issue of the Need for High-quality STEM Experiences in Rural Spaces at the Middle Grades Level.  Through weekly meetings and ongoing project communication and activities, the team engaged in analysis of the existing problems of practice of access to STEM enrichment and capacity deficits in rural areas to come up with the multi-faceted model for addressing those issues within the context of designing a STEM experience that serves the needs of both students and teachers.

Phase 2:  Development of new solutions informed by existing design principles

The second phase was addressed by the collaborative STEM SEALs team through engaging in iterative intervention development guided by the SAM model.

Design Team Collaboration for Curriculum Development.  A key task for the project was creating instructional content that leverages an additional and important group of collaborators – current middle school educators within the rural area.  To do this, a design team was comprised of nine rural educators with expertise in middle grades STEM instruction who hail from the five counties within the NCF service area.  Design team members attended three day-long meetings where they listened to STEM experts, offered input, engaged with materials, and provided recommended changes in the development of the student institute experience.  Design team members provided written and verbal feedback and observation of their use of the materials served as additional data to inform the design.  Evaluators and research members from the STEM SEALs team spoke with members of the design team to further understand and be responsive to the ideas raised.  The specific input received from content experts, best-practice research, and individuals who work with students in authentic education settings provided a strong foundation for the full scope of curriculum, although the details were subject to change during iterative development.

Engagement of Different Experts to Inform Curriculum Delivery Design.  Social psychology research provides a wealth of information on techniques and strategies for supporting underrepresented groups in STEM.  Led by Cynosure’s social science and education researchers, the research team, a subset of the STEM SEALs project team, identified opportune moments within the sequence of the module delivery to elicit conversations and address misconceptions related to who does STEM and what a STEM career looks like.  Additionally, activities supported by best practices in the education research literature were incorporated into the modules delivery to bolster the confidence and self-efficacy of rural students as well as develop a growth mindset and sense of identity within STEM.  For example, students will watch as the experts troubleshoot, seeing productive failure as an integral part of the engineering design process.

Consultation with External Advisors.   The project leaders also consulted with advisory board members to solicit feedback on the process of developing high quality materials and designing engaging STEM experiences.  Advisory board members brough a wealth of expertise on informal STEM and the intersection between faculty, practitioners, and rural populations.  Their feedback was used to refine drafts to maximize implementation with fidelity.

Phase 3:  Iterative cycles of testing and refinement of solutions in practice

Feasibility Testing with Local Educators.  To ensure the curriculum materials were understandable, appropriate, and useful for the intended population, middle grades teachers from the region were recruited to review and react to several of the developed modules planned for the institute.  Reviewers were given the physical supplies and curriculum materials associated with each module to work through independently (with a STEM SEALs team member available for questions and trouble shooting.) After each module review, the reviewers completed a form where they rated the module, reflected on the feasibility of the module within a real-world summer institute with middle school students, and provided big picture and detailed feedback along with recommendations to improve the materials.  Within the feedback, participants rated the intervention (from 1=Strongly disagree to 5=Strongly agree) to assess: (a) ease of use, (b) innovation, (c) value and need, (d) feasibility, (e) potential effectiveness for achieving intended goals, (f) usability, (g) advantages over existing methods; and (h) overall quality.  Educators also rated the degree to which they: (a) would recommend the proposed intervention to schools, (b) would use the intervention themselves, (c) believe the intervention would be effective for preparing students in STEM, and (d) recommend continued development and testing.

As part of the review process, all reviewers were invited to the NFC campus to take part in the module activities during a day-long, more in depth exploration of the materials with the content experts.  The purpose was two-fold.  First, by implementing pieces of the institute with middle school educators the STEM SEALs team could gather feedback on the appropriateness of the language used as well as the assumptions made about students’ pre-requisite content knowledge.  Implementers would glean a stronger understanding of the extent to which guidance is needed for handling lab equipment and hear strategies for helping students stay on task and support them in their learning.  Second, through the reactions and questions of teachers to the modules, implementers could assess the extent to which teachers have mastered the content and gain a better awareness of the content information that will be needed in module curriculum facilitator guides.

Lastly, review team members were invited to attend a focus group to gather quantitative and qualitative evaluation data.  These data helped the team ascertain whether educators (a) view the intervention as demonstrating high quality, innovation, and value; (b) advocate use of the intervention as feasible and needed for schools; and (c) recommend continued development and testing.  During the focus group sessions, the project team member also led group discussion to gather specific comments and suggestions, including review of the implementation guidelines to gather information on potential feasibility and fidelity challenges.

Data were analyzed by the research team to assess the degree to which the curriculum materials are acceptable.  If any module failed to meet the team’s standards, it was revised accordingly based upon feedback.  Examples of revisions included removing confusing elements or adding clarifying directions for equipment use or assembly, substitution of more simplified code, and addition of videos or other resources to extend the learning.

Field Testing with Educator Facilitators.  Local middle school educators were invited to serve as facilitators of the summer institute, under the direction and support of the community college content experts.  During and upon the conclusion of the summer institute, facilitators provided feedback on the experience.  Mechanisms were in place to collect data by researchers, evaluators, and content experts and designers.  These feedback mechanisms allowed for just-in-time adjustments to the experience and served as a record for changes and recommendations for future institutes.

Field Testing with Students.  To assess the usability of the materials, students in middle grades were recruited to participate in summer institutes that allowed them to engage with the STEM SEALs materials and culminate with a design challenge.  Feedback from students was gathered through informal interviews with students during the institute, observation of their affect and behaviors during the institute, a survey soliciting written ratings and recommendations at the end of the experience, and analyses of pre- and post-institute assessments of key anticipated outcomes.

The iterative design-develop-test process involved multiple testing cycles.  Early tests allowed the STEM SEALs team to enact the module curriculums (and design challenge) with the intention of gathering feedback to inform further revisions.  Later tests serve more as a pilot, that is, is a more formal testing of the revised modules (and design challenge) to examine whether the intervention elicits the intended outcomes.

Phase 4:  Reflection to produce design principles and enhance solution implementation

Review and Discussion of Data Analysis Findings.  Data from all the processes described were analyzed and then shared with the STEM SEALs team, who reflected and shifted as needed based on the data.  This phase involved the use of deliberate reflection activities to ensure that sensemaking happens routinely around the contributions, insights, recommendations, and lessons learned.

Surveying Project Members as Mechanism for Reflection.  For example, STEM SEALs team members completed reflection forms separately.  These data were analyzed, and common themes shared during team meetings to solicit further discussion and to form the basis for revisioning efforts.

Reviewing Outside Reactions to Further Stimulate Reflection.  Additionally, the convening of an advisory board and an Expert Teacher Material Reviewer helped to engage in further synthesis, sharing, and outside review and reflection based on experts from the field.

2. Method

While Design-based Research was a lens woven into the fabric of our study originally, its application was indispensable during the fast-shifting events that followed the advent of the COVID-19 pandemic.  The designers and implementers of the innovation had to make rapid decisions.  It was unclear whether the situation would be replicated, and it felt important to be mindful when cataloguing decisions and their rationale under the current context.  Suddenly, there were new questions raised, ones whose answers could quickly and meaningfully contribute to the field.  The researchers on the STEM SEALs team chose to adapt the Design-based Research approach to create a rapid learning process to align with the quick pace of the pivots made by those implementing.  The results were an approach that situated rapid cycles of iteration within a modified Phase IV Design-based Research strategy, with the aim of engaging in strategic data-informed efforts to successfully navigate the pandemic-mandated pivot from an in-person STEM enrichment event to a virtual offering.  What ultimately resulted was a series of rapid Design-based Research cycles.  Within each cycle the barrier is identified, solutions explored, alternatives analyzed, and a decision is made. See Figure 2.

Figure 2:  Design-based Research Rapid Cycle Iteration Process

Ultimately, Design-based Research, on overdrive, if you will, became the methodological mechanism for both informing and capturing the important learning during this time.

Thus, the over-arching research question guiding this effort was, “To what extent does a Design-based Research approach support the development and rapid adaptation needed to provide enriching rural STEM experiences?”

To answer this question, an exploratory study (described next under “Exploratory Approach”) was developed, guided by Design-based Research, in which existing data (that is, data collected for other purposes) were leveraged along with the collection of additional data, tailored to addressing the specific needs of the Design-based Research rapid cycle iterative design effort (described under “Data Sources”).

2.1.  Exploratory Approach

The findings presented in this article were generated utilizing a qualitative framework at the level of exploratory analysis with the goal of garnering insights on the emergent success of the approach at promoting adaptation and innovation associated with the project’s success at navigating the challenges and barriers presented by the sudden shutdown of schools and bans on gathering associated with the initial months of the COVID-19 pandemic.  Exploratory studies are a type of preliminary research that provides initial information and sense-making.  It serves as a foundation for later efforts poised to yield more conclusive findings [28].  Selecting an exploratory design offers several advantages aligned to the goals of this study, including: 1) affording researchers flexibility to adjust and adapt as research advances; 2) permitting researchers to recognize parts of a phenomenon that merit additional study earlier in the process; and 3) assisting other researchers by identifying potential causes of and solutions to a problem, that may be more intensively studied in the future.  Additionally, the potential to encourage rapid learning, increase uptake, and share knowledge with the field means the study approach can be responsive to the larger national need for expertise on how STEM enrichment providers can transition from traditional approaches to innovations such as virtual, asynchronous, or blended content delivery models.

2.2.  Design-based Research Rapid Cycle Data Sources

Data were collected from various sources to make explicit and unpack the STEM SEALs team’s reasoning before, during, and after making the decision to pivot to on online delivery.  The utilized data sources include the following:

Project Team Meeting Participation and Minutes

The STEM SEALs team convened on a weekly basis in order to document the project’s activities and accomplishments, discuss challenges and issues, make key decisions, and plot out upcoming tasks.  Discussions were recorded and meeting notes summarized to document important decisions and their rationale.

Project Team Reflection Survey Responses and Transcript

The STEM SEALs team members were each given a reflective survey that they completed individually.  These surveys were compiled and analyzed by the research team and shared formally through a group reflection activity.  The analysis and the transcript from the reflection activity meeting served as data sources for this study.

Internal Review of Virtual Delivery Methods and Materials

Instead of engaging an outside review team during the rapid cycle, the research team asked the STEM SEALs members who had developed the materials to engage in a virtual walk through of the rover experience.  This included facilitating a session where the research team coded the Micro:bit, assembled the rover, and manipulated the rovers to accomplish tasks.  The review was conducted through the online Zoom platform for two days, March 27^th and April 3^rd.  It served as a critical initial field test of the material implementation virtually.

Research Team Facilitated Group Review, Analysis and Decision-making

Data that were collected related to barriers identified were routinely analyzed and presented at STEM SEALs team meetings.  Typically, the analysis summaries were provided in advance and explored together during a facilitated virtual meeting.  Ultimately, the meetings ended with a decision on a solution or response to the identified barrier.

2.3.  Design-based Research Rapid Cycle Product Test Stage (the Camp) Data Sources

Data were leveraged from several sources to serve as evidence of success of the product of the Design-based Research rapid cycle process.  In this case, the product was the camp itself.  An inventory of these data sources included:

Observation of Virtual Camp Experiences

The virtual STEM SEALs experience was held July 8^th through 17^th of 2020.  The virtual camp involved resources cataloged and accessible to participants using the Google Classroom platform.  Support was offered by NFC staff through live in-person demonstrations and discussions and as-needed asynchronous support.  These support sessions were recorded as part of the observation data collection.

Data from Teachers

Data generated from pre- and post-camp surveys captured information on educator backgrounds and perceptions of STEM.  Teachers also completed reflection surveys at the end of activities during the STEM SEALs summer camp.  The purpose of these reflections was to better understand how teachers and their students experienced the materials and to collect any recommendations for improvement.

Data from Students

Data was also collected through student surveys.  These surveys asked students about their background and knowledge of STEM at the end and the beginning of the camp.  Students also completed end of module assessments to document their learning.  Lastly, students took part in an end-of-camp reflection exercise where they were asked to look back on their time at the camp and provide their impressions of the experience as well as offer any recommended changes.

2.4.  Data Analysis

This exploratory study, situated within the larger project, involved multiple layers of data analysis.  Three different perspectives were adopted to address the research question: 1) to delineate the role that emerged for Design-based Research within rapid applications, 2) to describe the contributions and outputs of the rapid Design-based Research cycles on the resulting product, and 3) to test the resulting product.  In addressing these aims, qualitative data analysis techniques were predominantly employed.  The data analysis strategy varied based on its purpose within the rapid Design-based Research cycle.  The data were synthesized systematically and then objectively analyzed using mechanisms that identified key themes.  In some cases, quantitative analysis techniques were further incorporated through summarizing program tracking and survey assessment data.  Survey data were analyzed with the statistical software Stata (Version 13) and descriptive statistics were computed.  Data verification strategies were incorporated within and across the data analysis activities.  Triangulation of findings was conducted such that meeting documentation, field records, and transcripts were referenced as the researchers on the team utilized an iterative process of detecting and categorizing emerging themes, then cross-checking those themes with the various sources of data for confirming evidence.

3. Results and Discussion

In early 2020, the STEM SEALs team began to finalize the first inaugural enrichment camp where student participants would be immersed in autonomous and remotely controlled robotic devices.  The event was scheduled to reside on the NFC campus and would serve up to 48 participants.  Then the pandemic struck.  The team looked to regroup with the uncertainty of how to implement the camp while also complying with safety provisions that required social distancing.  To even attempt, the STEM SEALs team narrowed in on the robotic rovers and decided to focus specifically on the activities that would culminate in a Land Challenge.  Figure 3 displays an image of the rover students would assemble, code, and operate.  To lessen obstacles in an already challenging time, recruitment for the first camp was restricted to those educators who had been involved with STEM SEALs as a member of the design or review team.  For those educators, the student participant pool was assembled.  The student pool was restricted to those individuals who resided with the educator or those whom the educator was in regular close contact (e.g., a grandchild).

Figure 3:  Image of Assembled Rover

The camp activities were organized into six modules (see Figure 4).  These modules included lessons that introduced students to the overarching engineering design challenge which was the focus for the weeklong summer experience.  Following the curriculum timeline, next students would be exposed to coding using the Micro:bit and then begin their construction of the rover.  Once built, the students would learn how to use the Micro:bit to control and navigate the rover.  The week culminated with a competition tied to the engineering design challenge.

STEM SEALs staff mailed to all participants kits that contained pieces of the rover and assembly tools, as well as binders that served as manuals for the camp.  The camp officially began on July 8^th and lasted until July 17^th.  A total of 29 teachers and students participated.  The camp included online meetings each day.  These meetings provided a space for students to receive help from the NFC expert team as desired.  Teachers were also available to support and aid students as best they could while also working through the STEM SEALs learning modules.

Figure 4:  STEM SEALs Camp Content Overview

Individually, students met at the competition site and had a chance to show off their rovers and compete in the challenge.

Feedback on the camp was offered in multiple ways, mutually supporting the value of the experience on student STEM learning.

For example, students said:

I personally liked learning the coding processes that went into coding the Micro:bit.  Learning the code and seeing it work was really satisfying.

I learned a bit more about the electromagnetic scale and got a more in-depth description of how radio waves communicate with each other.

I learned about how even computers use a simulated sense of echolocation to decide how far an object is from it and the patterns it uses to get around the obstacle.

3.1.  Findings

The findings section includes a presentation of the iterative design process outcomes along with documentation of the emergent Design-based Research roles and activities at each point in the process.  The goal was to document the key decisions at each point in the process and the underlying STEM best practices that were instrumental in the success of the pilot.  Ultimately, the findings are in service to a larger question of whether Design-based Research is a good fit for developing high quality STEM experiences, and especially so in situations that require rapid decision-making and significant pivots over time.

Making the tough call

Recognizing that it was now or never.  Eventually, the time came when a decision would be needed before the window of opportunity for holding a virtual camp would close.  Fortunately, the entire project team recognized that there was no more time to wait or debate.  They made the final decision to abandon the idea of an in-person event and instead, move forward with a virtual camp.

Although the pandemic forced US schools to close their doors in late March and early April of 2020, the idea that offering an in-person summer camp opportunity might be in jeopardy was not a consideration, at first.  The STEM SEALs team initially thought the county and schools would open in plenty of time to move forward with an in-person summer camp.  However, as the pandemic dragged on, the STEM SEALs team began to doubt whether an in-person event would be feasible.  When no break in the social distancing restrictions was visible on the horizon the team started to acknowledge that virtual might, in fact, be the only option available.  Reluctantly, the team accepted that the inaugural STEM SEALs camp offering would be held as a virtual event.

Early in this process the research team recognized that they were going to be entering into uncharted territory.  The traditional approach with well-laid out, pre-determined research activities would need to be paused in favor of strategies that would align to the rapid switch and emerging needs of the new design efforts.  As had been the structure of the partnership since the beginning of the grant period, the research team continued to engage in weekly meetings with NFC expert team.  It was through these meetings that the new Design-based Research approach began to provide support aligned to the rapidly shifting design efforts.  Within these meetings, the research team would identify a welcomed and impactful role as an external sounding board as the NFC expert team collectively began confronting the realization that the social distancing mandates put in place to slow the spread of COVID, might potentially prevent them from hosting an in-person camp as planned.  With so much work completed already and so much excitement building to kick off the seminal activities of the project, it was understandable how reluctant the NFC expert team was to accept that the event could not happen as planned.

The research team helped to nudge the STEM SEALs team toward what they recognized as a necessary pivot during the meetings, sharing about other groups that they were working with who had already made the decision to pivot and providing encouragement that it could be done with STEM SEALs as well.  Perhaps more importantly, the research team listened to the thoughts and concerns that were being voiced by the team members and adopted a formal role of providing formative feedback that redirected focus back to areas where issues had been raised, but not yet fully addressed.  Ultimately, it was in this role of re-voicing that the research team significantly contributed to the making of a timely decision.  The research team recognized that the logistical needs and timeline concerns frequently raised by the project manager needed to be highlighted.  This Design-based Research activity was simple, but it proved essential through helping to direct the NFC expert team’s focused attention on fast approaching deadlines before they passed.  For example, the research team brought up the required timeline needed to successfully engage in recruitment – along with highlighting the long list of activities that would need to occur beforehand.

Expectation setting

Avoiding the “Anything is Better than Nothing” Mindset.  Initially, when the STEM SEALs team realized that there was no way that the pandemic conditions would resolve in time to host an in-person event, two competing mindsets emerged: 1) If we can’t do it the way we envisioned, then there is no point in doing it, and 2) We need to do something and anything we do is better than nothing.  The team gradually began to embrace the notion that flexibility in the original vision was necessary.  However, the team was also firm in not wanting to water down the student experience or alter the main activities that had been so carefully selected.  Ultimately, it was the team’s deep-rooted commitment to find ways to preserve the foundational elements of STEM SEALs, that propelled them to be able to do what at times seemed impossible, rather than reluctantly shifting toward the second mindset.  The commitment to this mindset was an essential component of their persistence and willingness to innovate and adapt to find ways to engage students in a virtual experience that would afford opportunities for them to assemble a rover and to write and run the code that would allow it compete in a real challenge course.

As done previously, the research team engaged with the NFC expert team members weekly with the intention of documenting the process, collecting formative data, creating feedback loops for sharing back findings, and supporting the use of findings to inform continued adaptation and revision.  Consistently, but in a much more fast-paced manner, the Design-based Research team had to sacrifice some aspects of rigor to ensure that thoughts were shared in time for decisions to be made.  During this time, the research team listened and asked questions, trying to understand more about points of disconnect and indecision and quickly recognizing that the project had yet to engage in the best practices to establish a collective vision.  The steps taken to elicit that vision were ultimately very worthwhile, because while many similar efforts pivoted with the “anything is better than nothing” mindset guiding their work, avoiding that mindset was a very important goal of the STEM SEALs team.

The fear that their inaugural effort would be something that lacked the flavor and rigor of what the team had been excited to offer in the original format, was something that permeated their early planning conversations.  It was not until they recognized that there were central aspects of the STEM SEALs engagement that would have to be incorporated, or else the team preferred to abandon the idea rather than try to offer something less.  Through questions designed to identify the components of the vision that individuals valued and for which there was significant consensus, the research team supported the NFC expert team in identifying the elements that were believed to be fundamental to the project:

• Providing an opportunity for student participants to construct their own rovers,
• Teaching them how to program a Micro:bit to use in controlling the rover, and
• Allowing them to use their acquired skills to complete a challenge course in competition of some form with others in the camp.

Identification of the key challenges

Keeping Students on Track and Maintaining Pace.  Quickly the STEM SEALs team noted how the shift from an in-person to virtual delivery would affect student pace and their ability to note slowdowns and intervene.  In the face-to-face delivery as originally envisioned, students could work at their own pace, but that pace would be watched by the teachers and the NFC expert team so they might intervene with support when students got stuck or off-task.  The teacher facilitators could assist in making sure all students were successfully able to complete the camp’s essential elements.  But, the virtual environment did not have the same level of oversight.  The team wondered how the organization of the camp content could be structured in such a way that all students were able to follow along, stay on track, and proceed at their own pace.

Finding an Effective Substitute for In-Person Demonstrations.  In the original, in-person design, the NFC experts planned to provide additional scaffolding for the teacher facilitators and their students through on-demand demonstrations.  Thus, the STEM SEALs team had to grapple with the issue of defining and crafting further resources to set students up for success, but without the benefit of on-the-spot, step-by-step, in-person demonstrations.  Another related loss was that the in-person demonstrations also afforded students the opportunities to watch other participants and learn through watching and benefiting from other students’ efforts to progress through the process.  Finding ways for students to receive the instruction needed for them to build their rovers successfully in lieu of the in-person strategies would be a challenge the project would need to overcome.

Providing Students with the Personalized Support Needed to Stay Engaged.  The activities and learning tasks associated with the STEM SEALs modules are designed to be novel and challenging.  In a face-to-face setting, students would be monitored and supported, but the team recognized that the virtual format potentially left students unprepared and without sufficient support to experience the desired level of success.  Without a sufficient level of support, the STEM SEALs team feared the students would not stay engaged or be successful.  This facet was complicated because the team recognized that the rigor of STEM exposure that was envisioned provided opportunities for students to grapple and problem-solve and with appropriate support to thrive.  Without appropriate support most would likely end up frustrated or, even worse, quit in the face of what they perceived to be their failures.  Ensuring sufficient support and scaffolding for students engaging in STEM is challenging in any format, but during a pandemic, through a virtual interface, within a rural setting, and maintaining social distancing guidelines providing suitable support posed an enormous barrier.

As the team forged forward, much of the early conversations were around concerns about what would not work or what needs students would have that would not be easy to meet in a virtual format.  In some ways, these conversations sounded a bit like a listing of the top  ideas why there was no way to offer the STEM SEALs vision in a virtual format.  Eventually, there would be recognition that all most all their concerns and what ifs emanated from four primary hurdles that the team would need to overcome to have confidence that the vision for the STEM SEALs camp could be carried out successfully in a virtual format.

The research team continued to engage in routine conversations with the group, listening to the discussions about moving forward with an ear toward extracting the core set of actionable barriers that were being alluded to by the team members across the discussions.  Often in the team discussions, team members tended to talk about different challenges they could foresee:

• What if some students struggle and are not able to keep up while others may try to jump ahead before they are ready?
• We can’t be there to be one-on-one with the students and so how are they going to be supported when they struggle?
• Not everybody might be able to be online at the same time, so what if some of the students need to engage asynchronously?
• How are they going to be able to follow along with a virtual demonstration and either do it at the same time or go back and do it later?

Employing a qualitative lens in analyzing the list of challenges that had been voiced, the research team was able to extract a much smaller number of key challenges to maintaining the foundational elements of STEM SEALs as a virtual offering; ultimately helping distilling concerns to identify the four key challenges listed above.

Identifying promising strategies to address the core challenges

Placing Significantly Greater Importance on Student Curricular Materials.  The STEM SEALs team worked through each of the challenges, drawing on research, holding discussions, and providing time for reflection.  First, how would the team provide the foundational elements remotely?  There would need to be very close attention to the materials.  STEM SEALs team members would have to think like a student and what would be available to students.  For example, students will not have dual monitors.  If students were using the computer to talk with an expert or watch a video, they would need directions, durable directions, to follow and track their progress for the rover assembly.  The team used cards with detailed pictures to give students the support they would need for the technical assembly.

In the words of a dyad leader…

Activity cards pictures really helped clarify the steps of each activity and the coding required. Anywhere that more pictures can be added at the various stages of the rover building would help improve understanding of the required activity.  The students tended to use the images on activity cards instead of instructions. 

Taking Advantage of Existing Virtual Platforms.  Ensuring students stayed on track when working independently through online materials seemed like a large issue initially, but ultimately the team noted it was likely a familiar challenge for anyone offering remote learning.  Therefore, the team consulted existing platforms for virtual environments.  They landed on Google Classroom, which had the advantage of being designed for students of this age and which students may already be familiar.  As the team worked to integrate their materials into the Google Classroom platform, they noted a number of helpful features.  There were features for presenting content in modules, similar to the design on the original face-to-face camp, and it possessed tools for controlling how far students could progress (e.g., the team set controls for what students could see and click on) and built-in monitoring to track students efforts (e.g., the team could see what students have clicked on and viewed as well as set mini-assignments that showed what had been completed).

Provision of Numerous Videos.  Given the critical need for teachers and students to see and observe how to perform certain elements, the team turned first to existing, publicly available videos.  Videos would be needed to explain and extend the camp content and could be used to demonstrate the larger context and relevance of the camp by featuring real-world applications.  They culled videos from a number of various sources to create a video library for participants.  They would also need videos to demonstrate camp-specific activities, so they began shooting videos of themselves performing STEM SEALs activities.  These videos would be strategically placed within the STEM SEALs materials so that students could view the relevant videos as they navigated through the STEM SEALs experience.

In the words of dyad leaders …

The resources are very thorough. Dr. Maresch’s video on propulsion was very interesting but the manual programming of the pins was a little confusing considering the knowledge wasn’t necessary for the programming tasks in this module…The remainder of the video was very helpful, especially the speed and steer demo and explanation.

In fact, the research team received feedback that more and shorter videos would be helpful.  In their words,

To keep the student’s attention, the video length needs to be kept at a length of 10 minutes or less.  For those struggling, a suggested more detailed tutorial video can be uploaded at the end of each module.

More short videos to explain what is happening at each step and what to expect would be great if available to instructors.

Using Dyads.  Part of the goal of the STEM SEALs experience was to interest students in STEM careers.  To do so, it was important to get and keep students highly engaged throughout the camp.  Additionally, students’ perception of the camp might be unfavorable if they do not perform well in the culminating Land Challenge.  The camp activities were designed to build upon each other, so it was vital that students successfully complete every activity.  The STEM SEALs team also recognized that with most of the student participants being middle-school aged, they would need to have a consistent adult to consult with when problems arose and to intervene when their pace slowed.  In response to these factors, the team decided to use dyads, that is, each student was paired with an adult, preferably within their household, who could provide that more intimate, just-in-time support and encouragement, and who would help connect them with the experts and other resources needed to be successful.

Expert Office Hours.  While the selection of a dyad deliver model addressed many of the issues around student engagement, the teacher facilitators could not be expected to have the technical expertise to solve all the potential challenges that could surface.  Experts must be accessible to support the dyads, and available in a timely fashion for students to stay on pace.  The team had to think through how to provide the technical expertise and flesh out a schedule of their availability to align with the workflow of students.

In the words of a dyad leader on the strengths of the camp…

The willingness of the team to assist with correcting programming. It takes a village:)

Armed with a clearer understanding of the challenges that threatened their ability to successfully engage students in a virtual STEM SEALs event, the team was positioned to move forward in rapid fashion.  The research team leant their hands to the effort, increasing their role and contributions as design team members helping to search out aligned best practices, identifying existing web resources that could be leveraged, trying out PowerPoint for sharing instructional tutorials, and most notably, suggesting one of key strategies that would ultimately prove most critical to the success of the effort: the dyad model.  In December of 2019, when the project was happily proceeding as originally proposed, one of the research team leads, engaged in an observational visit to attend the final meeting of the design team that had been convened to support the NFC expert team in developing curriculum materials aligned to middle school standards and students.  At this meeting, a research team member spent the day interacting with the design team members and getting to know more about their backgrounds, interests, and motivations for participating.  Several of the members were motivated to participate both by professional and personal desires wanting to be able to provide the best opportunities for their students while also wanting to learn more about ways they could position their own children to be more STEM-able.  During the downtime, the researcher noted that several were looking at Amazon and placing orders for things that they had been using at the camp.   When asked about the purchases, the researcher learned that they were things they were buying because they thought they would be great for their own children.  Grounded in these observations, the research team recognized that one of the most significant threats to being able to provide a positive experience in a virtual camp was a lack of sufficient support for students as they grappled with very new and challenging content, would be best addressed by moving to a model that looked to recruit individuals who resembled the design team members.  Individuals who possessed the instructional ability to support students and a desire for their children to have greater access to STEM enrichment.  This practice of recruiting paired participation of a teacher/educator and a child was the foundation of the dyad approach that was utilized for the virtual offering.

Taking the virtual camp out for a test run

Having Mechanisms for the Expert Team to Troubleshoot would be Essential.  After figuring out the strategies and refining the materials, the project team was ready to try out some of the planned camp activities with a group of novices who could provide feedback about what was working and where more development efforts could be directed.  The process worked well and provided some good feedback.  The initial review, however, was much more influential in preparing the project team for success by making sure they were prepared to address a need that had not previously been fully illuminated, “How could the team troubleshoot in a virtual environment?”.

The research team, which typically includes expert review and feedback gathering as a key step in the Design-based Research process, recognized that providing an opportunity for the project team to engage in a test-run and collecting review feedback would go a long way toward strengthening the efficacy of the planned effort.  With very little time and significant barriers to getting outsiders materials, sufficient contextual information, and requisite content knowledge, the research team, who did not have prior firsthand exposure to the actual rover materials, signed up as novices to engage in a virtual run through of the rover construction process.  Starting with the same unassembled materials that would be provided to participants, the expert instructors led the research team members who were working at remote sites independent from one another in constructing the rovers.  The session proved immediately fruitful as one reviewer struggled to easily identify the front of the rover body frame from the back, and as the challenges of trying to screw in the tiny screws were addressed with some additional tips that would be added to the materials before the camp.  Gathering feedback of this nature is an explicit reason for engaging in the review and provided the value-add that was anticipated.

However, the ultimate value of the review would quickly surpass the value envisioned with disaster striking and from that disaster an extremely significant innovation would quickly take form.  During the test-run review session that was conducted, when one of the two reviewers was testing out recently programmed code that had been transferred to the Micro:bit controlling the rover, the reviewer had the rover placed on their desk in front of them, but as soon as the code sequence was initiated the rover’s motors controlling the wheels engaged and the rover quickly drove itself off the desk surface and came crashing to the floor.  The rover was rendered inoperable, and panic started to set in for both the expert team and the reviewer.  Solidified in this moment would be a need for the expert team and the participants to connect in ways that afforded opportunities to troubleshoot and come up with solutions when things did not work or go as planned.  In this case, the panic quickly subsided as the expert instructor who was connecting with the reviewer via Zoom instructed the reviewer to position their camera such that he could survey the damage.  After which, the instructor using the white board feature and his own camera position on his rover helped to talk the reviewer through the steps that would be needed to return the rover to working order.  Ultimately, the review effort helped to solidify a need and strategy for how expert instructors would be able to connect with participants to provide the one-on-one technical assistance that would be needed to help diagnose problems and scaffold solutions.

Fulfilling the promise of a design challenge event

Finding an answer for, “Where is the Challenge in that?”.  With the relief that strategies had been incorporated to address all the barriers to success that that had once made a virtual STEM SEALs seem destined to fail, the project leadership did not pause to celebrate, noting that the full STEM SEALs vision centered around the entire experience of learning being grounded in the pursuit of competing with peers in tackling a related design challenge.  So, they continued to iterate, recognizing that the virtual format would make it very challenging for students to compete on a similar course and really have the opportunity to apply and test out their new learning and skills in pursuit of a context embedded design challenge.  The design team noted that, to drive the rovers, students would need large, flat, smooth surfaces; something that is much harder to locate in rural areas.  The team was greatly concerned that building rovers that could not be successfully driven would leave students disheartened.  Ultimately, they believed that learning that does not culminate in a real-world application was below the rigor that they wished to ensure, and they further recognized that completing the camp and competing in isolation would further limit the impact of being part of a cohort of participants who had a shared experience.  With that in mind, the team identified a strategy to ensure every student was positioned to be able to successfully engage in the design challenge competition, they arranged for students to compete individually at a parking lot, but live streamed so that students could watch each other and share in the excitement.

Documenting the unintended benefits and lessoned learned

In this step, the focus was on reflective learning toward documenting, 1) emergent benefits that were beyond those originally targeted by the STEM SEALs summer enrichment activity, and 2) emergent best practices that have merit for moving beyond just the current offering and incorporated into future project design and development efforts.  As such, this process of taking stock to synthesize learning was one that was initiated and led by the research team which synthesized data from their observation of virtual camp activities, student surveys, dyad teacher surveys and focus groups, and project team reflections.  From this work, important outcomes in the form of emergent benefits and best practices were identified and documented.

Increased Versatility.  The shift to an online delivery model made the STEM SEALs land experience instantly more versatile and resulted in quicker progress towards leveraging virtual platforms.  Access to a STEM SEALs experience remotely would be very beneficial given its goal of appealing to students in rural areas.

Strengthening the K-12 Education Connection.  The dyad framework also had unintended advantages.  By providing STEM exposure and professional development through the virtual camp, had led to a stronger continued engaged by educators who maintain interested in the STEM SEALs grant and larger mission to build a STEM ecosystem in the area. By connecting directly with teachers and then connecting teachers to each other and the STEM SEALs development work, the project has made swifter progress in creating a core group of educators who can serve as ambassadors as the College looks to strengthen its connection to regional K-12 institutions.

Emergent best practices identified to include in future development efforts were:

Elevating the Framing of Failure.  Frustration can intensify quickly in virtual spaces when students are not able to observe others experiencing the same challenges or are spending more time grappling before they are able to receive support or technical assistance.  The move to an online offering placed a larger spotlight on addressing failure and led to more proactive work to frame failure as normal and positive.  In the words of a dyad leader …

Consider that middle schoolers’ need consistency across directions, platforms, and materials.  It is hard for them to have materials vary in information when in truth the purpose is for the material to match.  Many of them get so frustrated they shut down.  They love learning computer applications, doing their assignments online. coding is interesting to them; however, many get very frustrated.  They do not understand that frustration is good/ a part of real learning.

New Approach to Differentiation.  The STEM SEALs had known that the participants would be a diverse group, regardless of whether the camp was delivered in-person or remotely.  This diversity would mean that students would be bringing different sets of pre-existing knowledge and experience and would vary in the kind and amount of support needed.  Given the virtual format and much of delivery asynchronously, there was a need to keep instruction shorter, and more condensed.  While in-person delivery might have allowed instructors to shepherd students down a more common pathway, the student pathway through the materials would likely be more disparate in the pivot to online.  The team saw how alternate pathways that culminate in a singular outcome would need to be crafted.  With multiple pathways available, instructors would have the opportunity to steer students toward the appropriate pathway so all students could feel successful progressing in a way that was aligned to their particular prior knowledge and current expertise.

Need to Connect Students to Each Other.  The research team had noted the importance of belonging and community in fostering a sense of STEM identity and interest in STEM, leading to student choices to pursue STEM careers, a goal of the project.  But, supporting emerging friendships and creating bonds between participants in a virtual environment was an anticipated challenge.  While the team had great success in connecting students to facilitators with the dyad support model and in connecting students to experts with the videos and virtual meetings, they had less success connecting students to each other.  The team tried to use participant self-introduction videos, videos to showcase participant work, and other strategies to promote team building and student to student connectivity, but these attempts ultimately fell short.  New adaptations are currently being considered.  In the words of a dyad leader …

With the online program this year, there was not a lot of peer student interaction which many teachers felt would have helped the students to work through the many difficulties they had.  Most instructors agreed that the materials provided opportunities for students to express, clarify, justify, interpret, and represent their ideas (i.e., making thinking visible) and to respond to (some limited) peer and teacher feedback.

4. Conclusion

The contributions of this exploratory study of applying the Design-based Research approach within rapid change efforts in STEM enrichment resulted in important contributions to the field in two areas.  The first of which is advancing best practice strategies for taking in person hands on STEM enrichment into asynchronous virtual delivery applications.  The context of providing high quality STEM enrichment in isolated rural schools areas where connecting students with opportunities and expertise to promote rich engagement in STEM are limited and where creating greater access for rural students is further hampered by transportation and connectivity barriers.  Ultimately overcoming these barriers may necessitate expanding opportunities for rural student participation asynchronously in virtual STEM offerings.  Research reviews have shown that studies of both synchronous and asynchronous methods of online learning can be effective [29].  The findings of this exploratory study identify emerging best practices for offering virtual STEM opportunities in a rural setting that are able to meet this high standard have many important implications.  The findings lend evidence to support the success of STEM SEALs in offering a highly engaging and successful inaugural summer enrichment experience in an asynchronous and virtual format, and more importantly, provide rich insights to expand the literature on best practices for virtual STEM enrichment programming.

Secondly, the findings of this study are significant because they also provide initial evidence of the viability and value of expanding the knowledge base around best practices for embedding Design-based Research within rapid design efforts.  While Design-based researchers have established well-supported best practices and a strong evidence base to promote its routine inclusion in STEM enrichment design efforts, the application of Design-based Research in rapid cycle initial design efforts has not been as well researched [30, 31].  While the pandemic forced widespread pivoting throughout the STEM landscape, the hope is that it will remain an isolated event and that no similar mass need for pivoting will occur.  Yet, the rapidly advancing technological will constantly push for rapid development and rapid innovation within STEM enrichment offerings, amplifying the need for greater exploration of Design-based Research methods within rapid cycle design efforts similar to the example provided in this manuscript.       The STEM SEALs project efforts to engage Design-based Research to support rapid change efforts represents an important contribution to the field with the findings from the initial test of the offering strongly supporting its effectiveness.   The evidence gathered showed Design-based Research as a valuable tool for program improvement, even in the most extreme situation of a global pandemic, and in important contribution to best practices in developing STEM learning experiences.

Conflict of Interest

The authors declare no conflict of interest.

Acknowledgment

This material is based upon work supported by the National Science Foundation under grant no. 1812913.

1. R. Zulli Lowe, A. Smith, C. Prout, G.G. Maresch, C. Bacot, L. Murfee, & B. Eustace, “Taking STEM enrichment camps virtual: Strategies & reflections from quick pivot due to COVID-19” In 2021 Integrated STEM Education Conference, 83-90, 2021.
2. World Health Organization, “WHO Coronavirus Disease (COVID-19) Dashboard[Online],” https://COVID19.who.int/?gclid=CjwKCAjwiMj2BRBFEiwAYfTbCod9hYnKI6jtuqIoHEVnS tP_VrjFu3NFnfkKJSgh9rZcBuI7mtmfDhoCuYcQAvD_BwE, 2020.
3. K. Spencer, “Not all towns are created equal, digitally: How a Colorado school district struggles to give its students a technology boost (The Hechinger Report),” Teachers College at Columbia University, 2017.
4. National Science Board, “Science and engineering indicators 2014,” National Science Foundation, 2014.
5. R. Lapan, M. Aoyagi, M. Kayson, “Helping rural adolescents make successful postsecondary transitions: A longitudinal study,” Professional School Counseling, 10, 266-272, 2007, doi:10.5330/prsc.10.3.u6j3j64h48p27w25
6. J. Plucker, B. Harris, “Acceleration and economically vulnerable children,” In A nation empowered: Evidence trumps the excuses holding back America’s brightest students, Belin- Blank Center for Gifted Education and Talent Development, 181-188, 2015.
7. S.G. Assouline, L.M. Ihrig, D. Mahatmya, “Closing the excellence gap: Investigation of an expanded talent search model for student selection into an extracurricular STEM program in rural middle schools,” Gifted Child Quarterly, 61(3), 250 –261, 2017.
8. T. Kittleson, J.T. Morgan, “Schools in balance: Comparing Iowa physics teachers and teaching in large and small schools,” Iowa Science Teachers Journal, 39(1), 8-12, 2012.
9. J. Plucker, J. Giancola, G. Healey, D. Arndt, C. Wang, “Equal Talents, Unequal Opportunities: A Report Card on State Support for Academically Talented Low-Income Students,” Jack Kent Cooke Foundation, 2015.
10. J.S. Eccles, B.L. Barber, M. Stone, J. Hunt, “Extracurricular activities and adolescent development,” Journal of Social Issues, 59, 865-889, 2003, doi:10.1046/j.00224537.2003.00095.x
11. P. Olszewski-Kubilius, S.Y. Lee, “The role of participation in in-school and outside-of-school activities in the talent development of gifted students,” Journal of Secondary Gifted Education, 15(3), 107-123, 2004.
12. R. Gira, “The challenge: Preparing promising low-income students for college,” In Overlooked gems: A national perspective on low-income promising learners: Conference proceedings from the national leadership conference on low-income promising learners, National Association for Gifted Children, 69-74, 2007.
13. T. Anderson, J. Shattuck, “Design-based research: A decade of progress in education research?” Educational Researcher, 41(1), 16–25, 2012, doi:https://doi.org/10.3102/0013189X11428813
14. T. Štemberger, M. Cencič, “Design-based research in an educational research context,” Journal of Contemporary Educational Studies, 65, 90-104, 2014.
15. E. Oh, T.C. Reeves, “The implications of the differences between design research and instructional systems design for educational technology researchers and practitioners,” Educational Media International, 47(4), 263-275, 2010.
16. D. Zinger, A. Naranjo, I. Amador, N. Gilbertson, M. Warschauer, “A design-based research approach to improving professional development and teacher knowledge: The case of the Smithsonian learning lab.” Contemporary Issues in Technology and Teacher Education, 17(3), 388-410, 2017.
17. F. Wang, M.J. Hannafin, “Design-based research and technology-enhanced learning environments,” Educational technology research and development, 53(4), 5-23, 2005.
18. T. Amiel, T. Reeves. “Design-based research and educational technology: Rethinking technology and the research agenda,” Educational Technology & Society, 11, 29-40, 2008.
19. A. Bakker, “An introduction to design-based research with an example from statistics education,” Approaches to qualitative research in mathematics education: Examples of methodology and methods, 429-466, 2015, doi:10.1007/978-94-017-9181-6_16, 2014
20. S. Barab, K. Squire, “Design-based research: Putting a stake in the ground,” Journal of the Learning Sciences, 13, 1-14, 2004, doi:10.1207/s15327809jls1301_1.
21. C. Pardo-Ballester, J.C. Rodríguez, “Using design-based research to guide the development of online instructional materials.” Developing and evaluating language learning materials 86012,” 2009.
22. F.E. Terrazas-Arellanes, L.A. Strycker, E.D. Walden, C. Knox, “Development of a middle school online science curriculum: Lessons learned from a design-based research project.” In Handbook of Research on Diverse Teaching Strategies for the Technology-Rich Classroom, ICI Global, 2020, doi: 10.4018/978-0238-9.
23. D.M. Bressler, M. Shane Tutwiler, A.M. Bodzin, “Promoting student flow and interest in a science learning game: a design-based research study of school scene investigators,” Education Technology Research Development, 69, 2789–2811, 2021, doi:https://doi.org/10.1007/s11423-021-10039-y
24. A. Savran Gencer, H. Doğan, “The assessment of the fifth-grade students’ science critical thinking skills through design-based STEM education,” International Journal of Assessment Tools in Education, 7(4), 690-714, 2020, doi: 10.21449/ijate.744640
25. S. Fowler, S.N. Leonard, “Using design based research to shift perspectives: a model for sustainable professional development for the innovative use of digital tools,” Professional Development in Education, 1-13, 2021, doi: 10.1080/19415257.2021.1955732
26. M. Allen. Leaving ADDIE for SAM: An agile model for developing the best learning experiences, ASTD Press, 2012.
27. M.W. Easterday, D.R. Lewis, E.M. Gerber, “Design-based research process: Problems, phases, and applications,” International Society of the Learning Sciences, 2014.
28. C. Jackson, “The advantages of exploratory research design,” eHow, https://www.ehow.co.uk/info_8525088_advantages-exploratory-research-design.html, 2020.
29. F. Amiti, “Synchronous and asynchronous e-learning,” European Journal of Open Education and E-Learning Studies, 5(2), 2020.
30. C. Hoadley, F.C. Campos, “Design-based research: What it is and why it matters to studying online learning,” Educational Psychologist, 57(3), 207-220, 2022.
31. T.C. Reeves, L. Lin, “The research we have is not the research we need,” Education Technology Research Development, 68, 1991–200l, 2020, doi: https://doi.org/10.1007/s11423-020-09811-3

(Click Here)