Evaluation Method

Research and Evaluation Approach

Theoretical Frameworks

This program evaluation uses several theoretical frameworks. Hutchins' theory of distributed cognition (1991) highlights the importance of considering tools and artifacts in decision-making and problem-solving, shaping our understanding of knowledge. Experiential learning theory (Kolb, 2014) and open pedagogy (Hegarty, 2015) emphasize the importance of applying theories in the real world, which is dynamic and sometimes messy. Design-based research, proposed by Barab and Squire (2004), recognizes the flexibility and complexity of real-world practices, aiming to describe situations rather than control them. Bell (2004) supports this approach, arguing that many aspects of learning and cognition are too intricate to be simplified into one variable or definition. Design-based research also allows for ongoing design revisions, active participation, and analysis of tools, artifacts, and activities. This framework serves as a bridge between theory and practice, making it particularly effective for doctoral research (Goff & Getenet, 2017).

Evaluation Approach

Design-based research is closely linked to the developmental evaluation approach, which is particularly useful in dynamic and innovative environments (Patton, 2010). It emphasizes the role of evaluation in program workflows, allowing for quick feedback and continuous improvement (Patton, 2006). This approach is beneficial in complex or "messy" environments, enabling ongoing redesign during program implementation. The OER Lab is a prime example of such a dynamic program. Tools and workflows are constantly evolving, staffing changes with each new group of students, and OER policies and procedures are often unclear or subject to change. The developmental evaluation approach allows Lab staff and supervisors to adapt the program while incorporating feedback and evaluation findings.

Data Collection

The research questions, theoretical frameworks, and evaluation approach are best suited for a qualitative data collection method. We used a semi-structured interview format allows us to delve into participants' perspectives on key study areas while also allowing for personal experiences to shape the discussion and uncover additional themes.

Participants

The participants for this program evaluation were student employees who worked with the Ontario Tech OER Lab for the 2023-2024 academic year. These student employees provide the main workforce for the daily operations of the Lab. The funding for these students comes from the University Works program at Ontario Tech. Students who participate in the program must meet eligibility criteria set out by the university, which includes:

• Canadian citizenship or permanent residency

• Full-time registration in an undergraduate program at the University between September and April of the academic year

• A plan to return to full-time studies the following year (if following a summer employment term)

• A demonstration of financial need

• A GPA of at least 2.0 from the last full-time semester of study.

After confirming their eligibility and receiving final approval from the Office of the Registrar, students can apply for available jobs through the university's online jobs portal. Jobs are posted during the Winter semester and are typically available until all positions are filled. For the OER Lab, job postings have historically been posted past the program start date (beginning of May) for an additional three weeks until the positions are filled.

The University Works program receives a set amount of funding per full time enrolment (FTE), based on the Guidelines for Tuition Set Aside as outlined by the Ministry of Training, Colleges and Universities of Ontario. Tuition Set Aside is a portion of student tuition that must be retained and used to support access programs, such as University Works (Ministry of Training, Colleges, and Universities, 2019). As such, the number of available funded positions are in high demand. An annual application process takes place in the Winter semester for the next academic year. Interested departments and Faculties must create a job posting including:

• The name of the job and department it reports to

• A description of the job and its day-to-day responsibilities

• Required skills, education levels, and experience

• Preferred areas of study (optional)

• Rate of pay

The internal process of determining the number of funded positions allocated to each request varies from year to year. Historically, the OER Lab has had an increase in student positions since its inception (three students in 2020-2021, eight in 2021-2022, eight in 2022-2023 plus full-time Lab Coordinator).

Instruments and Analyses

Design-based research methods explore various factors like resources, collaborations, outcomes, and systems without trying to control them. Instead, they look at how these factors connect and interact (Barab & Squire, 2004). It's important to consider the different contexts in which these factors are studied because it can affect the study results. This study aims to provide helpful insights for other colleges and universities interested in setting up their own version of the OER Lab.

During the study, we used a semi-structured interview script to guide conversations with participants. The interviews were analyzed using deductive coding with a hierarchical frame. This method allowed us to not only look at the main themes but also how they interacted with each other, considering the context of the OER Lab ecosystem.