OATutor: An Open-source Adaptive Tutoring System and Curated Content Library for Learning Sciences Research

2.1 Open Educational Resources

Open Educational Resources (OERs), resources that are openly provisioned for use and adaptation by a community of users for non-commercial purposes [23], allow for broader access and dissemination of learning materials such as problem sets, videos, and other web-media resources. With OERs, educators can incorporate material into their syllabus at no charge while reducing their time spent authoring content. Most OERs, like free and open textbooks, focus on accessibility [50]. Others, like Khan Academy, have created alternative recognition systems for skills and educational achievements. Meanwhile, the Scratch software has focused on collective construction of knowledge, where the program is developed and repurposed by others in order to learn from one another on the platform [32]. Kolibri, an online and offline OER learning platform for educators across the globe that contains curated and openly licensed educational content libraries, focuses on including a wide range of curricula containing lessons, assessments, books, and games [24]. However, there is a key gap to be filled in the translation between OERs and computer tutoring systems. The lack of a common structure to OERs largely inhibits them from interoperating with one another and slotting into digital assessment systems [41].

2.2 Massive Open Online Courses

Massive Open Online Courses (MOOCs), open access instructional content released through an online platform, gained traction as digital versions of courses from well-known universities made available to the world [6]. As a result, platforms like Coursera, edX, and Udemy emerged as content providers, with some offering credits counting towards a future degree. Participants interact with assessments, videos, and in some cases teaching staff and their peers, in order to foster discussion and further thinking. Through MOOCs, learners can enroll in courses for free, opening participation to anyone with internet access. However, a certificate of completion in a MOOC costs money, and while MOOCs are open access, their content is propriety and not creative commons like an OER.

2.3 Open Datasets

A variety of educational technology platforms have released anony-mized data logs, promoting offline experiment replication and generalization of methods across datasets. Carnegie Learning has released dozens of datasets, which have contributed considerably to educational data mining research [17]. Most notably, they released a dataset in 2010 for the Association for Computing Machinery (ACM) Special Interest Group on Knowledge Discovery and Data-mining’s (KDD) annual competition. The dataset contained over 9 GB of student data, becoming the most often cited from an educational technology platform [48] and the largest dataset the competition had seen at the time. ASSISTments, a free web-based tutoring platform, has also been generous with the release of its datasets that have been used in over 100 papers ². More recently, large datasets from tutoring systems developed and deployed in South Korea have been released, such as EdNet [10].

2.4 Open Algorithms

Open algorithms remove barriers for the educational data mining community to expand and replicate research. For example, pyBKT provides the first python implementation of a popular algorithm for cognitive mastery estimation [5], which is the same algorithm used in the Cognitive Tutor/MATHia. Open-sourcing of these tools can garner considerable attention from the community, with pyBKT receiving 1,000 installs per month. Many other algorithms are open-sourced as a companion to a research paper. As an example, an adaptive assessment engine used within MOOCs was open-sourced³ and evaluated for learning gains within several courses [46].

2.5 Open Platforms

A combination of open resources, datasets, and algorithms in the form of an open platform has been rare. Cognitive Tutor’s Geometry 1997 dataset is public, but the system itself is not open and not free to use. ASSISTments, an open platform, encourages outside researcher collaboration to run A/B experiments on different content-based approaches [20], but does not release the content of its platform, nor its codebase. We hypothesize that the same acceleration of research outlined in the sections above could be seen in the domain of platform design with the open-sourcing of a full-featured tutoring system and content library. The open-source nature of a system also fosters trust, encouraging wider spread student, teacher, and institution adoptions [28].

This is not to say that partial implementations have not been made available. Several open-source tutor repositories adopting the moniker of either adaptive or intelligent can be found online (Table 1). All, however, were short-lived projects at the unfielded prototype stages, implementing a small subset of ITS features and with no development activity registered in the past three years. Most had implemented immediate feedback and hints (LogicITS⁴, Joots⁵, and ThesisITS⁶) but with only one, ThesisITS - a more mature learning management system and java coding interface - having a documented content abstraction or authoring tool allowing for content extension beyond the hard-coded example problems. CalcTutor offered procedural generation of integration and differentiation questions with answer checking. GnuTutor [34] and GuruTutor [35], the only projects with accompanying papers, were inspired by past, proprietary discourse-based tutor interfaces. Another system, GRAPPLE [8], provides an open-source and adaptive hypermedia framework, but is explicitly not a tutor, as it only supports passive resource use, not problem-solving activities. None of the systems, at this time, adhered to the majority of ITS principles, and namely lacked support to assess skill mastery and adapt instruction accordingly, a core capability of Intelligent Tutoring Systems.

2.6 Content Authoring Tools

Various authoring tools exist to populate tutoring environments with learning material. Cognitive Tutor Authoring Tools (CTAT) has been the most developed toolset for producing ITS content with only minimal programming background [52]. For this reason, many tutor developers have built upon the CTAT framework due to flexible content authoring, creating example tools for a variety of learning tasks and domains. A well-known drawback to content authoring for ITS has been the time taken for content creation. CTAT has made progress on this front, reducing the estimate of 200-300 hours of development to produce content for one hour of instruction [2] to only 50-100 hours [52].

ASSISTments is another example of a tutoring system that has focused on authoring tools. These tools also match the 50 hour estimate, having shown to be around four times faster than original ITS authoring tools, while also not requiring any programming background whatsoever [44]. However, a new user must spend time learning the ASSISTments builder interface, as it is an original design allowing for nested hints and scaffolds. A simpler “quick builder” also exists within ASSISTments that simplifies the interface but does not provide the ability for scaffolding and hinting to be added [21].

Additional authoring tools such as ASPIRE have also emerged, with a focus on easing the creation of ITS [33]. Such tools aim to simplify the process in a form that allows domain experts who are not experienced in the realm of ITS or software engineering to create their own curricular content [49]. The downside of such tools, however, lies in them not being able to overcome old time burdens for content creation, resulting in development times similar to those CTAT previously helped overcome [49].

2.7 Crowdsourcing and Learnersourcing

The content found in the aforementioned OERs is an example of crowdsourcing, a “participative online activity” involving a proposer who reaches out to a group of individuals in order to request a “voluntary undertaking of a task” [15]. In most forms of crowdsourcing, a financial or other benefit is provided to possible participants to encourage the undertaking of the task. Due to the common need for an incentive and the necessary effort to find individuals, crowdsourcing on a large scale over a significant time period can be challenging [31]. In order to create a sustainable crowdsourcing environment, it is critical to manage work-allocation in a flexible manner, and ensure the incentive is encouraging for participants to remain on the project [31].

In addition to OERs, certain tutoring systems feature their own pool of crowdsourced and learnersourced content. As mentioned above, ASSISTments allows teachers to view, use, and sometimes build upon content from other users. This type of content has been shown to be beneficial for students during the learning process and validates the reliability of sourcing educational content from users knowledgeable of the subject [42]. Educational crowdsourcing has also seen success in a real-time environment with TeacherASSIST, a feature created by ASSISTments to allow teachers to provide more on-demand crowdsourced help [39].

Micro-task markets such as Amazon’s Mechanical Turk have emerged as efficient crowdsourcing environments [27]. However, when it comes to educational content, crowdsourcing from such a platform could raise various concerns. Low quality material with errors could result in misconceptions, disengagement, and could even have negative socio-economic consequences by adversely impacting students’ grades in formal education environments. Thus, there is a significant requirement in order to qualify as a contributor to educational crowdsourced content (such as being a teacher, mentor, tutor, and having experience with the subject matter). With these criteria being very particular, it falls outside the scope of typical crowdsourcing platform communities. Where there are appropriate community members, the cost for having such specific criteria increases considerably. Thus, educational platforms tend to resort to known communities with the appropriate background.

To avoid possible financial barriers while seeking qualified participants, one can shift to a more focused category of crowdsourcing called learnersourcing. Learnersourcing is suitable for content authoring tasks as it is focused on featuring a recent or current learner of the related academic subject as the individual participating in the voluntary crowdsourced task [25]. Additionally, learnersourcing revolves around the creation of systems and tasks related to the subject being learned and incorporates the experience and creativity of the learner for the successful completion of the task.

In the process of learnersourcing, individuals can be identified into three separate groups: contributors, beneficiaries, and members of the instructional team [47]. Contributors are the individual learners who create the necessary learning artifacts, while beneficiaries are the learners on the receiving end of such artifacts. The incentive structure for contributors lies in the “Generation Effect”, students better recall information they have generated themselves [47]. Thus, the contributors also participate in learning, and strengthen their own understanding of the subject.

3.1 Research-based ITS design choices

Intelligent Tutoring Systems have been researched for many decades with a strong foundation of design principles [4] that have led to robust learning gain evaluations [37]. In Table 2, we detail the initial feature set adopted for OATutor and the justification for adoption based on past research.

Table 2:

Hints	Hints "provide instruction in the problem solving context" (ITS Principle 3) [4] with figures or text shown to students upon request [3]. They can be used to convey short declarative information. Alternatively, they can contain a full solution to a similar problem, called a worked example, which has been found to be effective for learning, particularly with tasks focused more on concepts than procedures [26].
Scaffolding	Scaffolding are similar to hints but include one or more questions posed to the student. Most often, scaffolding are part of a tutor strategy called tutored problem solving, which breaks a problem down into multiple steps and in doing so can "communicate the goal structure underlying the problem solving" (ITS Principle 2) [4]. Several studies have found this approach to produced higher learning gains than a worked example hint strategy [13, 19], particularly with learning tasks focused on procedural knowledge [26], while other studies have found no significant difference [43].
Mastery Estimation	ITS are adaptive in the amount of practice they prescribe to a student. The ability of the system to avoid under or over practice is dependent on how well it is able to estimate student mastery of a skill [9]. Knowledge Tracing [12], an algorithm based on a Hidden Markov Model, is used to estimate cognitive mastery for each skill in an ITS [45]. This assessment is important for the tutor to be able to withhold problems a student is not ready for yet (ITS Principle 8) and to advance them on to more abstract material when appropriate (ITS Principle 7) [4].
Data Format	The data logged from student use of ITS have been the subject of large data mining competitions [48] and has provided fertile ground for education data science [17]. We adopt a tutor logging format familiar to researchers, that creates a row for each student interaction with the tutor [30].
Knowledge Component Model	A Knowledge Component (KC) model [1, 29] is a definition of the skills expected to be assessed and tutored in the system and the problems associated with those skills. It is prescribed that the skills be coarse-grained enough to "promote an abstract understanding of the problem-solving knowledge" (ITS Principle 4) but fine-grained enough to "represent student competence as a production set" (ITS Principle 1) and inform the adaptive selection of the next problem [4].
Variabilization	Variabilization allows many instances of a problem template to be generated based on variables taking on a predefined set of values or ranges of values. Problem answer checking and tutoring can also be based on instantiating of variables, and can help save content authors time [44].
Bottom-out Hints	A bottom-out hint serves as the final hint for a problem and the final opportunity to "provide instruction in the problem solving context" (ITS Principle 3) [4] usually containing the answer to the problem. Bottom-out hints are usually provided to prevent the student from dwelling too long on a problem after completing the hints and scaffolds. They have been shown to improve student performance on problems following the problem where they were shown the bottom-out hint [18].
Immediate Feedback (Correctness)	Upon submitting an answer the student is immediately informed of its correctness. This feature helps "provide immediate feedback on errors" (ITS Principle 6) [4] and has been shown to have a statistically significant positive effect on learning gain [13, 14].

View Table

Table 2: OATutor features incorporated based on ITS principles and past research

3.2 Content Structure

Inspired by Intelligent Tutoring Systems, OATutor uses a hierarchical structure for its content pool to generalize the expressiveness of existing popular tutoring systems in the literature (e.g., ASSISTments, Cog Tutor). The base unit of content is referred to as a problem which contains a title, a body, and a collection of subparts of the problem referred to as steps. An example is shown in Fig. 1 instructing the student to find the surface area of a cylinder.

Figure 1:

View Figure

The step is the finest granular level, and the level at which students enter their answer. This structure inherently encourages content authors to provide varying levels of grain size of instruction. Each step contains a title, body, and an answer field which can be either a textbox (set to exact answer or algebra answer) or a multiple-choice selection. Steps are typically structured as independent subparts of the problem that can be solved in any order. To provide additional help, each step contains a collection of help items called a tutoring pathway. The step shown in Fig. 1 instructs the student to calculate the surface area of the cylinder. However, the student has a conceptual error with the order of operations and ignores the parentheses, which the system marks as an incorrect answer. The student can retry the problem again on their own or click on the raised hand icon to begin the step’s tutoring pathway.

In Fig. 2, a step is decomposed into a mixture of hints and scaffolding. This serves to communicate the underlying goal structure within a problem step and walk the student through the problem solving process. Hints are unlocked and shown one at a time to reduce cognitive load. The hint unlocking order structure is configurable, with hint dependencies specified in the tutoring pathway content files. If a student’s answer is incorrect, they can reattempt the problem or choose to open any number of help items. Students can leave the help items to (re)attempt the original step at any point. All questions have a bottom-out hint which displays the correct answer to help the student move on to the next question if the previous hints were insufficient. Bottom-out hints can be turned on or off in a global configuration file. In Fig. 2, the student is shown to have finished two hints and a scaffold. Upon solving the scaffold, the student realizes their error in the order of operations, skips the remaining hints, and correctly solves the step from Fig. 1.

Figure 2:

View Figure

The content pool exists as a modular folder within the OATutor repository, with problems and their tutoring represented in JSON format (Figures 3 and 4). Content authors do not need familiarity with JSON and instead can author using a Google Spreadsheet. An automated conversion script is included and responsible for converting content in these spreadsheets to the JSON format required by OATutor’s content pool.

Figure 3:

View Figure

Figure 4:

View Figure

Content is tagged with skills at the step level. For OpenStax content, the learning objective of the problem within the lesson was used. In order to support easy definition and redefinition of skill mappings for researchers to experiment with, OATutor places these tags in a centralized configuration file called the skill model, shown in Fig. 5. This avoids the need to edit the fields of every individual step in the content pool. The skill model represents a matrix of all the steps and the skills they are associated with. The skill model matrix was stored in a javascript file in an earlier iteration of the system, but was later changed to a JSON file to better support a researcher participant who had explored re-tagging all OATutor content with skills from the US Common Core and desired a more structured skill association file format to output to.

Figure 5:

View Figure

Courses are the top level of content organization in the OATutor interface (Fig. 6), which contain lessons (Fig. 7). Instead of lessons explicitly containing problems, each lesson defines a list of skills, also called learning objectives, in the tutor. Problems are then adaptively chosen based on their association with the skills of the lesson according to the skill mapping, or KC model configuration (Fig. 5). The course, lessons, and learning objectives structure is designed to follow the structure of a typical textbook, intended to be integrated into the syllabus of a formal course. Fig. 8 shows how courses and lessons are defined using JSON in the system. Lessons can be configured to recycle problems in the event that a student has exhausted all problems associated with the skills of a lesson, but has not yet reached the mastery threshold. Variabilization increases the value of recycled problems (see section 3.4), changing the numbers within a problem each time it is shown. Additional lesson toggles of giveStuFeedback (correctness feedback), giveStuHints (allowing hints to be requested), doMasteryUpdate (using mastery learning as a stopping criterion), and showStuMastery (showing students their mastery estimate) are available and turned on by default. Researchers and teachers may like to set these to false when creating lessons that serve as tests.

Figure 6:

View Figure

Figure 7:

View Figure

Figure 8:

View Figure

3.3 Mastery-based Adaptive Problem Selection

Bayesian Knowledge Tracing (BKT) [12] is used to model student mastery of skills. BKT models student knowledge using a Hidden Markov Model in which observations are the correctness of a student’s answer for a problem. Each skill has four model parameters which are stored in a file called bktParams: \(p_{\text{mastery}}^{(t)}\), p_guess, p_slip, p_transit. \(p_{\text{mastery}}^{(t)}\) represents the probability of the student having mastered the skill at time t, p_guess represents the probability of the student solving a problem without having mastered the skill, and p_slip represents the probability of the student making a mistake applying a skill they have mastered. p_transit is used to compute the probability a student will master a skill at each answer opportunity. OATutor uses the Bayesian update shown in Equation 1 depending on the correctness of the student answer, where mastery at t is the model’s prior belief before observing a response at time t and t|ans is the updated posterior after ovserving a student’s response: (1) \(\begin{equation} p_{\text{mastery}}^{(t|ans)} = {\left\lbrace \begin{array}{@{}l@{\quad }l@{}}\frac{p_{\text{mastery}}^{(t)} * (1 - p_{\text{slip}})}{p_{\text{mastery}}^{(t)} * (1 - p_{\text{slip}}) + (1 - p_{\text{mastery}}^{(t)}) * p_{\text{guess}}} &\text{Correct}\\ \frac{p_{\text{mastery}}^{(t)} * p_{\text{slip}}}{p_{\text{mastery}}^{(t)} * p_{\text{slip}} + (1 - p_{\text{mastery}}^{(t)}) * (1 - p_{\text{guess}})} & \text{Incorrect} \end{array}\right.} \end{equation} \)

Each problem is manually tagged by content authors with a skill (ex. evaluating_algebraic_expressions). Each lesson contains skills and a specified target mastery threshold for each skill. After a student finishes a problem, the system chooses a new problem that the student has not mastered yet using a configurable heuristic function. The default heuristic function iterates through all problems and chooses the one with the lowest average \(p_{\text{mastery}}^{(t)}\), averaged among all the skills the problem is tagged with.

Learning continues until all lesson skills have been mastered, at which point the system informs the student that they have mastered all skills. Progress is stored locally within the student’s browser as a cookie, avoiding the need for a backend. Although this means that OATutor lacks cross-device progress tracking, the lessons are designed to be completed in a 30 minute session, mitigating this need. If the student exhausts the entire problem pool without mastering the skills, completed problems can be recycled.

3.4 Variabilization and Answer Checking

Problem variabilization is a feature that reduces the burden on content creators by allowing content to be instantiated with different numbers, creating more versions of content. Content authors specify all variables in the problem and the possible sets of values the variable can take on, as seen in the example in Fig. 3. A random integer is chosen as the index for all of the variable possible value arrays (the modulo operator ensures there are no index out of bounds errors for varying length value arrays). OATutor uses predefined groupings of variables to improve the student experience, since different combinations of numbers can significantly change the difficulty of a problem. An example of this would be that certain randomly chosen numbers will not divide evenly into whole numbers for fraction problems. Answers will often need to be stated in terms of variables given earlier in the problem. Variabilization can be defined at the problem, step, or hint/scaffold scope, similar to the variable scope of a computer program. Variable values at lower granular levels take precedence in the case of a duplicated variable name.

OATutor also supports localization in the form of global variable sets, which allow templated words to be replaced in all problems to accommodate different contexts. For example, if a problem contains the word “elevator” as the default, it may be desirable to use the word “lift” for a UK audience. Localization offers to reduce language confusing without undue content duplication, improving the quality of students’ interaction with the platform.

The platform was first designed for math courses, so the system needed to allow content creators to typeset math expressions and also allow students to input math expressions. OATutor’s frontend was created using React, enabling us to reuse common UI components from open-source React libraries such as MaterialUI, saving programmer development hours. LaTeX rendering libraries were integrated to support rendering mathematical expressions. Custom markdown syntax was formulated to denote LaTeX and inline images. For problems with textbox input fields, student answers are automatically converted to LaTeX in real time. This allows students to input their answers in a more natural way that requires no knowledge of LaTeX.

Algebra type answer-field checking is performed using Khan Academy’s open-source computer algebra system (KAS) which can verify if two expressions are algebraically equivalent. This gives students greater flexibility in entering their answers into the system, avoiding the need for the system to specify input ordering or answer formats.

3.5 Extensibility and A/B Testing

OATutor was designed to be as modular and simple as possible in supporting randomized A/B testing. Researchers can conduct A/B testing by navigating to the desired logic within the source of the platform and inserting a conditional statement. Common A/B testing entry points exist in the platform as examples for researchers, such as A/B testing different BKT learning parameters and different tutoring pathways (Fig. 9). When a student visits OATutor, condition assignment is generated randomly and saved into the student’s browser. All subsequent student actions will then be logged with the current condition for future analysis.

Figure 9:

View Figure

3.6 Accessibility Standards

In an effort to make our system accessible to all users who wish to learn or conduct research with our platform, we have ensured that every page complies with the Web Content Accessibility Guidelines 2.0⁷ (level AA) and the federal Revised Section 508 Standards⁸. To do this, the website is composed of modular components that were built to be accessible. As demonstrated in Fig. 10, our code has Accessible Rich Internet Applications (ARIA) attributes embedded in the application. Additionally, we tag our graphical components with alternative texts so that screen readers can easily identify figures in problems or buttons in our interface (Fig. 10). OATutor’s accessibility compliance with the Revised Section 508 standards is documented using the Voluntary Product Accessibility Template (VPAT). Using the Section 508 version of the template, our publicly available document⁹ lays out the required components OATutor has for educational institutes that receive US federal funding to adopt the system.

Figure 10:

View Figure

Going beyond the codified accessibility standards, our system incorporates many of the modern best practices for complex input transformations, dynamic content notifications, and error handling. Since users are able to enter mathematical symbols, operands, and matrices, the system makes use of the Math-quill library and a custom equation interpreter to generate English text for screen readers to consume. In the event that a user using a screen reader submits an answer to be checked, they will also receive real-time feedback via a "role=alert" notification popup. To ensure that component malfunctions do not affect the overall user experience, every component is wrapped with at least one error boundary. In addition to logging an exception event, the error boundary allows us to customize a fallback view for the system.

3.7 Ease of Adoption

OATutor was designed with simplicity in mind. It takes two clicks to set up an entire learning system, run a pre-made AB test, and deploy the system as a github-pages website on GitHub.io¹⁰. By offloading data storage needs to the users’ browsers and making our models browser-friendly, OATutor does not need any servers beyond basic static file hosting. Nevertheless, OATutor is built with several features that could be enabled if the orchestrating user points the configuration file to their own or cloud hosted database.

3.7.2 Data for BKT.

The format of OATutor’s data makes it convenient to train knowledge tracing models, such as with pyBKT, an open-source Python library of BKT models [5]. The log data format can be easily used to fit parameters in just a few lines. In Fig. 11, data from OATutor is loaded as a CSV. Using pyBKT, a model is initialized. A BKT model is trained on all the unique skills in the dataset, and the learning parameters (prior, learns, guesses, slips, and forgets) are fit for each skill. The learned values of each parameter per skill are shown in Table 5.

Figure 11:

View Figure

Table 5:

pyBKT sample outputs
Skill	Prior	Learns	Guesses	Slips	Forgets
evaluating_algebraic_expressions	0.70324	0.02174	0.48241	0.17582	0.00000
finding_x_intercepts_and_y_intercepts	0.45737	0.31908	0.07976	0.32756	0.00000
classifying_a_real_number	0.29347	0.42512	0.78437	0.34095	0.00000
the_discriminant	0.11926	0.17466	0.67566	0.51459	0.00000
solving_a_rational_equation	0.677738	0.07468	0.46928	0.25739	0.00000

View Table

Table 5: Skill parameters after fitting to OATutor pilot data

3.7.3 LMS Interoperability.

OATutor supports the Learning Tools Interoperability (LTI) standard to integrate the tutor with classroom Learning Management Systems (LMS) such as Canvas. This allows teachers to view student mastery of skills within the tutor from their LMS gradebook. To support LTI, we created a middleware server implemented with Node.js to handle LTI authentication. Teachers simply specify the middleware as an external app when creating assignments, then link to a lesson on OATutor through a graphical selection screen. Students can then click on the assignment link when logged in to their LMS account to be redirected to the respective lesson on the platform. This removes the necessity for students to create a separate account on the platform and also simplifies the platform, avoiding storage and management of user credentials.

When OATutor is accessed by students via LTI, the same logging format applies; however, the additional fields of unique course_id and unique lms_user_id are populated, both of which are anonymized but persistent across devices and sessions as per the LTI protocol. The field course_name is also populated, which reflects the name of the course set by the teacher in the LMS.

Scores are passed back to the LMS as students finish problems. These scores are available for teachers to view in the gradebook of the LMS, which shows a detailed breakdown of the student’s mastery probability for each skill within the lesson. Each skill score is calculated by dividing the student’s mastery by the mastery threshold, capped at 100%. The overall score is calculated as the average of all skill component scores. An example grade report is shown in Fig. 12. This also shows a detailed log of actions in the lesson associated with the student to support the teacher in diagnosing a student’s engagement level and quality of understanding, by viewing the history of answers they have given and their pace of interactions with the system.

Figure 12:

View Figure

4.1 Selection of Creative Commons Source

In order to ensure the quality of the curated content, we established a methodology for selecting qualified resources for content curation. Our examination of various Open Educational Resources is shown in Table 6. Multiple OERs were selected through research on the web, including searching databases such as Amazing Educational Resources¹¹. First, we examined the license provided by each OER. The goal was to identify which resources had usable creative commons licenses that would allow for curation of their content. Then, it was critical to identify the types of resources commonly found in the OER. Our goal was to find an active pool of problem resources that could be transcribed. The next step was analyzing if the provided problem resources offered help. Already having some sort of answer or explanation could significantly reduce the time of curation, since less time would need to be spent on creating hints and scaffolds from scratch. Finally, we identified the types of community input available for the OER, which would allow us to better determine the quality of content available. After examining all these variables, we compiled a score for each resource we examined. We marked the top five scores in green, and the scores that came close to the top five with yellow, and in our case, we chose OpenStax as most suitable.

4.2 Building Community

Our system utilizes an aspect of crowdsourcing bordering on the definition of learnersourcing to create a sustainable content-authoring environment. Our process involves the recruitment of qualified learners to help with content curation of the desired respective subject (such as college-level math students to help with algebra content). We request a “voluntary undertaking of a task” while having our group of individuals who are contributors be former learners of the subject. However, we do not fully meet the definition of learnersourcing, as our contributors may not necessarily have recently learned the material they are authoring but instead may have mastered it previously in addition to having tutoring experience on the subject. Furthermore, the main benefit provided to said contributors is not rooted in learning. The qualified learners become involved through an Undergraduate Research Apprenticeship program providing course credit, and join with the motivation of collaborating with the sponsoring research lab.

After a learner has been recruited as a content creator, they undergo an introductory training course, consisting of an introduction to the system, a video guide on how to create problems, and an introductory “assignment” asking the creator to make five sample problems. The goal of the course is to prepare creators for their task in a time-efficient manner, while providing them with a hands-on introduction to the system through the sample problems.

In order to focus on fostering a healthy content curation environment, the content team held weekly half-hour meetings. This allowed authoring issues to be voiced and gave the opportunity for editors to directly respond to questions from the content creators.

Additionally, the community building that took place fostered a collaborative crowdsourcing environment to help meet the expectations of participant content authors while also allowing them to individually convey critiques and applicable feedback, a necessary factor in learnersourcing-like environments [7]. Furthermore, the meetings served as an integral method to keep the team engaged. Communication was a prime factor in showing project activity in addition to consistent feedback through emails, allowing further room for discussion. This in turn allowed emulation of a learnersourced environment as the crowdsourced community of content authors were given the foundations to interact with the content and apply their knowledge, though their incentives may not primarily be focused on learning.

4.3 Author Spreadsheets and Conversion

Content creators used Google Spreadsheets to author content, first transcribing OER problems to the sheet and then adding appropriate hints and scaffolding to them. A sheet consisted of the columns: Problem Name, Row Name, Title, Body Text, Answer, answerType, KC (skill), Image url (optional). Generally, a single tab of a sheet corresponded to a lesson in a book. Creator lesson tabs were copied to Editor sheets, which were copied to the Main Spreadsheet, which was populated with an entire book’s lessons. Using Google apps had the added benefit of creators being able to see each other’s work. OATutor’s authoring tools make use of a script to automate conversion from problem data in the tabular sheets to JSON format in the problem pool for the frontend to render, allowing creators to curate material without requiring CS knowledge. The sheet must be shared with a Google service account to allow the script to make google sheet API calls. The script also automatically converts plaintext math expressions to LaTeX using a library called pytexit.

Another feature of the script is automated feedback. The script runs a quality check daily to ensure that there are no errors in the sheet that would cause a runtime error in the tutor. Another series of heuristic-based checks are also performed, such as ensuring that multiple-choice questions have multiple answer options. The outputs from these quality checks are automatically placed inline with the problems in the google sheets, allowing content editors to recognize and correct them.

5.1 Participants and Study Context

Our content team consisted of 25 total undergraduate students participating for a semester or two over three years. The team was led by the third author, a sophomore majoring in Mathematics with a minor in Education at the time of joining, who continued as a Master’s student in Education in Fall 2022. He was an unpaid intern on the team for the first two months, then transitioned to an hourly paid position onwards. The 25 content team members joined through an Undergraduate Research Apprenticeship Program (URAP) at UC Berkeley (on a semester system), with the exception of two students who joined from the local community college under an internship funded by their college. The URAP program provides course credit for a semester to students accepted into research labs and allows for an extension to a subsequent semester.The content team held weekly half-hour meetings in order to ensure authoring issues were voiced, in addition to allowing editors to directly respond to questions from the content authors. At the beginning and end of the semester, the first author also joined the content meetings to sync on any tasks, blockers, insights, and high-level ideas for teaching content. Content team members’ feedback was crucial as they were the first to encounter the material in OATutor and identify anything that they believed was missing or confusing.

The software development team consisted of five members. The second author, a junior in EECS at time of joining, was lead developer from the start of 2020 through mid 2021. The fifth author, a sophomore in EECS at the time of joining, became lead developer from mid 2021 onwards. Two additional EECS students contributed additional code support and the first author set design objectives and supervised development.

We conducted four pilots over 7 course offerings with a local community college algebra teacher (T1). She was introduced to the team by the college’s Dean of Math, Science, and Engineering (D1) as a teacher who “always has their eyes and ears on the newest pedagogies using technology”. Our pilot teacher taught online math courses for the past 15 years and advocates for using open OERs in her classroom to alleviate the financial burden on students. Because of her experience with OERs and use of OpenStax Algebra textbooks, she was identified as an ideal teacher participant to give feedback.

Finally, we collaborated with four researchers whose academic and research interests aligned with OATutor. Two researchers (R1 and R2) were also recruited through URAP and distinguished by their research interests at the intersection of data science and education. The other two researchers were Ph.D. candidates, one in Education (R3) and another in Electrical Engineering and Computer Science (R4). R3 was co-advised by the first author and R4 was a past collaborator. Both had deep experience in computer tutoring but had not been involved in OATutor’s development. As first consumers of our data exports, the researchers’ feedback was critical in identifying valuable additions, challenges, and future design opportunities.

5.2 Data Collection

We collected data from our participants using a combination of interviews, meeting notes, system-submitted user feedback, and surveys.

6.1 Case Studies of OATutor Supporting Researchers

As our first case study, we interviewed R1 whose research involved producing the CSV export for BKT models to be trained by R2. It also involved producing analytics on the most accessed lessons by students in the pilot. As a result of her project, several attributes were added to the logging schema to make analysis easier. These changes included added attributes to easily identify when a question was being answered for the first time. This was important because BKT only considers first attempts at each question. Secondly, R1 previously had to access the content files in order to add lesson titles to the usage analytics. To avoid this inconvenience, the logging data schema changed and the database was back-populated with the lesson name and course name associated with each action.

As a second case study, we interviewed, R3, regarding the suitability of the system for conducting research related to their dissertation. They raised that it would be desirable to A/B test the system’s default tutorial help (static hints and scaffolds) against an experimental dynamic hint generation algorithm, similar to [36]. To implement and conduct A/B testing for this feature, only one line of core logic had to be edited to include a handler for executing functions (Fig. 15). An example of a generator that we can A/B test (Fig. 16) is demonstrated in Figure 17 where it uses the problem and answer context to give a hint dynamically generated by the black box function ’modelGenerateHints’.

Figure 15:

View Figure

Figure 16:

View Figure

Figure 17:

View Figure

6.2 Findings from Pilot Teacher Feedback

During OATutor’s deployment in our pilot teacher’s classes, T1 noted in our interview with her that the students who interacted with OATutor were generally the more motivated ones who wanted extra practice. Students would leave feedback on their experience with the system, including bugs and questions, on Canvas’s discussion board in the first two pilots. The content team lead would then relay the feedback to editors, who would subsequently make the necessary fixes. However, because students did not receive any credit for using the system, their usage waned as the term progressed. T1 recalled noticing frustration among some students as they experienced content bugs, particularly in the first few pilots. She remarked that some students’ frustrations were also due to not knowing how to enter certain operators in their answers, such as square root, sharing that, “Math is already frustrating. I didn’t want to require them to use the system and be frustrated with the technology too.” However, she recognized that as the system got used more and received more feedback, the interface and experience improved and she continues to use the system in her classes.

Our pilot teacher’s general feedback in her meetings with the monthly OATutor co-design team was to make the system’s features similar to other web systems, particularly learning management systems, for consistency and ease of use. The ability to see a student’s log of actions was a feature she observed in other systems. Additionally, other homework systems she has used with her students have rich text editors for students to input equations and symbols into their answers. She also remarked that all software she adopts in her classroom must be accessible and meet ADA requirements. The ability to see student progress on a teacher-facing dashboard was also important to her to track time spent on mastery per student. Finally, an expansion of question types supported such as graphing questions was expressed as helpful to include to allow for a wider variety of topics to be practiced. In her ideal world, she would like more control over the questions, in terms of adding, removing, and editing problems to cater to her students.

6.3 Findings from the Content Team

At each content team meeting, the content team lead recorded feedback and discussed suggestions with the first author. In addition, the content team lead released a survey at the end of the students’ research programs to collect feedback on their overall authoring experience. The form was sent to 25 members who had been part of the OATutor editor team. Six members were still part of the team at the time and 18 members had, historically, continued past their first semester. A total of 16 responses were received, one of which was excluded due to being incomplete. Survey responses were anonymously submitted. When asked what factors led the content team to join OATutor (multi-response allowed), 80% of respondents joined because of the social good mission of the project, 53.3% joined to add a UC Berkeley project to their CV, and 53.3% joined because of the open-source-free nature of what OATutor is producing. With the majority of respondents joining because they believed in the mission of the project, it was possible to count on their feedback and commitment to bettering the content library.

Figure 18:

View Figure

The form also asked the team members to estimate how long the introductory training course for the system took (free-response allowed, with time estimated to the nearest hour), as well as the average time to create one problem (free-response allowed, with time estimated to the nearest minute). Among the 15 measured responses, the average introductory course length was 2.27 hours (with a standard deviation of 1.10) and the average for problem creation was 11.03 minutes (with a standard deviation of 9.11). These numbers are in line with ASSISTments training and creation times, which cite four hours of training and problem creation time of 25.4 minutes [51].

During the weekly meetings, two themes arose from feedback that would inform iterations of the authoring framework. Two members of the team reflected on stylistic differences they would observe among problems within lessons. Among these were differences in terminology as well as problem-solving strategies used in similar problems stemming from each problem being authored by a different member of the team. While students being exposed to several approaches to the same problem could help them generalize, it also could be overwhelming when just starting to learn the subject. The second theme was content authors voicing a desire to see what problems looked like rendered in the system shortly after authoring content in order to help catch mistakes. At the time, content was only rendered every 4-7 days.