Essays, Posts & Presentations

Developing Framework for Adaptive Feedback

Anna Lager

Author

Anna Lager

Title

Developing Framework for Adaptive Feedback

Description

This project is the framework prototype for adaptive feedback tested on the given middle-school physics problem.

Developing Framework for Adaptive Feedback

Anna Lager

University of Helsinki

This project is the framework prototype for adaptive feedback tested on the given middle-school physics problem.The developed package: the student is asked a question, which may depend on randomised parameters, the system assesses whether the student’s answer is correct and, if not, whether it belongs to a list of anticipated errors specified by the author. It then responds adaptively: depending on the student’s response, there may be additional hints or supplementary questions. Next steps will be to include possibility of collecting student mistake data and possible implementation with the use of authoring environment.

Introduction

Feedback to an incorrect answer can convert the error into a basis for further exploration, and helps to build understanding. Following this idea we want to develop a framework of how adaptive feedback can be realized using WL. To make the task clearer: we decided to start with a particular middle-school kinematics task that usually causes errors (based on teaching experience). Based on this example we developed a possible framework for feedback.

Package

The core of our project is the developed package. The developed functions and the structure of questions are presented there. Some of the main excerpts of the package are explained below. The full package file can be found in the attachments.

Question structure

Each question is an association with several keys. The following table clarifies what are the keys for the question and what they stand for. The values for the keys for each question are to be input by the author.

In[]:=

Dataset[<|question0-><|questionName->"Name of the question",questionDestination->"The notebook where the question will be printed",questionText->"Question Text",parameters->"List of parameters this question requires, e.g.{a,b}",answerValue->"Correct answer, e.g. (a+b)/2",answerType->"Type of the answer, e.g. String",anticipatedErrors->"List of possible anticipated errors that could be made, e.g. a+b",gradingFunction->"The way we test the student's input against the correct answer/anticipated errors in this quesiton",correctAnswerFeedback->"Some text feedback: Well done!",anticipatedErrorFeedback->"Some text feedback: This is incorrect and we might know why!",unaticipatedErrorFeedback->"Some text feedback: This is incorrect, but we are not sure why.",correctAnswerSupplementaries->"Name of the question we move on to if the student answered correctly",IncorrectAnswerSupplemetaries->"List of the question names (hint-questions) to move on to for each possible anticipated error or for unticipated error"|>|>]

Out[]=

question0	questionName	Name of the question
	questionDestination	The notebook where the question will be printed
	questionText	Question Text
	parameters	List of parameters this question requires, e.g.{a,b}
	answerValue	Correct answer, e.g. (a+b)/2
	answerType	Type of the answer, e.g. String
	anticipatedErrors	List of possible anticipated errors that could be made, e.g. a+b
	gradingFunction	The way we test the student's input against the correct answer/anticipated errors in this quesiton
	correctAnswerFeedback	Some text feedback: Well done!
	anticipatedErrorFeedback	Some text feedback: This is incorrect and we might know why!
	unaticipatedErrorFeedback	Some text feedback: This is incorrect, but we are not sure why.
	correctAnswerSupplementaries	Name of the question we move on to if the student answered correctly
	IncorrectAnswerSupplemetaries	List of the question names (hint-questions) to move on to for each possible anticipated error or for unticipated error

Functions

The main functions from the package are presented below. The grading function is defined within a particular question and is to be input by the author. Possible grading functions for different type of questions are discussed in the Subsection “Grading Function”.

Setting the values:

SetValues[nb_,question_]:= Questions[question][destinationNotebook]=nb; Questions[question][currentValues]=ReleaseHold[Questions[question][parameterValues]]; Questions[question][currentAnswerValue]=Questions[question][answerValue]/.Thread[Questions[question][parameters]->Questions[question][currentValues]]; Questions[question][currentErrorValues]=Questions[question[anticipatedErrors]]/.Thread[Questions[question][parameters]->Questions[question][currentValues]];SetChildValues[nb_,parentQuestion_,childQuestion_]:= Questions[childQuestion][destinationNotebook]=nb; Questions[childQuestion][currentValues]=Questions[parentQuestion][currentValues]; Questions[childQuestion][currentAnswerValue]=Questions[childQuestion][answerValue]/.Thread[Questions[childQuestion][parameters]->Questions[childQuestion][currentValues]]; Questions[childQuestion][currentErrorValues]=Questions[childQuestion][anticipatedErrors]/.Thread[Questions[childQuestion][parameters]->Questions[childQuestion][currentValues]];

Asking the question:

AskQuestion[question_]:= NotebookWrite[Questions[question][destinationNotebook],Questions[question][questionText]/.Thread[Questions[question][parameters]->Questions[question][currentValues]]]; NotebookWrite[Questions[question][destinationNotebook],QuestionResponseCell[question]]; NotebookWrite[Questions[question][destinationNotebook],QuestionResponseButton[question]]

Checking if the input student answer is correct (the grading function differs depending on the question; more information on that in the next subsection):

Grade[question_,studentAnswer_]:=Questions[question][gradingFunction][ Questions[question][currentAnswerValue], If[Questions[question][answerType]=="Expression",ToExpression[studentAnswer],studentAnswer]]

Checking if the student input answer is one of the anticipated incorrect answers:

GradeError[question_,studentAnswer_]:=Map[Questions[question][gradingFunction][#,If[Questions[question][answerType]=="Expression",ToExpression[studentAnswer],studentAnswer]]&,Questions[question][anticipatedErrors]/.Thread[Questions[question][parameters]->Questions[question][currentValues]]]

Find what particular anticipated incorrect answer was input by the student:

FirstErrorIndex[question_,studentAnswer_]:=Position[GradeError[question,studentAnswer],True](*Basedontherealizedanticipatederrorchooseafeedback*)Feedback[question_,studentAnswer_]:=Module[{ge=GradeError[question,studentAnswer]},Which[ Grade[question,studentAnswer], Questions[question][correctAnswerFeedback], MemberQ[ge,True], Questions[question][anticipatedErrorFeedback][[First[Position[ge,True]]]], True, Questions[question][unanticipatedErrorFeedback]]/.Thread[Questions[question][parameters]->Questions[question][currentValues]]]

Choosing a supplementary question based on the input answer:

Supplementaries[question_,studentAnswer_]:=If[ Grade[question,studentAnswer], Questions[question][correctAnswerSupplementaries], Questions[question][incorrectAnswerSupplementaries]]

Response module:

QuestionResponseCell[question_]:=With[{symbol=Symbol["$CellContext`"<>Questions[question][questionName]<>"StudentAnswer"]},Cell[BoxData[InputFieldBox[Dynamic[symbol],String,FieldHint->"Enter your answer"]],"Text"]]QuestionResponseButton[question_]:=With[{symbol=Symbol["$CellContext`"<>Questions[question][questionName]<>"StudentAnswer"]},Cell[BoxData[ButtonBox["OK",Appearance->Automatic,ButtonFunction:>((SelectionMove[EvaluationBox[],All,Cell];SelectionMove[Questions[question][destinationNotebook],After,Cell];NotebookWrite[Questions[question][destinationNotebook],Feedback[question,symbol]];Do[SetChildValues[Questions[question][destinationNotebook],question,supplementary],{supplementary,Supplementaries[question,symbol]}];Do[AskQuestion[supplementary],{supplementary,Supplementaries[question,symbol]}])&),Evaluator->Automatic,Method->"Preemptive"]],"Text"]]

Grading function

Depending on what question is asked, the grading function may differ. Here are some basic options that have been developed and used in the project:
1) This grading function checks if the input expression is equal to the correct answer:

In[]:=

gradingfunction1=TrueQ[#1==#2]&;gradingfunction1[(20+30)/2,25]

Out[]=

True

2) Is the sentiment of the answer positive/negative? This function can be used for example in yes/no type of questions without limiting the student to these particular words .

In[]:=

gradingfunction2[inputAnswer_]:=TrueQ[Classify["Sentiment"][inputAnswer]=="Positive"];gradingfunction2["definitely"]gradingfunction2["no way"]

Out[]=

True

Out[]=

False

3) Does the text mention these specific words that form part of the names of physical constants or units? In case of short answer questions, we might want to check if the names of physical constants or units, that are crucial to the answer, are mentioned.

In[]:=

gradingfunction3=(Length[Intersection[#1,Normal[TextContents[#2,{"PhysicalConstant","Unit"}][All,1]]]]>0&);text1="the mass is 5 kg";text2="5 moles";answerset={"Avogadro constant","moles"};gradingfunction3[answerset,text1]gradingfunction3[answerset,text2]

Out[]=

False

Out[]=

True

Notebooks and Package

The current design suggests that the “start” notebook assesses the package and generates a new notebook (“student” notebook), where the questions are asked and where the student inputs his answers. The feedbacks and supplementary questions will be appearing in the “student” notebook. The “start” notebook and the package file should be in the same folder.
The code of the “start” notebook (commented here for purposes of keeping it within one Notebook):

(*QuitSetDirectory[NotebookDirectory[]];<<AssessExperiment`nb=CreateDocument[];SetValues[nb,question1];AskQuestion[question1];*)

Test Task

The full package file AssessExperiment.wl can be found in the attached supporting files. The values for the question keys are currently set by us for the test kinematics problem: A particle moves a distance of

metres; its speed over the first

s/2

metres is

metres per second, and its speed over the second

s/2

metres is

metres per second. What its its average speed? Parameters a and b will be randomized. Here is some illustration of the student Notebook with some of the answers and following feedback.

Concluding remarks

We have developed a prototype of framework design for adaptive feedback, that takes in a student answer to a given introductory physics problem and give an adaptive feedback based on the mistakes done. This project is just the very first step in developing the framework for adaptive feedback. Originally it was planned to use Forms and possibly use an authoring tool for more user-friendly interface, but we had some struggles with it and it was not realized during WSS 2021. It is one of the main steps in the future. Additionally, we will work on the possibility to implement grade collection and student mistake collection so we could collect and analyze data on how and what mistakes students make. One more thing to do in the future is develop more grading functions based on FreeForm input.

Keywords

◼

Adaptive Feedback

Acknowledgment

Huge thank you to Phil Ramsden: without his help and expertise I wouldn’t have gone very far with my project in these three weeks. Thank you to Paul Abbott and Sylvia Haas for their advice and insights, and lots of thanks to the whole WSS 2021 TA group for their fast and helpful support. Thank you WSS 2021 for the opportunity to participate in this school and get to meet so many great people!

References

◼

https://reference.wolfram.com/language/guide/FreeFormAndExternalInput.html

Cite this as: Anna Lager, "Developing Framework for Adaptive Feedback" from the Notebook Archive (2021), https://notebookarchive.org/2021-07-6gsbnju