Our Assessment Innovation Fund pilots: Progressay

Progressay

Progressay is an edtech company providing online-based learning assessment. As an award-winning edtech start-up, they have developed a patent-pending augmented essay-grading AI designed to enhance essay marking and accelerate learning.  

About the pilot

Led by Progressay in partnership with the NCFE, this Pilot addresses the potential role Artificial Intelligence could play in enhancing educational assessment processes. It specifically focuses upon Artificial Intelligence generated augmented feedback for English Language Functional Skills Reading and Writing. 

This study was designed to determine the impact of AI Augmented written feedback on a student’s perception of their efficacy in English Language writing and whether there exists a relationship between self-efficacy and achievement in English Language writing. This was done with a view to positively affect student performance and potentially impact upon wider issues such as social mobility and access to a larger range of career opportunities. 

The main focus of provision was: 

1. Real time data analytics

2. Real time diagnostic feedback

Objective

The primary goals of this research are as follows: 

1. Learner Attainment: To assess the impact of the pilot program on improving the reading and writing skills of participants in line with NCFE Reading and Writing Functional Skills English Level 2 criteria.

2. Learner Efficacy: To understand how participating students perceive their progress, confidence, and overall efficacy in reading and writing after completing the pilot program.

Method 

This research employed a mixed-methods approach to evaluate the effectiveness of the pilot program for NCFE Reading and Writing Functional Skills English Level 2 conducted on the Progressay platform. The study combines quantitative data collection and analysis to measure learner attainment and qualitative data collection and analysis to assess learner efficacy. 

The project was run using an Agile methodology, meaning project management involved breaking the project into phases with an emphasis upon continuous collaboration and improvement. The project followed a cycle of planning, executing, and evaluating. 

Results

This pilot of Progressay AI Augmented Marking provided valuable insights into the effective use of Artificial Intelligence augmented educational technology for the purposes of feedback and essay marking. The pilot entailed trialing two online courses that featured Progressay’s patent-pending automated essay marking technology. These courses relate two NCFE’s English Level 2 Functionals Skills for Reading and Writing. Headlines include: 

• Reading: 94.5% agreement rate based on training 30,491 essays 

• Writing: 94.7% agreement rate based on training 3,353 essays 

• Pupils improved by an average of 2.5 marks at Level 2 and 1.3 marks at Level 3. 

• Pupils with SEND saw an average increase of 1.5 marks across both L2 and L3. 

Analysis

The study emphasises the importance of feedback quality and specificity. In the post-trial survey students expressed a desire for detailed and precise guidance, indicating that feedback should not merely be "nice" but also direct and actionable. The focus group comments show that the objectivity of the feedback provided by AI technology was appreciated, highlighting its value in fostering fairness and impartiality in the educational process. 

71% said that they thought a system that gives marks and feedback as quickly as Progressay would help them to progress faster but only 49.4% said that they could see themselves using it. 

Of those who reported that they did not see themselves using a system like Progressay it seems that some of the reasoning behind this was due to their belief that it would be expensive and therefore unavailable to schools more widely. Addressing this divide is essential to ensure equitable access to educational technology and opportunities. 

Conclusion

Following the pilot project taking place, it is clear from the lessons and findings from it that Progressay AI Augmented marking has the potential to positively impact education by enhancing self-efficacy, providing timely feedback, increasing attainment, and promoting personalised learning experiences. By recognising the impact of specific, objective, and timely feedback on student confidence and academic achievement, Progressay offers teachers and schools an opportunity to tailor their approaches to empower students in taking ownership of their learning journeys. This can lead to increased student engagement, persistence, and resilience when facing academic challenges. 

Despite the potential benefits of such technology, the study also highlights challenges. The digital divide, with some students displaying more confidence in their IT skills than others, is a significant concern. 

It is clear that teachers remain pivotal in supporting students' self-efficacy and it is vital that they are supported when ensuring effective use of AI technology in educational contexts.