SkillSaige uses large language models (LLMs) to grade certain question types. These models evaluate responses based on two key inputs:
- Grading instructions
- Grading context
Grading instructions can often be simple and direct. For example, the following would be a strong starting point for a Critique question:
Any answer that identifies the main issue in this situation should receive a passing score. A high score should require identifying what should have been done differently.
The LLM is already aware of the exchange it is grading, but grading context makes the evaluation process more reliable and consistent.
A sample grading context might be:
This is part of a course on safety compliance. It is aimed at new hires who may not have worked in the industry before. This unit covers [topic], and this lesson focuses on [topic].
Together, grading instructions, grading context, and the details of the scene being evaluated allow the LLM to grade more reliably and provide more meaningful feedback.
These examples are meant to serve as a starting point. For especially nuanced material, or material the LLM struggles to grade consistently, more detailed instructions and context may be needed.