Epic 7.3.1 : Worksheet Generator - Squad Robo Rebels

[ApoorvThite]

Epic 7.3.1 Worksheet Generator

Squad – Robo Rebels

Project Manager : Shrishti Singh
Team Members : Asmi Panigrahi, Shanika Perera, Apoorv Thite, Shrihari Prasad
Gopalakrishnan, Zheng Wu, Bhanu Prakash, Rohith Jatla

Description and Formatting :

• We created worksheet generator located features/worksheet
• We used tool ID "2". from api/tools_config.json:

Here is the configuration of our input in the features/worksheet/metadata.json:

So our request body looks like:

User Information:

• "id": "string": Placeholder for the user's unique identifier.
• "fullName": "string": Placeholder for the user's full name.
• "email": "string": Placeholder for the user's email address.

Request Type:

• "type": "tool": Indicates the type of request, which is a tool-related request.

Tool Data:

• "tool_id": 2: Identifier for the specific tool being requested.
• "inputs": An array of input parameters for the tool, including:
• "name": "grade_level", "value": 12: Specifies the grade level as 12.
• "name": "topic", "value": "Physics": Specifies the topic as Physics.
• "name": "difficulty_level", "value": "high": Specifies the difficulty level as high.
• "name": "question_type", "value": Can specify the question type (from fill-in-the-blank, multiple-choice, open-ended) upto maximum 10 questions.
• "name": "language", "value": "EN": Specifies the language of the questions as English.

Key Features :

Worksheet Generation:

• WorksheetGenerator Class: Manages the creation of a worksheet by generating questions based on specified parameters (grade level, topic, difficulty level, question types, language).
• Question Types: Supports multiple-choice, fill-in-the-blank, and open-ended questions.
• Verbose Mode: Logs detailed information during the question generation process if enabled.

2. Question Building:

• Abstract Base Class (QuestionBuilder): Provides a blueprint for generating questions, including abstract methods for validating responses, transforming JSON data, and creating questions.

Specific Question Builders:

• MultiChoiceQuestionBuilder: Generates and validates multiple-choice questions.
• FillInBlankQuestionBuilder: Generates and validates fill-in-the-blank questions.
• OpenEndedQuestionBuilder: Generates and validates open-ended questions.
• Prompt and Model Configuration: Each question builder uses a specific prompt and Google Gemini model for generating questions.

3. Custom JSON Parsing and Validation:

• Custom Parsers: Uses Pydantic models (MultiChoiceQuestion, FillInBlankQuestion, OpenEndedQuestion) to parse and validate the structure of generated questions.
• JSON Schema Examples: Each Pydentic model includes JSON schema examples for the expected format of the questions.

4. Logging:

• Setup Logger: A logger is configured to log messages at various points in the code, especially for tracking question generation attempts and validation results.

5. Error Handling:

• Validation Checks: The code includes error handling for invalid inputs, such as missing parameters or improperly formatted responses.
• Retry Mechanism: If a question fails to generate or validate correctly, the system attempts to generate a new question, with a limit on the number of retries.

6. File Handling:

• Read Text Files: A helper function (read_text_file) is used to read prompt templates from text files based on a relative path.

7. Extensibility:

• Flexible Design: The system is designed to allow the addition of new question types by extending the QuestionBuilder abstract class and registering the new builder in the question_builder_classes mapping.

Code Structure:

• Logger Setup: We've initialized a logger using setup_logger() to handle logging throughout the function.
• Function Definition: We've defined a function named executor, which accepts parameters such as grade_level, topic, difficulty_level, question_types, lang, and an optional verbose flag.
• Debug Logging (Conditional): We've included a conditional debug log that activates if the verbose flag is set to True, logging the provided parameters (grade_level, topic, difficulty_level, question_types, lang).
• Parameter Setup: We've created a dictionary named params that contains the values for grade_level, topic, difficulty_level, question_types, lang, and verbose.
• Worksheet Generation: We've instantiated a WorksheetGenerator object using the params dictionary and called its create_worksheet method to generate and return the questions for the worksheet.
• Output Assignment: The generated worksheet questions are stored in the output variable.

For Testing purposes, we limited the num_questions to 10. We might increase the allowed limit of questions to make the system more optimised to handle larger number of questions as per requirement.

For generating questions, in our tools.py file, we created the following classes

The MultiChoiceQuestion class validates and structures multiple-choice question data, ensuring it includes the question text, a list of choices, and the correct answer key. It also provides an example schema for clarity and consistency in data formatting.

The FillInBlankQuestion class ensures that fill-in-the-blank question data includes both the question text with a blank and the correct answer. It also includes an example schema within json_schema_extra to demonstrate the expected data format for clarity and validation purposes.

The OpenEndedQuestion class ensures that open-ended question data includes both the question text and the correct answer. It also provides an example schema within json_schema_extra to demonstrate the expected data format for validation and consistency.

Testing:

Testing our code involved ensuring that each component functioned correctly and integrated seamlessly with the others.

• Given the various classes and methods involved, we first performed unit tests to validate the functionality of individual classes like WorksheetGenerator, MultiChoiceQuestionBuilder, FillInBlankQuestionBuilder, and OpenEndedQuestionBuilder.
• For each of these classes, we created test cases that simulated different input scenarios, such as varying grade levels, topics, difficulty levels, and question types.
• For instance, when testing the MultiChoiceQuestionBuilder, we checked whether it generated questions with the correct structure, validated the generated responses, and ensured the format of choices was accurate. Similarly, we tested the FillInBlankQuestionBuilder and OpenEndedQuestionBuilder to confirm they correctly generated and validated their respective question types.

Different use cases for testing included varying combinations of input parameters to ensure robustness.

• For example, we tested the system with edge cases like the maximum number of questions (e.g., 10), as well as invalid inputs, such as an unsupported question type or missing required parameters like grade_level or topic, in addition to the general responses of different topic, grade level, diffculty level, type of questions and language.
• Another use case involved testing the verbose mode to confirm that detailed logs were produced during question generation, helping us trace any issues that arose. Additionally, we simulated scenarios where the model might generate responses that were not valid JSON or lacked essential components (e.g., a question without an answer) to verify that our validation logic correctly identified and handled these cases.
• The expected outputs varied based on the inputs; for valid inputs, we expected the generated worksheet to contain a list of properly formatted questions, while for invalid inputs, we expected error messages or exceptions that were handled gracefully by the system.

Innovation and Future Scope:

• In our implementation, we incorporated a language input feature and allowed users to specify both the type and number of each question, a functionality unique to our squad within the epic.
• This capability enabled users to customize their worksheets not only by grade level, topic, and difficulty, but also by selecting the language in which the questions are generated, along with the precise mix of question types (e.g., multiple-choice, fill-in-the-blank, open-ended) and the quantity of each.
• This approach ensured that the worksheets could be tailored to diverse educational needs and contexts. We thoroughly tested this functionality across various use cases, such as generating worksheets in different languages and combinations of question types, and it consistently produced accurate, well-structured outputs that met user specifications.
• The flexibility of this implementation demonstrated its effectiveness in adapting to different educational settings and learning objectives. As for the future scope of this epic, we envision expanding the language support to include more languages and dialects, refining the models to better handle nuanced language differences, and introducing adaptive learning features that adjust the question difficulty based on the student's performance.
• Additionally, we could explore integrating this feature with other educational tools and platforms, enhancing its utility and reach within the educational technology ecosystem.

[hash2004]

No need to include this, core.py should use tool file

[hash2004]

Great job! Please review the changes requested