Epic 7.7.1: Syllabus Generator - Logic Legion

[chitoroagad]

Syllabus Generator

Summary:

• This tool aims to create a sample syllabus customised to the liking of the user.
• Users can specify a small number of specific inputs as well as vague instructions to the tool and receive a tailored syllabus.
• Users can specify which sections of the syllabus to include or exclude in their output using the options api parameter.
• The generated syllabus includes the following:
  • Title
  • Overview
  • Policies and Exceptions
  • Grading Policy
  • Assessment Components
  • Required/Recommend Materials

Changes:

1. syllabus_generator directory in features/ containing:
   • tools.py includes SyllabusBuilder class responsible for all logic involved in using the LLM to create the syllabus.
   • model.py includes the Pydantic model for validating output.
   • core.py includes the executor function.
   • metadata.json includes expected inputs for the api.
   • prompt/ includes various text files for prompt generation.
   • tests/ includes test_sb.py used for testing all components of the SyllabusBuilder as well as test.json which includes a sample input for the api.
2. Updated the tools_config.json to be able to access the syllabus_generator straight from the api.
3. Created development environment in flake.nix for better LSP support and simplified environment set up.
   • Activate using nix develop and --command <prefered shell> to enter env.
   • bash is default.

Example:

Input

    {
        "user": {
            "id": "string",
            "fullName": "string",
            "email": "string"
        },
        "type": "tool",
        "tool_data": {
            "tool_id": 2,
            "inputs": [
                {
                    "name": "subject",
                    "value": "Data Structures"
                },
                {
                    "name": "grade_level",
                    "value": "University"
                },
                {
                    "name": "course_overview",
                    "value": "This course covers the fundamental concepts and applications of data structures in computer science. Students will explore various data structures such as arrays, linked lists, stacks, queues, trees, and graphs. The course will emphasize both the theoretical and practical aspects of data structures, including their implementation and analysis of their efficiency. By the end of the course, students will be equipped with the knowledge to use data structures effectively in real-world applications and advanced computing challenges."
                },
                {
                    "name": "customisation",
                    "value": "I want the course to require proficiency in Python"
                },
                {
                    "name": "options",
                    "value": [
                        "all"
                    ]
                }
            ]
        }
    }

Output

{
  "data": {
    "title": "Data Structures",
    "overview": "This course covers the fundamental concepts and applications of data structures in computer science. Students will explore various data structures such as arrays, linked lists, stacks, queues, trees, and graphs. The course will emphasize both the theoretical and practical aspects of data structures, including their implementation and analysis of their efficiency. By the end of the course, students will be equipped with the knowledge to use data structures effectively in real-world applications and advanced computing challenges.",
    "objectives": [
      "Understand the fundamental concepts of data structures, including their definition, representation, and operations.",
      "Apply data structures to solve real-world problems in various domains.",
      "Analyze the efficiency of data structures in terms of time and space complexity.",
      "Implement data structures in Python and evaluate their performance.",
      "Develop a deep understanding of the strengths and weaknesses of different data structures."
    ],
    "policies_and_exceptions": {
      "attendance_requirements": "Regular attendance is expected and absences must be communicated in advance. More than two unexcused absences may result in a deduction of your final grade.",
      "make-up_work": "Missed assignments and exams must be made up within one week of the original due date. Make-up exams may be different from the original exam.",
      "academic_honesty": "All work submitted for this course must be original and your own. Any instances of plagiarism will result in severe penalties, including failing the course."
    },
    "grade_level_assessments": {
      "assessment_components": {
        "assignments": 20,
        "exams": 30,
        "projects": 30,
        "participation": 10,
        "quizzes": 10
      },
      "grade_scale": {
        "A": "90-100%",
        "B": "80-89%",
        "C": "70-79%",
        "D": "60-69%",
        "F": "Below 60%"
      },
      "performance_expectations": "Students are expected to demonstrate a high level of understanding of the course material, participate actively in class discussions, and complete all assignments and projects on time.",
      "feedback_and_improvement": "Feedback on assignments and exams will be provided within one week of submission. Students are encouraged to seek additional feedback during office hours or by appointment."
    },
    "required_materials": {
      "recommended_books": [
        "Data Structures and Algorithms in Python",
        "Data Structures and Algorithms Made Easy",
        "Introduction to Algorithms"
      ],
      "required_items": [
        "Laptop with Python installed",
        "Notebook and pens",
        "Access to online learning platform"
      ]
    },
    "additional_information": {
      "prerequisites": "Basic programming skills in Python",
      "office_hours": "Mondays and Wednesdays, 2-3 PM",
      "contact_information": "Email: [email protected], Phone: 123-456-7890"
    }
  }
}

Test Coverage

Input Validation Tests:

• test_invalid_input: Ensures that invalid inputs such as an empty subject or
missing grade level raise appropriate errors.

• test_valid_model and test_invalid_model: Validates the Pydantic model
(SyllabusModel) to ensure that it correctly accepts valid data and raises
errors for missing or incorrect fields.

Functionality Tests:

• test_create_prompt_temp_with_mocked_read_text_file: Checks if the
create_prompt_temp function correctly assigns the appropriate grading
policy based on the grade level, using mock inputs to simulate different
school levels.

• test_compile: Verifies that the compile function correctly generates the
prompt and model chain based on the type of compilation ("customization"
or "syllabus").

• test_validate_response_valid and test_validate_response_invalid: Checks
the validate_response method to ensure it correctly identifies valid and
invalid syllabus structures.

How Tests Were Conducted

The tests were conducted using the pytest framework, which provides a
robust environment for running test cases. This includes using fixtures for
setting up test cases (sb fixture for the SyllabusBuilder instance) and
mocking functions to simulate and control the behavior of dependencies.

Cases Covered

1. Input validation to ensure that the correct types and values are passed.

2. Prompt creation based on different grade levels to ensure the proper
   grading policies are applied.

3. Syllabus compilation to ensure that the model and prompt chains are
   correctly assembled and invoked.

4. Model validation to confirm that the syllabus structure adheres to the
   expected schema.

5. Response validation to make sure that the generated syllabus meets all
   necessary requirements.

Code Snippets

Input Validation Tests

test_invalid_input:

with pytest.raises(ValueError):
    SyllabusBuilder(subject='', grade_level=10)

with pytest.raises(ValueError):
    SyllabusBuilder(subject='Biology', grade_level=None)

test_valid_model and test_invalid_model:

valid_input = {
    'title': 'Sample Syllabus Title',
    'overview': 'Sample overview of the syllabus content.',
    'objectives': [
        'Define the key terms associated with...',
        'Describe the cardiac cycle...',
    ],
    'policies_and_exceptions': {
        'attendance_requirements': 'Sample attendance requirements...',
        'make_up_work': 'Sample make-up work policy...',
    },
    'grade_level_assessments': {
        'assessment_components': {
            'assignments': 20,
            'exams': 25,
            'projects': 25,
            'presentations': 15,
            'participation': 15,
        },
        'grade_scale': {
            'A': '90-100%',
            'B': '80-89%',
            'C': '70-79%',
            'D': '60-69%',
            'F': 'Below 60%',
        },
    },
    'required_materials': {
        'recommended_books': ['book 1', 'book 2', 'book 3'],
        'required_items': [
            'paint',
            'paintbrush',
            'pencil',
            'eraser',
            'notebooks', 'sharpies',
            'crayons',
            'black ink pen',
            'ruler',
        ],
    },
}

Functionality Tests

test_compile:

@patch.object(SyllabusBuilder, '\_create_custom_promptTemp')
@patch.object(SyllabusBuilder, '\_create_prompt_temp')
def test_compile(mock_create_prompt_temp, mock_create_custom_promptTemp):
    mock_custom_prompt_instance = MagicMock(spec=PromptTemplate)
    mock_create_custom_promptTemp.return_value = mock_custom_prompt_instance

    mock_syllabus_prompt_instance = MagicMock(spec=PromptTemplate)
    mock_create_prompt_temp.return_value = mock_syllabus_prompt_instance

    sb = SyllabusBuilder(subject='Mathematics', grade_level='Grade 5', verbose=True)

    chain_customisation = sb._compile('customisation')
    mock_create_custom_promptTemp.assert_called_once()
    chain_syllabus = sb._compile('syllabus')

    mock_create_prompt_temp.assert_called_once()

Testing documentation by @ElyasBinothman

Documentation made by team members

Syllabus Generator.pdf @YomnaEisa

[mikhailocampo]

Please see changes

[mikhailocampo]

Update the PDF to include a summary of what the tests were and what edge cases were accounted for

[mikhailocampo]

from pydantic import BaseModel, Field, model_validator
from typing import Optional
from typing_extensions import Annotation

class LLMOutput(BaseModel):
  title: str
  overview: str
  description: Annotation[str, Field(default="This is a default desc.")]
  assignments: Annotation[List[Assignment], Field(default_factory=list)] 
  ....

Pydantic for validation can be useful here for validating the output

[chitoroagad]

I couldn't find any documentation on Annotation but I found Annotated, I assume this is what you meant.
Should I use this over just Field in model.py?

[chitoroagad]

Is it still useful to have a default values if, depending on the input options, some fields are not returned at all and are typed as Optional?
Or is the current implementation sufficient?

[mikhailocampo]

If you need to validate response fields with Pydantic's model_validator

from pydantic import BaseModel, model_validator, Field

class MultipleChoiceQuestion(BaseModel):
  answer: str
  choices: List[str]
  
  @model_validator()
  def validate_answer(cls, v):
    # Checks to make sure answer exists in the choice 
    assert answer in choice

[chitoroagad]

I have replaced the response validation code by simply calling SyllabusModel(**response), which does all of the validation for me.
This should be enough for now.

[mikhailocampo]

Not sure if this is needed to be committed

[chitoroagad]

This makes sure everyone is using the same versions of packages.
Can be updated using nix flake update.

The whole flake.nix is more of a quality of life addition for those familiar with nix.

It can be extended to run the whole backend in a declarative environment using nix run.

[chitoroagad]

I watched the code review and I appreciate the feedback by the way 👍.