Feat: Email Validation #135

m2rads · 2024-12-24T09:27:17Z

This PR addresses issue #104

Changes

Add Mailosaur for email generation, email rendering, 2fa and content validation.
Add a new tool called sleep_miliseconds that helps AI to wait for a condition to be met before proceeding

…e met before AI contineuing execution

…ully inside it

… the peer dependencies are handled by the package managers

…ht browsers if it's not found

Merge main

Merge update-readme branch

vercel · 2024-12-24T09:27:22Z

The latest updates on your projects. Learn more about Vercel for Git ↗︎

Name	Status	Preview	Comments	Updated (UTC)
shortest	✅ Ready (Inspect)	Visit Preview	💬 Add feedback	Dec 26, 2024 9:45am

m2rads · 2024-12-24T09:35:01Z

@slavingia FYI I ended up using Mailosaur as it provides more features and is faster.

…ieved

… and render it in the new page for AI to see

…ypto to work with mailosaur

m2rads

Hi @slavingia . Here's my implementation for Email Validation. Below is a simple test case in Helper. I added explanation for important parts of the code for you. Let me know if you'd like me explain any parts in more details. Before I finalize my PR please let me know if you'd like to adjust something. Here's the demo:

helper.mp4

m2rads · 2024-12-26T03:30:41Z

packages/shortest/package.json

@@ -29,36 +29,38 @@
    "build:types": "tsc --emitDeclarationOnly --outDir dist/types && cp index.d.ts dist/",
    "build:js": "esbuild src/index.ts --bundle --platform=node --format=esm --outfile=dist/index.js --external:esbuild --external:punycode --external:playwright --external:@anthropic-ai/sdk --external:expect --external:dotenv",
    "build:cjs": "esbuild src/index.ts --bundle --platform=node --format=cjs --outfile=dist/index.cjs --external:esbuild --external:punycode --external:playwright --external:@anthropic-ai/sdk --external:expect --external:dotenv",
-    "build:cli": "esbuild src/cli/bin.ts src/cli/setup.ts --bundle --platform=node --format=esm --outdir=dist/cli --metafile=dist/meta-cli.json --external:fsevents --external:chokidar --external:glob --external:esbuild --external:events --external:path --external:fs --external:util --external:stream --external:os --external:assert --external:url --external:playwright --external:@anthropic-ai/sdk --external:expect --external:dotenv --external:otplib --external:picocolors --external:punycode",
+    "build:cli": "esbuild src/cli/bin.ts src/cli/setup.ts --bundle --platform=node --format=esm --outdir=dist/cli --metafile=dist/meta-cli.json --external:fsevents --external:chokidar --external:glob --external:esbuild --external:events --external:path --external:fs --external:util --external:stream --external:os --external:assert --external:url --external:playwright --external:@anthropic-ai/sdk --external:expect --external:dotenv --external:otplib --external:picocolors --external:punycode --external:https --external:http --external:net --external:tls --external:crypto --external:mailosaur",


Here we are just marking some more nodejs built ins as external so that they will work with Mailosaur implementation:. Here is the list:

https
http
net
tls
crypto

m2rads · 2024-12-26T03:37:02Z

packages/shortest/src/ai/tools/index.ts

+  {
+    name: "render_email",
+    description: "Fetch, decode and render received email in browser",
+    input_schema: {
+      type: "object",
+      properties: {
+        action: {
+          type: "string",
+          enum: ["render_email"],
+          description: "Render the received email in a new tab"
+        }
+      },
+      required: ["action"]
+    }
+  },
+  {
+    name: "sleep_milliseconds",
+    description: "Pause test execution for specified duration",
+    input_schema: {
+      type: "object",
+      properties: {
+        action: {
+          type: "string",
+          enum: ["sleep_milliseconds"],
+          description: "The action to perform"
+        },
+        duration: {
+          type: "number",
+          description: "Duration to sleep in milliseconds (e.g. 5000 for 5 seconds)",
+          minimum: 0,
+          maximum: 60000
+        }
+      },
+      required: ["action", "duration"]
+    }
+  },


Added two new tools here: render_email and sleep_miliseconds

render_email is used to execute Mailosaur email validation in browser-tools.ts file.
sleep_miliseconds is used to help AI to wait for a condition to be met. Either instructed by the user or based on AI judgement.

Why both of them needed in this PR:
getting the latest email requires AI to wait for a few seconds to so that the user receives the email, otherwise the test case might fail prematurely

m2rads · 2024-12-26T03:37:44Z

packages/shortest/src/browser/core/browser-tool.ts

+        case 'sleep_milliseconds': {
+          const defaultDuration = 1000;
+          const maxDuration = 60000;
+          let duration = input.duration ?? defaultDuration;
+
+          // Enforce maximum duration
+          if (duration > maxDuration) {
+            console.warn(`Requested sleep duration ${duration}ms exceeds maximum of ${maxDuration}ms. Using maximum.`);
+            duration = maxDuration;
+          }
+
+          // Convert to seconds for logging
+          const seconds = Math.round(duration / 1000);
+          console.log(`⏳ Waiting for ${seconds} second${seconds !== 1 ? 's' : ''}...`);
+
+          await this.page.waitForTimeout(duration);
+          output = `Finished waiting for ${seconds} second${seconds !== 1 ? 's' : ''}`;
+          break;
+        }


Implementation for sleep_miliseconds: max time is 60 seconds and min is 1 second

m2rads · 2024-12-26T03:39:36Z

packages/shortest/src/browser/core/browser-tool.ts

+        case 'render_email': {
+          if (!this.mailosaurTool) {
+            if (!this.config.mailosaur) {
+              throw new ToolError('Mailosaur configuration required');
+            }
+            this.mailosaurTool = new MailosaurTool(this.config.mailosaur);
+          }
+
+          const newPage = await this.page.context().newPage();
+
+          try {
+            const email = await this.mailosaurTool.getLatestEmail();
+
+            // Render email in new tab
+            await newPage.setContent(email.html, {
+              waitUntil: 'domcontentloaded'
+            });
+
+            await newPage.waitForLoadState('load', {
+              timeout: 5000
+            }).catch(e => {
+              console.log('⚠️ Load timeout, continuing anyway');
+            });
+
+            // Switch focus
+            this.page = newPage;
+
+            output = `Email received successfully. Navigated to new tab to display email: ${email.subject}`;
+            metadata = {
+              window_info: {
+                title: email.subject,
+                content: email.html,
+                size: this.page.viewportSize() || { width: this.width, height: this.height }
+              }
+            };
+
+            break;
+          } catch (error) {
+            await newPage.close();
+            throw new ToolError(`Failed to render email: ${error}`);
+          }
+        }
+


Implementation for render_email: This tool will grab the latest email from Mailosaur in HTML format and will render it in a new tab. It will then give the control back to AI to validate the email.

Why render it in a new tab?
Some users may want to test the links in the email. This gives AI a chance to interact with the email and click on it since it's in the browser.

Just a note that this is indeterministic. Perhaps we could implement a mechanism to manually compare the content of the email and the received email in the inbox.

m2rads · 2024-12-26T03:40:26Z

packages/shortest/src/browser/integrations/mailosaur.ts

This file is the implementation of Mailosaur in terms of grabbing the latest email from inbox. We can use this file to add more features like SMS validation and other types of email validations.

m2rads · 2024-12-26T03:41:03Z

packages/shortest/src/types/config.ts

+  mailosaur?: {
+    apiKey: string;
+    serverId: string;
+    emailAddress: string;
+  };


apiKey and serverId are necessary but not emailAddress. I will remove it.

… for validation

m2rads

Ready for review!

m2rads · 2024-12-26T09:40:40Z

packages/shortest/src/browser/core/browser-tool.ts

+            await newPage.close();            
+            const errorMessage = error instanceof Error ? error.message : String(error);
+
+            if (errorMessage.includes('Email content missing')) {
+              return {
+                output: `Email was found but content is missing. This might be due to malformed email. Moving to next test.`,
+                error: 'EMAIL_CONTENT_MISSING'
+              };
+            }
+
+            if (errorMessage.includes('Mailosaur email address is required')) {
+              return {
+                output: `Email address is required but was not provided.`,
+                error: 'EMAIL_ADDRESS_MISSING'
+              };
+            }
+
+            if (errorMessage.includes('No matching messages found')) {
+              return {
+                output: `No email found for ${input.email}. The email might not have been sent yet or is older than 1 hour. Moving to next test.`,
+                error: 'EMAIL_NOT_FOUND'
+              };
+            }
+
+            // Generic error case
+            return {
+              output: `Failed to fetch or render email: ${errorMessage}. Moving to next test.`,
+              error: 'EMAIL_OPERATION_FAILED'
+            };
+          }
+        }
+


A bit more robust error handling to handle validation failures more accurately.

m2rads · 2024-12-26T09:43:00Z

packages/shortest/CHANGELOG.md

+## [0.1.2] - 2024-12-26
+
+### Added  
+- Added support for Mailosaur email validation
+- Added email rendering feature in the browser
+- Added sleep_milliseconds tool to add delays in the test execution
+- Added more robust error handling for Mailosaur email validation


Adding the features that was added to in this PR. Since we have added a new feature, I used the wording Added. Maybe we can have a consistent wording for this file!?

m2rads · 2024-12-26T09:57:14Z

packages/shortest/src/ai/prompts/index.ts

+8. **Testing Email**:
+   - If you need to test a condition that involves seeing the contents of an email, use the "render_email" tool.
+   - This tool will grab the latest email from the email address given to you and will render it in a new tab for you to see.
+   - Once you are done with validating the email, navigate back to the original tab.
+   - You MUST pass the email address that is given to you to the tool as a parameter otherwise it will fail.
+   - If no email address is given to you for this test, you should fail the test.
+   - For email validation, you MUST always use 'Click' and 'Mouse' action instead of using keyboard shortcuts.
+


Add a new rule to system prompt to instruct AI how to use Email validation. Specifically instructing AI to pass email address as ActionInput and use 'Click' and 'Mouse' action instead of using keyboard shortcuts because shortcuts may cause unexpected events when dealing with an Inbox UI.

slavingia · 2024-12-26T11:19:01Z

This PR addresses issue #104 (In progress...)

Needs to be updated as not in progress?

slavingia · 2024-12-26T11:19:32Z

app/__tests__/dashboard.test.ts

@@ -3,19 +3,14 @@ import { clerk, clerkSetup } from "@clerk/testing/playwright";

 let frontendUrl = process.env.SHORTEST_TEST_BASE_URL ?? "http://localhost:3000";

-shortest.beforeAll(async ({ page }) => {


Self-review would be helpful to me. Why isn't this needed?

m2rads added 17 commits December 23, 2024 18:38

feat: add sleep_miliseconds tool to wait for a certain condition to b…

a6747d6

…e met before AI contineuing execution

refactor: Directly execute setup.js since we now handle errors gracef…

59d2252

…ully inside it

fix: installation of playwright chrome browser

bae44d5

update

b9c6b1f

refactor: simplify setup script to only install playwright browsers -…

7e9ceaf

… the peer dependencies are handled by the package managers

refactor: remvoe manual installation of playwright browser in workflow

3f1f59a

update changelogs

8a93770

update: add -D to installation guide and .shortest/ to .gitignore

854eca9

bump the version

2b58f52

Update: add exception handling to browser manager to install playwrig…

b2743d7

…ht browsers if it's not found

update

c9b73d4

update changelogs

b210f0f

install browsers before test execution to speed up the workflow

c06ce71

Merge branch 'main' into update-readme

e7f9a2f

Merge main

Merge branch 'update-readme' into fit/email-validation

342c75e

Merge update-readme branch

update prompt related to sleep_miliseconds tool

d80c979

feat: add rendering and email generation tool

4a8c07f

m2rads marked this pull request as draft December 24, 2024 09:27

m2rads added 10 commits December 24, 2024 10:06

update tests

eb6b468

merge main

9bd39fd

bump version

5db5747

Update config type to include Mailosaur apiKey and serverId

0d8b817

Add mailosaur sdk

ea7a770

add email address to the config file

e0bfaa0

remove generate email tool as we add email address in config tool now

17ce083

Feat: add new integration with mailosaur to grab the latest email rec…

639c87d

…ieved

update exports

2c9ac2f

update system prompt to include instructions for email validation

a074d90

m2rads added 2 commits December 24, 2024 21:58

Feat: add 'render_email' tool to grab the latest email from Mailosaur…

3697dc4

… and render it in the new page for AI to see

add a new test script to test the email feature

22db152

vercel bot deployed to Preview December 25, 2024 05:59 View deployment

m2rads added 2 commits December 25, 2024 19:20

update: marking node builts as external for https, http, net, tls, cr…

aaebf80

…ypto to work with mailosaur

update

c57dcfd

vercel bot deployed to Preview December 26, 2024 03:22 View deployment

m2rads commented Dec 26, 2024

View reviewed changes

m2rads requested a review from slavingia December 26, 2024 03:44

m2rads added 2 commits December 25, 2024 23:11

Add email address as action input

116b58b

remove the need of adding email address to the config

dff473b

m2rads linked an issue Dec 26, 2024 that may be closed by this pull request

Email support #104

Open

m2rads added 7 commits December 25, 2024 23:32

update: use emailAddress from ActionInput instead of the config file

bdb36fe

update emailAddress to be loosely typed

834d5da

Update: in case of a failure in validating an email, send error to AI…

c621ebe

… for validation

Update prompt to instruct AI to pass the email as ActionInput

88474bf

update: handle Mailsauor failrues seperately

dab4f71

update: add a safe gaurd for missing email address

1ac66cd

Update prompt to be more strcict about tool use

21be464

vercel bot had a problem deploying to Preview December 26, 2024 09:31 Failure

Update changelogs

858e023

vercel bot had a problem deploying to Preview December 26, 2024 09:40 Failure

remove extra space

f77a49d

vercel bot had a problem deploying to Preview December 26, 2024 09:42 Failure

update the test email address

a61ec17

vercel bot deployed to Preview December 26, 2024 09:45 View deployment

m2rads marked this pull request as ready for review December 26, 2024 09:53

m2rads commented Dec 26, 2024

View reviewed changes

slavingia requested changes Dec 26, 2024

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Feat: Email Validation #135

Feat: Email Validation #135

m2rads commented Dec 24, 2024 •

edited

Loading

vercel bot commented Dec 24, 2024 •

edited

Loading

m2rads commented Dec 24, 2024

m2rads left a comment

m2rads Dec 26, 2024

m2rads Dec 26, 2024

m2rads Dec 26, 2024

m2rads Dec 26, 2024

m2rads Dec 26, 2024

m2rads Dec 26, 2024

m2rads left a comment

m2rads Dec 26, 2024

m2rads Dec 26, 2024

m2rads Dec 26, 2024

slavingia commented Dec 26, 2024

slavingia Dec 26, 2024

		@@ -3,19 +3,14 @@ import { clerk, clerkSetup } from "@clerk/testing/playwright";

		let frontendUrl = process.env.SHORTEST_TEST_BASE_URL ?? "http://localhost:3000";

		shortest.beforeAll(async ({ page }) => {

Feat: Email Validation #135

Are you sure you want to change the base?

Feat: Email Validation #135

Conversation

m2rads commented Dec 24, 2024 • edited Loading

Changes

vercel bot commented Dec 24, 2024 • edited Loading

m2rads commented Dec 24, 2024

m2rads left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

m2rads left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

slavingia commented Dec 26, 2024

Choose a reason for hiding this comment

m2rads commented Dec 24, 2024 •

edited

Loading

vercel bot commented Dec 24, 2024 •

edited

Loading