POC: E2E test with Playwright

[a0m0rajab]

Type of feature

🍕 Feature

Current behavior

This is related to the latest stream I had with @nickytonline we could test the popup index page through the code below.

import { test } from './popupFixture';

test('popup page', async ({ page, extensionId }) => {
    await page.goto(`chrome-extension://${extensionId}/index.html`);
    await page.getByRole('button', { name: 'Login' }).click();
});

When I tried to work on the tests and login to the extension I could not do that due the inability to open a new tab and get the authentication from it, the playwright is testing the popups as HTML pages and not allowing to have the browser features that we need.

Here is an example from playwright:

This page is pretty similar to http://localhost:3000/ when you run the dev server, and that page does not allow you to login or open new window, I am not sure if this is possible to do not.

Related issue: #238

A solution to have is:

• test components in the extension, playwright docs
• test the link to the app if it's correct or not (to prevent the issue we faced before)
• testing the injection codes with evaluation functions

The downside for this would be:

• not being able to test if the extension has injected any code to the page, which is what we are doing.

Another thing we could try:

• using the browser to unpack the extension and use it from the browser but not sure if that is possible.

A good thing in playwright is that we could use the authentication feautre to login to github and opensauced, then we could use the cookies/auth information from that.

I think this is a feature request in the playwright but it's not implemented yet, related issues:
support for Chrome extension popup testing
Ability to click browser action buttons
Can I use playwright to test chrome extension?

Resources:
End-to-end testing Chrome extension, google docs

Suggested solution

No response

Additional context

No response

Code of Conduct

• I agree to follow this project's Code of Conduct

Contributing Docs

• I agree to follow this project's Contribution Docs

[a0m0rajab]

I just got stuck on this and did not know what to do, when I googled it I could not move forward.

[a0m0rajab]

I gave up on playwright, Selenium might have a better option for end to end test, https://www.selenium.dev/selenium/docs/api/py/webdriver_chrome/selenium.webdriver.chrome.options.html#selenium.webdriver.chrome.options.Options.add_extension

[nickytonline]

Selenium is too slow and older. We're already using Playwright in the app repo and we're keeping things uniform for tooling. Cypress was an option, but as mentioned, it doesn't work with popups in an extension.

What specific issues are you running into?

[a0m0rajab]

The main issue is that Playwright show the extension popup as HTML page rather than an extension, this limits the access to extension-related tools like: https://developer.chrome.com/docs/extensions/reference/storage/ which we use for signup etc.

I thought about either going to selenium, or testing the extension components and functions without having the E2E test.

[JabSYsEmb]

Hi all!

thank you @a0m0rajab and @nickytonline for your last live stream. I was able to proceed further with testing:

• I've destructed the context from callback function of test('popup page'), context
• waited for the popup to be opened await context.waitForEvent('page')
• fill the password/username fields and click on 'login' button.
• after being authorized we can close the popped up page and get back to index.html

I share also the code snippet for that but it still buggy, notice I have added dotenv package to be able to avoid hardcoding the password and username

// env.text.local 
GITHUB_USERNAME="password"
GITHUB_PASSWORD="username"

import { test } from "./popupFixture";
import dotenv from "dotenv";
import path from "path";
import { expect } from "playwright/test";

dotenv.config({ path: path.resolve(__dirname, "..", "env.test.local") });

test("popup page", async ({ page, extensionId, context }) => {
  await page.goto(`chrome-extension://${extensionId}/index.html`);
  await page.getByRole("button", { name: "Login" }).click();

  // wait for popup to open
  const authPage = await context.waitForEvent("page");
  // await authPage.waitForLoadState("domcontentloaded");
  await authPage
    .locator("#login_field")
    .fill(process.env.GITHUB_USERNAME ?? "username");
  await authPage
    .locator("#password")
    .fill(process.env.GITHUB_PASSWORD ?? "password");
  await authPage.locator('input[type="submit"]').click();

  // await  authPage.getByRole("button", { name: "Authorize" }).click();
  // await  authPage.getByAltText("User Avatar").focus();
  await Promise.all([
    authPage.getByRole("button", { name: "Authorize" }).click(),
    authPage.getByAltText("User Avatar"),
  ]);

 await authPage.close();
 
  await page.getByRole("button", { name: "Login" }).click();
  await expect(page.getByAltText("Open Sauced logo")).toBeVisible();
});

[a0m0rajab]

@JabSYsEmb thanks! that solves the problem I faced, I was expecting this to behave like the dev mode and not allow us to open pages!

just adding extra context from the DMs, playwright provides different screenshots for every page it opens in the time line:

the next code is made to consider two situations: if we logged in before to opensauced and if we did not.

  await Promise.all([
    authPage.getByRole("button", { name: "Authorize" }).click(),
    authPage.getByAltText("User Avatar"),
  ]);

I did a quick check right now, it works, thank you for unblocking me.
I will check this tomorrow on livestream with a test account and see how things will go, we might have a PR tomorrow,

I think the next step would be setting the auth up for the whole tests: https://playwright.dev/docs/auth

[a0m0rajab]

Update from the last stream,
the solution @JabSYsEmb provided seems to work. The only thing I wanted to do was to have the auth setup in different logic to keep the extension test logic separated. The playwright auth documentation provides a way to authenticate and get the cookies from the website, which seemed logical to try, but it did not work when I tried it with the setup we have for the extension right now. I need to dig a bit further to understand the reason behind that.

At the same time, I created a test account to login to Github and opensauced.

As for the next step, I feel to run the logic that @JabSYsEmb provided, get it to work then play around the cookies file and dependencies.

[a0m0rajab]

I ran the test after applying the requested changes in the #292 PR

I did play around with the tests and tried to check if we could find the injected code in the browser with the tests that we have. For that reason, I wrote an extra test block:

test('Test GitHub Page', async ({ page, extensionId, context }) => {
    const newPage = page;
    await newPage.goto(`https://github.com/open-sauced/ai/pull/292`);
    await newPage.waitForTimeout(10000);
    await newPage.getByText('Reviewers');
    await expect(newPage.locator('#add-pr-to-highlights-button')).toBeVisible()
});

screen shot

I am not sure why when I used the next code it did not work,

test('Test GitHub Page', async ({ page, extensionId, context }) => {
    await page.goto(`https://github.com/open-sauced/ai/pull/292`);
    await page.waitForTimeout(10000);
    // wait for an item with id:add-pr-to-highlights-button
    // wait for this getByText('Reviewers')
    await page.getByText('Reviewers');
    await expect(page.locator('#add-pr-to-highlights-button')).toBeVisible()
});

Screenshot, not sure why it's showing a failure

[a0m0rajab]

A worth to check link for the cookies is this one: https://playwright.dev/docs/test-fixtures#overriding-fixtures which adds the cookies directly to the fixture.