Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Bug]: e2e tests have become flaky #3482

Open
2 tasks done
MatissJanis opened this issue Sep 22, 2024 · 6 comments
Open
2 tasks done

[Bug]: e2e tests have become flaky #3482

MatissJanis opened this issue Sep 22, 2024 · 6 comments
Labels
bug Something isn't working help wanted Extra attention is needed regression This bug appeared after a recent release and needs to be fixed before the next release

Comments

@MatissJanis
Copy link
Member

Verified issue does not already exist?

  • I have searched and found no existing issue
  • I will be providing steps how to reproduce the bug (in most cases this will also mean uploading a demo budget file)

What happened?

The e2e tests have become increasingly more flaky in the past few days. Need to look into what's the issue and address it.

Where are you hosting Actual?

None

What browsers are you seeing the problem on?

No response

Operating System

None

@MatissJanis MatissJanis added bug Something isn't working help wanted Extra attention is needed regression This bug appeared after a recent release and needs to be fixed before the next release labels Sep 22, 2024
@joel-jeremy
Copy link
Contributor

joel-jeremy commented Sep 23, 2024

One this I noticed is that the rules page is crashing on VRT but when you open the page in the browser it works just fine.
VRT:
image

Browser:
image

@joel-jeremy
Copy link
Contributor

This change also appears randomly:
image

@UnderKoen
Copy link
Contributor

One thing I noticed with #3499 was that the css classes of glamour weren't applied with the same precedence as in the browser. This can be caused by the order of the css in the final css changing

@MatissJanis
Copy link
Member Author

MatissJanis commented Sep 25, 2024

Another issue I'm seeing semi-frequently: onboarding test.


Sample failing ci job: https://github.com/actualbudget/actual/actions/runs/11037117182/job/30657293317

Trace: trace.zip

View the trace here: https://trace.playwright.dev


Edit: it seems that there are a bunch of different redirects happening when clicking "start fresh". Some are redirecting to /budget and some to /accounts. Must be a race condition somewhere.

@MatissJanis
Copy link
Member Author

Both the rules & onboarding e2e should be more stable now. Haven't been able to reproduce the settings one yet. If anyone sees a failing CI job with it - please post it here.

Or if you also see other e2e ci jobs failing (that use the latest changes in master) - please post them here.

@MatissJanis
Copy link
Member Author

Related #3506

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working help wanted Extra attention is needed regression This bug appeared after a recent release and needs to be fixed before the next release
Projects
None yet
Development

No branches or pull requests

3 participants