Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Investigate test flakiness #293

Open
MMMalik opened this issue Sep 29, 2022 · 0 comments
Open

Investigate test flakiness #293

MMMalik opened this issue Sep 29, 2022 · 0 comments
Labels
status:confirmed An issue confirmed by the development team. type:task Any other issue (refactoring, typo fix, etc).

Comments

@MMMalik
Copy link
Contributor

MMMalik commented Sep 29, 2022

Are you reporting a feature request or a bug?

Task

Provide detailed reproduction steps (if any)

It happens occasionally that unit tests start failing without any apparent cause even if there were no code changes: #292.

While they are called "unit tests", in fact these tests don't really fulfill the requirements of unit testing:

  • There is a dependency on CDN
  • Singleton CKEDITOR namespace is used across all tests and there is an unreliable teardown for editor instances
  • Test cases are run sequentially
  • Test cases are run in predefined order

Current stack:

  • Karma is used because it allows to easily run tests in real browsers. Real browser is currently necessary due to fact that editor is loaded from CDN.
  • @testing-library/react and related tools are used because they are pretty much staples when it comes to testing React code. In a typical scenario it's usually used with Jest and jsdom but we need a real browser here.
  • Jasmine is used because it's a popular choice for Karma.

Perhaps our current approach can be improved?

Expected result

Unit tests are fast and reliable.

Actual result

Unit tests occasionally fail for unclear reasons.

@MMMalik MMMalik added type:failingtest A failing test. status:confirmed An issue confirmed by the development team. type:task Any other issue (refactoring, typo fix, etc). and removed type:failingtest A failing test. labels Sep 29, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
status:confirmed An issue confirmed by the development team. type:task Any other issue (refactoring, typo fix, etc).
Projects
None yet
Development

No branches or pull requests

1 participant