Implementing a Custom Test Runner for Shortest

[m2rads]

Hey Shortest community! 👋 I wanted to propose an idea: let's build a custom test runner instead of relying solely on Github Actions. Don't get me wrong, Github Actions are great, but they've got some limitations:

1. New repos without workflow files can't use Shortest right away
2. New unit tests won't run until a workflow is set up
3. Users have to bounce between Shortest and Github
4. Untested local changes can cause unexpected errors in the main branch

A custom test runner could solve these issues and more! It'd let users run quality checks through Shortest without committing changes. Plus, we could expand beyond unit tests to include UI regression, integration, and end-to-end testing. Users can rely on Shortest for software quality assurance.

What do you think? 🚀

[slavingia]

How would it work for various codebases? But open to the idea!

[m2rads]

We’d use a Jenkins pipeline to handle tasks like cloning the repo, checking out the feature branch, and running a .yml file for testing (AI could generate this if the user hasn’t provided one). A Vercel serverless API would bridge the client and pipeline, with future AI capabilities like codebase indexing.

Instead of committing test files, we send them to Jenkins. This gives us more control over testing and avoids introducing changes late in the process.

Looking ahead, we could expand to UI regression testing using Selenium to interact with web apps or APIs. While unit tests are essential, they don't capture the full scope since they rely on mocking. This setup could support more comprehensive testing, like running Selenium on dev environments.

If you are open to this design I can setup an early version of this idea and demo it to you.

[slavingia]

Very open to this! My biggest concern would be supporting diff kinds of codebases easily like Rails, but maybe we just focus on codebases that are fully in TypeScript for now. Also not sure how working w Postgres etc but down to see the demo!

[slavingia]

Should be much easier to do this given it's all browser tests, email, not sure how db would work.