Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add automated testing and improve test files to have better data sets #6

Open
mateusdeap opened this issue May 30, 2023 · 0 comments
Open
Labels
enhancement New feature or request help wanted Extra attention is needed

Comments

@mateusdeap
Copy link
Member

mateusdeap commented May 30, 2023

Testing this library isn't super fun right now.

The 2 main issues are:

  1. It's manual
  2. The sample json output file is quite limited

We need to come up with a way to automatically test that the given classified output of calling the library is correct for a given failure input file.

I think testing individual modules might be quite easy, but this main functionality test will require some thinking.

For one, we're working with RSpec output, specifically. I do want to expand this lib to work with other outputs from other testing frameworks (specifically elixir ones, for obvious reasons), but for now I think we can focus on RSpec.

We want the lib to grab test output files and group the errors by their root cause. Right now we actually group them by message similarity, which isn't great, but I think that the first thing is to:

  1. Come up with some mock test output file, or a few files
  2. Write out what we want the library to output
  3. Have our test framework compare the actual output with the expected output

Maybe there's a better way. Maybe we don't want to necessarily test the output but just test how the data is grouped. That way we can avoid string comparisons and leave the output testing to when we decide to test our formatting options.

@mateusdeap mateusdeap added enhancement New feature or request help wanted Extra attention is needed labels May 30, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request help wanted Extra attention is needed
Development

No branches or pull requests

1 participant