fix: --reporter flag to support multiple values

[timohuovinen]

Affected Components

• CLI
• Create CLI
• Test
• Docs
• Examples
• Other

Notes for the Reviewer

This allows the reporter flag to support multiple params
normally in the form

npx checkly test --reporter list --reporter json
npx checkly test -r dot -r json

This is also consistent with our existing -e and -i flag usage.

I've decided to use the existing flag parsing library function multiple: true to keep things consistent, instead of the suggested npx checkly test --reporter list,json, which would have affected a lot more code and made flag usage within the CLI inconsistent

Resolves #RED-242

Reasoning for multiple reporters:

The --reporter (-r) flag previously only accepted a single value. Users who wanted both human-readable terminal output and machine-parseable output in the same run had no way to achieve this, they'd have to choose one or the other

The code was already built to support multiple reporters, and looks like the flag was always intended to support multiple reporters, but not fully implemented, so generally this is mostly a fix for a missing but expected feature.

i.e. the createReporters already returned an array of Reporter instances and the runner iterated all of them, the limitation was purely in how the CLI flag was parsed and passed through.

Note that playwright's --reporter flag, which also supports multiple reporters, uses a CSV approach, as in --reporter list,json. https://playwright.dev/docs/test-reporters

[ferrandiaz]

LGTM

[sorccu]

Can you explain what the reasoning for needing multiple reporters is, and how it will affect the output? What is the intended use case? Add to PR description please.

[timohuovinen]

@sorccu added as much detail as I can, good now?

[sorccu]

Alright. If you also want to release, look at contributing.md.