K6: Improve threshold parsing and errors

0

Currently, threshold calculations are actually executed in mini JavaScript runtimes: https://github.com/loadimpact/k6/blob/master/stats/thresholds.go

That's hard to beat for flexibility, but in the long term, we probably should transition to a simple DSL that's easy to parse and verify. It will likely be needed for things like the distributed execution anyway, but it will definitely be useful for reducing user errors and confusion. For example, here are some thresholds that are wrong, but won't actually abort the k6 execution:

export let options = {
    thresholds: {
        "http_req_duration": [
            // this never fails...
            "avg=123",
            // this only emits a warnings every second
            "throw new Error('wat')",
            // this is never reached... but if it was, it will only emit a warning as well
            "test>=500",
        ],
    },
};

As a short-term fix, we likely should make the thresholds stop the script when there are errors. And it's worth investigating if we can make the values we set in the VM consts, so using = instead of == produce an error.

na-- picture na--  ·  15 Mar 2019

Most helpful comment

4

Hey @na--, thanks for watching!

In the demo they quickly noticed that there was an issue, since they were expecting an error.

Yep, I had chalked it up to me passing the wrong configuration, but if k6 did throw an error on startup in the case of threshold misconfiguration, that'd be a great dev experience!

I guess us using it as a mini case study probably wasn't its original purpose, but it was very useful for us nonetheless.

Happy it could be a helpful case study. :)

Feel free to ping us here or on Slack if you have seen any other bugs or if you have suggestions for improving k6 in any way!

And will do! Thanks. Really like what you guys have created!

cajames picture cajames  ·  27 Nov 2019

All comments

1

For the new threshold parsing code, we might consider this library: https://github.com/PaesslerAG/gval

I saw it when researching the jsonpath libraries @robingustafsson mentioned in https://github.com/loadimpact/k6/issues/992, the second one is based on it.

na-- picture na--  ·  9 Apr 2019
1

We stumbled on a k6 demo presentation and spotted a somewhat connected issue, using rate in the threshold for a Counter metric: https://youtu.be/Hu1K2ZGJ_K4?t=1103

Ideally, this should have resulted in a configuration error before the script even started, since combining these things doesn't make sense. Unfortunately, because of the current laizes-faire threshold implementation, the script ran successfully :disappointed: . In the demo they quickly noticed that there was an issue, since they were expecting an error. But in a real test setup, a k6 user might set up a threshold slightly incorrectly like that, and then run their script in a CI with the potentially false assurance that their system performs below the specified thresholds...

Thanks, @cajames, for your very good demo! I guess us using it as a mini case study probably wasn't its original purpose, but it was very useful for us nonetheless :sweat_smile: Feel free to ping us here or on Slack if you have seen any other bugs or if you have suggestions for improving k6 in any way!

na-- picture na--  ·  22 Nov 2019
4

Hey @na--, thanks for watching!

In the demo they quickly noticed that there was an issue, since they were expecting an error.

Yep, I had chalked it up to me passing the wrong configuration, but if k6 did throw an error on startup in the case of threshold misconfiguration, that'd be a great dev experience!

I guess us using it as a mini case study probably wasn't its original purpose, but it was very useful for us nonetheless.

Happy it could be a helpful case study. :)

Feel free to ping us here or on Slack if you have seen any other bugs or if you have suggestions for improving k6 in any way!

And will do! Thanks. Really like what you guys have created!

cajames picture cajames  ·  27 Nov 2019
0

https://github.com/loadimpact/k6/issues/1832 (or something like it) is a prerequisite for the current issue.

na-- picture na--  ·  1 Feb 2021