JBehave
  1. JBehave
  2. JBEHAVE-509

Alternate behavior for failing assertions

          Details

          • Type: Improvement Improvement
          • Status: Open Open
          • Priority: Major Major
          • Resolution: Unresolved
          • Affects Version/s: None
          • Fix Version/s: None
          • Component/s: Core
          • Labels:
            None
          • Environment:
            All
          • Number of attachments :
            0

            Description

            I've been thinking about how JBehave handles assertion failures.

            As it works today, JBehave behaves something like this when it encounters a failing assertion:

            Given I have searched for a round-trip flight
            When I view my trip summary
            Then I should see the origin and destination of each flight segment
            And I should see the fare cost (FAILED)
            And I should see the taxes and fees (NOT RUN)
            And I should see the on-time rating of each flight segment (NOT RUN)
            Test Failed!

            I'd like to have the results look like this:

            Given I have searched for a round-trip flight
            When I view my trip summary
            Then I should see the origin and destination of each flight segment
            And I should see the fare cost (FAILED)
            And I should see the taxes and fees
            And I should see the on-time rating of each flight segment (FAILED)
            Test Failed!

            I'm finding that most assertions in a functional test need not halt test execution and it would be nice to have the test run "until it can't." If the fare cost doesn't get fixed for weeks, that won't hide the broken on-time ratings.

            For JBehave, I believe that the change would entail recording and swallowing assertion exceptions while failing on other sorts of exceptions. If selenium can't find the continue button, it will throw a NoSuchElementException and halt the test.

            If this does get implemented, I think that it would be useful to have some sort of before/after step hooks, so that we can take screenshots for each failing assertion.

            It would be nice to have something like: @AfterStep(FAILING)

              Activity

              Hide
              Permalink
              Paul Hammant added a comment -

              I think there's a difference (as you allude) between RuntimeException and AsserionError derivatives.

              The former happening in THEN steps should cease further step invocation. The latter happening, seems to be part of the intent of testing, and it should be OK to continue with more THEN steps (but no further if there were another WHEN step after for the same scenario)

              I think if should be the default behavior for JBehave, and that somewhere (in config) it should be turned off.

              Show
              Paul Hammant added a comment - I think there's a difference (as you allude) between RuntimeException and AsserionError derivatives. The former happening in THEN steps should cease further step invocation. The latter happening, seems to be part of the intent of testing, and it should be OK to continue with more THEN steps (but no further if there were another WHEN step after for the same scenario) I think if should be the default behavior for JBehave, and that somewhere (in config) it should be turned off.

                People

                • Assignee:
                  Unassigned
                  Reporter:
                  Chris Kozak
                • Votes:
                  1 Vote for this issue
                  Watchers:
                  0 Start watching this issue

                  Dates

                  • Created:
                    Updated: