JBehave
  1. JBehave
  2. JBEHAVE-1064

Weird behaviour when stories are cancelled

    Details

    • Type: Bug Bug
    • Status: Open Open
    • Priority: Major Major
    • Resolution: Unresolved
    • Affects Version/s: 3.9.5
    • Fix Version/s: None
    • Component/s: Core
    • Labels:
      None
    • Testcase included:
      yes
    • Number of attachments :
      0

      Description

      I discovered that sometimes JBehave does not break the build in our CI system when some of the stories hang (and are cancelled because of the timeout). While trying to reproduce the issue, I saw many many different outcomes of the build including a build success when stories timed out (which makes them failed to me).
      Unfortunately not for all the cases I have a reproducible case but two of the issues I can reproduce.

      Sources for the test project: https://github.com/dimas/jbehave-cancellation-issue

      Issues it can demonstrate:

      Behave aborts the whole story so scenarios past the cancelled one are not even visible in the report

      This happens even despite doFailOnStoryTimeout(false) and doIgnoreFailureInStories(true) which to me should guarantee all scenarios to be processed before failing.
      If in the Test class constructor you set .useStoryTimeoutInSecs(1)), then it will kill the very first scenario and there is no signs of the second one to be present. The target/jbehave/view/reports.html report will look like

      Scenarios: Total=1, Successful=0, Pending=0, Failed=1

      The target/jbehave/view/reports.html shows just

      Scenario: slow and failing test
      Given a default state
      When I do something slow (FAILED)
      java.lang.InterruptedException: sleep interrupted
      STORY CANCELLED (DURATION 2 s)
      

      No second scenario

      JBehave may report cancelled scenarios as successful.

      Change .useStoryTimeoutInSecs to 2. The target/jbehave/view/reports.html report will look like

      Scenarios: Total=2, Successful=1, Pending=1, Failed=1

      Which seem to be just wrong given that target/jbehave/view/test.test.html shows:

      Scenario: slow and failing test
      Given a default state
      When I do something slow
      Then nothing should happen (FAILED)
      java.lang.InterruptedException: sleep interrupted
      
      Scenario: scenario description
      STORY CANCELLED (DURATION 3 s)
      

      So I see on failed scenario and one cancelled but which of them JBehave thinks is successful?

      Overall, it feels like the outcome hugely depends on what exact moment time out has expired. Which should not be like that.

      I would expect as a correct behaviour that cancelled scenario should be always treated as failed (definitely not successful) and all the scenarios after it should be processed but not invoked given that story timeout has expired but at least reported in full. Maybe all the steps should be listed as skipped/pending/something else. But they all should be present.
      Or otherwise if first scenario in each story files, lots of scenarios just disappear from the report without any traces...

        Activity

        Dmitry Andrianov made changes -
        Field Original Value New Value
        Description I discovered that sometimes JBehave does not break the build in our CI system when some of the stories hang (and are cancelled because of the timeout). While trying to reproduce the issue, I saw many many different outcomes of the build including a build success when stories timed out (which makes them failed to me).
        Unfortunately not for all the cases I have a reproducible case but two of the issues I can reproduce.

        Sources for the test project: https://github.com/dimas/jbehave-cancellation-issue

        Issues it can demonstrate:
        * Behave aborts the whole story so scenarios past the cancelled one are not even visible in the report despite doFailOnStoryTimeout(false) and doIgnoreFailureInStories(true). If in the Test class constructor you set .useStoryTimeoutInSecs(1)), then it will kill the very first scenario and there is no signs of the second one to be present. The target/jbehave/view/reports.html report will look like

        Scenarios: Total=1, Successful=0, Pending=0, Failed=1

        The target/jbehave/view/reports.html shows just

        {code}
        Scenario: slow and failing test
        Given a default state
        When I do something slow (FAILED)
        java.lang.InterruptedException: sleep interrupted
        STORY CANCELLED (DURATION 2 s)
        {code}

        No second scenario

        * JBehave may report cancelled scenarios as successful.

        Change .useStoryTimeoutInSecs to 2. The target/jbehave/view/reports.html report will look like

        Scenarios: Total=2, Successful=1, Pending=1, Failed=1

        Which seem to be just wrong given that target/jbehave/view/test.test.html shows:

        {code}
        Scenario: slow and failing test
        Given a default state
        When I do something slow
        Then nothing should happen (FAILED)
        java.lang.InterruptedException: sleep interrupted

        Scenario: scenario description
        STORY CANCELLED (DURATION 3 s)
        {code}

        So I see on failed scenario and one cancelled but which of them JBehave thinks is successful?
        I discovered that sometimes JBehave does not break the build in our CI system when some of the stories hang (and are cancelled because of the timeout). While trying to reproduce the issue, I saw many many different outcomes of the build including a build success when stories timed out (which makes them failed to me).
        Unfortunately not for all the cases I have a reproducible case but two of the issues I can reproduce.

        Sources for the test project: https://github.com/dimas/jbehave-cancellation-issue

        Issues it can demonstrate:
        .h3 Behave aborts the whole story so scenarios past the cancelled one are not even visible in the report
        This happens even despite doFailOnStoryTimeout(false) and doIgnoreFailureInStories(true) which to me should guarantee all scenarios to be processed before failing.
        If in the Test class constructor you set .useStoryTimeoutInSecs(1)), then it will kill the very first scenario and there is no signs of the second one to be present. The target/jbehave/view/reports.html report will look like

        Scenarios: Total=1, Successful=0, Pending=0, Failed=1

        The target/jbehave/view/reports.html shows just

        {code}
        Scenario: slow and failing test
        Given a default state
        When I do something slow (FAILED)
        java.lang.InterruptedException: sleep interrupted
        STORY CANCELLED (DURATION 2 s)
        {code}

        No second scenario

        .h3 JBehave may report cancelled scenarios as successful.

        Change .useStoryTimeoutInSecs to 2. The target/jbehave/view/reports.html report will look like

        Scenarios: Total=2, Successful=1, Pending=1, Failed=1

        Which seem to be just wrong given that target/jbehave/view/test.test.html shows:

        {code}
        Scenario: slow and failing test
        Given a default state
        When I do something slow
        Then nothing should happen (FAILED)
        java.lang.InterruptedException: sleep interrupted

        Scenario: scenario description
        STORY CANCELLED (DURATION 3 s)
        {code}

        So I see on failed scenario and one cancelled but which of them JBehave thinks is successful?
        Dmitry Andrianov made changes -
        Description I discovered that sometimes JBehave does not break the build in our CI system when some of the stories hang (and are cancelled because of the timeout). While trying to reproduce the issue, I saw many many different outcomes of the build including a build success when stories timed out (which makes them failed to me).
        Unfortunately not for all the cases I have a reproducible case but two of the issues I can reproduce.

        Sources for the test project: https://github.com/dimas/jbehave-cancellation-issue

        Issues it can demonstrate:
        .h3 Behave aborts the whole story so scenarios past the cancelled one are not even visible in the report
        This happens even despite doFailOnStoryTimeout(false) and doIgnoreFailureInStories(true) which to me should guarantee all scenarios to be processed before failing.
        If in the Test class constructor you set .useStoryTimeoutInSecs(1)), then it will kill the very first scenario and there is no signs of the second one to be present. The target/jbehave/view/reports.html report will look like

        Scenarios: Total=1, Successful=0, Pending=0, Failed=1

        The target/jbehave/view/reports.html shows just

        {code}
        Scenario: slow and failing test
        Given a default state
        When I do something slow (FAILED)
        java.lang.InterruptedException: sleep interrupted
        STORY CANCELLED (DURATION 2 s)
        {code}

        No second scenario

        .h3 JBehave may report cancelled scenarios as successful.

        Change .useStoryTimeoutInSecs to 2. The target/jbehave/view/reports.html report will look like

        Scenarios: Total=2, Successful=1, Pending=1, Failed=1

        Which seem to be just wrong given that target/jbehave/view/test.test.html shows:

        {code}
        Scenario: slow and failing test
        Given a default state
        When I do something slow
        Then nothing should happen (FAILED)
        java.lang.InterruptedException: sleep interrupted

        Scenario: scenario description
        STORY CANCELLED (DURATION 3 s)
        {code}

        So I see on failed scenario and one cancelled but which of them JBehave thinks is successful?
        I discovered that sometimes JBehave does not break the build in our CI system when some of the stories hang (and are cancelled because of the timeout). While trying to reproduce the issue, I saw many many different outcomes of the build including a build success when stories timed out (which makes them failed to me).
        Unfortunately not for all the cases I have a reproducible case but two of the issues I can reproduce.

        Sources for the test project: https://github.com/dimas/jbehave-cancellation-issue

        Issues it can demonstrate:
        h3. Behave aborts the whole story so scenarios past the cancelled one are not even visible in the report
        This happens even despite doFailOnStoryTimeout(false) and doIgnoreFailureInStories(true) which to me should guarantee all scenarios to be processed before failing.
        If in the Test class constructor you set .useStoryTimeoutInSecs(1)), then it will kill the very first scenario and there is no signs of the second one to be present. The target/jbehave/view/reports.html report will look like

        Scenarios: Total=1, Successful=0, Pending=0, Failed=1

        The target/jbehave/view/reports.html shows just

        {code}
        Scenario: slow and failing test
        Given a default state
        When I do something slow (FAILED)
        java.lang.InterruptedException: sleep interrupted
        STORY CANCELLED (DURATION 2 s)
        {code}

        No second scenario

        h3. JBehave may report cancelled scenarios as successful.

        Change .useStoryTimeoutInSecs to 2. The target/jbehave/view/reports.html report will look like

        Scenarios: Total=2, Successful=1, Pending=1, Failed=1

        Which seem to be just wrong given that target/jbehave/view/test.test.html shows:

        {code}
        Scenario: slow and failing test
        Given a default state
        When I do something slow
        Then nothing should happen (FAILED)
        java.lang.InterruptedException: sleep interrupted

        Scenario: scenario description
        STORY CANCELLED (DURATION 3 s)
        {code}

        So I see on failed scenario and one cancelled but which of them JBehave thinks is successful?
        Dmitry Andrianov made changes -
        Description I discovered that sometimes JBehave does not break the build in our CI system when some of the stories hang (and are cancelled because of the timeout). While trying to reproduce the issue, I saw many many different outcomes of the build including a build success when stories timed out (which makes them failed to me).
        Unfortunately not for all the cases I have a reproducible case but two of the issues I can reproduce.

        Sources for the test project: https://github.com/dimas/jbehave-cancellation-issue

        Issues it can demonstrate:
        h3. Behave aborts the whole story so scenarios past the cancelled one are not even visible in the report
        This happens even despite doFailOnStoryTimeout(false) and doIgnoreFailureInStories(true) which to me should guarantee all scenarios to be processed before failing.
        If in the Test class constructor you set .useStoryTimeoutInSecs(1)), then it will kill the very first scenario and there is no signs of the second one to be present. The target/jbehave/view/reports.html report will look like

        Scenarios: Total=1, Successful=0, Pending=0, Failed=1

        The target/jbehave/view/reports.html shows just

        {code}
        Scenario: slow and failing test
        Given a default state
        When I do something slow (FAILED)
        java.lang.InterruptedException: sleep interrupted
        STORY CANCELLED (DURATION 2 s)
        {code}

        No second scenario

        h3. JBehave may report cancelled scenarios as successful.

        Change .useStoryTimeoutInSecs to 2. The target/jbehave/view/reports.html report will look like

        Scenarios: Total=2, Successful=1, Pending=1, Failed=1

        Which seem to be just wrong given that target/jbehave/view/test.test.html shows:

        {code}
        Scenario: slow and failing test
        Given a default state
        When I do something slow
        Then nothing should happen (FAILED)
        java.lang.InterruptedException: sleep interrupted

        Scenario: scenario description
        STORY CANCELLED (DURATION 3 s)
        {code}

        So I see on failed scenario and one cancelled but which of them JBehave thinks is successful?
        I discovered that sometimes JBehave does not break the build in our CI system when some of the stories hang (and are cancelled because of the timeout). While trying to reproduce the issue, I saw many many different outcomes of the build including a build success when stories timed out (which makes them failed to me).
        Unfortunately not for all the cases I have a reproducible case but two of the issues I can reproduce.

        Sources for the test project: https://github.com/dimas/jbehave-cancellation-issue

        Issues it can demonstrate:
        h3. Behave aborts the whole story so scenarios past the cancelled one are not even visible in the report
        This happens even despite doFailOnStoryTimeout(false) and doIgnoreFailureInStories(true) which to me should guarantee all scenarios to be processed before failing.
        If in the Test class constructor you set .useStoryTimeoutInSecs(1)), then it will kill the very first scenario and there is no signs of the second one to be present. The target/jbehave/view/reports.html report will look like

        Scenarios: Total=1, Successful=0, Pending=0, Failed=1

        The target/jbehave/view/reports.html shows just

        {code}
        Scenario: slow and failing test
        Given a default state
        When I do something slow (FAILED)
        java.lang.InterruptedException: sleep interrupted
        STORY CANCELLED (DURATION 2 s)
        {code}

        No second scenario

        h3. JBehave may report cancelled scenarios as successful.

        Change .useStoryTimeoutInSecs to 2. The target/jbehave/view/reports.html report will look like

        Scenarios: Total=2, Successful=1, Pending=1, Failed=1

        Which seem to be just wrong given that target/jbehave/view/test.test.html shows:

        {code}
        Scenario: slow and failing test
        Given a default state
        When I do something slow
        Then nothing should happen (FAILED)
        java.lang.InterruptedException: sleep interrupted

        Scenario: scenario description
        STORY CANCELLED (DURATION 3 s)
        {code}

        So I see on failed scenario and one cancelled but which of them JBehave thinks is successful?

        Overall, it feels like the outcome hugely depends on what exact moment time out has expired. Which should not be like that.

        I would expect as a correct behaviour that cancelled scenario should be always treated as failed (definitely not successful) and all the scenarios after it should be processed but not invoked given that story timeout has expired but at least reported in full. Maybe all the steps should be listed as skipped/pending/something else. But they all should be present.
        Or otherwise if first scenario in each story files, lots of scenarios just disappear from the report without any traces...

          People

          • Assignee:
            Unassigned
            Reporter:
            Dmitry Andrianov
          • Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

            Dates

            • Created:
              Updated: