Details
-
Type: Bug
-
Status: Open
-
Priority: Major
-
Resolution: Unresolved
-
Affects Version/s: 3.9.5
-
Fix Version/s: None
-
Component/s: Core
-
Labels:None
-
Testcase included:yes
-
Number of attachments :
Description
I discovered that sometimes JBehave does not break the build in our CI system when some of the stories hang (and are cancelled because of the timeout). While trying to reproduce the issue, I saw many many different outcomes of the build including a build success when stories timed out (which makes them failed to me).
Unfortunately not for all the cases I have a reproducible case but two of the issues I can reproduce.
Sources for the test project: https://github.com/dimas/jbehave-cancellation-issue
Issues it can demonstrate:
Behave aborts the whole story so scenarios past the cancelled one are not even visible in the report
This happens even despite doFailOnStoryTimeout(false) and doIgnoreFailureInStories(true) which to me should guarantee all scenarios to be processed before failing.
If in the Test class constructor you set .useStoryTimeoutInSecs(1)), then it will kill the very first scenario and there is no signs of the second one to be present. The target/jbehave/view/reports.html report will look like
Scenarios: Total=1, Successful=0, Pending=0, Failed=1
The target/jbehave/view/reports.html shows just
Scenario: slow and failing test Given a default state When I do something slow (FAILED) java.lang.InterruptedException: sleep interrupted STORY CANCELLED (DURATION 2 s)
No second scenario
JBehave may report cancelled scenarios as successful.
Change .useStoryTimeoutInSecs to 2. The target/jbehave/view/reports.html report will look like
Scenarios: Total=2, Successful=1, Pending=1, Failed=1
Which seem to be just wrong given that target/jbehave/view/test.test.html shows:
Scenario: slow and failing test Given a default state When I do something slow Then nothing should happen (FAILED) java.lang.InterruptedException: sleep interrupted Scenario: scenario description STORY CANCELLED (DURATION 3 s)
So I see on failed scenario and one cancelled but which of them JBehave thinks is successful?
Overall, it feels like the outcome hugely depends on what exact moment time out has expired. Which should not be like that.
I would expect as a correct behaviour that cancelled scenario should be always treated as failed (definitely not successful) and all the scenarios after it should be processed but not invoked given that story timeout has expired but at least reported in full. Maybe all the steps should be listed as skipped/pending/something else. But they all should be present.
Or otherwise if first scenario in each story files, lots of scenarios just disappear from the report without any traces...
Activity
Field | Original Value | New Value |
---|---|---|
Description |
I discovered that sometimes JBehave does not break the build in our CI system when some of the stories hang (and are cancelled because of the timeout). While trying to reproduce the issue, I saw many many different outcomes of the build including a build success when stories timed out (which makes them failed to me).
Unfortunately not for all the cases I have a reproducible case but two of the issues I can reproduce. Sources for the test project: https://github.com/dimas/jbehave-cancellation-issue Issues it can demonstrate: * Behave aborts the whole story so scenarios past the cancelled one are not even visible in the report despite doFailOnStoryTimeout(false) and doIgnoreFailureInStories(true). If in the Test class constructor you set .useStoryTimeoutInSecs(1)), then it will kill the very first scenario and there is no signs of the second one to be present. The target/jbehave/view/reports.html report will look like Scenarios: Total=1, Successful=0, Pending=0, Failed=1 The target/jbehave/view/reports.html shows just {code} Scenario: slow and failing test Given a default state When I do something slow (FAILED) java.lang.InterruptedException: sleep interrupted STORY CANCELLED (DURATION 2 s) {code} No second scenario * JBehave may report cancelled scenarios as successful. Change .useStoryTimeoutInSecs to 2. The target/jbehave/view/reports.html report will look like Scenarios: Total=2, Successful=1, Pending=1, Failed=1 Which seem to be just wrong given that target/jbehave/view/test.test.html shows: {code} Scenario: slow and failing test Given a default state When I do something slow Then nothing should happen (FAILED) java.lang.InterruptedException: sleep interrupted Scenario: scenario description STORY CANCELLED (DURATION 3 s) {code} So I see on failed scenario and one cancelled but which of them JBehave thinks is successful? |
I discovered that sometimes JBehave does not break the build in our CI system when some of the stories hang (and are cancelled because of the timeout). While trying to reproduce the issue, I saw many many different outcomes of the build including a build success when stories timed out (which makes them failed to me).
Unfortunately not for all the cases I have a reproducible case but two of the issues I can reproduce. Sources for the test project: https://github.com/dimas/jbehave-cancellation-issue Issues it can demonstrate: .h3 Behave aborts the whole story so scenarios past the cancelled one are not even visible in the report This happens even despite doFailOnStoryTimeout(false) and doIgnoreFailureInStories(true) which to me should guarantee all scenarios to be processed before failing. If in the Test class constructor you set .useStoryTimeoutInSecs(1)), then it will kill the very first scenario and there is no signs of the second one to be present. The target/jbehave/view/reports.html report will look like Scenarios: Total=1, Successful=0, Pending=0, Failed=1 The target/jbehave/view/reports.html shows just {code} Scenario: slow and failing test Given a default state When I do something slow (FAILED) java.lang.InterruptedException: sleep interrupted STORY CANCELLED (DURATION 2 s) {code} No second scenario .h3 JBehave may report cancelled scenarios as successful. Change .useStoryTimeoutInSecs to 2. The target/jbehave/view/reports.html report will look like Scenarios: Total=2, Successful=1, Pending=1, Failed=1 Which seem to be just wrong given that target/jbehave/view/test.test.html shows: {code} Scenario: slow and failing test Given a default state When I do something slow Then nothing should happen (FAILED) java.lang.InterruptedException: sleep interrupted Scenario: scenario description STORY CANCELLED (DURATION 3 s) {code} So I see on failed scenario and one cancelled but which of them JBehave thinks is successful? |
Description |
I discovered that sometimes JBehave does not break the build in our CI system when some of the stories hang (and are cancelled because of the timeout). While trying to reproduce the issue, I saw many many different outcomes of the build including a build success when stories timed out (which makes them failed to me).
Unfortunately not for all the cases I have a reproducible case but two of the issues I can reproduce. Sources for the test project: https://github.com/dimas/jbehave-cancellation-issue Issues it can demonstrate: .h3 Behave aborts the whole story so scenarios past the cancelled one are not even visible in the report This happens even despite doFailOnStoryTimeout(false) and doIgnoreFailureInStories(true) which to me should guarantee all scenarios to be processed before failing. If in the Test class constructor you set .useStoryTimeoutInSecs(1)), then it will kill the very first scenario and there is no signs of the second one to be present. The target/jbehave/view/reports.html report will look like Scenarios: Total=1, Successful=0, Pending=0, Failed=1 The target/jbehave/view/reports.html shows just {code} Scenario: slow and failing test Given a default state When I do something slow (FAILED) java.lang.InterruptedException: sleep interrupted STORY CANCELLED (DURATION 2 s) {code} No second scenario .h3 JBehave may report cancelled scenarios as successful. Change .useStoryTimeoutInSecs to 2. The target/jbehave/view/reports.html report will look like Scenarios: Total=2, Successful=1, Pending=1, Failed=1 Which seem to be just wrong given that target/jbehave/view/test.test.html shows: {code} Scenario: slow and failing test Given a default state When I do something slow Then nothing should happen (FAILED) java.lang.InterruptedException: sleep interrupted Scenario: scenario description STORY CANCELLED (DURATION 3 s) {code} So I see on failed scenario and one cancelled but which of them JBehave thinks is successful? |
I discovered that sometimes JBehave does not break the build in our CI system when some of the stories hang (and are cancelled because of the timeout). While trying to reproduce the issue, I saw many many different outcomes of the build including a build success when stories timed out (which makes them failed to me).
Unfortunately not for all the cases I have a reproducible case but two of the issues I can reproduce. Sources for the test project: https://github.com/dimas/jbehave-cancellation-issue Issues it can demonstrate: h3. Behave aborts the whole story so scenarios past the cancelled one are not even visible in the report This happens even despite doFailOnStoryTimeout(false) and doIgnoreFailureInStories(true) which to me should guarantee all scenarios to be processed before failing. If in the Test class constructor you set .useStoryTimeoutInSecs(1)), then it will kill the very first scenario and there is no signs of the second one to be present. The target/jbehave/view/reports.html report will look like Scenarios: Total=1, Successful=0, Pending=0, Failed=1 The target/jbehave/view/reports.html shows just {code} Scenario: slow and failing test Given a default state When I do something slow (FAILED) java.lang.InterruptedException: sleep interrupted STORY CANCELLED (DURATION 2 s) {code} No second scenario h3. JBehave may report cancelled scenarios as successful. Change .useStoryTimeoutInSecs to 2. The target/jbehave/view/reports.html report will look like Scenarios: Total=2, Successful=1, Pending=1, Failed=1 Which seem to be just wrong given that target/jbehave/view/test.test.html shows: {code} Scenario: slow and failing test Given a default state When I do something slow Then nothing should happen (FAILED) java.lang.InterruptedException: sleep interrupted Scenario: scenario description STORY CANCELLED (DURATION 3 s) {code} So I see on failed scenario and one cancelled but which of them JBehave thinks is successful? |
Description |
I discovered that sometimes JBehave does not break the build in our CI system when some of the stories hang (and are cancelled because of the timeout). While trying to reproduce the issue, I saw many many different outcomes of the build including a build success when stories timed out (which makes them failed to me).
Unfortunately not for all the cases I have a reproducible case but two of the issues I can reproduce. Sources for the test project: https://github.com/dimas/jbehave-cancellation-issue Issues it can demonstrate: h3. Behave aborts the whole story so scenarios past the cancelled one are not even visible in the report This happens even despite doFailOnStoryTimeout(false) and doIgnoreFailureInStories(true) which to me should guarantee all scenarios to be processed before failing. If in the Test class constructor you set .useStoryTimeoutInSecs(1)), then it will kill the very first scenario and there is no signs of the second one to be present. The target/jbehave/view/reports.html report will look like Scenarios: Total=1, Successful=0, Pending=0, Failed=1 The target/jbehave/view/reports.html shows just {code} Scenario: slow and failing test Given a default state When I do something slow (FAILED) java.lang.InterruptedException: sleep interrupted STORY CANCELLED (DURATION 2 s) {code} No second scenario h3. JBehave may report cancelled scenarios as successful. Change .useStoryTimeoutInSecs to 2. The target/jbehave/view/reports.html report will look like Scenarios: Total=2, Successful=1, Pending=1, Failed=1 Which seem to be just wrong given that target/jbehave/view/test.test.html shows: {code} Scenario: slow and failing test Given a default state When I do something slow Then nothing should happen (FAILED) java.lang.InterruptedException: sleep interrupted Scenario: scenario description STORY CANCELLED (DURATION 3 s) {code} So I see on failed scenario and one cancelled but which of them JBehave thinks is successful? |
I discovered that sometimes JBehave does not break the build in our CI system when some of the stories hang (and are cancelled because of the timeout). While trying to reproduce the issue, I saw many many different outcomes of the build including a build success when stories timed out (which makes them failed to me).
Unfortunately not for all the cases I have a reproducible case but two of the issues I can reproduce. Sources for the test project: https://github.com/dimas/jbehave-cancellation-issue Issues it can demonstrate: h3. Behave aborts the whole story so scenarios past the cancelled one are not even visible in the report This happens even despite doFailOnStoryTimeout(false) and doIgnoreFailureInStories(true) which to me should guarantee all scenarios to be processed before failing. If in the Test class constructor you set .useStoryTimeoutInSecs(1)), then it will kill the very first scenario and there is no signs of the second one to be present. The target/jbehave/view/reports.html report will look like Scenarios: Total=1, Successful=0, Pending=0, Failed=1 The target/jbehave/view/reports.html shows just {code} Scenario: slow and failing test Given a default state When I do something slow (FAILED) java.lang.InterruptedException: sleep interrupted STORY CANCELLED (DURATION 2 s) {code} No second scenario h3. JBehave may report cancelled scenarios as successful. Change .useStoryTimeoutInSecs to 2. The target/jbehave/view/reports.html report will look like Scenarios: Total=2, Successful=1, Pending=1, Failed=1 Which seem to be just wrong given that target/jbehave/view/test.test.html shows: {code} Scenario: slow and failing test Given a default state When I do something slow Then nothing should happen (FAILED) java.lang.InterruptedException: sleep interrupted Scenario: scenario description STORY CANCELLED (DURATION 3 s) {code} So I see on failed scenario and one cancelled but which of them JBehave thinks is successful? Overall, it feels like the outcome hugely depends on what exact moment time out has expired. Which should not be like that. I would expect as a correct behaviour that cancelled scenario should be always treated as failed (definitely not successful) and all the scenarios after it should be processed but not invoked given that story timeout has expired but at least reported in full. Maybe all the steps should be listed as skipped/pending/something else. But they all should be present. Or otherwise if first scenario in each story files, lots of scenarios just disappear from the report without any traces... |