JBehave
  1. JBehave
  2. JBEHAVE-573

Step Marked as PENDING without preceding FAILED step - silent failure???

    Details

    • Type: Bug Bug
    • Status: Resolved Resolved
    • Priority: Major Major
    • Resolution: Not A Bug
    • Affects Version/s: None
    • Fix Version/s: 3.7.4, 3.8
    • Component/s: Core
    • Labels:
      None
    • Number of attachments :
      0

      Description

      In a pull of master today, but using that JBehave-Core with client code, we have a case where the scenario output looks like :-

      Given foo
      When bar
      And I never get to this line (PENDING)
      Then it does not matter about this line (NOT PERFORMED)

      We can 100% assert that "I never get to this line" matches a single well defined step, with no sound-alikes.

      We can't see a FAILED line, which is our usual experience with JBehave

      We can see a Selenium screenshot, but we don't know what that pertains to, and it does not show anything that looks like an error. We're not sure why it was produced.

        Activity

        Hide
        Cristiano Gavião added a comment -

        Hi David, could you please provide some code that reproduce the problem?
        thanks

        Show
        Cristiano Gavião added a comment - Hi David, could you please provide some code that reproduce the problem? thanks
        Hide
        David Batchelor added a comment -

        We've learned that "matching" Groovy methods only match because Java bytecode is generated for Groovy source (and JBehave picks up on them.

        Thus the following works:

         
           @When("I never get to this line")
            void neverBlahBlah()  {
              // etc
            }
        

        But this Groovy case fails because bytecode is not generated for it:

         
           @When("I never get to this line")
            void neverBlahBlah(String foo = "bar")  {
              // etc
            }
        

        This is because Groovy generates meta-methods for more sophisticated method signatures like that. Can JBehave peer into Groovy's meta-model more ? This is going to be a pitfall for others...

        Show
        David Batchelor added a comment - We've learned that "matching" Groovy methods only match because Java bytecode is generated for Groovy source (and JBehave picks up on them. Thus the following works: @When("I never get to this line") void neverBlahBlah() { // etc } But this Groovy case fails because bytecode is not generated for it: @When("I never get to this line") void neverBlahBlah(String foo = "bar") { // etc } This is because Groovy generates meta-methods for more sophisticated method signatures like that. Can JBehave peer into Groovy's meta-model more ? This is going to be a pitfall for others...
        Hide
        Mauro Talevi added a comment -

        What's the usecase for the different method signature?

        Show
        Mauro Talevi added a comment - What's the usecase for the different method signature?
        Hide
        Damon Smith added a comment -

        I had the same problem with this: @When("I enter '$value' into the form field '$controlName'")

        when my method turns out to be private instead of public.

        I can see that there's reasoning behind why failing steps say pending. It's so that people can "write steps before they are implemented".

        Meanwhile, out here in the real world people don't actually want a build that passes successfully before the test methods are implemented. That's not helping anyone. No one does that. That would actually just be crazy.

        What people actually do out here in the real world is mistype the text of the step or (in my case) put a step annotation in front of what turns out to be a private method. Or any other of a litany of minor development errors that might cause a step not to match a function. Then spend HOURS trying to why the F*#K jbehave isn't working.

        When this happens, jbehave's approach is to fail silently, and if the developer should scroll through their voluminous log output then they might find that says a step is "pending". Insofar as jbehave is little more than a library to parse some text and call some functions from said text, do you not think it might be useful to THROW ERROR MESSAGES WHEN ERRORS OCCUR?!

        And then perhaps we can all get back to doing our actual work. I've had work mates try to explain to me that "no no you don't understand, that's the way it's supposed to work; it says pending when it means failed." We also have this FailingUponPendingStep thing. It means the tests fail but it doesn't mean that it tells me I don't have a method for my function.

        You can probably tell that this is annoying me slightly. JBehave is a TOOL. To do a VERY SIMPLE JOB. It has to parse some frickin text and call some frickin functions from it. It's not rocket surgery. When there's an error, throw an error and exit. You will make everyone else's lives much less of a misery if you just write software that does one single job in an orderly and predictable fashion.

        I say it again NOTHING IS WORSE THAN SILENT FAILURE in a software library. I would recommend that you a) fix this particular one and b) don't embed the concept of silent failure into the very business logic of your software.

        Show
        Damon Smith added a comment - I had the same problem with this: @When("I enter '$value' into the form field '$controlName'") when my method turns out to be private instead of public. I can see that there's reasoning behind why failing steps say pending. It's so that people can "write steps before they are implemented". Meanwhile, out here in the real world people don't actually want a build that passes successfully before the test methods are implemented. That's not helping anyone. No one does that. That would actually just be crazy. What people actually do out here in the real world is mistype the text of the step or (in my case) put a step annotation in front of what turns out to be a private method. Or any other of a litany of minor development errors that might cause a step not to match a function. Then spend HOURS trying to why the F*#K jbehave isn't working. When this happens, jbehave's approach is to fail silently, and if the developer should scroll through their voluminous log output then they might find that says a step is "pending". Insofar as jbehave is little more than a library to parse some text and call some functions from said text, do you not think it might be useful to THROW ERROR MESSAGES WHEN ERRORS OCCUR?! And then perhaps we can all get back to doing our actual work. I've had work mates try to explain to me that "no no you don't understand, that's the way it's supposed to work; it says pending when it means failed." We also have this FailingUponPendingStep thing. It means the tests fail but it doesn't mean that it tells me I don't have a method for my function. You can probably tell that this is annoying me slightly. JBehave is a TOOL. To do a VERY SIMPLE JOB. It has to parse some frickin text and call some frickin functions from it. It's not rocket surgery. When there's an error, throw an error and exit. You will make everyone else's lives much less of a misery if you just write software that does one single job in an orderly and predictable fashion. I say it again NOTHING IS WORSE THAN SILENT FAILURE in a software library. I would recommend that you a) fix this particular one and b) don't embed the concept of silent failure into the very business logic of your software.
        Hide
        Mauro Talevi added a comment -

        I would say there's nothing worse than someone ranting at a tool when they are using it in the way it's not meant for.

        The executable methods are meant to be public, and the step is marked as pending because a matching step is not found.

        We can improve documentation or even try to find a way to improve the functionality. But a bit less attitude would help

        Show
        Mauro Talevi added a comment - I would say there's nothing worse than someone ranting at a tool when they are using it in the way it's not meant for. The executable methods are meant to be public, and the step is marked as pending because a matching step is not found. We can improve documentation or even try to find a way to improve the functionality. But a bit less attitude would help
        Hide
        Colin Vipurs added a comment -

        Some of us actually do work the way JBehave was intended and to suggest otherwise is disingenuous at best and plain arrogant at worst. When writing new stories we want them in source control and CI even though they haven't been implemented yet and absolutely do not want them breaking the build.

        Show
        Colin Vipurs added a comment - Some of us actually do work the way JBehave was intended and to suggest otherwise is disingenuous at best and plain arrogant at worst. When writing new stories we want them in source control and CI even though they haven't been implemented yet and absolutely do not want them breaking the build.
        Hide
        Cristiano Gavião added a comment -

        Poor Damon Smith, sounds like a crying baby...

        I would recommend you A) Know a little bit more about what PRIVATE means and use it properly B) Don't blame about something that you don't payed for this way C) be more polite !!!

        Show
        Cristiano Gavião added a comment - Poor Damon Smith, sounds like a crying baby... I would recommend you A) Know a little bit more about what PRIVATE means and use it properly B) Don't blame about something that you don't payed for this way C) be more polite !!!
        Hide
        Damon Smith added a comment -

        You're right, it was an unnecessary rant, I'm sorry. It was not a personal attack on anyone and I was purely venting my frustration at the design decisions that led to this particular framework failing silently when I make one of my many programming errors.

        Show
        Damon Smith added a comment - You're right, it was an unnecessary rant, I'm sorry. It was not a personal attack on anyone and I was purely venting my frustration at the design decisions that led to this particular framework failing silently when I make one of my many programming errors.
        Hide
        Mauro Talevi added a comment - - edited

        Hi Damon, it's all right, we've all been through the frustration of not understanding why something fails to work as expected.

        We welcome all feedback and always try to improve the tool and the user experience. But let's try to be constructive.

        As for the issue at hand: a step is not marked as "pending" after a failure, that is a "not performed step". Pending means that is has not been matched to any public method in the steps classes. It is true that we've not explicitly underlined that the method must be public, and I'll update the docs in this sense to make it clearer.

        So in this sense, there is no silent failure, just lack of matched steps.

        Incidentally, it wouldn't be difficult to allow non-public methods to be used, but I wonder what the usefulness of it would be. The idea we're trying to convey is to create an API of available steps, declared by annotations.

        Show
        Mauro Talevi added a comment - - edited Hi Damon, it's all right, we've all been through the frustration of not understanding why something fails to work as expected. We welcome all feedback and always try to improve the tool and the user experience. But let's try to be constructive. As for the issue at hand: a step is not marked as "pending" after a failure, that is a "not performed step". Pending means that is has not been matched to any public method in the steps classes. It is true that we've not explicitly underlined that the method must be public, and I'll update the docs in this sense to make it clearer. So in this sense, there is no silent failure, just lack of matched steps. Incidentally, it wouldn't be difficult to allow non-public methods to be used, but I wonder what the usefulness of it would be. The idea we're trying to convey is to create an API of available steps, declared by annotations.
        Hide
        Mauro Talevi added a comment -

        Updated docs: JBEHAVE-847

        Show
        Mauro Talevi added a comment - Updated docs: JBEHAVE-847
        Mauro Talevi made changes -
        Field Original Value New Value
        Status Open [ 1 ] Resolved [ 5 ]
        Fix Version/s 3.7.4 [ 18964 ]
        Resolution Not A Bug [ 6 ]
        Mauro Talevi made changes -
        Fix Version/s 3.8 [ 19104 ]

          People

          • Assignee:
            Unassigned
            Reporter:
            David Batchelor
          • Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved: