By Edwin Hernandez
Hello everyone. This time we would like to talk about BDD and SpecFlow. We have featured many types of test automation in this blog, from UI Tests for regression to Performance Testing with JMeter. We have also highlighted some models to organize your tests such as POM (Page Object Model).
In this post, we will first review from a concept standpoint, what is the BDD model and how to use it. Then we will review what Cucumber is just like a leg up into a tutorial/intro of SpecFlow for .NET.:
1. CONCEPTS
- What is BDD?
Behavior-Driven Development (BDD) is an agile software development process that encourages collaboration among developers, quality assurance testers, and customer representatives in a software project. It encourages teams to use conversation and concrete examples to formalize a shared understanding of how the application should behave (Wikipedia).
What this means is that BDD tries to link business requirements with technical specifications by writing requirements in plain English through something called Domain-Specific-Language (DSL).
A related concept is TDD, which is a methodology in which developers first write a Test, and only then do they write the code to make the test pass. It's an iterative process in which more tests are created which then results in more code or improvement to code.
BDD is an evolution of TDD. In BDD, behavioral specifications are written in plain English (DSL), and these specifications can be used to write test code, and the test code can be used to create app code.
This doesn't always happen like this. As a matter of fact, on most of the projects of which I've been part, the BDD Feature is created first, then a developer writes some code to implement it, and only then is a test written to validate the Feature. This is because I've mostly worked implementing API or UI testing, which is difficult to implement without a working UI or an API Swagger (but not impossible, you can just create stubs until endpoints are available).
Below is a sample Feature specification (also from Wikipedia). BDD practices tell us that a feature must have a Title, a Narrative, and Acceptance Criteria. You can have multiple scenarios for the same feature since you need to cover several angles:
Title: Returns and exchanges go to inventory.
As a store owner,
I want to add items back to inventory when they are returned or exchanged,
so that I can track inventory.
Scenario 1: Items returned for a refund should be added to inventory.
Given that a customer previously bought a black sweater from me
and I have three black sweaters in inventory,
when they return the black sweater for a refund,
then I should have four black sweaters in inventory.
Scenario 2: Exchanged items should be returned to inventory.
Given that a customer previously bought a blue garment from me
and I have two blue garments in inventory
and three black garments in inventory,
when they exchange the blue garment for a black garment,
then I should have three blue garments in inventory
and two black garments in inventory.
Once a Feature is written, tests can be created. Each sentence on the specification is supposed to correspond to a unit of test code. These sentences can pass arguments down to the test code too (this is mostly done with a test tool, more on this later).
It's important to point out that BDD does not replace TDD, it's an evolution of it. Also, BDD can have some overlap with other test design patterns/models, such as the ones we have covered in this blog like Page Object Model (POM), or Fluent Design Pattern. But BDD works at a slightly higher level than those design patterns, it works at the requirement specification layer of the solution. This means that you can (and maybe should) implement BDD on top of a design model like POM.
BDD has been around for some time (at least 2008-9), but it seems to have picked up in the early 2010s. BDD is however only a methodology, some tools make it easier to implement it in a Programming IDE. We will mention two of them.
- Cucumber
We will review what Cucumber is, only from a concept standpoint since there is already a tutorial in this blog that takes you step-by-step on how to use it. You can find it here: Get Started with Cucumber and Azure DevOps!
Having said that, Cucumber is an open-source tool created to implement BDD. It was created in Ruby and meant for Ruby testing. Nowadays it supports other languages such as Java and JavaScript. The open-source version of Cucumber is called Cucumber Open and can be found on GitHub. It seems to be supported by SmartBear who also owns a proprietary version called Cucumber Studio.
Cucumber implements Features files according to the BDD standard and uses a line-oriented design to link the sentences (lines) in the Feature file to test code. This approach is called Gherkin Language, a language used in Cucumber to define test cases (feature files).
Please check out the Cucumber Tutorial that is mentioned above for a step-by-step description on how to implement BDD with Cucumber using Eclipse for your IDE and even integrating your test solution with Azure DevOps.
- SpecFlow
As you read this Microsoft Community blog, you may be here to learn how to implement all of this for .NET and on Microsoft's stack. Well, SpecFlow is the most popular (maybe only one?) open-source BDD framework for .NET.
SpecFlow started as a GitHub project in 2009 when a group of developers decided to build a native .NET framework to automate Gherkin feature files. This then evolved into SpecFlow, which was funded by Tricentis in 2019.
Now, SpecFlow has great documentation. If you are interested in this technology, you should check out their Step by Step Getting Started Guide and their Blog.
In this series of posts, we will provide a similar installation/use guide but try to provide added value by updating the steps for Visual Studio 2022 (official documentation still shows 2019) and by focusing a bit more on Microsoft's stack and Azure DevOps integration. We will also try to show you how to avoid the normal bumps along the way and investigate a few C# examples of usual cases where BDD is used in the .NET world.
In the next post, we will also explore Spex, another tool that makes it easier to sync your Feature files with Azure DevOps Test Cases.
Please read ahead!
2. TUTORIAL
-
Installing SpecFlow for Visual Studio 2022
- Step 1. Installing Extension
- The setup starts by installing the SpecFlow Extension which will then install several item templates. For this, start Visual Studio 2022 without opening any project or code. Then from the top menu, go to Extensions>Manage Extensions, then search the Marketplace for SpecFlow for Visual Studio 2022:
- Hit download and restart VS. When it comes back again, go through the dialog to install the extension:
- Step 2: SpecFlow Project Template
- Create a new project. You will notice that new templates have been installed. Create a new SpecFlow Project (C#, Windows).
- Use Framework .NET 6.0, choose MSTest for Test Framework, and make sure to add the FluentAssertions Library.
- The template will create a starter SpecFlow solution that should look like this:
- Create a new project. You will notice that new templates have been installed. Create a new SpecFlow Project (C#, Windows).
You should now have everything you need to start writing Feature files. Let's review the methodology first.
- Writing Feature following the BDD guidelines
Let's try to follow the complete BDD process, which as we said, is an evolution of TDD. We will try to now anchor it on Agile planning and later we will integrate it with Azure DevOps.
- To simplify things, let's work with a simple Class Library project that we will use to simulate buying/selling stock.
- For this let's assume the following user story (tracked in e.g., Azure DevOps):
As a StockApp User
I want to purchase a given amount of stock at the latest value
So that I can increase the value of my portfolio
Step 1. Let's write a SpecFlow Feature file with a single scenario to test that story:
Please note that this example shows that you can use parameters in line, which will then be replaced by the values in the Examples table. This is a quick and smart way to implement Data-Driven Testing. Also, note that we are using the BDD keywords for Given, When, and Then to organize our narrative (And takes the value of the previous line). We are "translating" the user story into a BDD style Feature file with statements and validations to accomplish the acceptance criteria.
Step 2. Now comes the fun part, you can right-click each of the lines on the Feature file individually or right-click any space to scope all lines, and "Define Steps". This will create step definitions for each line.
- This process should create arguments for your methods or every parameter you are using in your Feature file.
- You can choose to copy the step definitions to the clipboard or create a new class (.cs) already containing them
If you generate steps for all lines, it should look like this:
Step 3. Now, following the principles of TDD, we have a test, but we don't have the code to test. Let's add a Class Library project into the solution and reference it in the test project:
- At the Solution level, right-click then Add>New Project. Select a C# Class Library project, name it and make sure to use .NET 6.0:
- It should end up looking something like this:
- Then make sure to reference the Class Library project in the Test Project. Right-click References>Add New Reference, then:
Step 4. The next step would be to create the stubs of some empty classes, empty methods, and in general, just reorganize the arguments in the step definitions:
- Please note the use of SpecFlow Fluent Assertions in this line:
newPortfolio.Value.Should().BeGreaterThan(initialPortfolioValue);
This is an assertion, even if we don't use the Assert class. If the comparison fails, it should generate an Exception and fail the test. These fluent assertions are very useful and more efficient at validating collections and objects with many properties for such cases as it is common for API Test solutions.
- Now, the classes, methods, and properties mentioned in this test code don't exist in the Class Library project yet. If you used Visual Studio's feature that lets you create empty stubs with Non-Implemented Exceptions, then if you run your test, they will fail of course. Let's move on to the next step.
Step 5. Following the TDD principles, let's write some app code in the Class Library so that we have something to test:
That's just some simple app code that fits our purpose. Make sure your solution is built, then let's move on to execute the tests.
- Running SpecFlow Tests in Visual Studio
If you have followed this tutorial down to this point, you should be good to go and open the Test Explorer from the top menu (Test>Test Explorer) and let it discover all tests. It should look like this:
- Please note that each line of Test Data (Examples) is treated as a different test. You can execute a single data set, the whole test method, even the whole test class. Feel free to Debug and run line by line if you are interested in seeing how the execution progresses.
- If you run all the tests, they should pass successfully:
That's it, you have your first passing SpecFlow tests. Not much data is provided, other than the fact that they passed and the duration. Now let's review how to improve the reporting capabilities.
- SpecFlow Reporting Capabilities
Reporting is a very useful capability of SpecFlow. This feature is similar to the Advanced Logger for UI Tests NuGet Package that we described how to implement in this blog but goes a few steps forward in that it provides a dashboard for your test run.
**Note: The way reporting was implemented before SpecFlow 3 was that Reporting was included within the SpecFlow framework and package. However, these reports have been deprecated along with the whole SpecFlow-Runner (check the announcement here). Those reports were also helpful and If you are interested in how these SpecFlow 2 reports used to work, check the documentation here. However, I think that the new reports are just as good, if not better.
The new way to go, starting on SpecFlow 3, is to use SpecFlow LivingDoc. This is a separate NuGet package that you should already have installed in your solution since you are using the SpecFlow Project Template:
SpecFlow LivingDoc can help you generate reports for Azure DevOps or locally. For now, let's review how to create a report locally:
Step 1. Build (and clean) your project.
Step 2. Execute all tests.
A file named TestExecution.json should have been created in this location:
<SolutionDirectory>\<TestProject>\bin\Debug\net6.0
Step 3. The HTML report can only be generated through Command Line (CLI). You will need SpecFlow LivingDoc CLI, but do NOT install the one from NuGet Packages. Instead, open the Package Manager Console and execute this:
dotnet tool install --global SpecFlow.Plus.LivingDoc.CLI
That should install the version that is compatible with your SDK (.NET 6.0).
Step 4. From the same Package Manager Console, execute LivingDoc by providing the folder of your test project and the location of the TestExecution.json file:
LivingDoc feature-folder C:\users\<yourUser>\source\repos\BlogDemo\BlogDemo -t C:\users\<yourUser>\source\repos\BlogDemo\BlogDemo\bin\debug\net6.0\TestExecution.json
That should generate an HTML file like this one:
- I find this report very useful because it includes every data set, all step definitions, scenario, and Feature descriptions and it is easy to filter. You can run this report for full execution of all the features of your application and it should include all in a single report/dashboard.
- Make sure to check out the Analytics tab, to find the summarization (i.e., dashboard) view:
This should be enough for a simple example that includes all the basic components of BDD and SpecFlow in Visual Studio for .NET 6.0.
This post will be the first in a SpecFlow series, please stay tuned for the next ones:
- How to integrate SpecFlow BDD Tests with Azure DevOps using LivingDoc and Spex
- How to create API Tests with SpecFlow BDD in Visual Studio
- How to create Selenium UI Tests with SpecFlow BDD in Visual Studio
Thanks for reading!
References
- https://www.infoq.com/news/2018/04/cucumber-bdd-ten-years/
- https://specflow.org/about/
- https://specflow.org/learn/bdd/
- https://en.wikipedia.org/wiki/Behavior-driven_development
