Azure DevOps: Not Just for Devs and Ops

I attended the DAQAA meeting last week where Scott Sauber and Max Zimmerman from Lean TECHniques presented on Azure DevOps and provided an overview of the various features.

Azure DevOps is the “new”, rebranded version of Microsoft Team Foundation Server (TFS) and Microsoft Test Manager (MTM), and Visual Studio Online / Visual Studio Team Services (VSTS). It's effectively a unified tool for managing your product dashboards, Scrum/Kanban boards, code commits and pull requests, release pipelines, and testing. It provides the same functionality as JIRA, with the CI/CD capabilities of Jenkins, and the ability to commit code to a number of different places, including GitHub. It's a single pane of glass…if you don't count the number of tabs I have open to use multiple features at the same time.

I've been using Azure DevOps since May of last year, and used TFS, MTM, and VSTS prior to that, so I have a lot of familiarity with the application. As a tester, Azure DevOps provides me with three main features: the ability to review code commits and pull requests, the ability to follow a release and know what's been released where, and a place where I can create tests to execute with a way to track the pass/fail status.

When I begin working on a new user story, I create a new test case directly from the story, which automatically links it to the story. You can create as many test cases as you want using the free version of Azure DevOps, but you won't be able to use the test runner and actually track your pass/fail results. I prefer using a tool that is built for the task, versus trying to track your results in a spreadsheet. It's easier to track test cases by user story, assign them to a tester, and track results if all the information is kept in the same place.

Repos: Commits & Pull Requests

Code changes can also be linked directly to a user story. When I'm creating a test case, it's important to know what the developers are actually working on. I wrote in a previous post how reviewing pull requests helps with updating regression test suites, and in identifying risk and potential scenarios that need to be tested. Testing should not wait until a story is done (Code Complete).

Pipelines: Pipelines & Releases

My biggest difficulty in using Azure DevOps has been getting familiar with the Release tracking feature. Not only does it differ a bit from my experience with Jenkins, but the release structure at the current client is also a bit different, where each build is actually broken into three separate stages–DB, API, and UI–and released to multiple environments. All three stages must successfully deploy to a single environment. If any one of them fails, the release fails, even though several stages of a release may be successful. It's important to know WHEN code is being deployed, and WHERE it is being deployed. I've tried to test something MANY times, only to find I'm testing in the wrong environment, or the code changes haven't been deployed to the environment yet. I want to avoid wasted effort.

Boards

I've used both virtual and physical Scrum/Kanban boards for a number of teams I've been on. The more I use the virtual board, the less I like it. If you have remote team members, or have a monitor where you can display your board in real-time, it makes more sense. I used to appreciate being able to check the virtual board throughout the day where I could view the tasks developers had moved across the board. It helped me to know what user stories would be ready next, and which ones I should focus on. If a story isn't being worked on, it doesn't make sense to focus on it ahead of something that will be ready sooner.

Azure DevOps allows you to create tasks on the virtual board and tie them to a user story. You may be able to capture a number of the tasks for a story during a grooming session, but as new tasks need to be created, they can easily be added while working on a story.

I've found however that the tasks on the virtual board create another level of obscurity. Code changes can be linked directly to a story, but they can also be linked to a task. I've noticed that the tasks that are linked to a story are not listed in any particular order. I can easily view a list of code changes when linked to a story and see the order in which they were made, but I can't look at a list of tasks and easily see the order in which each task was completed. If code changes are linked to a task, I have to view the story, select the correct task, then click the link to view the code changes, but those changes could be spread across multiple tasks. In my opinion, it's better to be able to view all those code changes in a single place where you can view them in order.

You can mark a user story as “Done” without first having marked every task as “Done”. You can also create tasks, attach no work to them at all, and mark them as “Done”. However, if nothing is attached, and no comments were added, how do you know the task is “Done”?

Test Plans

This is where I manage all of my test cases, and execute them using the test runner. I have a single test plan for my team. I create a new static suite under that for each sprint. For each sprint, I create a new requirement-based suite, allowing me to query all the stories assigned to a given sprint, select the ones I want, and add them to a test suite. As I create test cases from each user story, the test cases will display here for each story. Using the test runner, I can execute a test case step-by-step, marking each step as passed, then mark the test case as passed. I can have multiple test cases for a single story, and I can re-order my test cases as needed to put them in order, or group them together as desired. Once I've passed a test case, I usually open the test case from the Test Plans module, select the link to the story, and update the story as “Done”. This way I can verify that my test case was in a “Ready” state, it “Passed”, and the story is “Done”. Using the test runner, I can also create defects as they're discovered. When a new bug is created from a test case, it will automatically pull in the step-by-step details from the test case, including the point at which the failure occurred.

I've lived in the Microsoft bubble all my life. I've used a number of different Microsoft products, including Windows, Office (Word, Excel, Outlook, PowerPoint, Publisher, Visio), SQL Management Server, and PowerBI. The more I use these products, the less I like them. I wish Azure DevOps would provide a spreadsheet-feel when creating a new test case. I frequently copy steps from one test case to another, but I can't copy an entire row. I have to copy a step cell-by-cell. You can't right-click on a cell and select a “Copy” option either, as you get a different set of options.

I also reported to Microsoft a very annoying cosmetic bug when creating test cases in Azure DevOps. When you select a cell to copy, the entire row is shaded in blue. If you copy the text (Ctrl+C) into an empty cell, no additional formatting is copied. If you copy the text into a cell that is pre-populated, the blue shaded background of the copied text displays in the new cell. I find myself compelled to retype the same text, without the blue shading, defeating the purpose for having copied the text. Since it's nothing more than a cosmetic issue, there are no plans to fix it, but why does it work that way in the first place?

Tags

I haven't found Azure DevOps to be very customizable. You may want to customize your environments, root causes for defects, OS and Browser information, etc. You can create configurations to cover various OS and Browser combinations, then select for each test case which configurations apply. As you complete a test case for one configuration, the test runner will proceed to the next one until all configurations have been completed. For other information, I've used tags to indicate in what environment a defect was found, what the root cause was for a defect, or whether a test case is automated, ready to be automated, needs updated, or cannot be automated.

Summary

I've used a variety of tools for Agile Product Management, to create user stories, check in code, deploy code changes, and execute tests. I haven't seen a single tool that does it all besides Azure DevOps. There are certainly ways in which it might be improved (starting with the name), but in the end, you have to judge whether the whole of Azure DevOps is greater than the sum of its parts. Sometimes a single application tries to do too much. There may be other, better options for at least some of the features in Azure DevOps, but none I've found that can do it all.