This pathway is a tool to help guide your self development in continuous delivery testing. It includes a variety of steps that you may approach linearly or by hopping about to those that interest you most.
Each step includes:
- links to a few resources as a starting point, but you are likely to need to do your own additional research as you explore each topic.
- a suggested exercise or two, which focus on reflection, practical application and discussion, as a tool to connect the resources with your reality.
Take your time. Dig deep into areas that interest you. Apply what you learn as you go.
STEP – Removing release testing
Why does this pathway exist? Understand the key reasons to significantly shorten a release process, the arguments against release testing and why organisations aim to avoid batched releases in agile environments:
- 2012 – So, what is water-SCRUM-fall? – Sanjeev Sharma
- 2012 – Pattern language of flow – release steps – Gareth Evans
- 2014 – Organisation Antipattern: Release Testing – Steve Smith
- 2009 – Work in small batches – Eric Ries
- 2015 – Avoiding the most common pitfall of large scale agile – Gojko Adzic
[2 hours] Research your existing release process and talk to people within your organisation to find out whether there are any current initiatives to improve it.
STEP – Introduction to continuous delivery
What is the end goal? Discover the basics of continuous delivery and the theory of how it can be implemented in organisations.
- 2013 – Continuous Delivery: What it is and how to get started (PDF) – Puppet Labs eBook
- 2013 – Continuous Delivery – Martin Fowler
- 2014 – Visualizations of Continuous Delivery – Nhan Ngo
- 2015 – The Case for Continuous Delivery – Zend
- 2012 – Continuous Delivery (video) – Jez Humble
- 2014 – Implementing Continuous Delivery (5 slide decks) – Thought Works
- 2014 – A skeptics guide to continuous delivery (5 part series) – J. Paul Reed
- 2010 – Continuous Delivery vs Continuous Deployment – Jez Humble
- 2013 – Continuous Delivery vs Continuous Deployment – Yassal Sundman
[1 hour] Based on what you’ve read, try to explain the theory of continuous delivery in your own words to someone in your team. Describe what appeals to you about continuous delivery, what you disagree with, and things that you think will be difficult to implement in your organisation. Afterwards, if you have any remaining questions, raise these with a technical lead or coach for further discussion.
STEP – Experiences in continuous delivery
How are other organisations doing continuous delivery? There is a lot of variance in implementation and differing opinions about how to approach the theory. Understand the realities of the people, processes and tools of teams doing continuous delivery:
- 2014 – Testing in a Continuous Delivery World (video) – Amy Phillips
- 2015 – Testing in Continuous Deployment (video) – Kristian Karl
- 2015 – Enabling Continuous Delivery in Enterprises with Testing (video) – Anand Bagmar
- 2015 – Continuous Delivery without Test Automation (slides) – Maaret Pyhäjärvi
- 2013 – The continuous delivery process (slides) – Huib Schoots & Miel Donkers
- 2013 – Test Patterns for Continuous Delivery (slides) – Andy Singleton
- 2015 – Continuous Development at Vend (slides) – Nick Walker
- 2015 – Delivery continuous delivery, continuously – Phil Wills & Simon Hildrew
- 2013 – So it took a year – Tomas Riha
- 2013 – Deploying the Netflix API – Ben Schmaus
- 2014 – Practical Continuous Deployment – Steve Smith
- 2014 – Etsy’s Journey to Continuous Integration for Mobile Apps – Nassim Kammah
- 2015 – How Pivotal Performs Continuous Delivery of Mobile Services – Dwayne Forde
[2 hours] Compare the experiences shared in the links above and the theory of continuous delivery. Identify common themes, and areas where ideas or implementation details differ. Discuss your analysis with a technical lead or coach.
STEP – Starting with continuous integration
What is the first step? Understand the concept of continuous integration:
- Continuous Integration (summary) – Thought Works
- 2006 – Continuous Integration – Martin Fowler
- 2006 – Continuous Build is not Continuous Integration – Dan North
- 2014 – Automation, the Backbone of Continuous Integration – Melissa Bennett
- 2015 – Continuous Integration is Dead – Yegor Bugayenko
- 2015 – Our working test automation failed me – Maaret Pyhäjärvi
- 2015 – How Often Do You Really Fix A “Failing” Automated Check – Richard Bradshaw
- 2015 – Meet Jenkins – Kohsuke Kawaguchi
- About git branching and merging and distributed workflow
- Atlassian stash documentation home
[3 hours] At the start of this talk transcript, Jez Humble points out that most people aren’t doing continuous integration. How does the approach to continuous integration in your team differ to the theory? Talk to a developer to confirm your understanding of your branching strategy, the way you use source control management tools, and how you manage merging to master. If you use a continuous integration tool, create a list of the jobs that are used by your team during development, and be sure that you understand what each one does. Reflect on how quickly your team respond to build failures in these jobs, and who takes ownership for resolving these. Discuss this exercise with a technical lead or coach to collaboratively identify opportunities for improvement, then raise these ideas at your next team retrospective.
STEP – Theory of test automation
Continuous delivery puts a lot of focus on test automation. In order to support development of an effective pipeline it’s important to understand common strategies for automation, and the distinction between checking and testing:
- 2012 – Test Pyramid – Martin Fowler
- 2009 – The forgotten layer of the test automation pyramid – Mike Cohn
- 2011 – Yet another software testing pyramid – Alister Scott
- 2014 – Introducing the Software Testing Cupcake (Anti-Pattern) – Fabio Pereira
- 2009 – Testing vs Checking – Michael Bolton
- 2013 – Testing and Checking Refined – James Bach
- 1998 – When should a test be automated (PDF) – Brian Marick
[1 hour] Read through the automation strategy for your product. How well does your existing strategy for automation support your delivery pipeline? What opportunities exist to improve this strategy? Discuss your thoughts with a technical lead or coach.
STEP – A delivery pipeline
Understand how to construct delivery pipeline and the role of automation:
- 2010 – Anatomy of the deployment pipeline – Jez Humble & Dave Farley
- 2014 – The continuous delivery pipeline – Andrew Phillips
- 2014 – All about pipelines (slides) – Thought Works
- 2011 – Continuous Delivery Using Build Pipelines With Jenkins and Ant – James Betteley
- 2011 – When should we be doing automated testing – Cromulent Testing
- 2015 – Extensive post release testing is a sign of an unhealthy testing process – Alister Scott
- 2015 – Continuous Testing: The missing link in Continuous Delivery – Ophir Prusak
[3 hours] Create a visual representation of the current delivery pipeline for your product. Use a timeline format that shows the build jobs in your continuous integration tool at every stage from development through to production deploy, any test jobs that execute automated suites, and points where the tester is hands-on, exploring the product. Compare your pipeline to the simplified images by Yassal Sundman forcontinuous delivery and continuous deployment, then reflect on the following questions:
- How would your approach to testing change, or not, if we were able to deploy to production 10 times a day? How about 100 times a day?
- Does the coverage provided by your automation give you a degree of comfort or confidence? If not, what needs to change?
- Does your automation execute fast enough? How fast do you think it should be? How can you achieve this?
- Where in the pipeline would you want to retain hands-on testing? How would you justify this?
Discuss your ideas with a technical lead or coach. Work together to identify actions from your thinking and determine how to proceed in implementing change.
STEP – Non-functional testing in continuous delivery
Learn more about integrating security, performance, and other non-functional testing in a continuous delivery pipeline:
- 2015 – Continuous Security – Laura Bell
- 2015 – Security for continuous integration – Ian Bartholomew
- 2015 – Automated security testing in a continuous delivery pipeline – Stephen de Vries
- Security sandwich – Thought Works
- 2014 – An introduction to performance testing in continuous delivery – Caleb Billingsley
- 2012 – Stress testing application in continuous delivery environment – Eugene Dvorkin
- 2014 – Non-functional testing in continuous delivery – Katrina Clokie
[2 hours] Does your organisation have a non-functional testing “sandwich”? Having read more about organisations who integrate these activities earlier in the process, what opportunities do you see to improve the way that you work? What would the first steps be? Talk to a technical lead or coach about what you’d like to see change.
STEP – Cross-browser testing
For continuous delivery of a web application, it’s important to include cross-browser testing in the delivery pipeline. Discover strategies for cross-browser testing and the tools available to support it:
- 2012 – The Truth About Browser Compatibility, and What You Need To Do – Tom Ewer
- 2014 – Cross browser testing conundrum – Shailendra Matkar & Sandeep Maher
- 2014 – Strategy and Solutions – Shailendra Matkar, Rahul Pathak & Sandeep Maher
- 2014 – What I’ve Learned from Performing Cross-Browser Testing – Kendall Smith
- 2015 – Tips for better cross-browser testing – Justin Rohrman
- 2014 – 13 Essential Tools to Check Cross-Browser Compatibility – Grace Smith
- 2014 – My setup and tips for cross-browser testing – Emil Björklund
- 2013 – Techniques for mobile and responsive cross-browser testing – Jordan Lewis
[8 hours] Learn more about the common cross-browser tools that are available, understand the advantages and disadvantages of each option, then select a tool to trial. Create a prototype to execute existing browser-based automation for your product across multiple browsers. If successful on your local environment, attempt to create a prototype job in your continuous integration tool to verify that your chosen solution works as part of your continuous integration. Discuss what you learned about the tool and the results of your experiment with a technical lead or coach.
STEP – Test data & databases
Discover the additional considerations around test data in continuous delivery:
- 2013 – Test Data and the Continuous Delivery Pipeline – Monica Luke
- 2015 – Why test data must extend beyond sampling production data – Tom Pryce
- 2015 – Is your data keeping up with your code deployments? – Tom Pryce
- 2014 – Continuous delivery and the database – Phil Factor
- Liquibase (source control for databases)
[1 hour] Data is a constant headache for testers. Consider the limitations of the test data in use by your automation. How could you improve the data within your delivery pipelines? How could you improve the way that you locate data for testing? Talk through your ideas with a technical lead or coach.
STEP – Configuration management & environments
An effective delivery pipeline is supported by multiple test environments. Learn more about configuration management, environments and infrastructure services in continuous delivery:
- 2013 – Notes on Continuous Delivery: Configuration Management – Luis Atencio
- Configuration management for continuous delivery (Part 2) – Mark Burgess
- 2013 – Build a toolbox for continuous delivery – Aliza Earnshaw
- What is Puppet? – Puppet Labs
- What is Docker? – Docker
- 2014 – Environment Portability – Tomas Riha
- 2015 – Continuous Delivery of Infrastructure Services – Lindsay Holmwood
[1 hour] Talk to your operations or support team about how they provide test environments for continuous integration, the infrastructure required to support a delivery pipeline, and what their plans are for future changes in this space.
STEP – Testing in production
Understand A/B testing and feature toggles:
- 2010 – The ultimate guide to A/B testing – Paras Chopra
- 2014 – Three hard truths about A/B testing – Peep Laja
- 2014 – Most of Your A/B Test Results Are Illusory and That’s Okay – Will Kurt
- 2009 – Goodbye Google – Douglas Bowman
- 2010 – Feature Toggle – Martin Fowler
- 2012 – Dark launching software features – Charles Illya Krempeaux
- 2013 – Configuration flags: A love story – Noah Sussman
- 2014 – Feature toggles are one of the worst kinds of technical debt – Jim Bird
[1 hour] Talk to people in your organisation to find out how you currently use feature toggles and how you make decisions about what to keep based on user analytics. Could your approach be more responsive through targeted use of a monitoring tool like splunk? Share your thoughts with a technical lead or coach.