Note: You can find the English version after the Chinese one.
【中文版 Chinese version】
从质量维度来看体系化思维的构建，而质量是产品与生俱来的，做好软件产品的质量保障工作更为有效的方式就是要将质量构建到软件产品本身，提高软件产品与生俱来的质量。这也就是质量内建（Build Quality In）的概念。
将质量管理分为四个阶段，即Plan(计划)、Do(执行)、Check(检查) 和 Act(处理)。在质量管理活动中，要求把各项工作按照作出计划、计划实施、检查实施效果，然后将成功的纳入标准，不成功的留待下一循环去解决。
需求条目验收：跟启动是配对的，也叫用户故事验收（Desk check/Shoulder check/Story signoff）、结卡/验卡等。同样在开发机器前面完成，一般由开发将功能演示给业务分析和测试人员，大家确认需求中的正向路径是否都已经开发实现了。更多详情可参考文章《高效用户故事验收》。
代码评审：Code review，也叫代码回顾。有团队成员聚集在一起做回顾的，也有独立线下评审的模式。Thoughtworks同事伍斌道长的文章《Code Review: 超越“审、查、评”的代码回顾》有关于code review非常详细的讲解，请移步参考。
【英文版 English Version】
0.1 Three levels of Systematic Thinking for Testing
Do testing personnel lack systematic thinking?
How to build a quality assurance system for new product teams or projects?
Many of us have come across countless articles or training courses on testing and quality, which are not lacking in testing practices or technical content, but it is difficult to build our own testing system. Based on similar doubts from many friends and my own team practice and consulting experience over the years, I will discuss the construction of systematic thinking for testing at three different levels: basic, intermediate, and advanced.
- "Systems Thinking for Testing (Basics)"
- "Building Systematic Thinking for Testing (Intermediate)"
- "Building Systematic Thinking for Testing (Advanced)"
0.2 Review of the Basic Level Content
In January 2022, "Systems Thinking for Testing - Basics" was published, which introduced the corresponding practical activities and methods for the five basic responsibilities of testing.
- Understand and clarify business requirements
- Develop strategies and design tests
- Implement and execute tests
- Defect management and analysis
- Quality feedback and risk identification
0.3 Overview of Intermediate Level Content
This article is the intermediate level of the series and will discuss the systematic thinking from the perspective of software quality, going beyond testing.
- From testing to quality: Move beyond testing and consider the construction of testing systems from the perspective of quality.
- Quality assurance and quality built-in: Quality is inherent in products, and quality built-in is the fundamental guarantee for quality assurance.
- In-depth practice of quality built-in: How to implement quality built-in is the most concerned issue, and this article will introduce in-depth practices for reference.
01 From Testing to Quality
1.1 Why Move from Testing to Quality
In the "Basics" section, the five basic responsibilities of testers mentioned are the aspects that testers must understand to carry out testing work and lay the foundation for doing a good job in testing.
However, to do a good job of quality assurance for software products, it is far from enough to consider it from the perspective of doing a good job in testing. There are two main reasons for this:
- Software quality cannot be improved through testing. Famous quality management expert Deming once mentioned, "Quality cannot be detected. Once the software product is developed, the quality is already there."
- Software testing can verify whether the software quality meets the requirements, but as the complexity and uncertainty of the software ecosystem increases, it is impossible to predict the software behavior, and therefore it is impossible to verify all aspects of the software quality through testing.
Therefore, testing is a fundamental skill, but focusing only on testing is not enough. It is necessary to break out of testing, adjust the perspective to the quality of the entire software development life cycle, and focus on various dimensions of quality in order to better establish systematic thinking and do a good job of quality assurance for software products.
Before reading the following content, I hope that readers can spend 10 seconds thinking about the following question:
Question: To break out of testing and think from the perspective of software quality, what aspects do we need to focus on?
This is a question that has been talked about for a long time. The answer seems to be clear to everyone, but in reality, it is a problem that is easily overlooked and misunderstood.
1.2 Definition of software quality
Before answering the above question, let's first look at the definition of software quality on Wikipedia. As shown in the figure below:
The definition of software quality on Wikipedia includes two aspects:
- Software functional quality: Based on the functional requirements specification, it looks at whether the software provides corresponding functions. This is from the user's perspective and is the external quality of the software.
- Software structural quality: It looks at the satisfaction of non-functional requirements that support the delivery of functional requirements, such as robustness and maintainability. This is from the perspective of the internal structure of the software and is the internal quality of the software.
The definition of software quality on Wikipedia is consistent with what we usually mention, which includes external quality and internal quality. External quality is perceivable from outside the software and is the quality perceived by users from their perspective of using the software; internal quality involves the internal form of the software, including technical architecture and code quality, and is perceived from the perspective of the team of developers.
I believe that everyone can understand external quality, which is the quality that can be perceived from the user's perspective. However, internal quality is often easily overlooked. In fact, this is also because people generally tend to pursue short-term benefits and cannot see the long-term impact that internal quality brings to software products. Here, I want to show you a very familiar picture to enhance your attention to internal quality.
The code that is rich in technical debt without organization is like the tangled wires in the picture, seriously affecting the diagnosis and fixing of product defects, and the expansion of later functions.
1.3 Overlooked factors that affect quality
Having understood the definition of quality, what are the factors that can affect software quality in daily software development work? In fact, there are many factors that can affect software quality, which may include but are not limited to the following:
- Rushed requirement analysis process, or a relatively single role of the participating personnel, resulting in insufficient understanding of the business context, lack of consideration for key scenarios, and so on;
- Busy delivering more features, ignoring the focus on code quality, not refactoring where necessary, and continuing to add more code on top of the already unhealthy code, leading to confusion and building up high technical debt;
- Insufficient test coverage, making it easy for new code to break existing functionality;
- After developers submit code, they move on to developing new code and lack attention to the submitted code, which can cause delays in fixing failing tests on the continuous integration pipeline, affecting subsequent testing progress;
- Large-scale refactoring occurs on the eve of release, making it impossible to fully regress, resulting in great risk;
- Only a small number of user scenarios are considered in the early stages of the project, and as the business grows, system functionality becomes difficult to extend, resulting in serious performance bottlenecks;
- Technical selection errors, development difficulties, failure to improve in a timely manner, making mistakes repeatedly, and ultimately the problem becomes so serious that it cannot be remedied;
- Insufficient evaluation of third-party components, resulting in the inability to support the production environment;
- Lack of handling of exceptional situations in development, lack of exploration in the testing process, covering only the main paths, and corner cases may cause problems;
- Development or testing only considers the current functional module, lacking broader considerations;
- Lack of attention to cross-functional requirements, leading to serious security or performance issues;
- Insufficient understanding of the production environment, and insufficient logging information, making it difficult to locate issues in the production environment, leading to long downtime;
02 Quality Assurance and Quality Built-in
2.1 How to Ensure Quality
From the perspective of quality, building a systematic thinking is important, and quality is inherent in the product. To effectively ensure the quality of software products, it is important to build quality into the software product itself, and improve the inherent quality of software products. This is the concept of quality built-in.
However, to better explain the concept of quality built-in, let's first look at two concepts: defect exposure and defect prevention.
2.2 Defect Exposure and Defect Prevention
The concept of "defect exposure" is well-known and refers to the defects exposed by the software, usually discovered during testing or use.
If we can comprehensively identify all the possible places where defects may occur in a software product, we can say that the count of defects in the software is certain. (Of course, in reality, we cannot consider all factors comprehensively, and defects are difficult to exhaust.)
If some of these defects can be prevented through certain measures, then the number of defects exposed during testing or use by end-users will be reduced. This is defect prevention, which reduces the number of defects exposed to users.
Software development is a continuous and incremental process, and defect prevention is not a one-time thing. It needs to be carried out continuously throughout the entire software development process. Of course, preventing defects early will be better, as it can significantly reduce the cost of defect fixing and improve the return on investment. The curve in the following figure clearly shows the difference in the cost of fixing defects exposed at different stages, and the cost of fixing defects significantly increases as the software development cycle progresses.
2.3 How to Build Quality In
If we expand the concept of defects to include all issues related to software product quality, then preventing defects is equivalent to building quality in.
As mentioned earlier, the quality factors that are easily overlooked need to be given special attention to build quality in.
However, due to the complexity of the software development process, it is impossible to get everything right the first time and prevent all defects. Therefore, building quality in does not strictly mean that no defects will be exposed, but rather that things need to be done right as early as possible in the software development lifecycle, and deviations need to be corrected through continuous and rapid feedback.
We all know the famous PDCA cycle (also known as the "Deming cycle"):
Quality management is divided into four stages: Plan, Do, Check, and Act. In quality management activities, it is required to plan, implement the plan, check the implementation effect, and then incorporate successful practices into the standard and solve unsuccessful ones in the next cycle.
Building quality in can be understood as executing the PDCA cycle frequently throughout the entire software development lifecycle, taking small steps forward, continuously validating feedback, and making timely adjustments and improvements. As shown in the following figure:
03 Typical Practice of Building Quality In
The previous section mainly introduced the concepts and ideas of quality and building quality in, which are crucial for building systematic thinking and broadening perspectives to form a holistic view. However, these contents alone are not enough. We also need to understand specific implementation methods. Next, we will discuss the implementation plan of building quality in throughout the software development lifecycle from the perspective of related practices.
3.1 Core of Quality Built-in
Before introducing specific quality built-in practices, it is necessary to understand the core of quality built-in at an abstract level, which can be divided into four aspects:
- Testing Shift Left
- Fast Feedback
- Shift Right Testing
- Everyone's Participation
3.1.1 Testing Shift Left
Traditional software development is divided into several relatively independent stages such as analysis, design, development, testing, and release, and testing is a stage that occurs after development is completed.
Testing Shift Left is for traditional independent testing stages, which no longer emphasizes independent testing stages but shifts the testing activity to the earliest stage of the software development life cycle (the leftmost side) - the requirements stage, to do a good job of defect prevention from the requirements stage.
It is important to note that testing shift left does not mean testers shift left, but emphasizes the shift of testing activities:
- Moving to the left of the traditional independent testing phase is called shifting left, not necessarily to the requirements phase. Of course, according to the previous explanation of quality built in, the ideal situation is to move further to the left. If it is impossible to move completely to the left, moving one step is also progress.
- Testers do need to participate in corresponding activities. However, if testers participate in the requirements phase but do not clarify or validate requirements effectively, it is not an effective testing shift left. The key is to look at the effectiveness.
- Not only testers but also developers and other roles need to shift left. Because the validation of requirements effectiveness requires the experience of different roles and inputs from different perspectives, different roles have a responsibility to participate in testing activities in each stage.
- As long as there are corresponding testing activities from the requirements phase, it is not necessarily that testers participate. What is emphasized here is that there must be testing activities, and which role is not the most critical. Roles are just hats. We know that some teams do not have testers, but it does not mean that these teams do not have testing activities.
3.1.2 Fast Feedback
Fast feedback, which means frequent and continuous feedback, can be understood as a process of continuous testing.
Usually, fast feedback requires automation support, such as automated testing, automatic code scanning in the pipeline, etc. It can also be some manual participation in practice, such as manual testing, various types of reviews and acceptance, etc. By quickly obtaining feedback on corresponding work (possibly requirement analysis, development, etc.) through such practice activities, timely repairs, adjustments, or optimizations can be made according to the feedback results.
To ensure the effective implementation of fast feedback/continuous testing practice activities, it is usually necessary to increase the quality gate and ensure the strict execution of the quality gate.
3.1.3 Testing Shift Right
Testing Shift Right is the counterpart of Testing Shift Left, which is to move testing activities from independent testing phases to the production environment.
However, due to the special nature of the production environment, Testing Shift Right is not a simple shift of testing activities. It is through some practical activities to obtain quality-related information such as user behavior and logs in the production environment, and use this information to provide feedback to the previous requirement, development, and testing work, promote the optimization and improvement of corresponding work, and do a better job of quality building.
Testing Shift Right cannot directly build quality, but it can help to build quality better. Therefore, I believe that Testing Shift Right is also one of the indispensable core contents of quality building.
This part of the content is relatively long, and there has been a detailed introduction before. It will not be repeated here. You are welcome to refer to the article "Testing Shift Right - QA in Production" for more detailed information.
3.1.4 Everyone's Participation
Testing shift left, fast feedback, and testing shift right cannot be accomplished by a single role/person. It requires input from different roles with different skills and perspectives. Quality is everyone's responsibility and everyone's participation and the ability to assume quality responsibilities are necessary.
Everyone's participation is not only necessary for many practice activities, but also important for sufficient communication and information sharing within the team. Only when information is transparent, and everyone is clear about quality goals, quality status, and risks can the team truly engage with an entrepreneurial spirit and maximize each team member's value. Beware of the "mushroom planting" phenomenon within the team.
There have been many previous shares on this topic, please refer to articles such as "Whole Team is Responsible for Quality : Can I Avoid It?" ,"Agile Testing's Guiding Principles", and "Everyone is Responsible for Quality".
3.2 Typical Practices of Quality Built-in
In general, typical practices of quality built-in include (but are not limited to) the ones shown in the figure below:
Regardless of whether it is traditional development mode or agile iterative development mode, the software development life cycle includes three stages: requirements analysis, development implementation, and production. In agile mode, however, requirements and development are carried out iteratively and the stages are not clearly defined in terms of time.
For ease of description, these practices are divided into three stages. However, in fact, practice activities are not closely related to stages. The key lies in following the core concepts of testing shift left, fast feedback, testing shift right, and everyone's participation.
3.2.1 Requirements Analysis
Here, I emphasize four typical quality built-in practices that I think are closely related to requirements analysis, which are requirement clarification, Behavior-Driven Development (BDD), Specification by Example (SbE), and requirement review.
- Requirement Clarification: As one of the basic responsibilities of testers, the understanding and clarification of requirements have been introduced in detail in the "Basics" article. However, what needs to be added here is that not only testers but also other relevant roles, such as developers, technical leaders, and technical architects, need to participate in requirement clarification collaboratively.
- Behavior-Driven Development (BDD): Many people mistakenly think that BDD is a testing method, but in fact, BDD is mainly to achieve a consensus on requirements among the three roles. However, the requirements generated through the BDD approach can guide the development of automated testing very well. For more details, please refer to my previous article "What Comes to Your Mind When We Talk About BDD?".
- Specification by Example (SbE): The core idea of SbE is similar to BDD but emphasizes the use of examples to illustrate the requirements to discover missing requirements, make requirements more complete, and make them easier for multiple roles to understand and clarify.
- Requirement Review: Developers and testers need to discuss and confirm requirements with business analysts. It may be a formal meeting for the review of a specific feature (a large requirement), or it can be an offline review of a single requirement item or agile user story. If the discussion of the requirements during the earlier requirement clarification process is thorough, the review here will be relatively lightweight, and the testers (with developers as needed) can arrange the time for the review themselves.
These practices are all aimed at helping the team achieve a common understanding of the requirements in the requirements analysis phase, ensuring the quality of the requirements, and reducing rework and waste caused by requirements problems in the later stages.
3.2.2 Development and Implementation
Once the quality control of the requirement source is in place, the next step is to ensure the quality of each step of the work through fast feedback during the development and implementation stage. The following practices serve this purpose:
Technical solution discussion: It is not about how to do this practice, but to remind that testers should participate as much as possible in the discussion of technical solutions. On the one hand, testers can better design corresponding testing strategies and conduct more comprehensive testing by understanding the technical solution clearly. On the other hand, based on their understanding of the system, testers can provide input for the discussion of technical solutions, such as issues that the system needs to focus on. Some people may feel that their technical skills in testing are not good and that their participation is not valuable, but what I want to say is that there will be no progress if they do not participate. Since they participate, there will definitely be gains, and in the long run, it is very valuable for both individuals and the team as a whole.
Story kickoff: It is a re-clarification and confirmation of a single story's requirement before the developer starts coding, mainly to confirm whether the acceptance criteria are complete enough and whether everyone understands them consistently. It should be as lightweight as possible and can be completed in front of the developer's computer. Business analysis, developers, and testers should participate together, and the time should generally not exceed 15 minutes.
Desk check: It is paired with the story kickoff. It is also completed in front of the developer's machine. Generally, the developer demonstrates the function to business analysts and testers , and everyone confirms whether the positive path of the requirement has been developed and implemented. For more details, please refer to the article "Efficient Desk Check."
Test case design: There is a detailed introduction to test case design in the "Basics" article. From the perspective of quality built-in, the main point is to design test cases as early as possible, and the format of the test cases can be relatively flexible. Listing the test points that need to be focused on is the key. From the perspective of fast feedback, it is not recommended to pay too much attention to format and process.
Test-driven development: TDD, common types include unit test-driven development (UTDD) and acceptance test-driven development (ATDD), which drives design completeness by writing tests first. Liu Ran, a colleague from Thoughtworks, has written multiple articles about TDD. Please visit his website for reference.
Code review: It can be done by team members gathering together or independently offline. Wu Bin, a colleague from Thoughtworks, has a very detailed explanation of code review in his article "Code Review: Beyond the 'Auditing, Checking, and Review' " Please refer to it for more information.
Automated Regression Testing: According to test layering theory, automated testing may include unit testing, interface integration testing, and end-to-end testing. Usually, unit tests are written by developers, while interface integration and end-to-end integration tests can be completed through collaboration between developers and testers. There are numerous articles related to automated testing, and here are some recommended for reference:
Unit Tests Review: Unit tests review generally occurs in the desk check activity, for details please refer to the article "The Value of QA Reviewing Low-Level Tests".
Continuous Integration: The concept of Continuous Integration (CI) is familiar to everyone, but it is not easy to implement it effectively. Common anti-patterns include:
- Having a CI pipeline, but nothing on the pipeline, no code scanning, no automated testing, no quality gates;
- The pipeline has integrated code scanning, but problems found are not fixed in time;
- Automated tests are not frequently executed on the pipeline or fail to be fixed in time;
- Code is not frequently submitted;
For more information, it is recommended to refer to the Thoughtworks Insights articles "New Developments in Continuous Integration Theory and Practice" and "Answers to Common Questions About Continuous Integration Practices".
Automation/Continuous Deployment: Automated deployment not only improves efficiency but also reduces problems introduced by manual deployment, making deployment happen more frequently and continuously. For more information on this topic, please refer to the Thoughtworks Insight article "Continuous Integration and Delivery Pipeline Anti-patterns".
Technical Debt Management: Do you remember the picture with messy and tangled wires earlier? If technical debt introduced in the software development process is not effectively managed, the software internals will also become a mess. For technical debt management, we recommend the article "The Past and Present of Technical Debt in Software Architecture Governance" by our colleague Ma Guangguang.
Exploratory Testing: Exploratory testing is a highly recommended practice for discovering potential and less visible defects. For specific methods and suggestions, please refer to the "Basics" where exploratory testing is introduced in detail.
Showcase: Showcase is a demonstration of the features developed for customers and business stakeholders. Showcase has many common misconceptions, please refer to the article "Seven Sins of Agile Practice Showcase".
Defect Analysis: The main benefit of defect analysis for quality built-in is to find out the root cause of the problem through analysis and prevent more defects from being exposed in the subsequent development work. How to do good defect management and analysis? Please refer to the following articles:
Quality Status Report and Quality Risk Analysis: Please refer to the "Fifth Basic Responsibility: Quality Feedback and Risk Identification" section in the "Basics".
Retrospective Meeting: The retrospective meeting summarizes and analyzes the events that the team has experienced over a period of time, which is about the overall health status of the team. I listed retrospective meetings as a typical practice of quality build-in because the factors that affect quality involve many aspects and are naturally closely related to the health of the team. Making good use of this practice, discovering problems in time, and taking corresponding improvement measures will be very valuable. For an efficient retrospective meeting, please refer to the Thoughtworks Insight article "Seven-Step Agenda for Effective Retrospective".
3.2.3 Online Usage
When a software product is released online, quality-related practices are all related to testing shift right. This is a very large topic, and there have been a series of articles introducing relevant practices before. The following articles can be referred to:
- "Testing Shift Right - QA in Production"
- "Testing Shift Right - Log Collection and Monitoring"
- "Testing Shift Right: QA and Ops Working Together to Build Anti-Fragile Software Systems"
Testing is a fundamental skill that all testers must possess, but the systematic thinking for testing should not be limited to testing work itself. When we look at quality from a broader perspective, we see a different scene.
We hope that this "Intermediate" can help testers broaden their perspective from the quality perspective, increase their overall understanding, and make quality assurance work smoother.