2024 StarEast Conference Notes

Quality Engineering

Too often Quality Assurance is a task done at the end of the software development process. With manual testing it makes sense to do quality assurance at the end, how can one test something that isnt ready? However, Quality Assurance now contains automation of that testing. When testing is automated it can be placed at any step in the development pipeline. Testing can be shifted left and placed further up the pipeline. Automated tests can now be run before a ticket is even looked at by QA. Shifting qualty assurance processes left allows quality assurance to be a part of other development processes, and start to change from assuring quality of our product, to designing quality into our product.

Cargo Cult Agile

Cargo Cult Agile occurs when a team adopts Agile practices without understanding of the point of the processes. You can identify Cargo Cult Agile if the following occurs,

  • Metrics Over Meaning: Metrics become the measure of process.
  • Tool Overload: A reliance on tools and methodologies.
  • Resistance To Change: Lack of experimentation and recognizing failures.

If these properties are present on your team then you are likely experiencing stagnation and frustration around your product. Those issues lead to quality issues.

Metrics

Two important concepts effect how teams decide on metrics.

  • Goodhart's Law: "When a measure becomes a target, it ceases to be a good measure"
  • Survivorship Bias: Decisions based on successful outcomes.

Individual goals will shift to align with the teams chosen metrics. If a team chooses velocity as a measure of success, then completing an amount of points will be more important than shipping complete and functional features. Goals that are commonly selected are quantitative in nature, something thats easy to assign a number to and track. Goals that are only quantative is a quagmire, a mirage that lures teams into a false sense of security where they think they understand the health of their software. Think about what a quantitative goal like velocity or number of defects means. It means

Enterprise Test Data Management

At organizations with a large amount of interconnected services it becomes difficult to properly manage test data. This is due to five issues:

  • Large and Distributed Services
  • Polyglot
  • Various stages of deprication
  • Outdated documentation
  • Continuously changing interfaces

Performance Testing

Insuring that your application has complete and functional features is the important, but if those functional features are functionaly useless due to performance issues then you'll have just as big an issue on your hands. Performance testing can be difficult to quantify as you are essentially attemopting to validate a subjective experience. What is fast enough? What is usable? Time to load the first page, workflow response time, and time to first action are great starting metrics that work for all web applications.

Remember that performance testing of APIs is different than complete user workflows. Design your tests in a flexible way, because the workflows you target will move overtime as services change. Think of ways your data can be mocked or generated. API testing required knowledge of how the APIs work and how they differ, dont start creating tests until you have a decent understanding of them. Shift testing left, because Integration testing is hard to test because the integration is always moving.

Setting Up A QA Team

Setting up a QA team from scratch can feel daunting, but heres some key objectives that will help.

Quick Start Guide

Focus on what new team members need so they can be productive day one. Document the steps to get all applications setup, who to contact for various access, and where other documentation is located.

Test Case Software

Its necessary to track test cases for the application. Look for something that can handle requiresments and test cases. Excel and word documents are not adequate, it becomes very painful to use when theres more than a handful of test cases.

An MVP Test Set

A set of tests that cover at minimum the application's key features. Start shallow to gain broad coverage, then develop tests that explore more of the applications features. This way you develop a set of smoke and key regression tests that are immediatley impactful for the QA team.

Debugging Wiki

A collection of documentation that shares solutions for common issues.

Risk Management

A system for identifying areas in the application that are crucial to the application functioning. This is important because it guides QA to focus on certain areas more than others.

AI

Every facet of the tech industry is trying to see where AI can benefit their development workflows. AI products and integrations for QA are coming, but in the meantime QA is being told to wait and prepare for the incoming AI storm somehow. AI solutions are split into two categories, anaylsis and generation. Here is a list of AI products that are being shown:

  • Test Creators
  • Vision AI
  • Deep healing
  • Risk Analysis
  • Mobile AI
  • App Exploration
  • Security Testing
  • Code Generation

AI solutions that attemp to simplify, or replace, your tasks come with major caveots. As an example, if your application requires Deep Healing capabilities then something has gone wrong in your development process, and simple design choices or proceedures can address the issue rather than seek to mend the symptoms.
Better AI systems are ones that can help shift left the development process. Systems that can perform risk analysis, security testing, and mobile simulation of your application are useful.
I want to specifically call out an exciting idea that was shown which is Predictive Quality Models. Prodictive Quality Models seek to warn developers of areas in an application that commonly produce bugs when code is changed. This has the potential to allow teams to understand and better scope work when diving into areas that are error prone.

Ending

StarEast 2024 was a mixed bag of classes that had messages and classes that were effectively trojan horses for a company to show a product. Most of the lessons and info presented here was gathered from the helpful classes. Overall it was beneficial to attend the Virtual conference. I think the idea of shifting QA left and trying to utilize the tools we have to catch QA related issues before they get to development or testing is key to creating quality software.