Skip to main content

Regression Testing

What is Regression Testing?​

Regression testing is the process of re-running existing test suites after code changes to verify that previously working functionality has not been broken — intentionally or accidentally.

Every code change — no matter how small — is a potential regression risk. A 1-line config change once took down a production payment service for 45 minutes.

Why Regression Testing Matters​

Scenario Without Regression TestingConsequence
Developer fixes bug A, inadvertently breaks feature BFeature B fails silently in production
Dependency upgrade changes serialisation behaviourAPI responses differ; consumers break
Database migration drops a column still used in codeNullPointerException in production
New service added to the call chain changes latencySLAs breached without warning

Types of Regression Testing​

Full Regression​

Run the complete test suite across all modules. Done before every major release.

  • Slowest, most comprehensive
  • Catches the broadest range of issues
  • May take 30–90 minutes in CI

Targeted/Partial Regression​

Run only tests related to the changed area. Used for patch releases and hotfixes.

  • Faster feedback loop
  • Higher risk of missing indirect regressions
  • Use test impact analysis tools to determine scope

The regression suite is automated and runs in CI/CD on every merge to develop or release/**:

Developer merges PR
        ↓
CI triggers regression suite
        ↓
Unit tests → Integration tests → API tests → Smoke E2E
        ↓
Results reported to PR / Slack / Jira
        ↓
Gate: All passed? → Allow merge to release branch
      Any failed? → Block, notify, assign

Regression Suite Design​

What to Include​

  • All existing unit tests
  • All integration tests
  • API contract tests (Spring Cloud Contract / Pact)
  • Critical user journey smoke tests
  • Previously failing tests that were fixed (prevent re-occurrence)

What NOT to Include​

  • Exploratory tests (manual only)
  • Performance tests (separate pipeline)
  • Tests known to be flaky — fix or quarantine them

Regression in the Java/Spring Ecosystem​

Spring Cloud Contract — Contract-Based Regression​

Define producer-side contracts that are published as stubs for consumers:

// contracts/TransactionController/shouldReturnTransactionsForUser.groovy
Contract.make {
    description "GET /api/v1/transactions returns 200 with page of transactions"
    request {
        method GET()
        url('/api/v1/transactions') {
            queryParameters {
                parameter 'fromDate': '2024-01-01'
                parameter 'toDate': '2024-01-31'
            }
        }
        headers {
            authorization(matching('Bearer .+'))
        }
    }
    response {
        status 200
        headers {
            contentType(applicationJson())
        }
        body([
            totalElements: $(consumer(anyNumber()), producer(1)),
            content: [[
                id: $(consumer(anyUuid()), producer('3fa85f64-5717-4562-b3fc-2c963f66afa6')),
                amount: 99.99
            ]]
        ])
    }
}

These contracts auto-generate regression tests on both producer and consumer sides.

Regression Test Management​

Test Tagging Strategy​

@Tag("regression")
@Tag("transactions")
@SpringBootTest
class TransactionRegressionTest {
    // ...
}

Run tagged regression tests in CI:

mvn test -Dgroups="regression"

Handling Flaky Tests​

Flaky tests destroy trust in the regression suite. Treat them as P2 defects:

  1. Identify flaky test (fails intermittently)
  2. Quarantine: add @Disabled("JIRA-456: flaky — intermittent race condition")
  3. Create a Jira ticket to fix it
  4. Fix root cause (often timing issues, test pollution, or real race conditions)
  5. Re-enable and monitor

Metrics to Track​

MetricTarget
Regression pass rate≥ 99% on every release
Regression execution time< 30 minutes for full suite
Flaky test rate< 1% of total test count
Defect escape rate< 2% of defects reach production

Exit Criteria for Regression​

  • Full regression suite executed on the release candidate build
  • 100% of tests passed (or failures are investigated and accepted)
  • No new flaky tests introduced
  • Results documented in Test Summary Report
  • Tech Lead and QA Lead sign-off
Regression ≠ New Feature Testing

Regression testing validates that existing functionality still works. New features require their own dedicated test cases before being added to the regression suite.