Software testing overview

— with mparker17

Follow along at mparker17.github.io/talk–software-testing-overview

Who am I?

mparker17 on Drupal.org, Github, and Gitlab

I work for Digital Echidna, Brady’s Meat & Deli

Digital Echidna’s logo Brady’s Meat & Deli’s logo

(previously Environment Canada, Versabanq, UWaterloo, PeaceWorks, Myplanet, OpenConcept)

Who am I? (ctd.)

I am an Acquia Certified Drupal Developer

CS degree not completed at UWaterloo

Why test?

Tests tell us that…

Dictionary definitions

Testing (verb)

To evaluate1 [a system for correctness]

A test (noun)

A procedure leading to acceptance or rejection1 [of a system’s correctness]

But keep in mind…

“Testing only shows the presence, not the absence, of bugs” — Edsger W. Dykstra2

Approaching tests

  1. Manual tests (get a person to test)
  2. Automated tests (get the computer to test itself)

We want to automate as many tests as possible to save time and money, but it’s not possible / practical to automate everything.

Classifying tests

Classifying tests (ctd.)

  1. Exploratory tests
  2. Acceptance tests (broken down into System, Integration, and Component tests)
  3. Unit tests

Classifying tests (ctd.)

Ignoring exploratory tests for a moment…

Unit tests Acceptance/System tests
have Small scope have Large scope
are Fast to run are Slow to run
are Cheap to write are Expensive to write
are Numerous are Sparse in number

Unit tests

What? highly-isolated, low-level, clear-box tests for code; covering code paths and edge cases for small units and small group interactions

Who? by developer for developer; in language of SUT

Synonyms for individual units — (couldn’t find any) Synonyms for unit groups — interaction, integration; integration & testing (I&T)

Part of/covers functional, regression testing

Unit tests (ctd.)

Key tech

Acceptance tests

What? high-level, black-box tests to communicate; clarify; and understand when a requirement is done.4

Who business analyists write the “happy path”, QA analysts write the “unhappy path”, developers provide input into both4

Synonyms (term overloaded/overused)

Exploratory tests

What? high-level, black-box tests to find unexpected behaviors, confirm expected behaviors, and the system behaves well under human operation

Who? business, designers, developers, marketing, QA, etc.

Synonyms field, refinement tests; bug-hunting

Part of/covers A/B, end-user, penetration (pentesting), security, usability, user experience (UX) testing

Acceptance/Component

What? isolated, mid-level, grey-box tests for behavior; covering the happy-path for features and business rules (as well as obvious corner/alternate paths)

Who? by QA & business, for business; often in a behavioral DSL

Synonyms behavior/behavioral, confidence, validation; verification & validation (V&V)

Part of/covers functional, regression

Acceptance/Component (ctd.)

Key tech

Acceptance/Integration

What? mid-level, black-box tests for groups of components; covering how they communicate and work together

Only meaningful in larger systems with independent parts (e.g.: headless Drupal, federated search, endpoints beyond the web (COPE))

Acceptance/Integration (ctd.)

Who? by system architects or lead system designers, for developers; often in same language as component tests

Synonyms contract, choreography, conformance, plumbing; integration and testing (I&T)

Part of/covers performance, throughput

Acceptance/Integration (ctd.)

Key tech5

Acceptance/System

What? high-level, black-box tests; covering the whole system

Who? by system architects, lead designers, developers, QA

Synonyms end-to-end (E2E) testing

Part of/covers accessibility, functional, load, performance, regression, throughput, user acceptance (alpha-testing, beta-testing), verification, validation testing; Independent Verification & Validation (ISVV)

Acceptance/System (ctd.)

Key tech

Other terms

Continuous delivery (CD) — the practice of releasing people’s work several times per day

Continuous Integration (CI) — the practice of merging multiple people’s work together several times per day

Regression tests — a check that new changes haven’t broken old work

Other terms (ctd.)

Sanity tests — a check of a program’s logic/flow; or whether further testing is needed

Smoke tests — a check for obvious failure

Test coverage — a measure of how much of your application’s behavior is verified by tests

Test Driven Development (TDD) — the act of writing tests before writing code

Test fixture — something to satisfy pre-conditions for the thing you’re about to test

Test suite — some grouping of tests

Works cited

  1. Beck, Test Driven Development By Example, Addison-Wesley, 2003. p.123
  2. Martin, Clean Architecture, Prentice Hall, 2018. p.26
  3. Zaidman, An Overview of JavaScript Testing in 2019, WellDone Software / Medium, Feb. 2019.
  4. Martin, The Clean Coder, Prentice Hall, 2011. p.100,103,105-106,115-119
  5. Martin, Testing Strategies in a Microservice Architecture / MartinFowler.com, Nov. 2014

Thanks!