To see revisions of this document or browse other course outlines, please Log In

Software Testing, Quality Assurance, and Maintenance Winter 2023
ECE 453 / CS 647 / CS 447

Published Apr 28, 2023

Class Schedule

Please log in to view this content.

Instructor & TA (Teaching Assistant) Information

Please log in to view this content.

Course Description

CS 447 / ECE 453:

Introduces students to systematic testing of software systems. Software verification, reviews, metrics, quality assurance, and prediction of software reliability and availability. Related management issues. [Note: Lab is not scheduled and students are expected to find time in open hours to complete their work. Offered: W]

ECE 453: Prereq: ECE 250; Level at least 3A Computer Engineering or Electrical Engineering. Antireq: SE 465

CS 447: Prereq: CS 350; Computer Science students only. Antireq: SE 465

CS 647:

Introduces students to systematic testing of software systems. Software verification, reviews, metrics, quality assurance, and prediction of software reliability and availability. Related management issues.

This course will provide an introduction to software testing and quality assurance techniques. The students will learn a wide spectrum of techniques and tools that can be used to improve and evaluate software quality ranging from mature testing methodologies to cutting edge automated verification algorithms. Topics to be covered include: coverage criteria (statement and branch), symbolic execution (static, dynamic, concolic), constraint solving (SMT), inductive invariants, automatic deductive verification, automatic invariant synthesis, and Software Model Checking. Formal logic is key to understanding the second half of the content of this course and, although the course is nominally self-contained, you should be familiar with symbolic reasoning in general.

Learning Outcomes

By the end of this course students should be able to:

Tentative Course Schedule

You can find additional information about the course at the course webpage, https://patricklam.ca/stqam-1231, and about the project at https://patricklam.ca/stqam-1231-projects/. You can find lecture material at the Waterloo Gitlab repo: https://git.uwaterloo.ca/stqam-1231/pdfs

Wk

Lecture

Reading

1

Admin 

Introduction

Fault, Error, Failure

A Math Primer by Andrzej Wasowski
2

Foundations: Syntax and Semantics

Structural Coverage

 

H. Nielson and F. Nielson. Semantics with Applications: An Appetizer. (Chapter 1) (Chapter 2)

(Optional) C. Le Goues. The While and While3Addr Language

Staats et al. On the Danger of Coverage Directed Test Case Generation

3

AddressSanitizer

ATG with Fuzzing

K. Serebryany et al. AddressSanitizer: A Fast Address Sanity Checker

AddressSanitizer Algorithm 

AFL and libFuzzer

4

Symbolic Execution

 

J.C. King. Symbolic Execution and Program Testing

R. Baldoni et al. A Survey of Symbolic Execution Techniques

4Z3 SMT Solver

Z3Py Tutorial (Jupyter Notebook)

D.R. Cok. The SMT-LIBv2 Language and Tools: A Tutorial

Z3 API Documentation

z3.py on github

Z3 Python Examples

N. Bjorner et al. Programming Z3

D. Yurichev SAT/SMT by example

5Dynamic Symbolic Execution

Cadar et al. Symbolic Execution for Software Testing in Practice

T. Ball and J. Daniel. Deconstructing Dynamic Symbolic Execution

R. Baldoni et al. A Survey of Symbolic Execution Techniques

5Semantics of Symbolic ExecutionA. Gurfinkel. Symbolic Execution Semantics for WHILE Language
6

Quiz 1 (Wednesday)

Propositional Logic

U. Schoning. Logic for Computer Scientists (Chapter 1)

A. Bradley and Z. Manna. The Calculus of Computation (Chapter 1)

-Reading Week 
7First Order Logic

U. Schoning. Logic for Computer Scientists (Chapter 2)

A. Bradley and Z. Manna. The Calculus of Computation (Chapter 2)

L. O'Connor. Holbert: Reading, Writing and Proving in the Browser.

8Axiomatic Semantics

H. Nielson and F. Nielson. Semantics with Applications: An Appetizer. (Chapter 9)

C.A.R. Hoare. Proof of a program: FIND

9Quiz 2 (Wednesday) 
10

Deductive Verification with Dafny

Dafny: Tutorial

Dafny: VCGen

Dafny Tutorial (uwaterloo mirror) (in class examples)

J. Koening and K.R.M. Leino. Getting Started with Dafny: A Guide

K.R.M. Leino. Developing Verified Programs with Dafny

L. Herbert et al. Using Dafny, an Automatic Program Verifier

11

Verification Condition Generation

Dafny: Theorems

Dafny: Arrays

 
12Dafny: Sorting 

 

Texts / Materials

Title / Name Notes / Comments Required
No required textbook. Lecture slides, lecture notes, and reading material will be provided. No

Student Assessment

ECE 453/CS 447 Grading Scheme
Component Value
Assignments 35%
Quiz 1 (Wednesday, February 15, in class) 10%
Quiz 2 (Wednesday, March 15, in class) 10%
Final Exam 45%
CS 647 Grading Scheme
Component Value
Assignments 35%
Quiz 1 (Wednesday, February 15, in class) 10%
Quiz 2 (Wednesday, March 15, in class) 10%
Final Exam 25%
Project 20%

All assignments, quizzes, and tests that are submitted online will be returned online. Submissions will be by pushing to gitlab.

I anticipate that the final exam will be in-person, and that quizzes will be in-class in-person, though there are no guarantees. If we go to online instruction again, then quizzes will be take home, open book, closed internet. Each online test will be time limited and must be completed within 3 days. That is, you can pick the best time to take the test, but once you start, you have to finish within the given time limit.

Assignment Screening

We will use MOSS to detect potential plagiarism in your submissions. We are not going to use turnitin.

Administrative Policy

**Territorial Acknowledgement**. The University of Waterloo acknowledges that much of our work takes place on the traditional territory of the Neutral, Anishinaabeg and Haudenosaunee peoples. Our main campus is situated on the Haldimand Tract, the land promised to the Six Nations that includes six miles on each side of the Grand River.

## COVID-19 Contingency Policy

At this stage, we can hope that Winter 2023 will proceed in person without COVID-related disruptions. In the case of a disruption to in-person classes, I will record lectures and post them on YouTube. In the case of a disruption to in-person final exams, the final exam will be take-home as well.

All other course work is already being conducted in a way that is compatible with online interaction and no change should be necessary. However, I will include this blurb just in case. I don't expect to have to use it. "To provide contingency for unforeseen circumstances, the instructor reserves the right to modify course topics and/or assessments and/or weight and/or deadlines with due and fair notice to students. In the event of such challenges, the instructor will work with the Department/Faculty to find reasonable and fair solutions that respect rights and workloads of students, staff, and faculty."

If you get COVID, you can self-declare an absence (undergrads: https://uwaterloo.ca/quest/help/students/how-do-i/self-declare-absence-undergraduate-students; grads: https://uwaterloo.ca/quest/help/students/how-do-i/self-declare-absence-graduate-students) and we'll shuffle things around to make things work.

If I get COVID, I will cancel lectures until I get better, and (after I get better), I will record lectures covering the relevant period. (I think we all hope I don't get COVID.)

As much as we'd like it to be, the pandemic is still not over, and I urge you to think about the risks that you choose to take. You are probably not going to die of COVID, but you are definitely better off not getting it (or getting it again); every time you get it you run the risk of getting possible long-term aftereffects. An estimate by Dr. Katelyn Jetelina is that you have about a 1-in-180 chance of being unable to work following COVID. That is basically almost one person per Waterloo Engineering undergraduate class. Regardless of the eventual mask policy for the term, I strongly recommend the use of masks in class.

## Course Policies

By registering for this class, students agree to the following class policies:

**Independent work**. All work turned in will be that of the individual student unless stated otherwise. Violations would result in zero credit to all students concerned. [Policy 71](https://uwaterloo.ca/secretariat/policies-procedures-guidelines/policy-71) will be followed for any discovered cases of plagiarism.

**Lateness**. You have 2 days of lateness to use on assignment submissions throughout the term. Each day you hand in an assignment late consumes one of the days of lateness. If you consume all of your late days, assignments that are still late will get 0 marks. You can only hand in an assignment up to the time all assignments are returned. Missed assignments get 0 marks. For example, you may hand in A1 two days late and A2 on time, or you can hand in A1 one day late and A2 one day late.

**Missed Quizzes**. If you miss a quiz, you will receive 0 marks for the quiz. If you have a legitimate reason (at the discretion of the instructor) that you cannot take the quiz, and obtain permission from the instructor **a week** in advance, the percentage for the quiz may be shifted to the final. No alternative quiz time will be provided.

University Policy

Academic integrity: In order to maintain a culture of academic integrity, members of the University of Waterloo community are expected to promote honesty, trust, fairness, respect and responsibility. [Check the Office of Academic Integrity for more information.]

Grievance: A student who believes that a decision affecting some aspect of their university life has been unfair or unreasonable may have grounds for initiating a grievance. Read Policy 70, Student Petitions and Grievances, Section 4. When in doubt, please be certain to contact the department’s administrative assistant who will provide further assistance.

Discipline: A student is expected to know what constitutes academic integrity to avoid committing an academic offence, and to take responsibility for their actions. [Check the Office of Academic Integrity for more information.] A student who is unsure whether an action constitutes an offence, or who needs help in learning how to avoid offences (e.g., plagiarism, cheating) or about “rules” for group work/collaboration should seek guidance from the course instructor, academic advisor, or the undergraduate associate dean. For information on categories of offences and types of penalties, students should refer to Policy 71, Student Discipline. For typical penalties, check Guidelines for the Assessment of Penalties.

Appeals: A decision made or penalty imposed under Policy 70, Student Petitions and Grievances (other than a petition) or Policy 71, Student Discipline may be appealed if there is a ground. A student who believes they have a ground for an appeal should refer to Policy 72, Student Appeals.

Note for students with disabilities: AccessAbility Services, located in Needles Hall, Room 1401, collaborates with all academic departments to arrange appropriate accommodations for students with disabilities without compromising the academic integrity of the curriculum. If you require academic accommodations to lessen the impact of your disability, please register with AccessAbility Services at the beginning of each academic term.

Turnitin.com: Text matching software (Turnitin®) may be used to screen assignments in this course. Turnitin® is used to verify that all materials and sources in assignments are documented. Students' submissions are stored on a U.S. server, therefore students must be given an alternative (e.g., scaffolded assignment or annotated bibliography), if they are concerned about their privacy and/or security. Students will be given due notice, in the first week of the term and/or at the time assignment details are provided, about arrangements and alternatives for the use of Turnitin in this course.

It is the responsibility of the student to notify the instructor if they, in the first week of term or at the time assignment details are provided, wish to submit alternate assignment.