Functionality assurance

From a technology risk point of view, there are a number of long-tail risks (difficult to envisage) that might result in unacceptable application functionality status.

Anomalous application states include:

OS (Operating System) not functional and application 100% disabled
OS partially functional and application partially disabled
Application 100% disabled through internal fault
Application partially disabled through internal fault
OS or application vulnerability introduced

States that can produce reduced functionality

OS security update
Internal application error condition (bug) brought about by either
- Normal, but unlikely operating parameters
- Changes in the external application context (soup that the application is operating in)
Application updates

Assumptions regarding Funcionality Assurance

It is NOT acceptable to detect reduced functionality through user interaction (somebody reports an error)
It is cost beneficial both from a functionality and a risk management point of view to assure that the applications within scope operates at 100% functionality.

How to do functionality assurance

Functionality assurance is NOT automated vulnerability scanning as it cannot detect introduced or undetected vulnerabilities.
A two level approach is to be taken and a number of regression tests are to be developed by different teams:
- From an OS point of view, tests to verify required functionality (OS build team).
- From an application point of view, test to verify the application functionality (Application developers).
The regression tests should be layered and should focus on providing a system "green light" if all required functionality is present or if not, identify the subsystem that failed the tests.
Trouble shooting should be a separate programme (too long a piece of string to be contained in a programme like this and very dependent on maturity of software engineering team).
Software programmers should provide "call-back" functionality so that system monitors can verify the application functionality.
The operations management team develop regression tests to verify the status of the OS
The operations management team schedule the automated running of these regression tests to verify that both the application and the OS is still providing the required functionality after security updates, patch updates etc.