Automated Testing Caveats

Automated security testing technologies, such as those, which rely on scanning, fuzzing and sending arbitrary malicious data to detect security defects, can seriously damage the web applications they are used against. Therefore, it is often recommended to perform automated tests only against systems in demo, testing or pre-production environments.

Test Implications

To understand the implications when using automated security testing tools, consider the following scenario. If you target a web application, which performs many database operations, such as updating or inserting new records, an automated security tool may not only lead to unnecessary load resulting in a form of a Denial of Service attack on the production environment, but it may also cause unnecessary bogus records to be inserted into the database where other production data sits. In the case of the former it will be required an extensive database clean up, which may not always be a trivial task and may be associated with a huge financial cost and other implications.

Some of the side effects that may occur when using automated security testing tools are summarized in the list below:

  • Denial of Service (DoS).
  • Record of bogus information in databases and log files.
  • Trigger of unnecessary or false security alarms.
  • System crash, malfunction and general instability.

A combination of automated and manual testing is often the only way to minimize any damage during testing and maximize vulnerability detection coverage. Automated tools can be used pre-production or during the development. Continuous Delivery can also be used to enforce security quality controls on the application before deployment to avoid later stage testing.

The key is to test early and test often. Providing your development or quality assurance teams with tools to do their own security testing is preferable and generally leads to more education about web security and more quality code avoiding all the costs and risks associated with later stage testing when the application is already deployed in production.