Error Messages – A bug hunting party

Testing error messages is one of the most intensive but fun work to be done by a tester. A lot of bugs can be found in this area. I can say that. I’m a test veteran with almost 20 years of experience.

An example I experienced

In a SOAP request message I needed to enter numerical value for a variable which is an identifier for a subscriber and to be used in the database. I put in some number values, no problem. Now to work on some input to see the error messages. Below some input values and the response I got from the application

Input with characters
Input : ‘dummy123’
Response : Unmarshalling Error: Not a number: dummy123

This looks like a problem in the code. It is used in the response message. This is not the correct way to deal with this.

Input with ‘low’ boundary values
Input : ‘0’ or ‘-1’
Response : errorCode=”1″ errorDescription=”[id has empty mandatory elements]”

Difficult to understand. “id” is not found in the table I’m querying?

Input with ‘high’ boundary values (MAXINT for a 32 bits machine)
Input : ‘2147483648’
Response : Unmarshalling Error: cvc-maxInclusive-valid: Value ‘2147483648’ is not facet-valid with respect to maxInclusive ‘2147483647’ for type ‘int’.

This is useful information form me as a tester. For the end-user perhaps too, but he must have a technical background.

Input with “space” character
Input : ‘ ‘ (i.e. a space character)
Response : soap:ClientUnmarshalling Error: cvc-datatype-valid.1.2.1: ” is not a valid value for ‘integer’.

Make sense, but it is captured as a software error

Input : ‘ 1230656’
Response : errorCode=”101″ errorDescription=”No records found.”

Huh, a space was not a value for an integer and here it is neglected

I carried on with some other input values like hexadecimal (0x1230656), round (1230.656), special characters (12306*6). Response was same as for “input with character” (Unmarshalling Error: Not a number: #####)

These are just examples from a simple input id value

I discussed this with the developer and propose him the following; All integer input between 0 and 2147483647 are valid. These values are allowed to query the database. All other input is not allowed. The software must take care of that. The fix I got for this piece of software is working properly and all wrong input will be prompted with the following response

errorCode=”1″ errorDescription=”

” errorDate=“#####”

This is a correct error response which is useful for the end user.

What about this one:
I wanted to enter a value, already in the database. The application came back with the following response (It is a bit altered to hide some vulnerable information)

errorCode=”500″ errorDescription=”SQL exception: PreparedStatementCallback; SQL [INSERT into table(##, ##, ##) VALUES(?,?,?)]; Duplicate entry ‘9140179104#####-2′ for key ‘PRIMARY’; nested exception is ######: Duplicate entry ‘9140179104#####-2’ for key ‘PRIMARY'” errorDate=“#####”

A SQL statement with table details in the error response. A big security risk. A simple error response is proposed and implemented

errorCode=”500″ errorDescription=[id is already added to the subscriber profile]” errorDate=“####”

Looking at all these examples, one must note that an implicit requirement is here in place:

An error message must contain useful information for the end user. The error message must be self-explanatory. The user must be able to understand the message so he can act upon it.

Still I see error message responses which are …….


One Comment Add yours

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s