Testing error messages is one of the most intensive but fun work to be done by a tester. A lot of bugs can be found in this area. I can say that. I’m a test veteran with almost 20 years of experience.
An example I experienced
In a SOAP request message I needed to enter numerical value for a variable which is an identifier for a subscriber and to be used in the database. I put in some number values, no problem. Now to work on some input to see the error messages. Below some input values and the response I got from the application
Input with characters
Input : ‘dummy123’
Response : Unmarshalling Error: Not a number: dummy123
This looks like a problem in the code. It is used in the response message. This is not the correct way to deal with this.
Input with ‘low’ boundary values
Input : ‘0’ or ‘-1’
Response : errorCode=”1″ errorDescription=”[id has empty mandatory elements]”
Difficult to understand. “id” is not found in the table I’m querying?
Input with ‘high’ boundary values (MAXINT for a 32 bits machine)
Input : ‘2147483648’
Response : Unmarshalling Error: cvc-maxInclusive-valid: Value ‘2147483648’ is not facet-valid with respect to maxInclusive ‘2147483647’ for type ‘int’.
This is useful information form me as a tester. For the end-user perhaps too, but he must have a technical background.
Input with “space” character
Input : ‘ ‘ (i.e. a space character)
Response : soap:ClientUnmarshalling Error: cvc-datatype-valid.1.2.1: ” is not a valid value for ‘integer’.
Make sense, but it is captured as a software error
Input : ‘ 1230656’
Response : errorCode=”101″ errorDescription=”No records found.”
Huh, a space was not a value for an integer and here it is neglected
I carried on with some other input values like hexadecimal (0x1230656), round (1230.656), special characters (12306*6). Response was same as for “input with character” (Unmarshalling Error: Not a number: #####)
These are just examples from a simple input id value
I discussed this with the developer and propose him the following; All integer input between 0 and 2147483647 are valid. These values are allowed to query the database. All other input is not allowed. The software must take care of that. The fix I got for this piece of software is working properly and all wrong input will be prompted with the following response
errorCode=”1″ errorDescription=”” errorDate=“#####”
This is a correct error response which is useful for the end user.
What about this one:
I wanted to enter a value, already in the database. The application came back with the following response (It is a bit altered to hide some vulnerable information)
errorCode=”500″ errorDescription=”SQL exception: PreparedStatementCallback; SQL [INSERT into table(##, ##, ##) VALUES(?,?,?)]; Duplicate entry ‘9140179104#####-2′ for key ‘PRIMARY’; nested exception is ######: Duplicate entry ‘9140179104#####-2’ for key ‘PRIMARY'” errorDate=“#####”
A SQL statement with table details in the error response. A big security risk. A simple error response is proposed and implemented
errorCode=”500″ errorDescription=[id is already added to the subscriber profile]” errorDate=“####”
Looking at all these examples, one must note that an implicit requirement is here in place:
An error message must contain useful information for the end user. The error message must be self-explanatory. The user must be able to understand the message so he can act upon it.
Still I see error message responses which are …….