Collecting Metrics Using Surveys

One way to supplement quantifiable metrics is with client satisfaction surveys. For instance, instead of trying to measure the exact response time of an application against some service-level standard, you could simply ask your main users how satisfied they were with the application response time.
In the same way, let’s say you want to gather metrics that indicate the time it takes to resolve client problems. This could involve tracking when the initial request comes in, when you first responded to the client and when the request was resolved. On the other hand, you could simply send out surveys that ask your client if they were satisfied with the time it took to resolve the problem.

Surveys are by their nature qualitative; that is, they reflect the opinion of the person being surveyed. Therefore, you would not necessarily want to base your entire project success criteria on survey metrics. Some results are more easily obtained quantitatively. For instance, there is usually no reason to send out a survey to the finance department to ask them if your spending is within budget. You should have the facts available to you. However, for many other types of metrics, a qualitative survey question can be used as a substitute for the quantitative metric.

A survey can consist of questions and space for answers from the respondent. Many surveys ask for a combination of ratings feedback and written feedback. The numerical answers are used to drive the metrics, while the written feedback provides additional perspective that can be used in the metrics analysis. There are a number of advantages to a well-worded survey.

When you are trying to determine how people interact with each other or how a process interacts with people, perhaps the best way to gather feedback is to ask the affected people themselves. Getting verbal feedback is helpful, but does not allow you to quantify the information. Surveys give you that flexibility – input from the affected people in a numeric format.

They are a relatively inexpensive way to gain feedback from many stakeholders. For instance, a vendor may try to gather feedback from hundreds or thousands of customers. A survey would be an effective way to allow this mass of people to contribute their ideas.

The ratings feedback can be interpreted mathematically for precision and ease-of-use. One great advantage of surveys is that you can gather feedback from a tremendous number of people, and yet synthesize the results using simple math.

You can get “shades of gray” from the rating feedback. A survey allows you to receive answers based on a continuum or a range of possible results.

Of course, there are also a number of weaknesses with a survey approach. These include:

Unless you really push the matter, you will not typically end up with a high percentage of surveys returned. In fact, you should be happy to receive 50% back, and return rates of 25% and lower are not uncommon. If you get too few surveys returned, you may not have the confidence to know if the results you have are representative of the entire group.

Many, perhaps most, surveys are not very good, and therefore the information returned is suspect. Problems can include:

Making it hard for people to fill in answers, perhaps by not leaving enough space

Making the rating scale confusing or inconsistent

Giving multiple choice questions where all of the choices are not represented in the answer

Asking biased questions that lead the responses in a certain direction

Asking for one rating when a statement contains two or more implicit questions (For instance, are you happy with the timeliness and format of the report?)

When you create a survey, a common question is whether survey participants should identify themselves or not. Asking for identification may make it easier to ask follow-up questions to gain better clarity on the responses. However, it may also inhibit feedback and may result in some people not responding at all.

Be Cautious When Asking “Yes” / “No” Questions

Many inexperienced developers of surveys start with the assumption that they will just ask questions that can be answered “yes” or “no.” In this way, the answers are cut and dry and easy to interpret. If you ask ten people a yes/no question, you can easily tabulate the results to determine the overall preferences.

The problem with the simple “yes” or “no” answer is that it is black or white (all or nothing), and does not leave any room for shades of gray. It is often the case that the respondent has a more complex view. For instance, he may be happy about an item sometimes and unhappy at other times. The respondent may not be willing to be slotted into a simple “yes” or “no” answer.

The better method is to still ask questions that allow answers to be expressed in a range. For example, you can ask the sponsor “How satisfied are you with the overall success of the project?” and allow him or her to express his answer on a one through five scale (or one through ten). Now the sponsor has some discretion. If he is totally satisfied, he can score the project a five out of five. If the sponsor was happy about most things, but unhappy about some, he can rate the team a four out of five, or perhaps a seven out of ten. Again, these scores are rolled up and averaged to determine the overall success level.

Send Out Survey Requests Early and Often so That You Have Time for Improvement

A client survey can be sent out at the end of every project – small, medium or large. This survey will provide the final feedback from the client on how well the deliverables meet their needs and how well the project was executed. However, the team does not have a chance to improve if the only survey is gathered at the end of the project. For larger projects, the team should survey on a periodic basis. For instance, this could be halfway through the project as well as the end. If your project is large enough, you should survey at the completion of the major project phases or milestones. If you survey too much, you will risk having the surveys ignored since not much has changed from the last survey. However, if you do not survey enough, it is hard to understand your current performance level (from the client’s perspective) and to make improvements based on their feedback.