Surveys
From EvaluationWiki
Contents |
Types of Surveys[1]
Surveys can be divided into two general categories on the basis of their extensiveness. A complete survey is called a “census.” It involves contacting the entire group you are interested in -- the total population or “universe.” The other category is more common; it is a sample survey. A sample is a representative part of a whole group (universe). Thus a sample survey involves examining only a portion of the total group in which you are interested, and from it, inferring information about the group as a whole.
Advantages and Disadvantages of the Two Types of Surveys
One of the decisions to be made in surveying is whether or not to sample. Parten[2] presents a list of advantages and disadvantages of the sample survey. (These, in turn, imply the advantages and disadvantages of the census.) The three most important considerations for the surveyor are: speed, low cost, and increased accuracy and analysis of the data.
By sampling only a small portion of a large population, it is possible
to collect data in far less time than would be required to survey the entire
group. Not only is the data collection quicker, but the data processing and
analysis also require less time because fewer pieces of data need to be
handled. Rapidly changing conditions and the short turn-around time
imposed in many surveys make the efficient use of time a critical variable.
If an accurate snapshot of the attitudes of a particular group is desired,
currency is of paramount importance. Professional political pollsters make
their living by providing quick snapshots of the “political climate.” Results
of such polls lose their accuracy very quickly (sometimes in as little as 24
hours--particularly in the days preceding a major election). So, for these
pollsters, time is truly of the essence. It's probably a safe bet that those of
you reading this guide will not need that degree of speed. Nevertheless,
speed is essential to ensure the data are "fresh," especially when it comes to
assessing public opinion in a volatile or contentious area before they
change appreciably.
The smaller amount of data gathered by sampling as opposed to
surveying an entire population can mean large cost savings. By limiting the
group to be surveyed, less time, hence less cost, are involved in collecting,
formatting, and analyzing the data. In addition, if automated data
processing (ADP) equipment is being used to analyze data, your overall
time investment will be even less, as will be the overall cost. Sampling
allows you to do a credible job for a smaller investment of time and money.
Parten[2] also notes that sampling enables the surveyor “to
give more attention to each return received and to make certain that the
data are as accurate as possible”. This attention may lead to more
precise information than would a less careful collection of data from the
entire population. Nothing more than a rudimentary quality control is
possible for the great volume of raw data gathered in a census. The more
data collected, the greater the potential for making “accounting” errors.
The disadvantages of sampling are few, but important. The main
disadvantages stem from risk, lack of representativeness, and insufficient
sample size, each of which can cause errors. Inattention to any of these
potential flaws will invalidate survey results.
It is important to realize that using a sample from a population to
infer something about the entire population involves a risk. The risk results
from dealing with partial information. If risk is not acceptable in seeking
the solution to a problem or the answer to a question, then a complete
survey or census, rather than a sample survey, must be conducted.
Determining the representativeness of the sample is the surveyor's
greatest problem when sampling. By definition, "sample" means a
representative part of an entire group. To avoid the charge of using
“biased data,” it is necessary to obtain a sample that meets the requirement
of representativeness, and this is not an easy task. Without a representative
sample, a survey will, at best, produce results that are misleading and
potentially dangerous.
The final major problem in sampling is to determine the size of the
sample. The size of the sample you need for a valid survey depends on
many variables including the risk you are willing to accept and the
characteristics of the population itself. Here, it is sufficient to say that if sampling
becomes too complicated, or the required sample size becomes too large,
the easiest solution may be to survey the entire population.
The decision as to whether to survey the entire population or only a
sample of it is not based on the above advantages and disadvantages alone.
It is affected by many other variables.
To Survey or Not to Survey
Before attempting a survey, you should investigate some basic facts and answer some pertinent questions. The result of this investigation will be a greater realization of the work involved in producing a survey. Perhaps it will lead to a decision not to survey.
Surveys demand time (maybe more time than you have available).
The exact amount of time varies greatly from survey to survey depending
on the number of people to be surveyed and the content of the survey. A
survey of a few questions administered to the people in your office may
take only a day or so, whereas a larger survey administered to a great
number of people located worldwide can take over three months from the
time the survey is delivered to the printer. And this does
not include the time needed to design the survey and construct the
questionnaire. Moreover, coordination with officials and the customers of
the survey takes additional time. If your estimate of the time needed to
produce the survey exceeds your deadline date, you are likely to decide
you do not have the time to conduct a survey. A hurried survey wastes
both your time and that of your respondents. The results of a hurried effort
are questionable at best.
Surveys are expensive to produce. The solution to the problem or
the answer to the question may not be worth the cost to produce it. Even
if it would be worth the price, you may not be able to obtain the needed
funds, either from your own pocket or from your organizational budget.
Although no standard estimates of survey cost are available, some of the
items of expense can be examined. The primary expense is in time and
effort; the time you spend producing the survey could be spent on other
tasks. If other personnel are needed, they will have to be paid. Access to
typewriters, word processors, and calculating machines (or computer
resources) is a must. If you expect to gather a great deal of data, the cost
of renting ADP time and of purchasing ADP scanner sheets should be
examined. Surveys of more than 150-200 respondents cannot feasibly be
tabulated by hand. The same is true for groups of less than 150
respondents if the survey questionnaire is lengthy. The final cost involves
supplies. At a minimum you will need paper and envelopes. You may also
have to pay either the cost of printing the survey questionnaire or the
postage or both. Each of the above costs that applies to your survey
should be estimated and the total cost measured against the survey
requirement.
Since surveys are being used more and more, the information you
want may have already been gathered. So before you undertake a survey, first make sure the answer to your
problem does not already exist. Next, evaluate the time you will need and
determine the cost involved to produce the survey results, and then weigh
these findings against the importance of the survey. Undertake a survey
only if it is worth the time, effort, and cost to make it a good one.
Developing the Purpose, Hypotheses, & Survey Plan
The first steps in producing a survey are the most important. They determine where you are going (the purpose), how you will know when your are there - or what you expect to find (the hypotheses, objectives, or research questions), and by what route you will go (the survey plan). If these steps are not well planned, all the remaining steps will be wasted effort.
The Purpose
The first step in producing a survey is to define the purpose or objective of the survey. “A clear statement of purpose is necessary not only as a justification/explanation of the project, but also as a guideline to determine if future actions in the project are in support of the original purpose”.[3] Without knowledge of the exact nature of the problem (objective), you cannot decide exactly what kind of data to collect or what to do with it once you have it. Usually a staff officer is given a problem or objective; it seldom originates with him. But this does not relieve the individual of responsibility for insuring that:
- the problem is well stated
- the surveyor understands exactly what the problem is
- the stated problem is the real problem
The survey should be designed to answer only the stated problem.
Adding additional interesting objectives will lengthen and complicate the
survey while clouding the real issue.
The Hypothesis, Objective, or Research Question
Once the problem has been clearly stated, the next step is to form one or more hypotheses. The hypothesis is actually your educated guess about the answer to the problem. It should not be a capricious guess, however. It ought to be based on your prior experience related to the problem, or perhaps any knowledge you may have of previous research done on the topic. Without such a framework in which to make an educated guess, you really have no basis for making a guess at all. If you do not have a clear basis for formulating a hypothesis, you should instead develop one or more objectives or questions to frame the scope of your questionnaire.
For example, if a problem is identified on the base as declining use
of the Officers' Club, an immediately obvious question comes to mind:
“Are the officers on this base satisfied with the Officers' Club facilities?”
This would be suitable as a research question. It is possible, though
doubtful, if you could come up with a supportable hypothesis, or educated
guess, as to the answer to the problem. You may, for instance, have
gathered some anecdotal evidence (overhearing colleagues talking) of
dissatisfaction with the club's facilities. But, this may not be sufficient for
making an educated guess that this is the real reason for the decline in club
use. The problem could be seasonal; it could be related to a decline in the
officer population on the base; or a number of other possibilities. The
point is that without some credible evidence to support a hypothesis, you
should probably not formulate one.
If you formulated a hypothesis for the current example on the
basis of the anecdotal evidence available to you, you would naturally
construct a questionnaire to survey the opinions of officers regarding their
use, or lack thereof, of the Officers' Club and the reasons for it. You might
never think to gather data from the base military personnel office to see if
the officer population is lower now than usual or if there are seasonal
(cyclic) trends in the size of the officer population on the base. In other
words, establishing the hypothesis may blind you to collecting data on
other possible causes of the problem. This is why all researchers are
cautioned not to formulate hypotheses unless they have a solid base in
theory or previously gathered evidence that suggests the hypothesis is, in
fact, probable.
Hypotheses must be carefully written. They should not contain
moral judgments or biased statements such as “All pilots are good leaders.”
There are many ideas on what constitutes a good leader and your idea may
not be the same as those of the people you will contact. Avoid words like
should, best, good, bad, and ought.
Hypotheses should be as specific as possible. Avoid words such as
most and some. If by most you mean a majority, then say majority. A
survey can more easily be designed to test whether “more than 75 percent
approve” than whether “most approve.”
A well-formulated hypothesis, objective, or research question
translates the purpose into a statement that can be investigated
scientifically. The level of difficulty you will face in producing a valid
survey will increase dramatically if they are not well formulated. Take care
in doing this step, and it will save you much effort later in the survey
development process.
The Survey Plan
The next step after determining the purpose and hypotheses is constructing the survey plan. The purpose of the survey plan is to ensure that the survey results will provide sufficient data to provide an answer (solution) to the problem you are investigating. The survey plan is comprised of three different parts:
- data collection plan
- data reduction and reformatting plan
- analysis plan
None of these plans stands on its own. Decisions you make on how
you will analyze your data will affect your data collection plan. The type of
data reduction you do will affect not only the types of analyses you can do,
but also the amount and types of data you need to collect. Because these
plans are closely interrelated, they should be developed concurrently.
The Data Collection Plan
The purpose of the data collection plan is to ensure that proper data are collected in the right amounts. The appropriateness of the data is determined by your hypothesis and your data analysis plan. For example, if you plan to analyze your results by age group to test a hypothesis, then you must collect data from each age group whose opinions you want to know. The right amount applies to sample data. The use of sample data involves risk, and the amount of that risk is determined by the size of your sample. The amount of risk you are willing or able to accept should be stated in your analysis plan. Proper and right come together when your analysis plan involves both sampling and analyzing data by groups. You not only have to collect data from some members of each group you plan to analyze, but you also have to see that each group provides a response rate that is high enough to ensure your meeting your minimum risk level.
The Data Reduction and Reformatting Plan
The purpose of the data reduction and reformatting plan is to identify up front and to decrease as much as possible the amount of data handling (reduction and reformatting) you will have to do. This plan is highly dependent on the other two plans. As previously mentioned, if your collection plan calls for a great deal of data, you should plan to use a computer to analyze the data. If ADP scanner sheets are to be used to record respondents' answers, include the sheets with the questionnaire so the respondent can fill out the scanner sheet. This will save a great deal of time that you would have to spend if you transferred the survey data to the scanner sheets yourself. It also eliminates the possibility of your making errors in transferring data. You should coordinate in advance with the ADP personnel to make sure they will be able to scan your answer sheets and, if necessary, analyze your data within your timeframe. ADP shops are busy places. The prudent surveyor will “book” the scanning and analysis jobs well in advance with ADP personnel to ensure their resources are available when needed.
A strong potential for error and tedious corrective work lies in data
reduction and reformatting. Proper care in developing this plan can save a
great deal of time later and preclude error.
Open- and Closed-Ended Questions
The use of Automatic Data Processing (ADP) necessitates the use of closed-end questions -- a type of question you should consider even if you are hand-tabulating your data. A closed-end question lists possible answers from which the respondent picks the one he/she likes best. An example is the common multiple-choice question. The open-end question is one to which the respondents write the answer out in their own words. At first glance, the open-end question seems superior since respondents supply their answers rather than ones from your list of answers. But the wide variety of answers respondents generally provide to open-end questions turns out to be a great handicap later. For every open-end question, there are virtually an infinite number of possible answers. Since you cannot analyze an infinite number of answers, you must devise some means of categorizing this diversity of answers into a smaller, more manageable group. You will find yourself spending a tremendous amount of time reading, comparing, categorizing, and recording each answer. Much of this time can be saved if you use care in developing the questionnaire and constructing your own categories in advance. Construct each question so that every possible major category of response is contained in the answer list.
Then, later, all the computer will have to do is count the number of
answers in each category. By having the survey respondent, not you,
categorize the answer, you will collect data that is more valid, reliably, and
accurate than if you did the categorizing yourself.
Analysis Plan
Finally, an analysis plan ensures that the information produced by the analysis will adequately address the originally stated hypotheses, objectives, or questions. It also ensures an analysis that is compatible with the data collected during the survey. In the analysis plan, you determine which statistics you will use and how much risk you can take in stating your conclusions. Each of these decisions will affect the amount and type of data you collect and how you will reduce it. Novice researchers often misuse statistical analyses out of ignorance of the assumptions on which the statistics are based. The most often committed error in statistical analysis by novices is using a statistical technique with inappropriate data. The results of such analyses appear to be legitimate, but are actually impossible to interpret correctly.
Concluding Thoughts
Oppenheim[4] suggests that to make sure all these parts of the survey plan are correctly interlocked, you can simply approach the natural sequence of survey operations in reverse order. First determine what conclusions you are interested in; then decide what statistics and results will be needed to draw these conclusions. From this, the type of questions needed and the nature of the sample can be determined.
A conscientious survey plan will help you produce a well designed
survey. The proper data will be processed correctly and efficiently to
produce the information required to shed light on, and hopefully provide a
solution to, the original problem.
References
- ↑ Ross, Keneth C. (1996). Air University Sampling and Surveying Handbook: Guidelines for planning, conducting, and organizing surveys.Retrieved September 9, 2006, from http://www.au.af.mil/au/awc/awcgate/edref/smpl-srv.pdf.
- ↑ 2.0 2.1 Parten, M. (1950). Surveys, Polls, and Samples: Practical Procedures. New York: Harper and Brothers, p 109-110.
- ↑ Department of the Air Force. (October 1974). A Guide for the Development of the Attitude Opinion Survey. Washington, D.C., p 2.
- ↑ Oppenheim, A. N. (1966). Questionnaire Design and Attitude Measurement. New York: Basic Books, Inc.
