Latest Articles

More Web 2.0 Needed In Schools

An influential think-tank calling for more Web 2.0 use in school and technology experts agree, arguing that children should get used to collaborative tools before they enter the workplace.

more

UK Council for Child Internet Safety Launches

The UK Government launched its programme to help protect children from exposure to potentially harmful content on the Internet, including some forms of advertising. New Media Knowledge spoke to AOL, one of the companies involved, to see what real impact the new group would have.

more

US Presidential Election Gets Social

Last week, Twitter launched its US Presidential Election microblogging site and, with social media likely to play a big part in the outcome, politicians this side of the pond should be looking closely at its impact, experts say.

more

• i-design 08 - Brendan Dawes, My Interactive
• i-design 08 - Simon Waterfall
• YouTube gets into Top Gear
• Goldfarming for Fun and Profit
• Sky Appoints its First Head of Mobile Advertising: an Industry Viewpoint
• Industry Welcomes Intel/Yahoo! Internet TV Tie-In
• Media Landscape vs. Media Ghetto

Related Articles

• Marketers Cautious Over Social Media Spend

  Social media marketing has been high on the agenda for many communications experts. Most marketers can see how the Internet has revolutionised the way the general public communicates, but many are still unsure whether they can leverage this technology to push brand messages. more

• Mobile Surf's Up

  The mobile Web has long been thought of as the next step for the Internet. Location-based services, micro communications and instant picture sharing could potentially revolutionise the way the public interacts with the Web and the way they communicate with each other. more

• The Digital Savvy Customer

  A report by consumer and media research company, Scarborough Research has revealed that nearly 6 per cent of the US population are leading edge consumers. more

• Mobile Shopping Finally Here?

  According to figures from Juniper Research's Mobile Web 2.0: Leveraging Location, IM, Social Web & Search report, mobile 2.0 applications currently generate $5.5 billion. With the sum expected to rise to $22.4 billion by 2013, businesses are unsurprisingly looking to target this lucrative market. more

• Mobile Forecast

  There are nearly four billion mobile phone handsets in use worldwide, roughly four times more than the number of computers. As handsets become more advanced, it seems an obvious route for operators to look at revenue streams other than voicecalls. more

Related Events

Children Online (NMK)

When: April 17th, 2007
Location: Room C1:04, 1st Floor, University of Westminster, 115 New Cavendish Street, London, W1W6UW
Price: £85.00 Reduced to £55.00 if you are eligible for a discount.

more



---

Online Traffic Optimisation Conference

When: February 21st, 2007
Location: Central London
Price: Reduced to £0.00 if you are eligible for a discount.

more



---

Online Marketing Show

When: June 7th, 2006
Location: Royal Horticultural Halls, Westminster
Price: Reduced to £0.00 if you are eligible for a discount.

more



---

Online PR (London)

When: March 21st, 2006
Location: 76 Portland Place, London W1B 1NT.
Price: £395.00 Reduced to £345.00 if you are eligible for a discount.

more



---

Online PR (Manchester)

When: March 15th, 2006
Location: 117-119 Portland Street, Manchester
Price: £395.00 Reduced to £345.00 if you are eligible for a discount.

more



---

Conducting Online Research

An introduction to the principles and practice of designing, delivering and analysing online surveys.

At an NMK seminar in February 2004, presented in association with Radiant (www.radiant-digital.com), Suzi Bentley of Radiant, Rich Alexandre of SurveyShack and Richard Naylor of BOP presented an introduction to the principles and practice of designing, delivering and analysing online surveys.

Suzi Bentley, Radiant

Suzi Bentley of Radiant got the evening off to a great start with a comprehensive overview of how to run a successful online market research project, which provided an excellent summary of the issues.

Not so long ago, Suzi explained, online surveys and research were viewed with suspicion by the 'serious' market research community. The online population was not considered sufficiently representative of the population at large, so unless you were trying to survey the attitudes of young, male IT developers and computer game fanatics, an online survey wasn't likely to tell you very much about your overall audience or customer base.

As internet penetration has increased, online research has become increasingly prominent, and can now hold its own alongside other methods such as mail surveys or face-to-face and telephone interviews. As Suzi points out, all research methods are subject to inherent flaws and biases, advantages and disadvantages, and it is the researcher's job both to be aware of these, and to employ strategies to minimise or overcome potential distortions.

So, when should you consider using online research, and when are other methods more appropriate?

What is online research good for?

• Measuring perception of different products or brands
• Measuring satisfaction
• Evaluating websites
• Measuring awareness
• Obtaining feedback about future plans
• Gathering thoughts on sensitive data
• Gaining business-to-business feedback

When should you stick to traditional methods?

• When non-written feedback is required
• When conducting large-scale national studies
• For tracking market share or revenue data
• When trying to reach older audiences (despite what you hear about 'silver surfers')
• If audio or video stimuli are present

The advantages of online research are that it is cost effective, great for quantitative research, quick to deploy and can provide a good breadth of responses. It is not so good if you need to collect in-depth responses, or conduct qualitative research.

Qual or Quant?

Quantitative research is concerned with measurement of numbers, such as the number of people who would buy a particular product, the percentage of people who agree with a certain statement, or the satisfaction levels of your customers. Online research is well-suited to this kind of work.

Qualitative research is softer, and explores why people act or think the way they do, and is most effective when 'open ended', as in focus groups or in-depth interviews.

Many research projects start with quant, and then dig a little deeper by deploying a bit of qual.

How to recruit your sample

Once you decide to embark on your online survey, you need to think about who you would like to complete the questionnaire, and how you can persuade them to do so. If you have a website, surveying visitors to the site or your database of subscribers is straightforward, and effective if you only want to find out about your current user base.

If you want to cast your net wider, buying email lists can be effective, but only if the recipients correspond to your target market or audience. Suzi warned that some lists are sold on a 'one-use only' basis, and that disreputable sellers can provide inaccurate lists. You should also ensure that your list provider has complied with data protection regulations – check the eight principles at www.informationcommissioner.gov.uk.

A good alternative is to promote your survey through 'affinity sites' – websites used by the same kind of audience that you're trying to research, and which complement your brand or organisation well. This avoids the necessity of buying an email list, and will potentially secure a larger sample size.

Suzi warned against using sites that specialise solely in encouraging people to fill in surveys, as these are likely to distort your sample by attracting inappropriate respondents, who are motivated by the chance to win prizes and competitions.

Designing your questionnaire

Before you design a questionnaire, you must have a clear idea of the purpose of your research. What will the results be used for? Who will be answering the questions? Once you've worked that out, you can start to put your questions together.

The questions you ask should be clear and unambiguous. Ask spontaneous, more general questions first ("what advertisements did you see on TV last night?"), and work towards prompted and more specific ones ("do you remember seeing an advertisement for product X?").

Avoid asking questions that:

• Are too difficult to answer: "Would you buy product X in 10 years' time?"
• Are unclear in their meaning: "Do you think relationships will change the world?"
• Have double meanings: "How hot was your chilli?"
• Contain two questions: "Do you think handguns and marijuana should be legalised?"
• Lead in some way: "When did you stop beating your wife?"

Always test surveys on a small sample of users before sending them out – this should highlight any glaring problems in the questionnaire design.

Build and Distribution

If you have web-design skills in-house, it should not be too difficult to build your own bespoke online questionnaire. Alternatively, you can use an offline survey design software package, or an online ASP survey service (see below), many of which are relatively inexpensive.

Once built, surveys can be distributed in an email, as a link in a newsletter or banner ad, or as a pop-up on a webpage.

Incentives

Offering incentives, such as a special offer or the chance to win a prize, can be a good way to encourage a good response to your survey. The value of the incentives you offer does not have to be particularly high if they are relevant to your target audience, and it is often better to offer a number of smaller prizes rather than one big one, since potential respondents will consider themselves more likely to win.

Another option, which works well with some target audiences, is to offer to make a small donation to a specified charity for every survey that is completed.

However, Suzi warns that incentives can also have a downside, in that they tend to increase the number of people completing the survey carelessly or inaccurately, in an effort to get their hands on a prize or gift. If the target audience is sufficiently interested in the subject of your survey, they will not need much incentive.

Other forms of 'digital' research

Although the internet is best suited to quantitative surveys, Suzi ended her presentation by mentioning some other ways of using digital media for research purposes, including online focus groups using instant messenger, and SMS research – which is good for polling the under-24 audience.

Richard Naylor, Burns Owens Partnership

As Suzi demonstrated, online questionnaires can be very effective for conducting market research and customer satisfaction surveys. For Richard Naylor of Burns Owens Partnership, the challenge was to use online surveys to carry out an in-depth industry sectoral study. Would the techniques and technology stand-up to the rigorous methodological requirements of this kind of research?

Richard began by providing some more detail about the methodological principles underlying the kind of work he is usually involved in. In an ideal world, he explained, it would be possible to take a census approach to research, and survey everyone in the population universe that you wish to examine. In practice, it is never possible to achieve a 100% response rate, not even when the Government conducts an official census. Instead, you must work with a survey sample that accurately represents the population universe.

Survey results are typically 'grossed-up' to provide figures for the whole population. For example, the BARB figures recording UK TV audiences rely on just 52 thousand interviews per year to estimate viewing figures for more than 24 million TV owning households.

The tricky part is that unless you know the make up of your population universe, you won't be able to structure a representative sample. But how reliable and statistically significant does your sample need to be? Professional researchers have written many an academic paper investigating the minutiae of this question, but if you want to keep things relatively simple, Richard highlights two crucial criteria to keep in mind.

Two Magic Numbers

• 95% Confidence Interval. Confidence intervals (reflecting error levels tolerated) are indicators of reliability, and surveys should be able to demonstrate a confidence interval of 95% or better. According to the National Statistics website, "a 95% confidence interval is a range within which the true population would fall for 95% of the times the sample survey was repeated. It is a standard way of expressing the statistical accuracy of a survey based estimate." The larger your sample size, the more accurate you can expect your survey to be.
• 100. This is the minimum sample size for which you can achieve a 95% confidence interval.

Weighting

An accurate sample is not only determined by its size, but also the degree to which it reflects the characteristics of the true universe. The purpose of weighting is to reduce biases in a survey and make a sample more like the universe, but as Richard explained, if you get the weighting wrong you could increase distortions rather than reduce them. Weighting can influence the degree of significance that is given to particular sub-sections of your sample. Weights need to be applied to:

• Statistically significant groupings
• Homogenous groupings – 'like with like'

For example, Richard carried out a survey of the skills needs of businesses in South West England, including the 'M4 corridor' and rural Somerset. It would not have been accurate to have given equal statistical status to all of the companies surveyed, from sole-traders in rural villages to larger firms in office developments off the motorway, because there are a greater number of companies in the M4 area. These companies tend to be larger, employ more people and have a greater impact on the regional economy. The survey sample and results need to reflect such considerations.

Samples

When planning a survey of this kind, and structuring a sample, it helps to produce a sample frame, identifying groupings within your population universe in tabular or matrix form, together with any other significant criteria (such as company size in the above example) that will influence the number of responses you wish to obtain from each specific subsection of respondents. Given that standard response rates for surveys tend to be between 10 and 30% of those invited to take part, it is logical that your intended sample size has to be several times smaller than the size of the survey universe.

In Practice

That's the theory, but what about in practice? The conventional wisdom is that, for best results, you should keep online surveys short and simple (see Jakob Nielsen's advice on the subject). But the kind of research that BOP conducts is invariably complex and detailed, so reducing questionnaires to one screen and three questions is simply not an option.

On a recent project, Richard constructed a 6-page, 46-question online questionnaire, and recorded a 15% drop-out rate. That means that 85% of respondents completed the survey – which is good enough to suggest that more ambitious surveys can be successfully carried out online (but not as good as phone interviews, for example, which tend to have a negligible drop-out rate). One problem with such long surveys is that respondents skip questions, but in Richard's view this can't always be avoided. The alternative is to make more of your questions mandatory, but doing so is likely to increase drop-out rates.

Sometimes, no data exists for the population universe that you wish to survey, making accurate sampling impossible - as was the case on one of BOP's recent jobs, mapping audiovisual archives in the south-east of England. This would be a problem whatever method of research was used, but Richard found that conducting the survey online helped him to reach a wider selection of relevant respondents than expected, because news of the survey spread virally by email. Despite not knowing the extent of the population universe at the outset, the research team estimated that the survey was ultimately completed by over 50% of eligible respondents.

Using Online Research for In-Depth Surveys

Do consider using online surveys when:

• the universe is very small and/or unknown
• the sample is not intended to be weighted or grossed up to represent the universe
• respondents are likely to be highly motivated
• qualitative dimension is nil or limited

Consider alternatives when:

• the survey is long and/or has a strong qualitative dimension

Rich Alexandre, SurveyShack

In the early days of online research, it was necessary to design, code, build and serve your own website questionnaire each time you conducted a new survey. Nowadays it is much quicker and easier to implement a survey. According to Rich Alexandre of SurveyShack, there are three main technology options available to online researchers:

• Custom Development. In-house or outsourced developer builds and runs online survey, hosts it, collects and manages data gathered.
• Purchase Software. Install an off-the-shelf software package on your own servers, enabling you to design, build and host surveys and manage the resulting data.
• Use an ASP solution. You design and manage your own surveys over the internet, but the software and hosting is provided by an ASP (application service provider).

ASP solutions are typically low-cost (or free for basic or trial options) and can often be set up in a matter of minutes. Popular examples include SurveyMonkey, Zoomerang and Rich's own company SurveyShack, the highest profile example to be based in the UK.

Most ASPs feature a range of templates and question types (multiple choice, radio buttons, written responses etc.) to help you design your surveys, and tools enabling you to analyse, manage and export your data.

Advantages of Using an ASP

• No software to install
• No updates to install – automatic updates
• No local hardware updates
• Lower cost of entry
• Choose the functionality you need
• Available from anywhere in the world where you have access to the internet and a web browser
• Customisation

Disadvantages of Using an ASP

• Customer is reliant on service provider

Because ASP solutions are cost effective, easy to use and quick to implement, they enable a wider range of people to take advantage of online surveys, and to use them in response to a wider variety of situations. Rich cited the example of a major ecommerce firm that was able to improve its delivery service, and reduce the number of customer service complaints received by 300 per day (30%), after surveying its customers online. To do so they surveyed 1000 randomly selected customers per week, and ran three separate questionnaires, each of five questions.

Other case studies included an MBA research project, which saved hundreds of pounds on stationery, postage and data entry costs when it switched from a postal to an online survey, and the HR department of a large housing organisation, which saves £10,000 per year by conducting its annual staff survey online. With examples such as these, it is no surprise that a host of major companies and organisations now regularly use ASPs when conducting research.

In fact, if there's one possible danger facing online surveys, it could be that they become so widespread that potential respondents start to suffer from survey fatigue. Suzi Bentley warns of the dangers of getting 'survey happy', and asking your customers to complete questionnaires more than once every quarter. And as online surveys become increasingly commonplace, it's important not to get sloppy; as Richard Naylor reminds us, just because online surveys make the practical aspects of research easier, you shouldn't spend any less time or effort thinking about the conceptual stuff.

View original event description.