What are Research Buyers being Taught about Data Analysis? about managing Agencies?

Kathryn Korostoff in 'How to Hire and Manage Market Research Agencies” is teaching new research users how to select and managing research agencies, what exactly is she teaching them? I've taken a look here at what she's advocating through her business Rockstar Research (UK)...

Firstly she covers the basics of project types and leverage existing research then launches into discussion of the agency selection process

A true full service from an agency will be the best fit when you come up with objectives and they service you against key milestones – anything less than this and your involvement becomes more timely and crucial to reducing costs
If no outside help can be obtained, the organization can still learn a great deal by applying the methods and analysing results themselves. However, there is a strong chance that data about the strengths and weaknesses of a product, service or program will not be interpreted fairly if the data are analysed by the people responsible for ensuring the product, service or program is a good one.
“If you want your MR project to precisely meet your needs, you will need a precise RFP. The RFP is necessary to document your research needs and expectations. It also gives you an opportunity to ask for free advice.”
Produce a scorecard to narrow the field, base it on criteria such as Fee, Insight into project objectives, Presentation skills, References, Relationship style fit, Time line commitment.

Making insights that can live in the business world

Where is starts to get interesting is her tips in managing the internal audience, and how to analyse/interpret results. Here's an overview.

1. Always start with your research goals

This will help you organise your data and focus your analysis.
If you wanted to improve a program by identifying its strengths and weaknesses, you can organise data into program strengths, weaknesses and suggestions to improve the program.
If you wanted to fully understand how your program works, you could organise data in the chronological order in which customers or clients go through your program.
If you are conducting a performance improvement study, you can categorise data according to each measure associated with each overall performance result, e.g., employee learning, productivity and results.

2. Basic analysis of "quantitative" information

Make copies of your data and store the master copy away. Use the copy for making edits.
Tabulate the information, i.e., add up the number of ratings, rankings, yes's, no's for each question.
For ratings and rankings, consider computing a mean, or average, for each question, this will generally provide you with reams of significant analysis.

3. Basic analysis of "qualitative" information

Organise comments into similar categories, e.g.concerns, suggestions, strengths, weaknesses, similar experiences, program inputs, recommendations, outputs, outcome indicators, etc.
Label the categories or themes, e.g., concerns, suggestions, etc.
Attempt to identify patterns, or associations and causal relationships in the themes, e.g., all people who attended programs in the evening had similar concerns, most people came from the same geographic area, most people were in the same salary range, what processes or events respondents experience during the program, etc.
Keep all commentary for several years after completion in case needed for future reference.

4. Interpreting information

Attempt to put the information in perspective, e.g., compare results to what you expected, promised results; management or program staff; any common standards for your products or services; original goals (especially if you're conducting a program evaluation); indications or measures of accomplishing outcomes or results (especially if you're conducting an outcomes or performance evaluation); description of the program's experiences, strengths, weaknesses, etc. (especially if you're conducting a process evaluation).
Consider recommendations to help employees improve the program, product or service; conclusions about program operations or meeting goals, etc.
Record conclusions and recommendations in a report, and associate interpretations to justify your conclusions or recommendations.

5. Reporting Results

The level and scope of content depends on to whom the report is intended, e.g., to funders / bankers, employees, clients, customers, the public, etc.
Be sure stakeholders have a chance to carefully review and discuss the report. Translate recommendations to action plans, including who is going to do what about the research results and by when.
Stakeholders will likely require a report that includes a strong description of the problem and a explanation of the research goals, methods, and analysis procedures; listing of conclusions and recommendations; and any relevant attachments, e.g., inclusion of research questionnaires, interview guides, delivered as a short presentation and report.
Be sure to record the research plans and activities in a research plan which can be referenced when a similar research effort is needed in the future.

Some Pitfalls to Avoid

Don't baulk at research because it seems far too "scientific." It's not. Usually the first 20% of effort will generate the first 80% of the plan, and this is far better than nothing.
There is no "perfect" research design. Don't worry about the research design being perfect. It's far more important to do something than to wait until every last detail has been tested.
Work hard to include some interviews in your research methods. Questionnaires don't capture "the story," and the story is usually the most powerful depiction of the benefits of your products, services or programs.
Don't interview just the successes. You'll learn a great deal by understanding failures and unsatisfied customers etc.
Don't throw away research results once a report has been generated. Results don't take up much room, and they can provide precious information later when trying to understand changes in the product, service or program.

Interesting results, all buyers certainly operate from a learning paradigm, looking to researchers for an information flow.

Back to Best Practices Library