Simple Changes in Design Can Lead to Big Changes in Data
- Posted by Jarod Johnson
- On April 4, 2022
Last year Brookmark updated our survey templates to expand the size of open-end text boxes. It was a relatively easy adjustment specifically aimed at improving the quality of open-end responses. Provided below is the before and after view as seen by survey respondents.
This is an example of an open end with the out-of-the-box format:
This is the same example with the new format:
Since the change, we closely monitored the verbatim responses on a few of our ongoing studies.
As our team reviewed the open-ends for one of these studies, they mentioned they seemed more in depth this year. We decided to dig deeper to quantify how much more in depth they were. For such a seemingly small change, the results are impressive.
- The average response length for that open end increased from 59.6 characters to 82.4, an increase of almost 40%.
- The number of responses for that open end that were categorized into multiple codes increased from 43.6% to 55.2%, indicating that not only were the responses longer, but higher quality.
Looking at other questions across multiple studies, we found similar results in almost every case: Longer and more codifiable responses. In the most striking case, the average length increased by 84% and had almost 3 times as many multi-code responses.
Based on these observations, we were able to confirm that the larger open text boxes not only lead to longer, more robust open-ends, but also more codable data. But the broader take-away is the extent to which your default settings can affect the quality of your data. Maybe you’ve looked at hundreds of surveys using these settings, so you don’t even think to change them. Or maybe you’ve noticed them but thought “that’s the way it’s always been done.” But every once-in-a-while, it’s good to look at your surveys with a fresh set of eyes, and look for easy fixes that can improve the respondent experience, and in turn improve your data.
About Brookmark Research:
Brookmark is a data driven consultancy focused on developing powerful marketing strategies to fuel the growth of our clients’ businesses. We leverage primary marketing research, 1st party, 3rd party and proprietary data sources along with proprietary tools and strategy frameworks to develop, test and evaluate marketing driven strategies that deliver exponential ROI. Whether you are an early-stage growth company or a Fortune 1000 company, we help successful companies accelerate, and stagnate companies invigorate growth.