Does Craigslist provide a more diverse group of research participants than are typically used? Yes says researchers Kathleen Alto, Keiko McCullough, and Dr. Ronald Levant.
Scientists must consider the methods in which they recruit participants and how said methods influence whose voices are heard in research. From 2000 to 2008, it was revealed that 61% of studies in Psychology of Men and Masculinity utilized college samples as the foundation for their findings (Wong, Steinfeldt, Speight, & Hickman, 2010). This reflects a broader trend in which psychologists largely utilize convenience samples of undergraduate college students for research. This practice brings the concern of generalizability to the forefront, as the findings of these studies may only be applicable to a small subset of the general population. Convincing evidence supports the idea that Internet samples tend to be more diverse than convenience samples on a multitude of variables (e.g., socioeconomic status, location, race, gender, age) (Gosling, Vazire, Srivastavam, & John, 2004; Lewis, Watson, & White, 2009; Mathy, Schillace, Coleman, & Berquist, 2002). Thus, we believe that using Internet samples alone or supplementary to college samples can help researchers improve the applicability of their results to the real world. We explored Craigslist as a way of gathering participants online because it is an accessible and free service. This is different from other methods of online data collection which usually charge fees or require payment. Anyone can go to Craigslist.org and post an advertisement seeking research participants for their study. Individuals can post in any area across the U.S. In our lab, we used Craigslist to recruit for two studies we were conducting at the time. After data collection was over, we were curious how recruiting through Craigslist affected our sample. We were also interested in seeing how the demographics of our Craigslist sample compared to the convenience, college student sample we also collected for these two studies.
We posted fairly frequently and offered an incentive (the chance to win a $50 gift card), and found that Craigslist was a speedy way to gain participants. Over ten weeks, we recruited 273 participants through Craigslist alone, 105 of which were men (38.5%). The Craigslist sample had a higher percentage of men than the convenience college sample (18.5%), though there was no significant difference in the proportion of men between the two samples. Thus, men in general were equally difficult to recruit using both methods. Be that as it may, Craigslist may be uniquely accessible to men of color when targeted posts are used. As we wanted to hear specifically from racial and ethnic minority individuals, we tailored the content of some posts to reflect that need. When comparing the Craigslist sample in which we targeted specific racial/ethnic minority groups to the Craigslist sample that did not target these groups, we found a significant difference. The targeted posts were effective in recruiting significantly more Hispanic/Latino American and Asian American men. However, targeted posts were seemingly ineffective in reaching a greater number of African American men. Interestingly, the Craigslist sample that did not target specific racial/ethnic minority groups still included significantly more Asian American and Hispanic/Latino participants, and fewer African Americans than the college sample.
In regard to other demographic variables, we found that Craigslist participants were significantly older, more educated, reported lower income and socioeconomic status, and were less economically and socially conservative. The Craigslist sample had significantly more non-Christian participants and greater sexual minority individuals than the college sample. Overall, it seems that the Craigslist sample was generally more diverse than the college sample. Knowing this, we were also interested in knowing which sample was more representative of the national population. We decided to compare the Craigslist samples from both studies, the college sample, and the Craigslist and college sample combined to represent what it may look like if other researchers were to recruit participants using both methods as we did.
When analyzing our results, we found that no sample emerged as the clear winner in regard to “best representativeness.” Even without using targeted posts, the Craigslist sample contained a higher proportion of Asian Americans than is present in the U.S. population. We found the college sample was closer to national representation. However, for Latino/Hispanic Americans, the Craigslist sample (without targeted posts) was more accurately representative than the college sample. For both Asian Americans and Latino/Hispanic Americans, when we use targeted posts, we oversample from these groups. For African Americans, even though targeted posts were ineffective, the Craigslist sample without targeted posts was most reflective of the representation of this group in the U.S. As one might expect, the Craigslist samples were more aligned with the general population in regard to age and educational attainment than the college sample. Religiously, the college sample was more consistent with most Americans as it was predominantly Christian. Despite these differences, we also found some similarities across all of the samples. For example, most participants identified as middle class, consistent with most Americans. All of the samples were more representative of sexual minorities, held more liberal attitudes, and underrepresented men.
We found Craigslist to be useful tool for recruiting diverse voices for our research. Particularly, it seems targeted recruitment efforts via Craigslist are generally successful and Craigslist as a platform produces diverse samples in regard to race/ethnicity, religion, and sexual orientation. Though we cannot claim Craigslist samples are undoubtedly more representative than college samples, on a number of factors the Craigslist sample did appear favorable in representativeness. We did find combining our Craigslist and college samples to be helpful in increasing the representativeness of our studies and believe other masculinity researchers can benefit from this strategy as well.
For more detailed results, see our paper here
Keiko McCullough is a first year doctoral student in the counseling psychology program at Indiana University Bloomington. Her primary research interests are men of color and masculinities, biracial/multiracial individuals’ stressors and wellness, and the intersection of gender, race, and new media broadly. Keiko is also passionate about feminist issues and social justice advocacy.
Kathleen Alto is a fourth year doctoral student at the University of Akron’s Collaborative Program in Counseling Psychology. She received her Bachelor’s in Science in Human Development from Cornell University in 2011. She is a passionate advocate for LGBTQ and women’s rights. Her research focuses on how gender and gender roles influence sexual prejudice attitudes.
Ronald F. Levant, Ed.D., ABPP, is Professor of Psychology, The University of Akron, where he served as Dean of the Buchtel College of Arts and Sciences. He also served as the 2005 President of APA, and as Editor of the Psychology of Men and Masculinity. Dr. Levant has authored, co-authored, edited or co-edited 16 books and over 200 peer-reviewed refereed journal articles and book chapters. His research focus is in the psychology of men and masculinity.