Starjournal Quotation Hartmann, Jochen, Huppertz, Juliana, Schamp, Christina, Heitmann, Mark. 2019. Comparing automated text classification methods. International Journal of Research in Marketing. 36 (1), 20-38.




Online social media drive the growth of unstructured text data. Many marketing applications require structuring this data at scales non-accessible to human coding, e.g., to detect communication shifts in sentiment or other researcher-defined content categories. Several methods have been proposed to automatically classify unstructured text. This paper compares the performance of ten such approaches (five lexicon-based, five machine learning algorithms) across 41 social media datasets covering major social media platforms, various sample sizes, and languages. So far, marketing research relies predominantly on support vector machines (SVM) and Linguistic Inquiry and Word Count (LIWC). Across all tasks we study, either random forest (RF) or naive Bayes (NB) performs best in terms of correctly uncovering human intuition. In particular, RF exhibits consistently high performance for three-class sentiment, NB for small samples sizes. SVM never outperform the remaining methods. All lexicon-based approaches, LIWC in particular, perform poorly compared with machine learning. In some applications, accuracies only slightly exceed chance. Since additional considerations of text classification choice are also in favor of NB and RF, our results suggest that marketing research can benefit from considering these alternatives.


Press 'enter' for creating the tag

Publication's profile

Status of publication Published
Affiliation External
Type of publication Journal article
Journal International Journal of Research in Marketing
Citation Index SSCI
WU Journalrating 2009 A+
Starjournal Y
Language English
Title Comparing automated text classification methods
Volume 36
Number 1
Year 2019
Page from 20
Page to 38
Reviewed? Y
Open Access Y
Open Access Link


Schamp, Christina (Details)
Hartmann, Jochen (Universität Hamburg, Germany)
Heitmann, Mark (Universität Hamburg, Germany)
Huppertz, Juliana (Universität Hamburg, Germany)
Research areas (ÖSTAT Classification 'Statistik Austria')
5320 Marketing (Details)
Google Scholar: Search