Abstract
Large Language Models (LLMs) have become increasingly incorporated into
everyday life for many internet users, taking on significant roles as advice
givers in the domains of medicine, personal relationships, and even legal
matters. The importance of these roles raise questions about how and what
responses LLMs make in difficult political and moral domains, especially
questions about possible biases. To quantify the nature of potential biases in
LLMs, various works have applied Moral Foundations Theory (MFT), a framework
that categorizes human moral reasoning into five dimensions: Harm, Fairness,
Ingroup Loyalty, Authority, and Purity. Previous research has used the MFT to
measure differences in human participants along political, national, and
cultural lines. While there has been some analysis of the responses of LLM with
respect to political stance in role-playing scenarios, no work so far has
directly assessed the moral leanings in the LLM responses, nor have they
connected LLM outputs with robust human data. In this paper we analyze the
distinctions between LLM MFT responses and existing human research directly,
investigating whether commonly available LLM responses demonstrate ideological
leanings: either through their inherent responses, straightforward
representations of political ideologies, or when responding from the
perspectives of constructed human personas. We assess whether LLMs inherently
generate responses that align more closely with one political ideology over
another, and additionally examine how accurately LLMs can represent ideological
perspectives through both explicit prompting and demographic-based
role-playing. By systematically analyzing LLM behavior across these conditions
and experiments, our study provides insight into the extent of political and
demographic dependency in AI-generated responses.