Estimating Effects of Long-Term Treatments
EC '23: Proceedings of the 24th ACM Conference on Economics and Computation, July 2023, Pages 907
Estimating the effects of long-term treatments in A/B testing presents a significant challenge. Such treatments - including updates to product functions, user interface designs, and recommendation algorithms - are intended to remain in the system for a long period after their launches. On the other hand, given the constraints of conducting long-term experiments, practitioners often rely on short-term experimental results to make product launch decisions. It remains an open question how to accurately estimate the effects of long-term treatments using short-term experimental data. To address this question, we introduce a longitudinal surrogate framework. We show that, under standard assumptions, the effects of long-term treatments can be decomposed into a series of functions, which depend on the user attributes, the short-term intermediate metrics, and the treatment assignments. We describe the identification assumptions, the estimation strategies, and the inference technique under this framework. Empirically, we show that our approach outperforms existing solutions by leveraging two real-world experiments, each involving millions of users on WeChat, one of the world's largest social networking platforms.
AI versus Friends in Recommending Online Content: A Large-scale Field Experiment
Content platforms rely intensively on artificial intelligence (AI) or social networks (friends) to recommend personalized online content. Nevertheless, little is known about the different impacts of these two dominant recommendation mechanisms on user engagement with the content and the platform. We, therefore, conduct a large-scale randomized field experiment that involves over 2.1 million users and over 6.8 million items (online content) recommended by AI or friends (i.e., contacts) on WeChat. We randomly assign users into three groups: Users in Group I are exposed to the content recommended by AI only; those in Group II are exposed primarily to the content recommended by friends with the display of social cues; and in Group III, users experience the same design as Group II but without the display of social cues. We find that AI recommender systems lead to a higher content clickthrough rate and dwell time, whereas friend recommendations result in a larger content share rate and platform retention. The content difference is the main driver for the higher content clickthrough rate and dwell time of AI recommender systems and for the larger content share rate of friend recommendations. The presence of social cues (influence) contributes to the higher platform retention associated with friend recommendations. We further demonstrate that the performance differences between AI and friend recommendations vary across characteristics of users (user activeness) and content (homophily level and the number of displayed social cues). Our findings have rich implications for the mechanism and management of online content recommendations.
Emotions in Online Content Diffusion
Social media-transmitted online information, which is associated with emotional expression, shapes our thoughts and actions. In this study, we incorporate social network theories and analyses and use a computational approach to investigate how emotional expression, particularly \textit{negative discrete emotional expression} (i.e., anxiety, sadness, anger, and disgust), leads to differential diffusion of online content in social media networks. We quantify diffusion cascades' structural properties (i.e., size, depth, maximum breadth, and structural virality) and analyze the individual characteristics (i.e., age, gender, and network degree) and social ties (i.e., strong and weak) involved in the cascading process. In our sample, more than six million unique individuals transmitted 387,486 randomly selected articles in a massive-scale online social network, WeChat. We detect the expression of discrete emotions embedded in these articles, using a newly generated domain-specific and up-to-date emotion lexicon. Different model specifications are used to robustly demonstrate the relationships between negative discrete emotions and online content diffusion. We find that articles with more expression of anxiety spread to a larger number of individuals and diffuse more deeply, broadly, and virally. Expression of anger and sadness, however, reduces cascades' size and maximum breadth. We further show that the articles with different degrees of negative emotional expression tend to spread differently based on individual characteristics and social ties. Our results shed light on content generation, diffusion, and regulation, utilizing negative emotional expression.
Customer-Product Matches in Online Social Referrals: A Graph Embedding Approach
New social media has been enlarging the impact of social referrals on product diffusion. There is, however, no direct evidence on how effectively referrers match products with referred customers that, in turn, leads to high returns for online social referrals. To answer this question, we analyzed 137,622 referrals of 20,169 instant apps from 11,668 referrers to 84,166 recipients (those referred), and 1,141,363 (100 per referrer) unique contacts randomly selected from referrers' local social networks excluding the recipients. We leveraged a state-of-the-art graph embedding framework to estimate users' latent preferences for the apps based on a massive amount of user-app historical usage sequences. We find that referrers recommend products to contacts with substantially stronger preferences for the referred products than the non-referred contacts have. Referrers' contacts have significantly greater preferences for the referred products than do their non-contacts. These results confirm the effectiveness of active (actively screening and selecting contacts to match products) and passive (homophily-based) matching in online social referrals. We also find that referrers exhibit the strongest preferences for the referred products. The matching outperforms among narrow-appeal products and for more engaged referrers and those with a smaller local social network. Our findings shed light on the mechanisms and management of social referrals.
Exposure to Novel Information Mediated by Algorithms vs. Social Ties
This project aims to identify the differential impact of algorithms and social media networks on individuals’ exposure and consumption (click-through) of online novel information. We are among the first to investigate this question, as the related evidence has been limited by empirical challenges. Specifically, we focus on four questions: First, how do algorithms or social media networks limit users’ exposure to novel information? Second, how do individuals choose to consume novel information given the exposure mediated by different technologies? Third, how does social influence (i.e., the effects of social cues), unique to social media networks, affect individuals’ novel information consumption? Fourth, how do these impacts differ across the characteristics of individuals and networks? Answering these questions will have rich implications for online users, content platforms, and regulators to effectively acquire, manage, and regulate online content.
We examine how users interact with algorithm-recommended versus socially shared content using data from a large-scale randomized field experiment involving 2.1 million WeChat users and more than 6 million recommended items (content). We conduct\footnote{The experiment was conducted on WeChat earlier this year. Data collection, cleaning, and analysis have not yet begun, except for randomization checks to verify the validity of experimental randomization. } experiment on WeChat Top Stories. It separately applies algorithms or social media networks to deliver various online content for users, providing a unique context for studying our questions. In Control Group, users are primarily exposed to the information shared by their WeChat friends. In Treatment Group One, users are only exposed to the information delivered only by algorithms. In Treatment Group Two, users are in the same condition as in Control Group, except social cues are hidden from users. Our findings will add to the literature on social networks, the economic implications of algorithms, and online information diffusion.
Social Advertising Effectiveness Using a Large-Scale Randomized Field Experiment
Study 1: Social Advertising Effectiveness Across Products Marketing Science, 2020
Study 2: Public versus Private Responses to Social Advertising
Study 1:
Most of the empirical evidence on social advertising effectiveness focuses on a single product at a time. As a result, little is known about how the effectiveness of social advertising varies across product categories or product characteristics. We, therefore, collaborated with a large online social network to conduct a randomized field experiment measuring social ads effectiveness across 71 products in 25 categories among more than 37 million users. We found some product categories, like clothing, cars and food exhibited significantly stronger social advertising effectiveness than other categories like financial services, electrical appliances, and mobile games. More generally, we found that status goods, which rely on status-driven consumption, displayed strong social advertising effectiveness. Meanwhile, social ads for experience goods, which rely on informational social influence, did not perform any better or worse than their theoretical counterpart search goods. Social advertising effectiveness also significantly varied across the relative characteristics of ad viewers and their friends shown in ads. Understanding the heterogeneous effects of social advertising across products can help marketers differentiate their social advertising strategies and lead researchers to more nuanced theories of social influence in product evaluation.
Study 2:
One unique feature of social advertising is the coexistence of public and private consumer responses. Social media allows users’ certain responses (e.g., likes) to be publicly revealed to one's social network. More importantly, public responses can also influence private responses (e.g., clicks). We, therefore, use data from a large-scale field experiment with a major social media platform (WeChat Moments) to investigate how the display of social cues (friends' likes) affects users' public (likes) and private responses (clicks) to social ads. We find that, on average, displaying the first social cue significantly enhances the liking rate and the clickthrough rate. Nevertheless, although showing additional social cues can further increase users' tendency to like an ad, it does not further increase the clickthrough rate. This empirical pattern is consistent with the interplay between informational and normative social influence in social advertising. Overall, we find that the coexistence of the two forces can enhance the conformity effect on the public liking response. However, when normative social influence dominates, a crowding-out effect on the private clicking response may occur. Our results have rich implications for advertisers and social media platforms in regard to the design of social advertising policies and social networks.
