Problem

The rapid integration of artificial intelligence (AI) into mobile applications has transformed the digital landscape, yet there is a limited understanding of how everyday users perceive, evaluate, and engage with these AI-powered features. Existing research often relies on controlled laboratory experiments, focuses on organizational AI adoption, or examines specific AI systems in isolation, failing to capture naturalistic user feedback from real-world consumer applications. Furthermore, while users generally desire AI, they also express concerns, and many studies assume users consciously recognize AI features, an assumption that needs scrutiny given AI's increasingly invisible operation within applications. This research addresses three key questions: how AI-featuring applications differ from non-AI apps in terms of user satisfaction, what specific concerns and benefits users express when discussing AI, and how the reception of AI features varies across different application categories and mobile platforms.

Method

This study employed a large-scale observational design, analyzing 1,484,633 publicly available user reviews from 422 mobile applications (200 AI-featuring, 222 control) on both the iOS App Store and Google Play Store. Data was collected between January 2024 and January 2025. Applications were systematically selected based on consistent presence in top charts, availability on both platforms, and a minimum of 1,000 user reviews. AI-featuring applications were identified through explicit AI-related terminology in descriptions, analysis of release notes, and verification via technology news. Control applications were carefully matched by category, user base size, and release timeframe. Data extraction utilized specialized Python libraries for application metadata and review content. Ethical standards were maintained by replacing reviewer names with hashed identifiers and redacting personal information. AI-related content was detected using a dictionary of 74 terms and phrases. Sentiment analysis combined rule-based methods with a fine-tuned DistilBERT model. Content analysis categorized concerns and benefits through pattern matching against validated dictionaries. Statistical analyses included independent samples t-tests, chi-square tests, hierarchical regression, logistic regression, and ANOVA. Topic modeling using Latent Dirichlet Allocation (LDA) with 10 topics was applied to AI-mentioning reviews to identify thematic patterns, preceded by extensive text preprocessing.

Results

The study identified a significant "AI Invisibility Effect": despite nearly half of the analyzed applications featuring AI capabilities, only 11.9% of user reviews explicitly mentioned AI (24% in AI apps versus 0.8% in control apps). Initially, AI-featuring applications received substantially lower average ratings (3.28 vs. 3.87) than control applications. However, a critical finding from hierarchical regression revealed a reversal: after controlling for explicit AI mentions and review characteristics, AI applications showed higher odds of five-star ratings. This suggests that the initial negative association was a suppression effect, where dissatisfaction was concentrated among users who *noticed and critiqued* AI. Among reviews mentioning AI, privacy/data concerns were paramount (34.8%), followed by accuracy/errors (22.1%), while efficiency/time-saving (42.3%) and helpfulness/utility (28.7%) were the primary perceived benefits. Reviews expressing only concerns were significantly lower-rated than those expressing only benefits. Temporal analysis indicated increasing privacy concerns. AI reception also showed significant heterogeneity across platforms and categories: iOS users were more receptive to AI features than Android users. Assistant and Creative applications showed strong positive effects from AI integration, whereas Entertainment applications exhibited a significant negative AI effect, and Utility applications showed virtually no effect.

Implications

The findings advance human-AI interaction theory by introducing the concept of "unconscious adoption," where users interact with AI features without explicit awareness or evaluation, challenging traditional technology acceptance models. The observed suppression effect indicates that negative user evaluations are driven by the explicit recognition of AI and potential expectation violations, rather than the mere presence of AI itself. For application developers and technology companies, these results highlight the need for improved, context-specific communication about AI capabilities. Successful AI integration is contingent on careful tailoring to category-specific user needs, proving most beneficial in Assistant and Creative applications, while potentially detrimental in Entertainment and neutral in Utility apps. Addressing privacy concerns is crucial for building user trust, alongside managing expectations about AI accuracy. Platform-specific optimization strategies are also suggested, given the differences in AI reception between iOS and Android users. Future research should qualitatively investigate the awareness gap, experimentally test the impact of AI framing on user evaluation, and examine AI within broader technological ecosystems.