Problem

Vision-Language Models (VLMs) are widely used in critical applications but rely on image preprocessing, such as scaling, for efficiency. This reliance creates a significant, often overlooked, security vulnerability. Existing adversarial attacks that exploit scaling operations are typically static and lack adaptability, making them brittle and easily detectable in dynamic VLM workflows. These static attacks fail to account for real-time model feedback and have not adequately addressed the severe implications for multi-step agentic systems, where compromised downstream decisions are critical. This gap leaves modern multimodal AI systems vulnerable to sophisticated, evolving threats.

Method

The researchers propose Chameleon, a novel, adaptive adversarial framework designed to exploit scaling vulnerabilities in production VLMs. Unlike static approaches, Chameleon employs an iterative, agent-based optimization mechanism that dynamically refuses image perturbations based on the target model’s real-time feedback. The process begins with a clean image to which an initial perturbation is added. This perturbed image is then downscaled and fed into the VLM. The VLM's response provides signals such as confidence scores, predicted classes, and a binary success indicator. A scalar reward function, balancing attack efficacy and stealth, drives the adaptation of the perturbation. Two optimization approaches are explored: a greedy local search, which updates perturbations only if they yield a higher reward, and a population-based genetic algorithm, which encourages diversity and exploration using crossover and mutation. The goal is to craft highly robust adversarial examples that remain imperceptible to humans but become active semantic instructions after standard downscaling operations, effectively hijacking downstream execution in multimodal AI systems.

Results

Chameleon was evaluated against the Gemini 2.5 Flash model, achieving an impressive Attack Success Rate (ASR) of 84.5% across varying scaling factors, significantly outperforming static baseline attacks, which averaged only 32.1%. The genetic algorithm slightly outperformed hill-climbing, achieving a 4% higher success rate and maintaining lower visual distortion (0.0693 normalized L2 distance vs. 0.0847). Hill-climbing, however, converged faster, requiring fewer iterations and API calls. The attacks effectively compromised agentic pipelines, reducing decision-making accuracy by over 45% in multi-step tasks, and decreased model confidence by 0.18-0.21 on average. The framework demonstrated strong generalization, with high success rates (86–92%) across different downsampling methods and various prompts (84–93%), indicating that it exploits fundamental properties of VLM preprocessing rather than method-specific artifacts. Perturbation magnitudes were consistently low (11.8 to 14.2 pixel shift), ensuring visual imperceptibility.

Implications

The findings demonstrate that image scaling operations represent a fundamental and underexplored security vulnerability in Vision-Language Models, with significant implications for deployed multimodal agentic systems. Chameleon's ability to achieve high attack success rates with imperceptible perturbations highlights a critical security gap that cannot be addressed by simple visual inspection. The framework's black-box applicability and computational efficiency make it a practical and immediate threat to real-world VLM deployments. This research underscores the urgent need for scaling-aware security evaluations and the development of robust defense mechanisms, such as adversarial training with scaled images, multi-scale consistency checks, and architectural innovations that ensure scaling invariance to create more trustworthy AI systems.