Nearly Zero-Cost Protection Against Mimicry by Personalized Diffusion Models

FastProtect achieves real-time protection against diffusion models, addressing the critical issue of latency in this task for the first time. By integrating perturbation pre-training with adaptive inference schemes, FastProtect meets all requirements for a practical protection solution. As illustrated in the figures above, the strengths of the proposed method are as follows: (a) FastProtect shows unprecedented speed in protection against diffusion models. On an A100 GPU, FastProtect achieves real-time latency even for processing 2048-by-2048-px image, while others require substantially longer time. (b) In terms of the trade-off between protection efficacy (FID, ↑ is better) and invisibility (DISTS, ↓ is better), FastProtect exhibits improvement over other protection methods.

Recent advancements in diffusion models revolutionize image generation but pose risks of misuse, such as replicating artworks or generating deepfakes. Existing image protection methods, though effective, struggle to balance protection efficacy, invisibility, and latency, thus limiting practical use. We introduce perturbation pre-training to reduce latency and propose a mixture-of-perturbations approach that dynamically adapts to input images to minimize performance degradation. Our novel training strategy computes protection loss across multiple VAE feature spaces, while adaptive targeted protection at inference enhances robustness and invisibility. Experiments show comparable protection performance with improved invisibility and drastically reduced inference time.

Schematic illustration of the proposed FastProtect. (a) Current iterative optimization approaches lack a training phase and perform optimization during inference, resulting in extremely slow protection. (b) Universal adversarial perturbation (UAP) introduces pre-training of perturbations, but their image-agnostic nature leads to degraded protection efficacy. (c) Combining the advantages of both paradigms, FastProtect adopts a pre-training approach similar to UAP but with a novel mixture-of-perturbation scheme and multi-layer protection loss to enhance protection efficacy. At inference, adaptive targeted protection further boosts protection efficacy with minimal additional cost, and adaptive protection strength improves invisibility.

Quantitative comparison results. The left side of the table presents the inference speed of each method, while the right side summarizes the invisibility and protection performance across four distinct domains. These quantitative results demonstrate that FastProtect achieves substantially faster inference, while maintaining comparable invisibility and robust protection effectiveness compared to existing methods.

Qualitative comparison results. The results indicate that FastProtect induces relatively less image quality degradation than the baselines. Furthermore, through the results generated by personalized LoRA, we confirmed that the protected images effectively retained the intended target characteristics, demonstrating the qualitative effectiveness of FastProtect.