企业官网建设流程全解析

在 🤗 Diffusers 中使用 Kandinsky 系列管线：文生图、图生图、Inpainting、插值与 ControlNet 实战指南

【免费下载链接】diffusers🤗 Diffusers: State-of-the-art diffusion models for image, video, and audio generation in PyTorch.项目地址: https://gitcode.com/GitHub_Trending/di/diffusers

Kandinsky 2.1 是 🤗 Diffusers 官方支持的多模态扩散模型，它继承自 DALL·E 2 与 Latent Diffusion 的最佳实践，并通过 CLIP 文本/图像编码器配合扩散图像先验（prior）在 CLIP 模态的隐空间之间建立映射，从而显著提升视觉质量，并支持图像混合与文本引导的图像编辑。本文基于 Kandinsky API 文档，完整讲解 Kandinsky 2.1/2.2/3 系列管线在文本生成图像、图生图、Inpainting、隐空间插值以及 Kandinsky 2.2 ControlNet 深度条件生成中的端到端用法，并结合 src/diffusers/pipelines/kandinsky 下的源码剖析 prior 管线与解码管线的工作机制，最后给出推理优化建议。读完本文，你将能够独立搭建 Kandinsky 生成流程并针对不同任务灵活切换管线。

前置知识：Kandinsky 的"先验 + 解码"双管线架构

Kandinsky 2.1/2.2 与常见扩散模型的最大区别在于必须先用 prior 管线编码提示词、生成图像嵌入（image embeddings），再用第二个管线把隐变量解码成图像。从源码结构看，这一设计对应仓库中的两类文件：

• prior 管线：pipeline_kandinsky_prior.py 中的KandinskyPriorPipeline，核心组件为PriorTransformer（unCLIP 先验）、CLIP 文本编码器CLIPTextModelWithProjection、CLIP 图像编码器CLIPVisionModelWithProjection与UnCLIPScheduler；
• 解码管线：pipeline_kandinsky.py 中的KandinskyPipeline，核心组件为多语言 CLIP 文本编码器MultilingualCLIP、XLMRobertaTokenizer、条件 U-Net（UNet2DConditionModel）与 MoVQ 解码器（VQModel），默认搭配DDIMScheduler。

在 pipeline_kandinsky.py 中可以看到，解码管线的 CPU offload 顺序定义为text_encoder->unet->movq，其 MoVQ 缩放因子由2 ** (len(self.movq.config.block_out_channels) - 1)计算得到，并在get_new_h_w中把用户指定的高宽向上对齐到缩放因子的整数倍，保证送入 U-Net 的隐变量尺寸合法。

版本差异提醒：Kandinsky 2.1 与 2.2 的用法非常相似，唯一区别是Kandinsky 2.2 解码时不接收prompt，只接收image_embeds；而Kandinsky 3 架构更精简，不需要 prior 模型，其用法与 Stable Diffusion XL 等其他扩散模型完全一致。

安装依赖（Colab 中取消注释执行）：

#!pip install -q diffusers transformers accelerate

文本生成图像（Text-to-image）

无论执行哪种任务，第一步都是搭建 prior 管线，将提示词编码并生成图像嵌入。prior 管线同时还会生成与负提示词""对应的negative_image_embeds。如果想获得更好效果，可以给 prior 管线传入真实的negative_prompt，但这会使 prior 管线的有效 batch size 翻倍。

Kandinsky 2.1：两阶段调用

from diffusers import KandinskyPriorPipeline, KandinskyPipeline import torch prior_pipeline = KandinskyPriorPipeline.from_pretrained("kandinsky-community/kandinsky-2-1-prior", dtype=torch.float16).to("cuda") # 或 "mps"、"xpu"、"cpu" pipeline = KandinskyPipeline.from_pretrained("kandinsky-community/kandinsky-2-1", dtype=torch.float16).to("cuda") prompt = "A alien cheeseburger creature eating itself, claymation, cinematic, moody lighting" negative_prompt = "low quality, bad quality" # 负提示词可选，但通常能改善结果 image_embeds, negative_image_embeds = prior_pipeline(prompt, negative_prompt, guidance_scale=1.0).to_tuple()

接着把提示词与两组嵌入一并传给KandinskyPipeline生成图像：

image = pipeline( prompt, image_embeds=image_embeds, negative_prompt=negative_prompt, negative_image_embeds=negative_image_embeds, height=768, width=768, ).images[0] image

从 pipeline_kandinsky.py 的__call__实现看，KandinskyPipeline会把image_embeds与negative_image_embeds拼接后与文本嵌入一起作为 U-Net 的added_cond_kwargs条件输入，并执行无分类器引导（CFG）：当guidance_scale > 1.0时，噪声预测与方差预测会先按通道拆分、再按 CFG 公式noise_pred_uncond + guidance_scale * (noise_pred_text - noise_pred_uncond)合并，最后经 MoVQ 解码并归一化到 0～1 输出 PIL 图像。

Kandinsky 2.2：解码阶段不接收 prompt

from diffusers import KandinskyV22PriorPipeline, KandinskyV22Pipeline import torch prior_pipeline = KandinskyV22PriorPipeline.from_pretrained("kandinsky-community/kandinsky-2-2-prior", dtype=torch.float16).to("cuda") # 或 "mps"、"xpu"、"cpu" pipeline = KandinskyV22Pipeline.from_pretrained("kandinsky-community/kandinsky-2-2-decoder", dtype=torch.float16).to("cuda") prompt = "A alien cheeseburger creature eating itself, claymation, cinematic, moody lighting" negative_prompt = "low quality, bad quality" image_embeds, negative_image_embeds = prior_pipeline(prompt, guidance_scale=1.0).to_tuple() image = pipeline(image_embeds=image_embeds, negative_image_embeds=negative_image_embeds, height=768, width=768).images[0] image

Kandinsky 3：无需 prior，直接生成

from diffusers import Kandinsky3Pipeline import torch pipeline = Kandinsky3Pipeline.from_pretrained("kandinsky-community/kandinsky-3", variant="fp16", dtype=torch.float16) pipeline.enable_model_cpu_offload() prompt = "A alien cheeseburger creature eating itself, claymation, cinematic, moody lighting" image = pipeline(prompt).images[0] image

端到端合并管线与 AutoPipeline

🤗 Diffusers 还提供端到端 API：KandinskyCombinedPipeline与KandinskyV22CombinedPipeline，二者会自动加载 prior 模型与解码器，无需分别加载。若想单独调整 prior 阶段的采样，仍可通过prior_guidance_scale与prior_num_inference_steps参数控制——在 pipeline_kandinsky_combined.py 的实现中，合并管线正是把这两个参数透传给内部 prior 管线的__call__（分别映射为guidance_scale与num_inference_steps）。

使用AutoPipelineForText2Image即可在底层自动调用合并管线：

Kandinsky 2.1：

from diffusers import AutoPipelineForText2Image import torch pipeline = AutoPipelineForText2Image.from_pretrained("kandinsky-community/kandinsky-2-1", dtype=torch.float16) pipeline.enable_model_cpu_offload() prompt = "A alien cheeseburger creature eating itself, claymation, cinematic, moody lighting" negative_prompt = "low quality, bad quality" image = pipeline(prompt=prompt, negative_prompt=negative_prompt, prior_guidance_scale=1.0, guidance_scale=4.0, height=768, width=768).images[0] image

Kandinsky 2.2：

from diffusers import AutoPipelineForText2Image import torch pipeline = AutoPipelineForText2Image.from_pretrained("kandinsky-community/kandinsky-2-2-decoder", dtype=torch.float16) pipeline.enable_model_cpu_offload() prompt = "A alien cheeseburger creature eating itself, claymation, cinematic, moody lighting" negative_prompt = "low quality, bad quality" image = pipeline(prompt=prompt, negative_prompt=negative_prompt, prior_guidance_scale=1.0, guidance_scale=4.0, height=768, width=768).images[0] image

图像到图像（Image-to-image）

图生图需要把初始图像与文本提示词同时传给管线。先加载 prior 管线：

Kandinsky 2.1：

import torch from diffusers import KandinskyImg2ImgPipeline, KandinskyPriorPipeline prior_pipeline = KandinskyPriorPipeline.from_pretrained("kandinsky-community/kandinsky-2-1-prior", dtype=torch.float16, use_safetensors=True).to("cuda") # 或 "mps"、"xpu"、"cpu" pipeline = KandinskyImg2ImgPipeline.from_pretrained("kandinsky-community/kandinsky-2-1", dtype=torch.float16, use_safetensors=True).to("cuda")

Kandinsky 2.2：

import torch from diffusers import KandinskyV22Img2ImgPipeline, KandinskyPriorPipeline prior_pipeline = KandinskyPriorPipeline.from_pretrained("kandinsky-community/kandinsky-2-2-prior", dtype=torch.float16, use_safetensors=True).to("cuda") # 或 "mps"、"xpu"、"cpu" pipeline = KandinskyV22Img2ImgPipeline.from_pretrained("kandinsky-community/kandinsky-2-2-decoder", dtype=torch.float16, use_safetensors=True).to("cuda")

Kandinsky 3：不需要 prior，直接加载图生图管线：

from diffusers import Kandinsky3Img2ImgPipeline from diffusers.utils import load_image import torch pipeline = Kandinsky3Img2ImgPipeline.from_pretrained("kandinsky-community/kandinsky-3", variant="fp16", dtype=torch.float16) pipeline.enable_model_cpu_offload()

下载一张用于条件约束的图像（load_image来自diffusers.utils，返回 PIL 图像并自动转为 RGB）：

from diffusers.utils import load_image # 下载示例图像 url = "https://raw.githubusercontent.com/CompVis/stable-diffusion/main/assets/stable-samples/img2img/sketch-mountains-input.jpg" original_image = load_image(url) original_image = original_image.resize((768, 512))

用 prior 管线生成image_embeds与negative_image_embeds：

prompt = "A fantasy landscape, Cinematic lighting" negative_prompt = "low quality, bad quality" image_embeds, negative_image_embeds = prior_pipeline(prompt, negative_prompt).to_tuple()

把原始图像、提示词与嵌入全部传给管线生成新图像：

Kandinsky 2.1：strength=0.3表示保留较多原始图像结构（数值越大，去噪越彻底、与输入图差异越大）：

from diffusers.utils import make_image_grid image = pipeline(prompt, negative_prompt=negative_prompt, image=original_image, image_embeds=image_embeds, negative_image_embeds=negative_image_embeds, height=768, width=768, strength=0.3).images[0] make_image_grid([original_image.resize((512, 512)), image.resize((512, 512))], rows=1, cols=2)

Kandinsky 2.2：同样不接收prompt：

from diffusers.utils import make_image_grid image = pipeline(image=original_image, image_embeds=image_embeds, negative_image_embeds=negative_image_embeds, height=768, width=768, strength=0.3).images[0] make_image_grid([original_image.resize((512, 512)), image.resize((512, 512))], rows=1, cols=2)

Kandinsky 3：

image = pipeline(prompt, negative_prompt=negative_prompt, image=image, strength=0.75, num_inference_steps=25).images[0] image

同样存在端到端合并管线KandinskyImg2ImgCombinedPipeline与KandinskyV22Img2ImgCombinedPipeline，并可用AutoPipelineForImage2Image一键调用：

Kandinsky 2.1：

from diffusers import AutoPipelineForImage2Image from diffusers.utils import make_image_grid, load_image import torch pipeline = AutoPipelineForImage2Image.from_pretrained("kandinsky-community/kandinsky-2-1", dtype=torch.float16, use_safetensors=True) pipeline.enable_model_cpu_offload() prompt = "A fantasy landscape, Cinematic lighting" negative_prompt = "low quality, bad quality" url = "https://raw.githubusercontent.com/CompVis/stable-diffusion/main/assets/stable-samples/img2img/sketch-mountains-input.jpg" original_image = load_image(url) original_image.thumbnail((768, 768)) image = pipeline(prompt=prompt, negative_prompt=negative_prompt, image=original_image, strength=0.3).images[0] make_image_grid([original_image.resize((512, 512)), image.resize((512, 512))], rows=1, cols=2)

Kandinsky 2.2：

from diffusers import AutoPipelineForImage2Image from diffusers.utils import make_image_grid, load_image import torch pipeline = AutoPipelineForImage2Image.from_pretrained("kandinsky-community/kandinsky-2-2-decoder", dtype=torch.float16) pipeline.enable_model_cpu_offload() prompt = "A fantasy landscape, Cinematic lighting" negative_prompt = "low quality, bad quality" url = "https://raw.githubusercontent.com/CompVis/stable-diffusion/main/assets/stable-samples/img2img/sketch-mountains-input.jpg" original_image = load_image(url) original_image.thumbnail((768, 768)) image = pipeline(prompt=prompt, negative_prompt=negative_prompt, image=original_image, strength=0.3).images[0] make_image_grid([original_image.resize((512, 512)), image.resize((512, 512))], rows=1, cols=2)

图像修补（Inpainting）

⚠️ 重要变更：Kandinsky 模型使用白色像素（而非黑色像素）表示遮罩区域。若在KandinskyInpaintPipeline的生产环境中使用旧掩码，需要将其反转为白色：

# PIL 输入 import PIL.ImageOps mask = PIL.ImageOps.invert(mask) # PyTorch 与 NumPy 输入 mask = 1 - mask

Inpainting 需要三样东西：原始图像、标记待替换区域的掩码、以及描述修补内容的文本提示词。先加载 prior 管线：

Kandinsky 2.1：

from diffusers import KandinskyInpaintPipeline, KandinskyPriorPipeline from diffusers.utils import load_image, make_image_grid import torch import numpy as np from PIL import Image prior_pipeline = KandinskyPriorPipeline.from_pretrained("kandinsky-community/kandinsky-2-1-prior", dtype=torch.float16, use_safetensors=True).to("cuda") # 或 "mps"、"xpu"、"cpu" pipeline = KandinskyInpaintPipeline.from_pretrained("kandinsky-community/kandinsky-2-1-inpaint", dtype=torch.float16, use_safetensors=True).to("cuda")

Kandinsky 2.2：

from diffusers import KandinskyV22InpaintPipeline, KandinskyV22PriorPipeline from diffusers.utils import load_image, make_image_grid import torch import numpy as np from PIL import Image prior_pipeline = KandinskyV22PriorPipeline.from_pretrained("kandinsky-community/kandinsky-2-2-prior", dtype=torch.float16, use_safetensors=True).to("cuda") # 或 "mps"、"xpu"、"cpu" pipeline = KandinskyV22InpaintPipeline.from_pretrained("kandinsky-community/kandinsky-2-2-decoder-inpaint", dtype=torch.float16, use_safetensors=True).to("cuda")

加载初始图像并构造掩码（下面示例把猫头上方区域置 1，表示待修补位置）：

init_image = load_image("https://huggingface.co/datasets/hf-internal-testing/diffusers-images/resolve/main/kandinsky/cat.png") mask = np.zeros((768, 768), dtype=np.float32) # 遮住猫头上方的区域 mask[:250, 250:-250] = 1

用 prior 管线生成嵌入（prior_output是KandinskyPriorPipelineOutput，可直接用**解包）：

prompt = "a hat" prior_output = prior_pipeline(prompt)

把初始图像、掩码、提示词与嵌入传给 Inpaint 管线：

Kandinsky 2.1：num_inference_steps=150提供更充分的去噪迭代：

output_image = pipeline(prompt, image=init_image, mask_image=mask, **prior_output, height=768, width=768, num_inference_steps=150).images[0] mask = Image.fromarray((mask*255).astype('uint8'), 'L') make_image_grid([init_image, mask, output_image], rows=1, cols=3)

Kandinsky 2.2：

output_image = pipeline(image=init_image, mask_image=mask, **prior_output, height=768, width=768, num_inference_steps=150).images[0] mask = Image.fromarray((mask*255).astype('uint8'), 'L') make_image_grid([init_image, mask, output_image], rows=1, cols=3)

同样可以使用端到端的KandinskyInpaintCombinedPipeline/KandinskyV22InpaintCombinedPipeline，通过AutoPipelineForInpainting在底层串联 prior 与解码器：

Kandinsky 2.1：

import torch import numpy as np from PIL import Image from diffusers import AutoPipelineForInpainting from diffusers.utils import load_image, make_image_grid pipe = AutoPipelineForInpainting.from_pretrained("kandinsky-community/kandinsky-2-1-inpaint", dtype=torch.float16) pipe.enable_model_cpu_offload() init_image = load_image("https://huggingface.co/datasets/hf-internal-testing/diffusers-images/resolve/main/kandinsky/cat.png") mask = np.zeros((768, 768), dtype=np.float32) # 遮住猫头上方的区域 mask[:250, 250:-250] = 1 prompt = "a hat" output_image = pipe(prompt=prompt, image=init_image, mask_image=mask).images[0] mask = Image.fromarray((mask*255).astype('uint8'), 'L') make_image_grid([init_image, mask, output_image], rows=1, cols=3)

Kandinsky 2.2：

import torch import numpy as np from PIL import Image from diffusers import AutoPipelineForInpainting from diffusers.utils import load_image, make_image_grid pipe = AutoPipelineForInpainting.from_pretrained("kandinsky-community/kandinsky-2-2-decoder-inpaint", dtype=torch.float16) pipe.enable_model_cpu_offload() init_image = load_image("https://huggingface.co/datasets/hf-internal-testing/diffusers-images/resolve/main/kandinsky/cat.png") mask = np.zeros((768, 768), dtype=np.float32) # 遮住猫头上方的区域 mask[:250, 250:-250] = 1 prompt = "a hat" output_image = pipe(prompt=prompt, image=init_image, mask_image=mask).images[0] mask = Image.fromarray((mask*255).astype('uint8'), 'L') make_image_grid([init_image, mask, output_image], rows=1, cols=3)

隐空间插值（Interpolation）

插值允许你在图像嵌入与文本嵌入之间的隐空间中进行探索，能够直观看到 prior 模型的中间产物。加载 prior 管线与两张想插值的图像：

Kandinsky 2.1：

from diffusers import KandinskyPriorPipeline, KandinskyPipeline from diffusers.utils import load_image, make_image_grid import torch prior_pipeline = KandinskyPriorPipeline.from_pretrained("kandinsky-community/kandinsky-2-1-prior", dtype=torch.float16, use_safetensors=True).to("cuda") # 或 "mps"、"xpu"、"cpu" img_1 = load_image("https://huggingface.co/datasets/hf-internal-testing/diffusers-images/resolve/main/kandinsky/cat.png") img_2 = load_image("https://huggingface.co/datasets/hf-internal-testing/diffusers-images/resolve/main/kandinsky/starry_night.jpeg") make_image_grid([img_1.resize((512, 512)), img_2.resize((512, 512))], rows=1, cols=2)

Kandinsky 2.2：

from diffusers import KandinskyV22PriorPipeline, KandinskyV22Pipeline from diffusers.utils import load_image, make_image_grid import torch prior_pipeline = KandinskyV22PriorPipeline.from_pretrained("kandinsky-community/kandinsky-2-2-prior", dtype=torch.float16, use_safetensors=True).to("cuda") # 或 "mps"、"xpu"、"cpu" img_1 = load_image("https://huggingface.co/datasets/hf-internal-testing/diffusers-images/resolve/main/kandinsky/cat.png") img_2 = load_image("https://huggingface.co/datasets/hf-internal-testing/diffusers-images/resolve/main/kandinsky/starry_night.jpeg") make_image_grid([img_1.resize((512, 512)), img_2.resize((512, 512))], rows=1, cols=2)

指定要插值的文本或图像，并为每一项设置权重（权重之和不必为 1，但会影响融合比例，建议多尝试不同组合）：

images_texts = ["a cat", img_1, img_2] weights = [0.3, 0.3, 0.4]

调用interpolate方法生成嵌入，再传给解码管线生成图像：

Kandinsky 2.1：

# prompt 可以留空 prompt = "" prior_out = prior_pipeline.interpolate(images_texts, weights) pipeline = KandinskyPipeline.from_pretrained("kandinsky-community/kandinsky-2-1", dtype=torch.float16, use_safetensors=True).to("cuda") # 或 "mps"、"xpu"、"cpu" image = pipeline(prompt, **prior_out, height=768, width=768).images[0] image

Kandinsky 2.2：

# prompt 可以留空 prompt = "" prior_out = prior_pipeline.interpolate(images_texts, weights) pipeline = KandinskyV22Pipeline.from_pretrained("kandinsky-community/kandinsky-2-2-decoder", dtype=torch.float16, use_safetensors=True).to("cuda") # 或 "mps"、"xpu"、"cpu" image = pipeline(prompt, **prior_out, height=768, width=768).images[0] image

从 pipeline_kandinsky_prior.py 的实现可以看到interpolate的底层逻辑：它遍历images_and_prompts列表，字符串经 prior 扩散生成image_embeds，PIL 图像或张量则直接经image_processor预处理后由CLIPVisionModelWithProjection编码，最后把所有嵌入按权重加权求和（image_emb * weight后sum）得到融合嵌入；同时以空字符串为条件生成负嵌入（zero_image_emb），返回KandinskyPriorPipelineOutput。这也解释了为何插值完成后解码阶段可以把prompt留空。

Kandinsky 2.2 ControlNet：深度图条件生成

⚠️ 注意：ControlNet 仅支持 Kandinsky 2.2！

ControlNet 允许用深度图、边缘检测等额外输入来约束大型预训练扩散模型。例如，给 Kandinsky 2.2 提供深度图，模型就能理解并保留深度图像的结构信息。

首先加载一张图像并提取其深度图：

from diffusers.utils import load_image img = load_image( "https://huggingface.co/datasets/hf-internal-testing/diffusers-images/resolve/main/kandinskyv22/cat.png" ).resize((768, 768)) img

用 🤗 Transformers 的depth-estimationpipeline 处理图像、提取深度图（make_hint把深度图复制成三通道并归一化到 0～1，再转为 CHW 格式）：

import torch import numpy as np from transformers import pipeline def make_hint(image, depth_estimator): image = depth_estimator(image)["depth"] image = np.array(image) image = image[:, :, None] image = np.concatenate([image, image, image], axis=2) detected_map = torch.from_numpy(image).float() / 255.0 hint = detected_map.permute(2, 0, 1) return hint depth_estimator = pipeline("depth-estimation") hint = make_hint(img, depth_estimator).unsqueeze(0).half().to("cuda") # 或 "mps"、"xpu"、"cpu"

深度条件文生图

加载 prior 管线与KandinskyV22ControlnetPipeline：

from diffusers import KandinskyV22PriorPipeline, KandinskyV22ControlnetPipeline prior_pipeline = KandinskyV22PriorPipeline.from_pretrained( "kandinsky-community/kandinsky-2-2-prior", dtype=torch.float16, use_safetensors=True ).to("cuda") # 或 "mps"、"xpu"、"cpu" pipeline = KandinskyV22ControlnetPipeline.from_pretrained( "kandinsky-community/kandinsky-2-2-controlnet-depth", dtype=torch.float16 ).to("cuda")

从提示词与负提示词生成图像嵌入（这里使用固定随机种子保证结果可复现）：

prompt = "A robot, 4k photo" negative_prior_prompt = "lowres, text, error, cropped, worst quality, low quality, jpeg artifacts, ugly, duplicate, morbid, mutilated, out of frame, extra fingers, mutated hands, poorly drawn hands, poorly drawn face, mutation, deformed, blurry, dehydrated, bad anatomy, bad proportions, extra limbs, cloned face, disfigured, gross proportions, malformed limbs, missing arms, missing legs, extra arms, extra legs, fused fingers, too many fingers, long neck, username, watermark, signature" generator = torch.Generator(device="cuda").manual_seed(43) # 或 "mps"、"xpu"、"cpu" image_emb, zero_image_emb = prior_pipeline( prompt=prompt, negative_prompt=negative_prior_prompt, generator=generator ).to_tuple()

最后把图像嵌入与深度图一起传给KandinskyV22ControlnetPipeline生成图像：

image = pipeline(image_embeds=image_emb, negative_image_embeds=zero_image_emb, hint=hint, num_inference_steps=50, generator=generator, height=768, width=768).images[0] image

深度条件图生图

图生图 + ControlNet 需要两个管线：

• KandinskyV22PriorEmb2EmbPipeline：根据文本提示词与初始图像生成图像嵌入；
• KandinskyV22ControlnetImg2ImgPipeline：根据初始图像与图像嵌入生成新图像。

先用depth-estimation处理初始猫图得到深度图：

import torch import numpy as np from diffusers import KandinskyV22PriorEmb2EmbPipeline, KandinskyV22ControlnetImg2ImgPipeline from diffusers.utils import load_image from transformers import pipeline img = load_image( "https://huggingface.co/datasets/hf-internal-testing/diffusers-images/resolve/main/kandinskyv22/cat.png" ).resize((768, 768)) def make_hint(image, depth_estimator): image = depth_estimator(image)["depth"] image = np.array(image) image = image[:, :, None] image = np.concatenate([image, image, image], axis=2) detected_map = torch.from_numpy(image).float() / 255.0 hint = detected_map.permute(2, 0, 1) return hint depth_estimator = pipeline("depth-estimation") hint = make_hint(img, depth_estimator).unsqueeze(0).half().to("cuda") # 或 "mps"、"xpu"、"cpu"

加载两个管线：

prior_pipeline = KandinskyV22PriorEmb2EmbPipeline.from_pretrained( "kandinsky-community/kandinsky-2-2-prior", dtype=torch.float16, use_safetensors=True ).to("cuda") # 或 "mps"、"xpu"、"cpu" pipeline = KandinskyV22ControlnetImg2ImgPipeline.from_pretrained( "kandinsky-community/kandinsky-2-2-controlnet-depth", dtype=torch.float16 ).to("cuda")

把文本提示词与初始图像传给 prior 管线生成嵌入（strength控制原始图像被扰动/覆盖的程度，正嵌入与负嵌入可分别设置）：

prompt = "A robot, 4k photo" negative_prior_prompt = "lowres, text, error, cropped, worst quality, low quality, jpeg artifacts, ugly, duplicate, morbid, mutilated, out of frame, extra fingers, mutated hands, poorly drawn hands, poorly drawn face, mutation, deformed, blurry, dehydrated, bad anatomy, bad proportions, extra limbs, cloned face, disfigured, gross proportions, malformed limbs, missing arms, missing legs, extra arms, extra legs, fused fingers, too many fingers, long neck, username, watermark, signature" generator = torch.Generator(device="cuda").manual_seed(43) # 或 "mps"、"xpu"、"cpu" img_emb = prior_pipeline(prompt=prompt, image=img, strength=0.85, generator=generator) negative_emb = prior_pipeline(prompt=negative_prior_prompt, image=img, strength=1, generator=generator)

运行KandinskyV22ControlnetImg2ImgPipeline从初始图像与嵌入生成新图像：

image = pipeline(image=img, strength=0.5, image_embeds=img_emb.image_embeds, negative_image_embeds=negative_emb.image_embeds, hint=hint, num_inference_steps=50, generator=generator, height=768, width=768).images[0] make_image_grid([img.resize((512, 512)), image.resize((512, 512))], rows=1, cols=2)

推理优化技巧

Kandinsky 的独特之处在于需要 prior 管线生成映射、再由第二个管线把隐变量解码为图像。优化应聚焦在第二个管线，因为大部分计算量都集中在这里。以下是提升推理性能的几条建议：

1. 启用 xFormers（PyTorch < 2.0 时）：

from diffusers import DiffusionPipeline import torch pipe = DiffusionPipeline.from_pretrained("kandinsky-community/kandinsky-2-1", dtype=torch.float16) + pipe.enable_xformers_memory_efficient_attention()

2. 使用torch.compile（PyTorch >= 2.0 时），自动利用缩放点积注意力（SDPA）：

pipe.unet.to(memory_format=torch.channels_last) + pipe.unet = torch.compile(pipe.unet, mode="reduce-overhead", fullgraph=True)

这等价于显式把注意力处理器设置为AttnAddedKVProcessor2_0：

from diffusers.models.attention_processor import AttnAddedKVProcessor2_0 pipe.unet.set_attn_processor(AttnAddedKVProcessor2_0())

3. CPU offload，避免显存不足（OOM）：

from diffusers import DiffusionPipeline import torch pipe = DiffusionPipeline.from_pretrained("kandinsky-community/kandinsky-2-1", dtype=torch.float16) + pipe.enable_model_cpu_offload()

4. 更换调度器：默认文生图管线使用DDIMScheduler，可以替换为DDPMScheduler等，观察推理速度与图像质量之间的权衡：

from diffusers import DDPMScheduler from diffusers import DiffusionPipeline scheduler = DDPMScheduler.from_pretrained("kandinsky-community/kandinsky-2-1", subfolder="ddpm_scheduler") pipe = DiffusionPipeline.from_pretrained("kandinsky-community/kandinsky-2-1", scheduler=scheduler, dtype=torch.float16, use_safetensors=True).to("cuda") # 或 "mps"、"xpu"、"cpu"

需要说明的是，在 pipeline_kandinsky_prior.py 中，prior 管线将prior排除在 CPU offload 之外（_exclude_from_cpu_offload = ["prior"]），且其 offload 顺序为text_encoder->prior；而解码管线 pipeline_kandinsky.py 的 offload 顺序为text_encoder->unet->movq，说明两个阶段的显存压力点不同，offload 策略也应区别对待。

相关管线速查表

Kandinsky 系列在仓库中的实现位于 src/diffusers/pipelines/kandinsky（2.1）、src/diffusers/pipelines/kandinsky2_2（2.2）与 src/diffusers/pipelines/kandinsky3（3），对应测试覆盖在 tests/pipelines/kandinsky（含test_kandinsky.py、test_kandinsky_prior.py、test_kandinsky_combined.py、test_kandinsky_img2img.py、test_kandinsky_inpaint.py），社区脚本还包括 examples/community/unclip_image_interpolation.py 与 examples/community/unclip_text_interpolation.py 等可参考用例。关于调度器速度与质量的权衡，可进一步阅读 Schedulers 指南；关于如何在多条管线间复用组件，可参考 复用跨管线组件 一节。

【免费下载链接】diffusers🤗 Diffusers: State-of-the-art diffusion models for image, video, and audio generation in PyTorch.项目地址: https://gitcode.com/GitHub_Trending/di/diffusers