企业官网建设流程全解析

目录

Qwen3-VL-8B

1. FP8（vLLM / SGLang /transformers，NVIDIA 显卡首选）

环境安装：

server封装：

客户端请求例子：

2. GGUF（llama.cpp/ Ollama / LM Studio，CPU、Mac、低显存显卡）

📌 社区高质量量化（备选）

GGUF 更多档位（bartowski /unsloth，档位极多 IQ2~Q6_K）

AWQ 4bit（vLLM，追求速度显存平衡）

MLX（Apple Silicon，Mac M 系列）

判断图像相似度

Qwen3-VL-8B

• cyankiwi 版：仓库cyankiwi/Qwen3-VL-8B-Instruct-AWQ-4bit，是 Hugging Face 上较活跃的 AWQ 4-bit 量化版本之一。-1

• MLliu6 版：仓库MLliu6/Qwen3-VL-8B-Instruct-AWQ-W4A16，采用 AWQ W4A16 量化，并明确排除了视觉模块（.*visual.*）的量化，有助于保留看图能力。

1. FP8（vLLM / SGLang /transformers，NVIDIA 显卡首选）

国内镜像，速度最好，官方 FP8，几乎无损，显存32个G

• Instruct 版：Qwen3-VL-8B-Instruct-FP8 - ModelScope 魔搭社区
• Thinking 版（强推理）：Qwen3-VL-8B-Thinking-FP8 - ModelScope 魔搭社区ModelScope

适合：RTX40 系、50 系、A10、A100；vLLM 直接加载，不用自己转量化

环境安装：

pip install qwen-vl-utils Install the latest version of vLLM 'vllm>=0.11.0'

server封装：

# -*- coding: utf-8 -*- import os os.environ["TORCH_COMPILE_DISABLE"] = "1" os.environ['VLLM_WORKER_MULTIPROC_METHOD'] = 'spawn' import time import uuid import asyncio import tempfile from contextlib import asynccontextmanager import torch from fastapi import FastAPI, UploadFile, File, Form, HTTPException from fastapi.responses import JSONResponse from pydantic import BaseModel, Field from qwen_vl_utils import process_vision_info from modelscope import AutoProcessor from vllm import LLM, SamplingParams # ==================== 配置 ==================== CHECKPOINT_PATH = os.environ.get( "QWEN_VL_CKPT", "/data/feature/0lbg/models/Qwen_Qwen3-VL-8B-Thinking/" ) GPU_MEMORY_UTILIZATION = float(os.environ.get("QWEN_VL_GPU_MEM", "0.70")) # 短句问答优化参数 MAX_OUTPUT_TOKENS = 64 # 输出上限 VISION_TOKEN_BUDGET = 512 # 视觉 token 预算 MAX_MODEL_LEN = 1024 # 模型上下文上限（包含输入+输出，留足余量） MAX_NUM_BATCHED_TOKENS = 1024 # 预填充批大小 MAX_NUM_SEQS = 16 # 并发序列上限 # 是否启用 thinking（Thinking 模型设为 False 可大幅提速） ENABLE_THINKING = False DEFAULT_QUESTION = ( "请仔细看图，直接给出简短答案，不要输出思考过程：\n" "1. 图中有人吗？\n" "2. 如果有，能看清头、躯干、四肢吗？\n" "3. 这个人有没有拿什么东西？如果拿了，是什么？\n" "请分点简洁回答。" ) # ============================================= # 全局资源 app_state = {} def prepare_inputs_for_vllm(messages, processor): # 关闭 thinking 加速（若支持） try: text = processor.apply_chat_template( messages, tokenize=False, add_generation_prompt=True, enable_thinking=ENABLE_THINKING, ) except TypeError: # 老版本 processor 不支持 enable_thinking text = processor.apply_chat_template( messages, tokenize=False, add_generation_prompt=True ) image_inputs, video_inputs, video_kwargs = process_vision_info( messages, image_patch_size=processor.image_processor.patch_size, return_video_kwargs=True, return_video_metadata=True, ) mm_data = {} if image_inputs is not None: mm_data["image"] = image_inputs if video_inputs is not None: mm_data["video"] = video_inputs return { "prompt": text, "multi_modal_data": mm_data, "mm_processor_kwargs": video_kwargs, } def build_messages(image_input, question: str): return [ { "role": "user", "content": [ { "type": "image", "image": image_input, "max_pixels": VISION_TOKEN_BUDGET * 32 * 32, }, {"type": "text", "text": question}, ], } ] @asynccontextmanager async def lifespan(app: FastAPI): print(f"[startup] loading processor from {CHECKPOINT_PATH} ...") processor = AutoProcessor.from_pretrained(CHECKPOINT_PATH) print("[startup] loading vLLM engine ...") llm = LLM( model=CHECKPOINT_PATH, trust_remote_code=True, gpu_memory_utilization=GPU_MEMORY_UTILIZATION, enforce_eager=False, tensor_parallel_size=torch.cuda.device_count(), seed=0, max_model_len=MAX_MODEL_LEN, max_num_batched_tokens=MAX_NUM_BATCHED_TOKENS, max_num_seqs=MAX_NUM_SEQS, limit_mm_per_prompt={"image": 1, "video": 0}, # 单图任务，节省显存 ) # 构建 stop token ids stop_token_ids = [] tok = processor.tokenizer if tok.eos_token_id is not None: stop_token_ids.append(tok.eos_token_id) for special in ("<|im_end|>", "<|endoftext|>"): tid = tok.convert_tokens_to_ids(special) if isinstance(tid, int) and tid >= 0 and tid != tok.unk_token_id: if tid not in stop_token_ids: stop_token_ids.append(tid) sampling_params = SamplingParams( temperature=0, max_tokens=MAX_OUTPUT_TOKENS, top_k=-1, stop_token_ids=stop_token_ids, ) app_state["processor"] = processor app_state["llm"] = llm app_state["sampling_params"] = sampling_params app_state["lock"] = asyncio.Lock() print(f"[startup] ready. max_output_tokens={MAX_OUTPUT_TOKENS}, " f"vision_budget={VISION_TOKEN_BUDGET}, " f"enable_thinking={ENABLE_THINKING}, stop_ids={stop_token_ids}") yield app_state.clear() app = FastAPI(title="Qwen3-VL Inference Server", lifespan=lifespan) # ==================== 请求 / 响应模型 ==================== class InferByPathRequest(BaseModel): image_path: str = Field(..., description="服务器本地图片绝对路径") question: str = Field(DEFAULT_QUESTION, description="提问文本") class InferResponse(BaseModel): text: str elapsed_sec: float # ====================================================== def _run_inference(image_input, question: str) -> str: processor = app_state["processor"] llm = app_state["llm"] sampling_params = app_state["sampling_params"] messages = build_messages(image_input, question) inputs = [prepare_inputs_for_vllm(messages, processor)] outputs = llm.generate(inputs, sampling_params=sampling_params) return outputs[0].outputs[0].text @app.get("/health") async def health(): return {"status": "ok", "model_loaded": "llm" in app_state} @app.post("/infer_path", response_model=InferResponse) async def infer_path(req: InferByPathRequest): """用服务器本地图片路径推理。""" if not os.path.isfile(req.image_path): raise HTTPException(status_code=400, detail=f"image not found: {req.image_path}") t0 = time.time() async with app_state["lock"]: text = await asyncio.to_thread(_run_inference, req.image_path, req.question) elapsed = time.time() - t0 return InferResponse(text=text, elapsed_sec=round(elapsed, 3)) @app.post("/infer_upload", response_model=InferResponse) async def infer_upload( file: UploadFile = File(..., description="上传的图片文件"), question: str = Form(DEFAULT_QUESTION), ): """上传图片文件推理。""" if file.content_type and not file.content_type.startswith("image/"): raise HTTPException(status_code=400, detail=f"not an image: {file.content_type}") suffix = os.path.splitext(file.filename or "")[1] or ".jpg" tmp_path = os.path.join(tempfile.gettempdir(), f"qwen_vl_{uuid.uuid4().hex}{suffix}") try: with open(tmp_path, "wb") as f: f.write(await file.read()) t0 = time.time() async with app_state["lock"]: text = await asyncio.to_thread(_run_inference, tmp_path, question) elapsed = time.time() - t0 return InferResponse(text=text, elapsed_sec=round(elapsed, 3)) finally: if os.path.exists(tmp_path): os.remove(tmp_path) @app.exception_handler(Exception) async def global_exc_handler(request, exc): return JSONResponse(status_code=500, content={"detail": f"{type(exc).__name__}: {exc}"}) if __name__ == "__main__": import uvicorn uvicorn.run( app, host="0.0.0.0", port=7200, workers=1, log_level="info", )

客户端请求例子：

import requests url = "http://127.0.0.1:7200/infer_path" payload = { "image_path": "/data/feature/0lbg/project/qianwen_vl/data/guangong.png", "question": "图中有人吗？有的话能看到头、躯干、四肢吗？此人手里拿了什么？", } payload1 = { "image_path": "/data/feature/0lbg/project/qianwen_vl/data/dizuo.png", "question": "图中有人吗？有的话能看到头、躯干、四肢吗？此人手里拿了什么？或者脚底下有什么东西", } resp = requests.post(url, json=payload, timeout=120) resp.raise_for_status() data = resp.json() print("模型回答：") print(data["text"]) print(f"耗时：{data['elapsed_sec']}s")

2. GGUF（llama.cpp/ Ollama / LM Studio，CPU、Mac、低显存显卡）

HF 官方 GGUF 仓库： https://huggingface.co/Qwen/Qwen3-VL-8B-Instruct-GGUF

• 内置：FP16 / Q8_0 /Q4_K_M（最均衡，约 5GB）
• 注意：Qwen3-VL 的 GGUF 是LLM 权重 + 独立 mmproj 视觉编码器两个文件，不要漏 mmproj
• 国内加速镜像：Qwen3-VL-8B-Instruct-GGUF - 开源模型 - Qwen - OpenCSG - image text to text - Gguf

📌 社区高质量量化（备选）

GGUF 更多档位（bartowski /unsloth，档位极多 IQ2~Q6_K）

• unsloth：https://huggingface.co/unsloth/Qwen3-VL-8B-Instruct-GGUF
• bartowski：https://huggingface.co/bartowski/Qwen_Qwen3-VL-8B-Instruct-GGUF

AWQ 4bit（vLLM，追求速度显存平衡）

cyankiwi 这个做的比较稳：https://huggingface.co/cyankiwi/Qwen3-VL-8B-Instruct-AWQ-4bit

MLX（Apple Silicon，Mac M 系列）

lmstudio-community：https://huggingface.co/lmstudio-community/Qwen3-VL-8B-Instruct-MLX-4bit

判断图像相似度

pc上推理一次需要8秒。

import time from modelscope import AutoModel import torch import torch.nn.functional as F from transformers.utils.versions import require_version from multiprocessing import freeze_support, Process require_version( "transformers<4.52.0", "The remote code has some issues with transformers>=4.52.0, please downgrade: pip install transformers==4.51.3" ) if __name__ == '__main__': freeze_support() # 模型 gme = AutoModel.from_pretrained( "iic/gme-Qwen2-VL-2B-Instruct", torch_dtype="float16", device_map="cuda", trust_remote_code=True ) for i in range(20): start=time.time() # 两个图像 URL images = [ r"D:\project\yolov12-main_new\data\0_kong\1120_1151_1.jpg", r"D:\project\yolov12-main_new\data\0_kong\1120_1151_0.jpg", ] # --- 获取图像 Embedding --- emb = gme.get_image_embeddings(images=images) # shape: (2, D) # --- Cosine Similarity --- # F.cosine_similarity 每行与每行比对 sim = F.cosine_similarity(emb[0].unsqueeze(0), emb[1].unsqueeze(0)) print("两张图片相似度：", float(sim),f'time:{time.time()-start}')