Align-DS-V是DeepSeek-R1-Distill-Llama-8B中的一个实验性视觉语言模型，由PKU-Alignment团队和香港科技大学共同开发，重点是通过全模态对齐增强推理能力。

性能

VQA 任务

作为一种视觉语言模型，Align-DS-V 在各种 VQA 聊天和推理任务中表现出色。

	Align-DS-V (8B)	GPT-4o
MathVista	27.0	30.4
MathVision	63.8	62.2
LLaVA-Bench-COCO	105.3	104.9
A-OKVQA	83.7	87.9

数学任务

此外，我们惊喜地发现，将DeepSeek-R1-Distill-Llama-8B扩展到视觉模态的Align-DS-R1也在原有文本模态推理能力的基础上取得了显著提高。

	Align-DS-V (8B)	DeepSeek-R1-Distill-Llama-8B
ARC (5-shot)	34.2	32.7
ARC-Challenge (5-shot)	40.5	21.4
BigBench-Hard (3-shot)	73.4	72.2

快速启动

我们将演示如何使用 Align-DS-V 解决下图所示的数学问题。

import requests
from PIL import Image

import torch
from transformers import AutoProcessor, LlavaForConditionalGeneration

model_id = "PKU-Alignment/Align-DS-V"
model = LlavaForConditionalGeneration.from_pretrained(
    model_id, 
    torch_dtype=torch.float16, 
    low_cpu_mem_usage=True, 
).to(0)

processor = AutoProcessor.from_pretrained(model_id)

# Define a chat history and use `apply_chat_template` to get correctly formatted prompt
# Each value in "content" has to be a list of dicts with types ("text", "image") 
conversation = [
    {

      "role": "user",
      "content": [
          {"type": "text", "text": "What is the result of this problem?"},
          {"type": "image"},
        ],
    },
]
prompt = processor.apply_chat_template(conversation, add_generation_prompt=True)

image_file = "./assets/demo.jpg" # in this repo
raw_image = Image.open(image_file)
inputs = processor(images=raw_image, text=prompt, return_tensors='pt').to(0, torch.float16)

output = model.generate(**inputs, max_new_tokens=4096, do_sample=False)
print(processor.decode(output[0], skip_special_tokens=True))

# <think>To solve the problem, I will first interpret the image to understand what
# mathematical operation is being represented. Then, I will perform the calculation
# based on the numbers provided in the image and confirm the result. The image shows
# a chalkboard with the equation \(18 + 23 = 41\) written on it. The numbers 18 and
# 23 are in light blue, and the result 41 is in light green. The equation \(18 + 23 = 41\)
# is presented on the chalkboard. To solve this, I will add the two numbers on the
# left side of the equation: 18 and 23. Adding these together, \(18 + 23\), I calculate
# that the sum is 41. This matches the number on the right side of the equation,
# confirming its correctness.</think>41

实验重现

引用

Align-DS-V 的重现脚本将在 align-anything 软件库中发布。

如果您发现该版本库中的模型或代码有用，请引用该版本库😊。

@inproceedings{ji2024align,
  title={Align Anything: Training All-Modality Models to Follow Instructions with Language Feedback},
  author={Jiaming Ji and Jiayi Zhou and Hantao Lou and Boyuan Chen and Donghai Hong and Xuyao Wang and Wenqi Chen and Kaile Wang and Rui Pan and Jiahao Li and Mohan Wang and Josef Dai and Tianyi Qiu and Hua Xu and Dong Li and Weipeng Chen and Jun Song and Bo Zheng and Yaodong Yang},
  year={2024},
  url={https://arxiv.org/abs/2412.15838}
}