Home>Programming related>Other source code
Download
InternVL

InternVL Family: Closing the Gap to Commercial Multimodal Models with Open-Source Suites —— A Pioneering Open-Source Alternative to GPT-4o

379689418-930e6814-8a9f-43e1-a284-118a5732daa4.png

1730545950913244.png

News

  • 2024/10/21: We release the Mini-InternVL series. These models achieve impressive performance with minimal size: the 4B model achieves 90% of the performance with just 5% of the model size. For more details, please check our project page and document.

  • 2024/08/01: The Chartmimic team evaluated the InternVL2 series models on their benchmark. The InternVL2-26B and 76B models achieved the top two performances among open-source models, with the InternVL2 76B model surpassing GeminiProVision and exhibiting comparable results to Claude-3-opus.

  • 2024/08/01: InternVL2-Pro achieved the SOTA performance among open-source models on the CharXiv dataset, surpassing many closed-source models such as GPT-4V, Gemini 1.5 Flash, and Claude 3 Sonnet.

  • 2024/07/24: The MLVU team evaluated InternVL-1.5 on their benchmark. The average performance on the multiple-choice task was 50.4%, while the performance on the generative tasks was 4.02. The performance on the multiple-choice task ranked #1 among all open-source MLLMs.

  • 2024/07/18: ?? InternVL2-40B achieved SOTA performance among open-source models on the Video-MME dataset, scoring 61.2 when inputting 16 frames and 64.4 when inputting 32 frames. It significantly outperforms other open-source models and is the closest open-source model to GPT-4o mini.

  • 2024/07/18: ? InternVL2-Pro achieved the SOTA performance on the DocVQA and InfoVQA benchmarks.

  • 2024/07/04: ? We release the InternVL2 series. InternVL2-Pro achieved a 62.0% accuracy on the MMMU benchmark, matching the performance of leading closed-source commercial models like GPT-4o. The free API of this model can be applied by filling (application form) / (申请表). Other models are available at HF link.

  • 2024/06/19: We propose Needle In A Multimodal Haystack (MM-NIAH), the first benchmark designed to systematically evaluate the capability of existing MLLMs to comprehend long multimodal documents.

  • 2024/05/30: We release ShareGPT-4o, a large-scale dataset that we plan to open-source with 200K images, 10K videos, and 10K audios with detailed descriptions.

  • 2024/05/28: Thanks to the lmdeploy team for providing AWQ quantization support. The 4-bit model is available at OpenGVLab/InternVL-Chat-V1-5-AWQ.

  • 2024/05/13: InternVL 1.0 can now be used as the text encoder for diffusion models to support multilingual generation natively in over 110 languages worldwide. See MuLan for more details.

  • 2024/04/18: InternVL-Chat-V1-5 has been released at HF link, approaching the performance of GPT-4V and Gemini Pro on various benchmarks like MMMU, DocVQA, ChartQA, MathVista, etc.

  • 2024/02/27: InternVL is accepted by CVPR 2024 (Oral)! ?

  • 2024/02/21: InternVL-Chat-V1-2-Plus achieved SOTA performance on MathVista (59.9), MMBench (83.8), and MMVP (58.7). See our blog for more details.

  • 2024/02/12: InternVL-Chat-V1-2 has been released. It achieves 51.6 on MMMU val and 82.3 on MMBench test. For more details, please refer to our blog and SFT data. The model is now available on HuggingFace, and both training / evaluation data and scripts are open-sourced.

  • 2024/01/24: InternVL-Chat-V1-1 is released, it supports Chinese and has stronger OCR capability, see here.

  • 2024/01/16: We release our customized mmcv/mmsegmentation/mmdetection code, integrated with DeepSpeed, which can be used for training large-scale detection and segmentation models.

TODO List

  • Support vLLM and Ollama

  • Rebuild documents using readthedocs

  • Support fine-tuning different LLMs with LoRA

  • Support video and PDF input in online demo

  • Release InternVL2 with VisionLLMv2 integration

  • Release requirements.txt for InternVL2

  • Release training / evaluation code for InternVL2 series

  • Release Streamlit web UI for InternVL1.5 and InternVL2

Documents

  • Get Started

    • Installation: [Environment]  [requirements.txt]

    • Evaluation Data Preparation: [InternVL Evaluation]

    • Chat Data Format: [Meta File]  [Pure Text]  [Single-Image]  [Multi-Image]    [Video]

    • InternVL-Chat API: [InternVL2-Pro]

    • Local Chat Demo: [Streamlit Demo]  [Gradio Demo]  [LMDeploy Demo]

    • Tutorials: [Enhancing InternVL2 on COCO Caption Using LoRA Fine-Tuning]

  • InternVL Family

    • InternVL 2.0: [Introduction]  [Quick Start]  [Finetune]  [Evaluation]  [Deployment]

    • InternVL 1.5: [Introduction]  [Quick Start]  [Finetune]  [Evaluation]  [Deployment]

    • InternVL 1.2: [Introduction]  [Quick Start]  [Finetune]  [Evaluation]

    • InternVL 1.1: [Introduction]  [Quick Start]  [Evaluation]

    • InternVL 1.0: [Classification]  [CLIP-Benchmark]  [Segmentation]  [InternVL-Chat-LLaVA]  [InternVL-G]

Compared with SOTA VLLMs

1730545963285305.png

Model Zoo

Multimodal Large Language Model (InternVL 2.0)

Model NameVision PartLanguage PartHF LinkMS LinkDocument
InternVL2‑1BInternViT‑300M‑448pxQwen2‑0.5B‑Instruct? link? link? doc
InternVL2‑2BInternViT‑300M‑448pxinternlm2‑chat‑1‑8b? link? link? doc
InternVL2‑4BInternViT‑300M‑448pxPhi‑3‑mini‑128k‑instruct? link? link? doc
InternVL2‑8BInternViT‑300M‑448pxinternlm2_5‑7b‑chat? link? link? doc
InternVL2‑26BInternViT‑6B‑448px‑V1‑5internlm2‑chat‑20b? link? link? doc
InternVL2‑40BInternViT‑6B‑448px‑V1‑5Nous‑Hermes‑2‑Yi‑34B? link? link? doc
InternVL2-Llama3-76BInternViT‑6B‑448px‑V1‑5Hermes‑2‑Theta‑
Llama‑3‑70B		? link? link? doc

InternVL2-Pro API

We welcome everyone to use our API for research. For better management, please submit (application form) / (申请表) to obtain free API access.

Multimodal Large Language Model (InternVL 1.0-1.5)

ModelDateHF LinkMS LinkNote
Mini‑InternVL‑Chat‑4B‑V1‑52024.05.28? link? link?? 16% of the model size, 90% of the performance
Mini‑InternVL‑Chat‑2B‑V1‑52024.05.19? link? link? 8% of the model size, 80% of the performance
InternVL‑Chat‑V1‑52024.04.18? link? linksupport 4K image; super strong OCR; Approaching the performance of GPT-4V and Gemini Pro on various benchmarks like MMMU, DocVQA, ChartQA, MathVista, etc.
InternVL‑Chat‑V1‑2‑Plus2024.02.21? link? linkmore SFT data and stronger
InternVL‑Chat‑V1‑22024.02.11? link? linkscaling up LLM to 34B
InternVL‑Chat‑V1‑12024.01.24? link? linksupport Chinese and stronger OCR
InternVL‑Chat‑19B2023.12.25? link? linkEnglish multimodal dialogue
InternVL‑Chat‑13B2023.12.25? link? linkEnglish multimodal dialogue

Vision Foundation Model (InternVL 1.0-1.5)

ModelDateHF LinkMS LinkNote
InternViT‑300M‑448px2024.05.25? link? linkdistilled small vision foundation model with 300M parameters (?new)
InternViT‑6B‑448px‑V1‑52024.04.20? link? linksupport dynamic resolution and super strong OCR feature extraction capability by incremental pre-training (?new)
InternViT‑6B‑448px‑V1‑22024.02.11? link? linksupport 448 resolution by incremental pre-training
InternViT‑6B‑448px‑V1‑02024.01.30? link? linksupport 448 resolution by incremental pre-training
InternViT‑6B‑224px2023.12.22? link? linkthe first version of InternViT-6B, extracted from InternVL‑14B‑224px

Vision-Language Foundation Model (InternVL 1.0)

ModelDateHF LinkMS LinkNote
InternVL‑14B‑224px2023.12.22? link? linkvision-language foundation model, InternViT-6B + QLLaMA, can be used for image-text retrieval like CLIP

What can InternVL do?

Visual Perception (click to expand)

  • Linear-Probe Image Classification [see details]

    ViT-22B uses the private JFT-3B dataset.

    method#paramIN-1KIN-ReaLIN-V2IN-AIN-RIN-Sketch
    OpenCLIP-G1.8B86.289.477.263.887.866.4
    DINOv2-g1.1B86.589.678.475.978.862.5
    EVA-01-CLIP-g1.1B86.589.377.470.587.763.1
    MAWS-ViT-6.5B6.5B87.8-----
    ViT-22B*21.7B89.590.983.283.887.4-
    InternViT-6B (ours)5.9B88.290.479.977.589.869.1

  • Semantic Segmentation [see details]

    methoddecoder#param (train/total)crop sizemIoU
    OpenCLIP-G (frozen)Linear0.3M / 1.8B51239.3
    ViT-22B (frozen)Linear0.9M / 21.7B50434.6
    InternViT-6B (frozen)Linear0.5M / 5.9B50447.2 (+12.6)
    ViT-22B (frozen)UperNet0.8B / 22.5B50452.7
    InternViT-6B (frozen)UperNet0.4B / 6.3B50454.9 (+2.2)
    ViT-22BUperNet22.5B / 22.5B50455.3
    InternViT-6BUperNet6.3B / 6.3B50458.9 (+3.6)

  • Zero-Shot Image Classification [see details]

    methodIN-1KIN-AIN-RIN-V2IN-SketchObjectNet
    OpenCLIP-G80.169.392.173.668.973.0
    EVA-02-CLIP-E+82.082.194.575.771.679.6
    ViT-22B*85.990.196.080.9-87.6
    InternVL-C (ours)83.283.895.577.373.980.6

  • Multilingual Zero-Shot Image Classification [see details]

    EN: English, ZH: Chinese, JP: Japanese, Ar: Arabic, IT: Italian

    methodIN-1K (EN)IN-1K (ZH)IN-1K (JP)IN-1K (AR)IN-1K (IT)
    Taiyi-CLIP-ViT-H-54.4---
    WuKong-ViT-L-G-57.5---
    CN-CLIP-ViT-H-59.6---
    AltCLIP-ViT-L74.559.6---
    EVA-02-CLIP-E+82.0---41.2
    OpenCLIP-XLM-R-H77.055.753.137.056.8
    InternVL-C (ours)83.264.561.544.965.7

  • Zero-Shot Video Classification

    method#frameK400K600K700
    OpenCLIP-G165.966.159.2
    EVA-02-CLIP-E+169.869.363.4
    InternVL-C (ours)171.071.365.7
    ViCLIP875.773.566.4
    InternVL-C (ours)879.478.871.5
Cross-Modal Retrieval (click to expand)

  • English Zero-Shot Image-Text Retrieval [see details]

    modelFlickr30KCOCOavg
    image-to-texttext-to-imageimage-to-texttext-to-image
    R@1R@5R@10R@1R@5R@10R@1R@5R@10R@1R@5R@10
    OpenCLIP-G92.999.399.879.595.097.167.386.992.651.474.983.085.0
    EVA-02-CLIP-E+93.999.499.878.894.296.868.887.892.851.175.082.785.1
    EVA-CLIP-8B95.699.699.980.895.597.670.389.393.953.076.083.486.2
    InternVL-C (ours)94.799.699.981.796.098.270.689.093.554.177.384.686.6
    InternVL-G (ours)95.799.799.985.097.098.674.991.395.258.681.388.088.8

  • Chinese Zero-Shot Image-Text Retrieval [see details]

    modelFlickr30K-CNCOCO-CNavg
    image-to-texttext-to-imageimage-to-texttext-to-image
    R@1R@5R@10R@1R@5R@10R@1R@5R@10R@1R@5R@10
    CN-CLIP-ViT-H81.697.598.871.291.495.563.086.692.969.289.996.186.1
    OpenCLIP-XLM-R-H86.197.599.271.090.594.970.091.597.066.190.896.087.6
    InternVL-C (ours)90.398.899.775.192.996.468.892.096.768.991.996.589.0
    InternVL-G (ours)92.999.499.877.794.897.371.493.997.773.894.498.190.9

  • Multilingual Zero-Shot Image-Text Retrieval on XTD [see details]

    methodENESFRZHITKORUJPaverage
    AltCLIP95.494.192.995.194.294.491.891.793.7
    OpenCLIP-XLM-R-H97.396.194.594.796.090.293.994.094.6
    InternVL-C (ours)97.395.795.195.696.092.293.395.595.1
    InternVL-G (ours)98.697.796.596.796.995.194.896.196.6
Multimodal Dialogue

See "Compared with SOTA VLLMs" section.

Quick Start with HuggingFace

using InternViT-6B for visual feature extraction (click to expand)
import torchfrom PIL import Imagefrom transformers import AutoModel, CLIPImageProcessormodel = AutoModel.from_pretrained('OpenGVLab/InternViT-6B-448px-V1-5',torch_dtype=torch.bfloat16,low_cpu_mem_usage=True,trust_remote_code=True).cuda().eval()image = Image.open('./examples/image1.jpg').convert('RGB')image_processor = CLIPImageProcessor.from_pretrained('OpenGVLab/InternViT-6B-448px-V1-5')pixel_values = image_processor(images=image, return_tensors='pt').pixel_valuespixel_values = pixel_values.to(torch.bfloat16).cuda()outputs = model(pixel_values)
using InternVL-C(ontrastive) and InternVL-G(enerative) for cross-modal retrieval (click to expand)
import torchfrom PIL import Imagefrom transformers import AutoModel, CLIPImageProcessorfrom transformers import AutoTokenizermodel = AutoModel.from_pretrained('OpenGVLab/InternVL-14B-224px',torch_dtype=torch.bfloat16,low_cpu_mem_usage=True,trust_remote_code=True).cuda().eval()image_processor = CLIPImageProcessor.from_pretrained('OpenGVLab/InternVL-14B-224px')tokenizer = AutoTokenizer.from_pretrained('OpenGVLab/InternVL-14B-224px', use_fast=False, add_eos_token=True)tokenizer.pad_token_id = 0  # set pad_token_id to 0images = [Image.open('./examples/image1.jpg').convert('RGB'),Image.open('./examples/image2.jpg').convert('RGB'),Image.open('./examples/image3.jpg').convert('RGB')
]prefix = 'summarize:'texts = [prefix + 'a photo of a red panda',  # Englishprefix + '一张熊猫的照片',  # Chineseprefix + '二匹の猫の写真'  # Japanese]pixel_values = image_processor(images=images, return_tensors='pt').pixel_valuespixel_values = pixel_values.to(torch.bfloat16).cuda()input_ids = tokenizer(texts, return_tensors='pt', max_length=80,                      truncation=True, padding='max_length').input_ids.cuda()# InternVL-Clogits_per_image, logits_per_text = model(image=pixel_values, text=input_ids, mode='InternVL-C')probs = logits_per_image.softmax(dim=-1)# tensor([[9.9609e-01, 5.2185e-03, 6.0070e-08],#         [2.2949e-02, 9.7656e-01, 5.9903e-06],#         [3.2932e-06, 7.4863e-05, 1.0000e+00]], device='cuda:0',#        dtype=torch.bfloat16, grad_fn=)# InternVL-Glogits_per_image, logits_per_text = model(image=pixel_values, text=input_ids, mode='InternVL-G')probs = logits_per_image.softmax(dim=-1)# tensor([[9.9609e-01, 3.1738e-03, 3.6322e-08],#         [8.6060e-03, 9.9219e-01, 2.8759e-06],#         [1.7583e-06, 3.1233e-05, 1.0000e+00]], device='cuda:0',#        dtype=torch.bfloat16, grad_fn=)# please set add_eos_token to False for generationtokenizer.add_eos_token = Falseimage = Image.open('./examples/image1.jpg').convert('RGB')pixel_values = image_processor(images=image, return_tensors='pt').pixel_valuespixel_values = pixel_values.to(torch.bfloat16).cuda()tokenized = tokenizer("English caption:", return_tensors='pt')pred = model.generate(pixel_values=pixel_values,input_ids=tokenized.input_ids.cuda(),attention_mask=tokenized.attention_mask.cuda(),num_beams=5,min_new_tokens=8,
)caption = tokenizer.decode(pred[0].cpu(), skip_special_tokens=True).strip()# English caption: a red panda sitting on top of a wooden platform
using InternVL-Chat for multimodal chat (click to expand)

Here, we take the smaller OpenGVLab/InternVL2-8B as an example:

import numpy as npimport torchimport torchvision.transforms as Tfrom decord import VideoReader, cpufrom PIL import Imagefrom torchvision.transforms.functional import InterpolationModefrom transformers import AutoModel, AutoTokenizerIMAGENET_MEAN = (0.485, 0.456, 0.406)IMAGENET_STD = (0.229, 0.224, 0.225)def build_transform(input_size):MEAN, STD = IMAGENET_MEAN, IMAGENET_STDtransform = T.Compose([T.Lambda(lambda img: img.convert('RGB') if img.mode != 'RGB' else img),T.Resize((input_size, input_size), interpolation=InterpolationMode.BICUBIC),T.ToTensor(),T.Normalize(mean=MEAN, std=STD)
    ])return transformdef find_closest_aspect_ratio(aspect_ratio, target_ratios, width, height, image_size):best_ratio_diff = float('inf')best_ratio = (1, 1)area = width * heightfor ratio in target_ratios:target_aspect_ratio = ratio[0] / ratio[1]ratio_diff = abs(aspect_ratio - target_aspect_ratio)if ratio_diff < best_ratio_diff:best_ratio_diff = ratio_diffbest_ratio = ratioelif ratio_diff == best_ratio_diff:if area > 0.5 * image_size * image_size * ratio[0] * ratio[1]:best_ratio = ratioreturn best_ratiodef dynamic_preprocess(image, min_num=1, max_num=12, image_size=448, use_thumbnail=False):orig_width, orig_height = image.sizeaspect_ratio = orig_width / orig_height# calculate the existing image aspect ratiotarget_ratios = set(
        (i, j) for n in range(min_num, max_num + 1) for i in range(1, n + 1) for j in range(1, n + 1) ifi * j <= max_num and i * j >= min_num)target_ratios = sorted(target_ratios, key=lambda x: x[0] * x[1])# find the closest aspect ratio to the targettarget_aspect_ratio = find_closest_aspect_ratio(aspect_ratio, target_ratios, orig_width, orig_height, image_size)# calculate the target width and heighttarget_width = image_size * target_aspect_ratio[0]target_height = image_size * target_aspect_ratio[1]blocks = target_aspect_ratio[0] * target_aspect_ratio[1]# resize the imageresized_img = image.resize((target_width, target_height))processed_images = []for i in range(blocks):box = (
            (i % (target_width // image_size)) * image_size,
            (i // (target_width // image_size)) * image_size,
            ((i % (target_width // image_size)) + 1) * image_size,
            ((i // (target_width // image_size)) + 1) * image_size)# split the imagesplit_img = resized_img.crop(box)processed_images.append(split_img)assert 
Expand
Additional Information
Related Applications
Recommended for You
Related Information All
User Comments