fg-clip-base
FG-CLIP is a vision-language model that aligns images and text at fine-grained (patch and region) levels, not just global image-caption pairs. It uses a two-stage training approach: first with global captions, then with region-level descriptions. The model is open-source under Apache 2.0, has ~150M parameters, and supports zero-shot image classification and image-text retrieval. It requires GPU (CUDA) and custom code for loading.
Key facts
Objective fields from the source. Values we can't verify are shown as “Unknown” rather than guessed.
| Field | Value |
|---|---|
| Developer | qihoo360 |
| Parameters | 150M |
| Context window | Unknown |
| License | apache-2.0 — OSI-approved |
| Modality / task | zero-shot-image-classification |
| Gated on HuggingFace | No |
| Downloads | 36.5k |
| Likes | 10 |
| Last updated | 2025-10-09 |
| Source | qihoo360/fg-clip-base |
What fg-clip-base is
FG-CLIP is a CLIP-variant model (transformers-based, safetensors format) from Qihoo360, published at ICML 2025 (arXiv:2505.05071). It extends standard CLIP by training on both global and region-level alignment signals, enabling dense feature similarity visualization. The model accepts images (224×224) and text captions up to 77 tokens (short) or 248 tokens (long), outputs normalized embeddings, and requires trust_remote_code=True. Context length is unknown for text generation tasks.
Run fg-clip-base locally
Load the open weights with 🤗 Transformers and generate — the same model, self-hosted.
from transformers import pipelinepipe = pipeline("text-generation", model="qihoo360/fg-clip-base")out = pipe("Explain retrieval-augmented generation in one sentence.", max_new_tokens=128)print(out[0]["generated_text"])Swap in vLLM or Ollama for production-grade serving. DEV.co can stand up the inference stack.
How you'd run it
A typical self-hosted path — open weights, an inference server, your application.
DEV.co builds each layer — from GPU infrastructure to the application.
Best use cases
Running & fine-tuning it
GPU with CUDA support required. Estimated VRAM: 600 MB–1.2 GB for inference (fp32, batch size 1–4). For larger batches or training/fine-tuning, 8 GB+ recommended. Exact requirements depend on batch size and sequence length (77 vs. 248 tokens); verify with your hardware before deployment.
No explicit fine-tuning instructions in the card. Model architecture uses AutoModelForCausalLM wrapper, suggesting potential for LoRA/QLoRA adaptation, but requires custom implementation. Region-level caption alignment would require curated datasets; feasibility for domain-specific fine-tuning is Unknown without access to training code.
When to avoid it — and what to weigh
- Need off-the-shelf CPU inference — Model requires GPU/CUDA acceleration. CPU inference will be extremely slow for production workloads.
- Strict reproducibility or determinism required — Custom code execution (trust_remote_code=True) and region-level training details may hinder reproducibility across environments or versions.
- Constrained latency or edge deployment — 150M parameters plus dense feature extraction overhead; not optimized for mobile, edge, or sub-100ms latency requirements.
- Production systems without custom code audit capability — Model loading requires trust_remote_code=True; security review of remote code is necessary before deploying in regulated or high-security environments.
License & commercial use
Licensed under Apache License 2.0. The license covers the project code and model weights. Apache 2.0 is a permissive OSI-approved license allowing commercial use, modification, and distribution with attribution and license notice.
Apache 2.0 is a permissive license; commercial use is permitted. However, the model card notes: 'This project utilizes certain datasets and checkpoints that are subject to their respective original licenses.' You must audit and comply with licenses of all training datasets and any included checkpoints. Without explicit dataset provenance in the card, conduct due diligence before commercial deployment.
DEV.co evaluation signals
Editorial assessment — not user reviews. Directional, with an explicit confidence level.
| Signal | Assessment |
|---|---|
| Maintenance | Moderate |
| Documentation | Adequate |
| License clarity | Clear |
| Deployment complexity | Moderate |
| DEV.co fit | Good |
| Assessment confidence | High |
Model requires trust_remote_code=True for loading, meaning arbitrary Python code from the remote repository is executed. Review the GitHub repository for security before using in production. No security audit results or vulnerability disclosures are mentioned. Datasets and checkpoints used are not fully enumerated; audit data sources for sensitive or biased content if deploying in sensitive domains.
Alternatives to consider
OpenAI CLIP
Mature, widely deployed vision-language baseline. No region-level fine-graining, but better battle-tested and no custom code requirement. Trade-off: less fine-grained alignment, but simpler deployment.
DINOv2 + text encoders
Alternative for fine-grained visual features; can pair with separate text encoder. More modular and avoids custom code, but requires manual fusion logic.
LLaVA or Qwen-VL
VLM approaches offering richer language understanding and reasoning. Better for open-ended captioning, but heavier (>7B parameters) and focused on generation rather than pure embedding alignment.
Ship fg-clip-base with senior software developers
FG-CLIP enables fine-grained image-text retrieval and region-level alignment without retraining. Audit dataset licenses for your use case, verify GPU availability, and review custom code security before production deployment. Contact our team to integrate FG-CLIP into custom LLM apps or RAG systems.
Talk to DEV.coRelated open-source tools
Surfaced by semantic similarity across the DEV.co open-source index.
Related on DEV.co
Explore the category and the services that help you build with it.
fg-clip-base FAQ
Can I use FG-CLIP commercially?
What GPU do I need to run FG-CLIP?
Do I need to use trust_remote_code=True?
Can I fine-tune FG-CLIP for my domain?
Work with a software development agency
DEV.co is a software development agency delivering custom software development services to companies building on open source. Our software developers and web developers design, integrate, and ship production systems — spanning web development, APIs, AI, data, and cloud. If fg-clip-base is part of your open-source llms roadmap, our team can implement, customize, migrate, and maintain it.
Explore FG-CLIP for Your Vision-Language Needs
FG-CLIP enables fine-grained image-text retrieval and region-level alignment without retraining. Audit dataset licenses for your use case, verify GPU availability, and review custom code security before production deployment. Contact our team to integrate FG-CLIP into custom LLM apps or RAG systems.