***
## 环境准备
“`bash
# Oracle A1 Ubuntu 22.04 ARM 环境
sudo apt update && sudo apt install -y python3-pip git
# 建议使用虚拟环境
python3 -m venv venv-reranker
source venv-reranker/bin/activate
“`
***
## 路径一：Transformers 标准部署（推荐）
这是官方推荐的本地推理方式，基于 `transformers` 库直接加载模型 。[1]
### Step 1：安装依赖
“`bash
pip install transformers torch numpy
# Oracle A1 是 ARM 架构，使用 CPU-only PyTorch
pip install torch –index-url https://download.pytorch.org/
“`
### Step 2：下载模型（首次运行自动缓存）
“`python
# download_model.py —— 提前下载，避免推理时超时
from transformers import AutoModel
print(“正在下载 jina-reranker-v3（约 1.2GB）…”)
model = AutoModel.from_pretrained(
“jinaai/jina-reranker-v3″,
dtype=”auto”,
trust_remote_code=True, # Jina v3 使用自定义架构，必须开启
cache_dir=”./models/jina-
)
print(“下载完成。”)
“`
> **关于 `trust_remote_code=True`**：
### Step 3：封装为服务类
“`python
# local_reranker.py
from transformers import AutoModel
import torch
from typing import List, Dict
class JinaRerankerV3:
def __init__(
self,
model_path: str = “jinaai/jina-reranker-v3”,
cache_dir: str = “./models/jina-reranker-v3”,
device: str = “cpu”
):
self.model = AutoModel.from_pretrained(
model_path,
dtype=”auto”,
trust_remote_code=True,
cache_dir=cache_dir,
local_files_only=True # 下载后启用，禁止出站网络请求
)
self.model.eval()
self.device = device
self.model.to(device)
print(f”
jina-reranker-v3 已加载到 {device}”)
def rerank(
self,
query: str,
documents: List[str],
top_n: int = 8,
threshold: float = 0.5,
batch_size: int = 16 # v3 最多支持 64 文档同时处理
) -> List[Dict]:
“””
返回格式：[{“index”: int, “document”: str, “relevance_score”: float}, …]
按分数降序排列，仅返回 >= threshold 的结果
“””
with torch.no_grad():
results = self.model.rerank(
query,
documents,
max_length=1024, # 单段落最大长度
top_n=top_n,
batch_size=batch_size
)
# 应用置信度门槛
filtered = [r for r in results if r[“relevance_score”] >= threshold]
return filtered
def check_hard_threshold(
self, reranked: List[Dict], hard_threshold: float = 0.6
) -> bool:
“””Top-1 分数低于 hard_threshold 则拦截，不进入 LLM”””
if not reranked:
return False
return reranked[0][“relevance_score”] >= hard_threshold
“`
### Step 4：集成到 FastAPI 后端
“`python
# main.py
from fastapi import FastAPI
from pydantic import BaseModel
from contextlib import asynccontextmanager
from local_reranker import JinaRerankerV3
reranker: JinaRerankerV3 = None
@asynccontextmanager
async def lifespan(app: FastAPI):
global reranker
reranker = JinaRerankerV3() # 启动时加载模型，避免每次请求重新加载
yield
app = FastAPI(lifespan=lifespan)
class RerankRequest(BaseModel):
query: str
documents: list[str]
top_n: int = 8
threshold: float = 0.5
class RerankResponse(BaseModel):
results: list[dict]
intercepted: bool # True = 触发硬性拦截，不应调用 LLM
@app.post(“/v1/rerank”, response_model=RerankResponse)
async def rerank_endpoint(req: RerankRequest):
results = reranker.rerank(
req.query, req.documents, req.top_n, req.threshold
)
intercepted = not reranker.check_hard_threshold(
return RerankResponse(results=
@app.get(“/health”)
async def health():
return {“status”: “ok”, “model”: “jina-reranker-v3”}
“`
“`bash
# 启动服务
uvicorn main:app –host 0.0.0.0 –port 8001 –workers 1
“`
***
## 路径二：GGUF 量化部署（内存受限时）
若 Oracle A1 内存不足（低于 4GB），可使用官方提供的 GGUF 量化版本，体积仅 **1.1GB（BF16）** 。[2]
### Step 1：编译 llama.cpp（推荐 Hanxiao fork）
“`bash
git clone https://github.com/hanxiao/
cd llama.cpp
make -j4 # ARM CPU 编译
“`
### Step 2：下载 GGUF 文件
“`bash
pip install huggingface_hub
python3 -c ”
from huggingface_hub import hf_hub_download
# 下载量化模型（BF16，1.1GB）
hf_hub_download(
repo_id=’jinaai/jina-reranker-
filename=’jina-reranker-v3-
local_dir=’./models/jina-
)
# 下载 MLP 投影层（3MB）
hf_hub_download(
repo_id=’jinaai/jina-reranker-
filename=’projector.
local_dir=’./models/jina-
)
”
“`
### Step 3：使用 GGUF Reranker
“`python
# 官方提供 rerank.py，直接从 HuggingFace 仓库下载后使用
from rerank import GGUFReranker # 来自 jinaai/jina-reranker-v3-GGUF
reranker = GGUFReranker(
model_path=”./models/jina-
projector_path=”./models/jina-
llama_embedding_path=”./llama.
)
results = reranker.rerank(
query=”保罗在罗马书里怎么说称义的”,
documents=[“段落1内容…”, “段落2内容…”],
top_n=8
)
“`
***
## 路径三：vLLM 高并发部署（多用户场景）
若系统需支持多用户并发，可通过 vLLM 将 v3 作为推理服务部署 。[3]
“`bash
pip install vllm
# 启动 vLLM 服务（兼容 OpenAI Rerank API 格式）
python -m vllm.entrypoints.openai.api_
–model jinaai/jina-reranker-v3 \
–trust-remote-code \
–task score \
–port 8001
“`
***
## 三条路径资源对比
| 指标 | Transformers | GGUF | vLLM |
|—|—|—|—|
| 模型大小 | ~2.4GB (FP32) | ~1.1GB (BF16) | ~2.4GB |
| Oracle A1 推理延迟 | ~1–2s / 15 段落 | ~0.5–1s / 15 段落 | ~0.3–0.8s（并发优化） |
| 内存占用 | ~3–4GB | ~1.5–2GB | ~4–5GB |
| 安装复杂度 | 低 | 中（需编译 llama.cpp） | 低 |
| 推荐场景 | **单服务器首选** | 内存受限环境 | 多用户并发 |
对于你的《生命读经》RAG 系统，**路径一（Transformers）是首选**：
Sources
[1] jinaai/jina-reranker-v3 – Hugging Face https://huggingface.co/jinaai/
[2] jinaai/jina-reranker-v3-GGUF – Hugging Face https://huggingface.co/jinaai/
[3] Supported Models – vLLM https://docs.vllm.ai/en/
[4] jina-reranker-v3 – Search Foundation Models https://jina.ai/models/jina-
[5] `trust_remote_code=true` for Jinaai 8k models · Issue #2352 – GitHub https://github.com/
[6] Trust remote code = True, even for fine-tuned local model? https://huggingface.co/jinaai/
[7] jina-reranker-v3: Last but Not Late Interaction for Document … – arXiv https://arxiv.org/html/2509.
[8] Is my Data safe when using trust_remote_code? : r/LocalLLaMA https://www.reddit.com/r/
[9] jina-rerankers on Elastic Inference Service – Elasticsearch Labs https://www.elastic.co/search-
[10] jina-ai/mlx-retrieval: Train embedding and reranker models … – GitHub https://github.com/jina-ai/
[11] How to deploy a Hugging Face model that requires … https://forum.opensearch.org/
[12] Reranker API – Jina AI https://jina.ai/reranker/
[13] jinaai/jina-reranker-v2-base-
[14] Jina-Reranker-V3: Last But Not Late Interaction … – YouTube https://www.youtube.com/watch?
[15] Pooling models – vLLM https://docs.vllm.ai/en/v0.11.
