搜索系统设计详解

本章导读

搜索系统是现代互联网应用的核心功能之一，无论是电商平台、内容平台还是企业内部系统，都离不开高效的搜索能力。本章将深入剖析倒排索引原理、Elasticsearch架构、搜索排序算法等核心技术，帮助你设计毫秒级响应、高相关性的搜索系统。

学习目标：

• 目标1：掌握倒排索引原理与Elasticsearch核心架构
• 目标2：理解TF-IDF、BM25等搜索排序算法的实现原理
• 目标3：能够设计支持海量数据、高并发的分布式搜索服务

前置知识：熟悉数据结构与算法，了解分布式系统基础

阅读时长：约 55 分钟

一、知识概述

搜索系统是现代互联网应用的核心功能之一，无论是电商平台、内容平台还是企业内部系统，都离不开高效的搜索能力。一个优秀的搜索系统需要具备毫秒级响应、高相关性排序、丰富的检索功能以及强大的扩展性。

本文将深入分析搜索系统的设计要点，包括倒排索引原理、Elasticsearch架构、搜索排序算法、分布式检索优化等核心问题，并提供完整的Java实现方案。

搜索系统的核心挑战

1. 检索性能：毫秒级响应海量数据
2. 相关性排序：精准匹配用户意图
3. 实时性：数据变更快速生效
4. 扩展性：支持数据量和查询量增长
5. 多样性：支持多种检索场景（全文检索、精准匹配、范围查询等）

---

二、知识点详细讲解

2.1 倒排索引原理

倒排索引结构

正向索引（文档 -> 词）：
文档ID    文档内容
1         "Java并发编程实战"
2         "Java虚拟机原理"
3         "Spring Boot实战"

倒排索引（词 -> 文档）：
关键词        文档ID列表
Java         [1, 2]
并发         [1]
编程         [1]
虚拟机       [2]
原理         [2]
Spring       [3]
Boot         [3]
实战         [1, 3]

Java实现示例

package com.example.search.index;

import lombok.Data;
import lombok.extern.slf4j.Slf4j;

import java.util.*;
import java.util.concurrent.ConcurrentHashMap;
import java.util.stream.Collectors;

/**
 * 倒排索引实现
 */
@Slf4j
@Data
public class InvertedIndex {
    
    // 倒排索引：词 -> 文档ID列表
    private final Map<String, List<Posting>> index = new ConcurrentHashMap<>();
    
    // 正向索引：文档ID -> 文档内容
    private final Map<Long, Document> documents = new ConcurrentHashMap<>();
    
    /**
     * 添加文档到索引
     */
    public void addDocument(Document document) {
        Long docId = document.getDocId();
        documents.put(docId, document);
        
        // 分词
        List<String> terms = tokenize(document.getContent());
        
        // 统计词频
        Map<String, Integer> termFreq = new HashMap<>();
        for (String term : terms) {
            termFreq.merge(term, 1, Integer::sum);
        }
        
        // 更新倒排索引
        for (Map.Entry<String, Integer> entry : termFreq.entrySet()) {
            String term = entry.getKey();
            int freq = entry.getValue();
            
            Posting posting = new Posting();
            posting.setDocId(docId);
            posting.setTermFrequency(freq);
            posting.setPositions(getPositions(terms, term));
            
            index.computeIfAbsent(term, k -> new CopyOnWriteArrayList<>())
                 .add(posting);
        }
        
        log.debug("添加文档到索引: docId={}, terms={}", docId, termFreq.size());
    }
    
    /**
     * 删除文档
     */
    public void deleteDocument(Long docId) {
        documents.remove(docId);
        
        // 从倒排索引中移除
        for (List<Posting> postings : index.values()) {
            postings.removeIf(p -> p.getDocId().equals(docId));
        }
    }
    
    /**
     * 搜索单个词
     */
    public List<Posting> search(String term) {
        return index.getOrDefault(term, Collections.emptyList());
    }
    
    /**
     * 搜索多个词（AND查询）
     */
    public List<Long> searchAnd(List<String> terms) {
        if (terms.isEmpty()) {
            return Collections.emptyList();
        }
        
        List<List<Posting>> postingLists = terms.stream()
            .map(this::search)
            .collect(Collectors.toList());
        
        // 取交集
        Set<Long> result = postingLists.get(0).stream()
            .map(Posting::getDocId)
            .collect(Collectors.toSet());
        
        for (int i = 1; i < postingLists.size(); i++) {
            Set<Long> docIds = postingLists.get(i).stream()
                .map(Posting::getDocId)
                .collect(Collectors.toSet());
            result.retainAll(docIds);
        }
        
        return new ArrayList<>(result);
    }
    
    /**
     * 搜索多个词（OR查询）
     */
    public List<Long> searchOr(List<String> terms) {
        Set<Long> result = new HashSet<>();
        
        for (String term : terms) {
            List<Posting> postings = search(term);
            for (Posting posting : postings) {
                result.add(posting.getDocId());
            }
        }
        
        return new ArrayList<>(result);
    }
    
    /**
     * 分词（简单实现，实际应使用分词器）
     */
    private List<String> tokenize(String content) {
        if (content == null || content.isEmpty()) {
            return Collections.emptyList();
        }
        
        // 简单分词：按空格和标点分割
        return Arrays.stream(content.toLowerCase()
            .split("[\\s\\p{Punct}]+"))
            .filter(s -> !s.isEmpty())
            .collect(Collectors.toList());
    }
    
    /**
     * 获取词在文档中的位置
     */
    private List<Integer> getPositions(List<String> terms, String target) {
        List<Integer> positions = new ArrayList<>();
        for (int i = 0; i < terms.size(); i++) {
            if (terms.get(i).equals(target)) {
                positions.add(i);
            }
        }
        return positions;
    }
    
    /**
     * 获取索引统计信息
     */
    public Map<String, Object> getStats() {
        Map<String, Object> stats = new HashMap<>();
        stats.put("documentCount", documents.size());
        stats.put("termCount", index.size());
        stats.put("totalPostings", index.values().stream()
            .mapToInt(List::size)
            .sum());
        return stats;
    }
    
    /**
     * Posting（倒排项）
     */
    @Data
    public static class Posting {
        private Long docId;           // 文档ID
        private int termFrequency;     // 词频
        private List<Integer> positions; // 词在文档中的位置
    }
    
    /**
     * 文档
     */
    @Data
    public static class Document {
        private Long docId;
        private String title;
        private String content;
        private Map<String, Object> fields; // 其他字段
    }
    
    /**
     * 测试
     */
    public static void main(String[] args) {
        InvertedIndex invertedIndex = new InvertedIndex();
        
        // 添加文档
        invertedIndex.addDocument(createDoc(1L, "Java并发编程实战", 
            "Java并发编程是Java开发的核心技能"));
        invertedIndex.addDocument(createDoc(2L, "Java虚拟机原理", 
            "深入理解Java虚拟机原理"));
        invertedIndex.addDocument(createDoc(3L, "Spring Boot实战", 
            "Spring Boot快速开发框架"));
        
        // 搜索测试
        System.out.println("搜索'Java': " + invertedIndex.search("Java"));
        System.out.println("搜索AND['Java', '并发']: " + 
            invertedIndex.searchAnd(List.of("java", "并发")));
        System.out.println("搜索OR['Java', 'spring']: " + 
            invertedIndex.searchOr(List.of("java", "spring")));
        
        // 统计信息
        System.out.println("索引统计: " + invertedIndex.getStats());
    }
    
    private static Document createDoc(Long id, String title, String content) {
        Document doc = new Document();
        doc.setDocId(id);
        doc.setTitle(title);
        doc.setContent(content);
        return doc;
    }
}

2.2 Elasticsearch架构

Elasticsearch核心概念

/**
 * Elasticsearch核心概念映射
 */
public class ElasticsearchConcepts {
    
    /**
     * 索引（Index）
     * 相当于关系数据库的"数据库"
     */
    // 示例：product_index（商品索引）
    
    /**
     * 类型（Type）
     * ES 7.x后已废弃，相当于关系数据库的"表"
     */
    // 示例：product（商品类型）
    
    /**
     * 文档（Document）
     * 相当于关系数据库的"行"
     */
    @Data
    public static class ProductDocument {
        private Long id;              // 商品ID
        private String name;          // 商品名称
        private String description;   // 商品描述
        private BigDecimal price;     // 价格
        private Integer categoryId;   // 分类ID
        private String categoryName;  // 分类名称
        private String brand;         // 品牌
        private Integer stock;        // 库存
        private Integer sales;        // 销量
        private Float rating;         // 评分
        private List<String> tags;    // 标签
        private LocalDateTime createTime;
        private LocalDateTime updateTime;
    }
    
    /**
     * 字段（Field）
     * 相当于关系数据库的"列"
     */
    
    /**
     * 映射（Mapping）
     * 相当于关系数据库的"表结构定义"
     */
}

索引映射定义

package com.example.search.mapping;

import org.springframework.data.annotation.Id;
import org.springframework.data.elasticsearch.annotations.*;

import java.math.BigDecimal;
import java.time.LocalDateTime;
import java.util.List;

/**
 * 商品索引文档
 */
@Document(indexName = "product_idx")
@Setting(settingPath = "elasticsearch/product-setting.json")
public class ProductIndex {
    
    @Id
    private Long id;
    
    /**
     * 商品名称
     * 使用IK分词器
     */
    @Field(type = FieldType.Text, analyzer = "ik_max_word", searchAnalyzer = "ik_smart")
    private String name;
    
    /**
     * 商品描述
     */
    @Field(type = FieldType.Text, analyzer = "ik_max_word", searchAnalyzer = "ik_smart")
    private String description;
    
    /**
     * 价格（精确值，用于范围查询和排序）
     */
    @Field(type = FieldType.ScaledFloat, scalingFactor = 100)
    private BigDecimal price;
    
    /**
     * 分类ID
     */
    @Field(type = FieldType.Long)
    private Long categoryId;
    
    /**
     * 分类名称（keyword用于精确匹配和聚合）
     */
    @Field(type = FieldType.Keyword)
    private String categoryName;
    
    /**
     * 品牌
     */
    @Field(type = FieldType.Keyword)
    private String brand;
    
    /**
     * 库存
     */
    @Field(type = FieldType.Integer)
    private Integer stock;
    
    /**
     * 销量
     */
    @Field(type = FieldType.Integer)
    private Integer sales;
    
    /**
     * 评分
     */
    @Field(type = FieldType.Float)
    private Float rating;
    
    /**
     * 标签列表
     */
    @Field(type = FieldType.Keyword)
    private List<String> tags;
    
    /**
     * 是否上架
     */
    @Field(type = FieldType.Boolean)
    private Boolean onSale;
    
    /**
     * 创建时间
     */
    @Field(type = FieldType.Date, format = DateFormat.date_hour_minute_second)
    private LocalDateTime createTime;
    
    /**
     * 更新时间
     */
    @Field(type = FieldType.Date, format = DateFormat.date_hour_minute_second)
    private LocalDateTime updateTime;
    
    /**
     * 嵌入式对象
     */
    @Field(type = FieldType.Object)
    private ShopInfo shop;
    
    // getters and setters...
}

/**
 * 店铺信息
 */
@Data
public class ShopInfo {
    private Long shopId;
    private String shopName;
    private Integer shopLevel;
}

/**
 * 索引设置（product-setting.json）
 */
/*
{
  "index": {
    "number_of_shards": 3,
    "number_of_replicas": 1,
    "refresh_interval": "1s"
  },
  "analysis": {
    "analyzer": {
      "ik_max_word": {
        "type": "custom",
        "tokenizer": "ik_max_word"
      },
      "ik_smart": {
        "type": "custom",
        "tokenizer": "ik_smart"
      }
    }
  }
}
*/

2.3 搜索查询实现

基础搜索

package com.example.search.service;

import org.elasticsearch.index.query.*;
import org.elasticsearch.search.sort.SortBuilders;
import org.elasticsearch.search.sort.SortOrder;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.data.domain.Page;
import org.springframework.data.domain.PageRequest;
import org.springframework.data.elasticsearch.core.*;
import org.springframework.data.elasticsearch.core.query.NativeSearchQuery;
import org.springframework.data.elasticsearch.core.query.NativeSearchQueryBuilder;
import org.springframework.stereotype.Service;

import java.util.List;

/**
 * 商品搜索服务
 */
@Service
public class ProductSearchService {
    
    @Autowired
    private ElasticsearchOperations elasticsearchOperations;
    
    /**
     * 关键词搜索
     */
    public Page<ProductIndex> search(String keyword, int page, int size) {
        // 构建查询条件
        BoolQueryBuilder boolQuery = QueryBuilders.boolQuery();
        
        // 多字段匹配
        MultiMatchQueryBuilder multiMatchQuery = QueryBuilders
            .multiMatchQuery(keyword, "name", "description")
            .type(MultiMatchQueryBuilder.Type.BEST_FIELDS)
            .minimumShouldMatch("75%");
        
        boolQuery.must(multiMatchQuery);
        
        // 过滤条件：只搜索上架商品
        boolQuery.filter(QueryBuilders.termQuery("onSale", true));
        
        // 构建搜索请求
        NativeSearchQuery searchQuery = new NativeSearchQueryBuilder()
            .withQuery(boolQuery)
            .withPageable(PageRequest.of(page, size))
            .withSort(SortBuilders.scoreSort().order(SortOrder.DESC))
            .build();
        
        // 执行搜索
        SearchHits<ProductIndex> searchHits = 
            elasticsearchOperations.search(searchQuery, ProductIndex.class);
        
        // 转换结果
        return SearchHitSupport.searchPageFor(searchHits, searchQuery.getPageable());
    }
    
    /**
     * 多条件组合搜索
     */
    public Page<ProductIndex> advancedSearch(ProductSearchRequest request, int page, int size) {
        BoolQueryBuilder boolQuery = QueryBuilders.boolQuery();
        
        // 关键词搜索
        if (request.getKeyword() != null && !request.getKeyword().isEmpty()) {
            MultiMatchQueryBuilder matchQuery = QueryBuilders
                .multiMatchQuery(request.getKeyword(), "name", "description")
                .type(MultiMatchQueryBuilder.Type.BEST_FIELDS);
            boolQuery.must(matchQuery);
        }
        
        // 分类过滤
        if (request.getCategoryId() != null) {
            boolQuery.filter(QueryBuilders.termQuery("categoryId", request.getCategoryId()));
        }
        
        // 品牌过滤
        if (request.getBrand() != null && !request.getBrand().isEmpty()) {
            boolQuery.filter(QueryBuilders.termQuery("brand", request.getBrand()));
        }
        
        // 价格区间
        if (request.getMinPrice() != null || request.getMaxPrice() != null) {
            RangeQueryBuilder rangeQuery = QueryBuilders.rangeQuery("price");
            if (request.getMinPrice() != null) {
                rangeQuery.gte(request.getMinPrice());
            }
            if (request.getMaxPrice() != null) {
                rangeQuery.lte(request.getMaxPrice());
            }
            boolQuery.filter(rangeQuery);
        }
        
        // 标签过滤
        if (request.getTags() != null && !request.getTags().isEmpty()) {
            boolQuery.filter(QueryBuilders.termsQuery("tags", request.getTags()));
        }
        
        // 只搜索上架商品
        boolQuery.filter(QueryBuilders.termQuery("onSale", true));
        
        // 构建搜索请求
        NativeSearchQueryBuilder queryBuilder = new NativeSearchQueryBuilder()
            .withQuery(boolQuery)
            .withPageable(PageRequest.of(page, size));
        
        // 排序
        if (request.getSortBy() != null) {
            switch (request.getSortBy()) {
                case "price_asc":
                    queryBuilder.withSort(SortBuilders.fieldSort("price").order(SortOrder.ASC));
                    break;
                case "price_desc":
                    queryBuilder.withSort(SortBuilders.fieldSort("price").order(SortOrder.DESC));
                    break;
                case "sales_desc":
                    queryBuilder.withSort(SortBuilders.fieldSort("sales").order(SortOrder.DESC));
                    break;
                default:
                    queryBuilder.withSort(SortBuilders.scoreSort().order(SortOrder.DESC));
            }
        }
        
        NativeSearchQuery searchQuery = queryBuilder.build();
        
        // 执行搜索
        SearchHits<ProductIndex> searchHits = 
            elasticsearchOperations.search(searchQuery, ProductIndex.class);
        
        return SearchHitSupport.searchPageFor(searchHits, searchQuery.getPageable());
    }
    
    /**
     * 自动补全（前缀匹配）
     */
    public List<String> autoComplete(String prefix, int size) {
        NativeSearchQuery searchQuery = new NativeSearchQueryBuilder()
            .withQuery(QueryBuilders.matchPhrasePrefixQuery("name", prefix))
            .withMaxResults(size)
            .build();
        
        SearchHits<ProductIndex> searchHits = 
            elasticsearchOperations.search(searchQuery, ProductIndex.class);
        
        return searchHits.stream()
            .map(hit -> hit.getContent().getName())
            .distinct()
            .limit(size)
            .collect(java.util.stream.Collectors.toList());
    }
    
    /**
     * 高亮搜索
     */
    public Page<ProductIndex> searchWithHighlight(String keyword, int page, int size) {
        // 高亮配置
        org.elasticsearch.search.fetch.subphase.highlight.HighlightBuilder.Field[] highlightFields = {
            new org.elasticsearch.search.fetch.subphase.highlight.HighlightBuilder.Field("name")
                .preTags("<em class='highlight'>")
                .postTags("</em>"),
            new org.elasticsearch.search.fetch.subphase.highlight.HighlightBuilder.Field("description")
                .preTags("<em class='highlight'>")
                .postTags("</em>")
                .fragmentSize(100)
                .numOfFragments(2)
        };
        
        NativeSearchQuery searchQuery = new NativeSearchQueryBuilder()
            .withQuery(QueryBuilders.multiMatchQuery(keyword, "name", "description"))
            .withHighlightFields(highlightFields)
            .withPageable(PageRequest.of(page, size))
            .build();
        
        SearchHits<ProductIndex> searchHits = 
            elasticsearchOperations.search(searchQuery, ProductIndex.class);
        
        // 处理高亮结果
        searchHits.forEach(hit -> {
            ProductIndex product = hit.getContent();
            List<String> nameHighlight = hit.getHighlightField("name");
            if (!nameHighlight.isEmpty()) {
                product.setName(nameHighlight.get(0));
            }
        });
        
        return SearchHitSupport.searchPageFor(searchHits, searchQuery.getPageable());
    }
}

/**
 * 商品搜索请求
 */
@Data
public class ProductSearchRequest {
    private String keyword;
    private Long categoryId;
    private String brand;
    private BigDecimal minPrice;
    private BigDecimal maxPrice;
    private List<String> tags;
    private String sortBy;
}

2.4 搜索排序算法

TF-IDF算法

package com.example.search.ranker;

import java.util.*;

/**
 * TF-IDF相关性计算
 */
public class TfIdfCalculator {
    
    /**
     * 计算TF（词频）
     * TF = 词在文档中出现次数 / 文档总词数
     */
    public double calculateTF(List<String> documentTerms, String term) {
        long count = documentTerms.stream().filter(t -> t.equals(term)).count();
        return (double) count / documentTerms.size();
    }
    
    /**
     * 计算IDF（逆文档频率）
     * IDF = log(总文档数 / 包含该词的文档数)
     */
    public double calculateIDF(List<List<String>> allDocuments, String term) {
        long docCount = allDocuments.stream()
            .filter(doc -> doc.contains(term))
            .count();
        
        if (docCount == 0) {
            return 0;
        }
        
        return Math.log((double) allDocuments.size() / docCount);
    }
    
    /**
     * 计算TF-IDF
     */
    public double calculateTFIDF(List<String> documentTerms, 
                                  List<List<String>> allDocuments, 
                                  String term) {
        double tf = calculateTF(documentTerms, term);
        double idf = calculateIDF(allDocuments, term);
        return tf * idf;
    }
    
    /**
     * 计算文档向量（用于相似度计算）
     */
    public Map<String, Double> getDocumentVector(List<String> documentTerms,
                                                   List<List<String>> allDocuments,
                                                   Set<String> vocabulary) {
        Map<String, Double> vector = new HashMap<>();
        
        for (String term : vocabulary) {
            vector.put(term, calculateTFIDF(documentTerms, allDocuments, term));
        }
        
        return vector;
    }
    
    /**
     * 计算余弦相似度
     */
    public double cosineSimilarity(Map<String, Double> vector1, 
                                    Map<String, Double> vector2) {
        Set<String> allTerms = new HashSet<>();
        allTerms.addAll(vector1.keySet());
        allTerms.addAll(vector2.keySet());
        
        double dotProduct = 0;
        double norm1 = 0;
        double norm2 = 0;
        
        for (String term : allTerms) {
            double v1 = vector1.getOrDefault(term, 0.0);
            double v2 = vector2.getOrDefault(term, 0.0);
            
            dotProduct += v1 * v2;
            norm1 += v1 * v1;
            norm2 += v2 * v2;
        }
        
        if (norm1 == 0 || norm2 == 0) {
            return 0;
        }
        
        return dotProduct / (Math.sqrt(norm1) * Math.sqrt(norm2));
    }
    
    /**
     * 示例
     */
    public static void main(String[] args) {
        TfIdfCalculator calculator = new TfIdfCalculator();
        
        // 文档集合
        List<List<String>> documents = Arrays.asList(
            Arrays.asList("java", "并发", "编程", "实战"),
            Arrays.asList("java", "虚拟机", "原理"),
            Arrays.asList("spring", "boot", "实战")
        );
        
        // 词汇表
        Set<String> vocabulary = new HashSet<>();
        documents.forEach(vocabulary::addAll);
        
        // 计算每个文档的向量
        List<Map<String, Double>> vectors = new ArrayList<>();
        for (List<String> doc : documents) {
            vectors.add(calculator.getDocumentVector(doc, documents, vocabulary));
        }
        
        // 计算文档相似度
        System.out.println("文档相似度:");
        for (int i = 0; i < vectors.size(); i++) {
            for (int j = i + 1; j < vectors.size(); j++) {
                double similarity = calculator.cosineSimilarity(vectors.get(i), vectors.get(j));
                System.out.printf("Doc %d vs Doc %d: %.4f%n", i + 1, j + 1, similarity);
            }
        }
    }
}

BM25算法

package com.example.search.ranker;

import java.util.*;

/**
 * BM25排序算法
 * Elasticsearch默认使用BM25
 */
public class BM25Calculator {
    
    // BM25参数
    private final double k1 = 1.2;  // 词频饱和参数
    private final double b = 0.75;  // 文档长度归一化参数
    
    /**
     * 计算BM25分数
     * 
     * @param termFreq 词在文档中的频率
     * @param docLength 文档长度
     * @param avgDocLength 平均文档长度
     * @param docCount 文档总数
     * @param docFreq 包含该词的文档数
     * @return BM25分数
     */
    public double calculateScore(int termFreq, int docLength, double avgDocLength,
                                  int docCount, int docFreq) {
        // IDF计算
        double idf = Math.log((docCount - docFreq + 0.5) / (docFreq + 0.5) + 1);
        
        // TF部分计算
        double tf = (termFreq * (k1 + 1)) / 
                    (termFreq + k1 * (1 - b + b * docLength / avgDocLength));
        
        return idf * tf;
    }
    
    /**
     * 计算文档总分（多词查询）
     */
    public double calculateDocumentScore(List<String> queryTerms,
                                          List<String> documentTerms,
                                          List<List<String>> allDocuments) {
        double totalScore = 0;
        
        int docCount = allDocuments.size();
        double avgDocLength = allDocuments.stream()
            .mapToInt(List::size)
            .average()
            .orElse(1);
        
        Map<String, Integer> termFreqInDoc = getTermFrequencies(documentTerms);
        
        for (String term : queryTerms) {
            int termFreq = termFreqInDoc.getOrDefault(term, 0);
            int docFreq = (int) allDocuments.stream()
                .filter(doc -> doc.contains(term))
                .count();
            
            if (docFreq > 0) {
                totalScore += calculateScore(
                    termFreq, 
                    documentTerms.size(), 
                    avgDocLength, 
                    docCount, 
                    docFreq
                );
            }
        }
        
        return totalScore;
    }
    
    /**
     * 获取词频统计
     */
    private Map<String, Integer> getTermFrequencies(List<String> terms) {
        Map<String, Integer> freq = new HashMap<>();
        for (String term : terms) {
            freq.merge(term, 1, Integer::sum);
        }
        return freq;
    }
    
    /**
     * 示例
     */
    public static void main(String[] args) {
        BM25Calculator calculator = new BM25Calculator();
        
        List<List<String>> documents = Arrays.asList(
            Arrays.asList("java", "并发", "编程", "实战"),
            Arrays.asList("java", "虚拟机", "原理"),
            Arrays.asList("spring", "boot", "实战")
        );
        
        List<String> query = Arrays.asList("java", "实战");
        
        System.out.println("BM25打分结果:");
        for (int i = 0; i < documents.size(); i++) {
            double score = calculator.calculateDocumentScore(
                query, documents.get(i), documents);
            System.out.printf("文档 %d: %.4f%n", i + 1, score);
        }
    }
}

自定义排序策略

package com.example.search.ranker;

import org.elasticsearch.script.Script;
import org.elasticsearch.script.ScriptType;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.data.elasticsearch.core.ElasticsearchOperations;
import org.springframework.data.elasticsearch.core.query.NativeSearchQueryBuilder;
import org.springframework.stereotype.Component;

import java.util.HashMap;
import java.util.Map;

/**
 * 自定义排序策略
 */
@Component
public class CustomSortStrategy {
    
    @Autowired
    private ElasticsearchOperations elasticsearchOperations;
    
    /**
     * 综合排序（搜索相关性 + 销量 + 评分）
     * Score = _score * 0.4 + sales_score * 0.3 + rating_score * 0.3
     */
    public org.elasticsearch.search.sort.ScriptSortBuilder buildCompositeSort() {
        Map<String, Object> params = new HashMap<>();
        params.put("sales_weight", 0.3);
        params.put("rating_weight", 0.3);
        params.put("max_sales", 10000);
        params.put("max_rating", 5.0);
        
        String scriptCode = 
            "double salesScore = doc['sales'].value / params.max_sales;" +
            "double ratingScore = doc['rating'].value / params.max_rating;" +
            "return _score * 0.4 + salesScore * params.sales_weight + " +
            "ratingScore * params.rating_weight;";
        
        Script script = new Script(
            ScriptType.INLINE,
            "painless",
            scriptCode,
            params
        );
        
        return org.elasticsearch.search.sort.SortBuilders.scriptSort(script, "number")
            .order(org.elasticsearch.search.sort.SortOrder.DESC);
    }
    
    /**
     * 地理位置排序（距离加权）
     * Score = _score / (1 + distance_km * 0.1)
     */
    public org.elasticsearch.search.sort.ScriptSortBuilder buildGeoSort(
            double lat, double lon) {
        
        Map<String, Object> params = new HashMap<>();
        params.put("lat", lat);
        params.put("lon", lon);
        
        String scriptCode = 
            "double distance = doc['location'].arcDistance(params.lat, params.lon) / 1000;" +
            "return _score / (1 + distance * 0.1);";
        
        Script script = new Script(
            ScriptType.INLINE,
            "painless",
            scriptCode,
            params
        );
        
        return org.elasticsearch.search.sort.SortBuilders.scriptSort(script, "number")
            .order(org.elasticsearch.search.sort.SortOrder.DESC);
    }
    
    /**
     * 时间衰减排序（更近的内容权重更高）
     * Score = _score * exp(-decay * hours)
     */
    public org.elasticsearch.search.sort.ScriptSortBuilder buildTimeDecaySort(
            double decayFactor) {
        
        Map<String, Object> params = new HashMap<>();
        params.put("decay", decayFactor);
        params.put("now", System.currentTimeMillis());
        
        String scriptCode = 
            "long createMs = doc['createTime'].value.toInstant().toEpochMilli();" +
            "double hours = (params.now - createMs) / (1000 * 60 * 60);" +
            "return _score * Math.exp(-params.decay * hours);";
        
        Script script = new Script(
            ScriptType.INLINE,
            "painless",
            scriptCode,
            params
        );
        
        return org.elasticsearch.search.sort.SortBuilders.scriptSort(script, "number")
            .order(org.elasticsearch.search.sort.SortOrder.DESC);
    }
}

2.5 分布式检索优化

分片策略

package com.example.search.shard;

import lombok.extern.slf4j.Slf4j;
import org.springframework.stereotype.Component;

import java.util.*;

/**
 * 分片路由策略
 */
@Slf4j
@Component
public class ShardRoutingStrategy {
    
    /**
     * 按用户ID分片
     * 确保同一用户的数据在同一分片
     */
    public int routeByUserId(Long userId, int shardCount) {
        return (int) (userId % shardCount);
    }
    
    /**
     * 按时间范围分片
     * 适用于时序数据
     */
    public String getShardIndexByTime(Date time, String baseIndexName) {
        Calendar cal = Calendar.getInstance();
        cal.setTime(time);
        
        int year = cal.get(Calendar.YEAR);
        int month = cal.get(Calendar.MONTH) + 1;
        
        return String.format("%s_%d_%02d", baseIndexName, year, month);
    }
    
    /**
     * 自定义路由键
     */
    public int routeByCustomKey(String routingKey, int shardCount) {
        int hash = routingKey.hashCode();
        return Math.abs(hash) % shardCount;
    }
    
    /**
     * 一致性哈希分片
     * 用于动态扩容场景
     */
    public static class ConsistentHashSharding {
        
        private final TreeMap<Integer, String> ring = new TreeMap<>();
        private final int virtualNodes = 150;
        
        public ConsistentHashSharding(List<String> shards) {
            for (String shard : shards) {
                for (int i = 0; i < virtualNodes; i++) {
                    String virtualNode = shard + "#" + i;
                    int hash = hash(virtualNode);
                    ring.put(hash, shard);
                }
            }
        }
        
        public String route(String key) {
            if (ring.isEmpty()) {
                return null;
            }
            
            int hash = hash(key);
            Map.Entry<Integer, String> entry = ring.ceilingEntry(hash);
            
            if (entry == null) {
                entry = ring.firstEntry();
            }
            
            return entry.getValue();
        }
        
        private int hash(String key) {
            return key.hashCode() & Integer.MAX_VALUE;
        }
    }
    
    /**
     * 示例
     */
    public static void main(String[] args) {
        // 一致性哈希示例
        List<String> shards = Arrays.asList("shard1", "shard2", "shard3");
        ConsistentHashSharding chs = new ConsistentHashSharding(shards);
        
        System.out.println("一致性哈希路由:");
        for (int i = 0; i < 10; i++) {
            String key = "user_" + i;
            String shard = chs.route(key);
            System.out.println(key + " -> " + shard);
        }
    }
}

查询优化

package com.example.search.optimize;

import org.elasticsearch.action.search.SearchRequest;
import org.elasticsearch.action.search.SearchResponse;
import org.elasticsearch.client.RequestOptions;
import org.elasticsearch.client.RestHighLevelClient;
import org.elasticsearch.common.unit.TimeValue;
import org.elasticsearch.index.query.QueryBuilders;
import org.elasticsearch.search.builder.SearchSourceBuilder;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.stereotype.Component;

import java.io.IOException;
import java.util.concurrent.TimeUnit;

/**
 * 查询优化器
 */
@Component
public class QueryOptimizer {
    
    @Autowired
    private RestHighLevelClient esClient;
    
    /**
     * 滚动查询（Scroll API）
     * 适用于大数据量导出
     */
    public void scrollQuery(String indexName, int batchSize, 
                            ScrollCallback callback) throws IOException {
        
        SearchRequest searchRequest = new SearchRequest(indexName);
        SearchSourceBuilder sourceBuilder = new SearchSourceBuilder();
        sourceBuilder.query(QueryBuilders.matchAllQuery());
        sourceBuilder.size(batchSize);
        
        searchRequest.source(sourceBuilder);
        searchRequest.scroll(TimeValue.timeValueMinutes(1L));
        
        SearchResponse response = esClient.search(searchRequest, RequestOptions.DEFAULT);
        String scrollId = response.getScrollId();
        
        // 处理第一批结果
        callback.process(response);
        
        // 循环获取后续批次
        while (true) {
            org.elasticsearch.action.search.SearchScrollRequest scrollRequest = 
                new org.elasticsearch.action.search.SearchScrollRequest(scrollId);
            scrollRequest.scroll(TimeValue.timeValueMinutes(1L));
            
            response = esClient.scroll(scrollRequest, RequestOptions.DEFAULT);
            
            if (response.getHits().getHits().length == 0) {
                break;
            }
            
            callback.process(response);
        }
        
        // 清除滚动上下文
        org.elasticsearch.action.search.ClearScrollRequest clearRequest = 
            new org.elasticsearch.action.search.ClearScrollRequest();
        clearRequest.addScrollId(scrollId);
        esClient.clearScroll(clearRequest, RequestOptions.DEFAULT);
    }
    
    /**
     * 批量查询（Multi Search）
     * 一次请求执行多个查询
     */
    public org.elasticsearch.action.search.MultiSearchResponse multiSearch(
            List<SearchRequest> requests) throws IOException {
        
        org.elasticsearch.action.search.MultiSearchRequest multiRequest = 
            new org.elasticsearch.action.search.MultiSearchRequest();
        
        for (SearchRequest request : requests) {
            multiRequest.add(request);
        }
        
        return esClient.msearch(multiRequest, RequestOptions.DEFAULT);
    }
    
    /**
     * 查询建议优化
     */
    public SearchSourceBuilder optimizeQuery(SearchSourceBuilder builder) {
        // 1. 限制返回字段
        builder.fetchSource(new String[]{"id", "name", "price"}, null);
        
        // 2. 禁用不需要的特性
        builder.explain(false);
        builder.version(false);
        builder.trackScores(false);
        
        // 3. 设置超时
        builder.timeout(new TimeValue(3, TimeUnit.SECONDS));
        
        // 4. 使用缓存
        builder.requestCache(true);
        
        return builder;
    }
    
    /**
     * 滚动回调接口
     */
    public interface ScrollCallback {
        void process(SearchResponse response);
    }
}

---

三、可运行Java代码示例

完整的搜索服务实现

package com.example.search;

import org.springframework.boot.SpringApplication;
import org.springframework.boot.autoconfigure.SpringBootApplication;
import org.springframework.data.elasticsearch.repository.config.EnableElasticsearchRepositories;

/**
 * 搜索服务启动类
 */
@SpringBootApplication
@EnableElasticsearchRepositories
public class SearchServiceApplication {
    
    public static void main(String[] args) {
        SpringApplication.run(SearchServiceApplication.class, args);
    }
}

package com.example.search.repository;

import com.example.search.mapping.ProductIndex;
import org.springframework.data.domain.Page;
import org.springframework.data.domain.Pageable;
import org.springframework.data.elasticsearch.annotations.Query;
import org.springframework.data.elasticsearch.repository.ElasticsearchRepository;

import java.util.List;

/**
 * 商品索引Repository
 */
public interface ProductIndexRepository extends ElasticsearchRepository<ProductIndex, Long> {
    
    /**
     * 按名称查询
     */
    Page<ProductIndex> findByName(String name, Pageable pageable);
    
    /**
     * 按分类查询
     */
    Page<ProductIndex> findByCategoryId(Long categoryId, Pageable pageable);
    
    /**
     * 按品牌查询
     */
    List<ProductIndex> findByBrand(String brand);
    
    /**
     * 按价格区间查询
     */
    @Query("{\"range\": {\"price\": {\"gte\": ?0, \"lte\": ?1}}}")
    Page<ProductIndex> findByPriceRange(Double minPrice, Double maxPrice, Pageable pageable);
    
    /**
     * 多字段模糊查询
     */
    @Query("{\"multi_match\": {\"query\": \"?0\", \"fields\": [\"name\", \"description\"]}}")
    Page<ProductIndex> searchByKeyword(String keyword, Pageable pageable);
}

package com.example.search.service;

import com.example.search.mapping.ProductIndex;
import com.example.search.repository.ProductIndexRepository;
import lombok.extern.slf4j.Slf4j;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.data.domain.Page;
import org.springframework.data.domain.PageRequest;
import org.springframework.scheduling.annotation.Async;
import org.springframework.stereotype.Service;

import java.util.List;

/**
 * 商品索引服务
 */
@Slf4j
@Service
public class ProductIndexService {
    
    @Autowired
    private ProductIndexRepository repository;
    
    @Autowired
    private ProductSearchService searchService;
    
    /**
     * 创建或更新索引
     */
    public void save(ProductIndex product) {
        repository.save(product);
        log.info("保存商品索引: productId={}", product.getId());
    }
    
    /**
     * 批量保存
     */
    public void saveAll(List<ProductIndex> products) {
        repository.saveAll(products);
        log.info("批量保存商品索引: count={}", products.size());
    }
    
    /**
     * 删除索引
     */
    public void delete(Long productId) {
        repository.deleteById(productId);
        log.info("删除商品索引: productId={}", productId);
    }
    
    /**
     * 批量删除
     */
    public void deleteAll(List<Long> productIds) {
        repository.deleteAllById(productIds);
        log.info("批量删除商品索引: count={}", productIds.size());
    }
    
    /**
     * 同步商品数据到ES
     */
    @Async
    public void syncProducts(List<Long> productIds) {
        // 从数据库查询商品
        // List<Product> products = productRepository.findAllById(productIds);
        
        // 转换为索引文档
        // List<ProductIndex> indices = products.stream()
        //     .map(this::convertToIndex)
        //     .collect(Collectors.toList());
        
        // 批量保存
        // saveAll(indices);
        
        log.info("同步商品数据完成: count={}", productIds.size());
    }
    
    /**
     * 全量重建索引
     */
    @Async
    public void rebuildIndex() {
        log.info("开始全量重建索引");
        
        // 删除旧索引
        repository.deleteAll();
        
        // 分批同步
        int page = 0;
        int batchSize = 1000;
        
        while (true) {
            // 从数据库分批查询
            // Page<Product> productPage = productRepository.findAll(
            //     PageRequest.of(page, batchSize));
            
            // if (productPage.isEmpty()) {
            //     break;
            // }
            
            // 转换并保存
            // List<ProductIndex> indices = productPage.getContent().stream()
            //     .map(this::convertToIndex)
            //     .collect(Collectors.toList());
            // saveAll(indices);
            
            page++;
        }
        
        log.info("全量重建索引完成");
    }
    
    /**
     * 转换为索引文档
     */
    private ProductIndex convertToIndex(Object product) {
        // 实际实现
        return new ProductIndex();
    }
}

---

四、实战应用场景

场景一：电商搜索

/**
 * 电商搜索服务
 */
@Service
public class EcommerceSearchService {
    
    @Autowired
    private ProductSearchService productSearchService;
    
    /**
     * 商品搜索（带过滤和排序）
     */
    public SearchResult searchProducts(SearchRequest request) {
        // 1. 构建查询条件
        ProductSearchRequest searchReq = new ProductSearchRequest();
        searchReq.setKeyword(request.getKeyword());
        searchReq.setCategoryId(request.getCategoryId());
        searchReq.setBrand(request.getBrand());
        searchReq.setMinPrice(request.getMinPrice());
        searchReq.setMaxPrice(request.getMaxPrice());
        searchReq.setTags(request.getTags());
        searchReq.setSortBy(request.getSortBy());
        
        // 2. 执行搜索
        Page<ProductIndex> productPage = productSearchService.advancedSearch(
            searchReq, request.getPage(), request.getSize());
        
        // 3. 构建返回结果
        SearchResult result = new SearchResult();
        result.setProducts(productPage.getContent());
        result.setTotal(productPage.getTotalElements());
        result.setPage(request.getPage());
        result.setSize(request.getSize());
        
        // 4. 获取聚合信息
        result.setAggregations(getAggregations(searchReq));
        
        return result;
    }
    
    /**
     * 获取聚合信息（品牌、分类等）
     */
    private Map<String, Object> getAggregations(ProductSearchRequest request) {
        // 实际实现：使用ES聚合功能
        return new HashMap<>();
    }
}

场景二：搜索建议

/**
 * 搜索建议服务
 */
@Service
public class SearchSuggestService {
    
    @Autowired
    private ElasticsearchOperations elasticsearchOperations;
    
    /**
     * 搜索提示
     */
    public List<String> suggest(String prefix, int size) {
        NativeSearchQuery query = new NativeSearchQueryBuilder()
            .withQuery(QueryBuilders.matchPhrasePrefixQuery("name", prefix))
            .withMaxResults(size)
            .build();
        
        SearchHits<ProductIndex> hits = 
            elasticsearchOperations.search(query, ProductIndex.class);
        
        return hits.stream()
            .map(hit -> hit.getContent().getName())
            .distinct()
            .limit(size)
            .collect(Collectors.toList());
    }
    
    /**
     * 热门搜索词
     */
    public List<String> getHotSearchTerms(int size) {
        String key = "search:hot:terms";
        // 从Redis获取
        // return redisTemplate.opsForList().range(key, 0, size - 1);
        return List.of("Java", "Python", "Spring", "MySQL");
    }
    
    /**
     * 记录搜索词
     */
    public void recordSearchTerm(String term) {
        String key = "search:hot:terms";
        // redisTemplate.opsForList().leftPush(key, term);
        // redisTemplate.opsForList().trim(key, 0, 99); // 只保留前100个
    }
}

---

五、总结与最佳实践

5.1 架构设计要点

设计维度	推荐方案	说明
索引设计	合理分片	根据数据量选择分片数
映射设计	精确类型	Text vs Keyword选择
查询优化	Filter缓存	过滤条件使用filter
排序策略	BM25 + 自定义	结合业务权重
数据同步	异步 + 批量	减少索引压力

5.2 性能优化建议

1. 索引优化：合理设置分片数、副本数
2. 查询优化：使用filter缓存、限制返回字段
3. 硬件优化：SSD存储、足够内存
4. JVM调优：Heap Size不超过32GB

5.3 监控指标

• 查询性能：平均响应时间、P99延迟
• 索引性能：索引速率、索引延迟
• 集群健康：节点状态、分片状态
• 资源使用：CPU、内存、磁盘IO

5.4 扩展性考虑

1. 水平扩展：增加节点分摊压力
2. 冷热分离：历史数据归档
3. 多集群：读写分离集群
4. 混合检索：结合向量检索

---

六、思考与练习

思考题

1. 基础题：倒排索引相比正向索引有哪些优势？为什么搜索引擎普遍采用倒排索引？
2. 进阶题：BM25算法相比TF-IDF有哪些改进？为什么Elasticsearch默认使用BM25？
3. 实战题：设计一个支持千万级商品的电商搜索系统，需要支持关键词搜索、分类筛选、价格排序、销量排序等功能，请给出完整的索引设计和查询优化方案。

编程练习

练习：实现一个简易的全文搜索引擎，要求：

1. 基于倒排索引实现文档索引和检索
2. 支持AND、OR、NOT布尔查询
3. 实现TF-IDF或BM25相关性排序
4. 支持高亮显示匹配词

章节关联

• 前置章节：《订单系统设计详解》
• 后续章节：分布式缓存设计
• 扩展阅读：《Elasticsearch权威指南》、《信息检索导论》、Lucene官方文档

---

📝 下一章预告

至此，系统设计系列文章已全部完成。接下来将进入分布式系统进阶内容，包括分布式缓存、消息队列、分布式事务等核心主题。

---

本章完