我的征尘是星辰大海。。。
The dirt and dust from my pilgrimage forms oceans of stars...
-------当记忆的篇章变得零碎，当追忆的图片变得模糊，我们只能求助于数字存储的永恒的回忆
作者:黄教授
手机视频列表
从NGram到Transformer轻量级语义模糊检索的实践感悟
视频
音频
原始脚本
从 InGram 到 Transformer，轻量级语义检索的实践感悟与技术脉络。 引言，理解的本质在于实践后的通透。 NLP 领域的技术演进从来不是空中楼阁式的理论创新，而是问题驱动、加工程妥协、加硬件迭代共同作用的结果。 最近在做轻量级中文文本模糊检索的概念验证时。 我重新梳理了从早期关键字匹配到 Transformer 大模型的技术脉络，才真正理解了一个道理，有些技术，听别人讲一千遍不如自己动手跑一遍。 就像学生时代听老师讲公式，总觉得听懂了，但到了考场才发现，真正的理解是能把知识拆解成解决问题的步骤。 NLP 技术的眼镜也是如此，从语法规则到统计模型，再到全局语义理解，每一步都踩在当时技术能落地的基石上，每一次突破都离不开硬件算力的支撑。 本文想分享的正是这次实践中关于轻量级检索定位、技术眼镜逻辑、工程化取舍的思考。 一、技术眼镜的核心矛盾，语法规则 vs 统计规律。 自然语言处理的起点是模仿编程语言的解析思路。 早期研究者们认为，既然 C 加加、Python 这类编程语言可以通过上下文无关文法 CFG 定义语法规则。 用 Parser 逐词解析，自然语言也能照此办理。 他们试图为中文、英文制定一套主谓宾的严格规则，用左匹配、右匹配的方式拆解句子，就像编译器解析代码一样。 但很快大家发现这条路走不通，自然语言的容错性和模糊性是编程语言的语法规则无法覆盖的。 比如中文里的倒装句，饭吃了没？你不符合主谓宾顺序，却不影响理解。 再比如你吃饭了没和你有没有吃饭，表达方式不同，语义却完全一致。 更不用说每天都在诞生的网络新词、方言俚语，规则库永远追不上语言的变化。 这个阶段的检索停留在关键字匹配加正则表达式的层面，差一个字就匹配失败，无法处理同一替换，规则维护成本高到离谱。 我之前尝试的 E L M 方案，本质上就属于这个阶段，依赖字符级的精准匹配，模糊检索能力极弱。 矛盾的转折点出现在统计模型的崛起。 研究者们发现，人类理解语言时，其实更关注关键词出现的频率，而非严格的语法顺序。 有个经典实验很能说明问题，把顺序写成序顺，人们快速扫描时依然能识别出正确的语义，这正是磁带模型 bag of words 的核心假设。 文本的语义主要由高频词的贡献关系决定，于是，Ingram 家族应运而生。 Unigram 统计单词频率，Bigram 统计相邻两个词的贡献频率，Trigram 扩展到三个词。 他们不再纠结语法是否正确，而是通过统计词与词相邻出现的概率，捕捉局部上下文关联。 比如，举头望明月这句话，Bigram 会关注举头望望明月的贡献关系，哪怕句子顺序颠倒成明月望举头，模型也能通过高频词的关联判 断出两者语义相近。 这个阶段的技术终于让检索从字符匹配升级到了语义模糊匹配。 而我们现在实践的 CPP 结巴加 Bigram 组合正是这个阶段的优化方案。 CPP 结巴解决中文分词的边界问题。 用词典加词频加动态规划的方式，把句子拆成符合人类习惯的词。 Backgram 则在分词结果上捕捉词与词的局部关联，让检索具备了模糊匹配的能力。 二、从局部关联到全局语义，词向量与 Transformer 的突破。 Ingram 模型解决了局部上下文的问题，但它的局限也很明显，上下文窗口是固定的。 Bigram 只能看相邻的两个词，无法处理长距离关联。 比如他说的那个词，就是昨天提到的 dico 里的 you，you 和 dico 的关联跨越了整句话，Ingram 模型根本捕捉不到。 于是分布式词向量模型登场了，以 Word RVec、Google 2013年开源和 FastText Facebook 开源为代表。 他们的核心思路是把每个词映射到一个低维向量空间，让语义相近的词向量距离更近。 比如举头和抬头，在向量空间里是邻近的点。 苹果手机和吃苹果里的苹果，虽然暂时无法区分，但已经能通过向量计算、余弦相似度实现精准的模糊检索。 Word R Vec 用 Skip-Gram 或 CBOW 模型学习词在上下文里的关联。 FastText 则引入了 subword 机制。 把词拆成更小的字符片段，解决了生僻词、未登录词的问题。 这一步让检索技术实现了第二次飞跃，从基于规则的局部匹配。 升级到了基于统计的隐性语义匹配，但词向量模型依然没有跳出磁带模型的本质。 句子向量是词向量的加权平均，完全忽略了词的顺序和全局上下文。 举头望明月和明月望举头的向量，在模型看来几乎是一样的。 真正的革命性突破来自2017年那篇划时代的论文。 Attention is all you need Transformer 架构横空出世。 Transformer 解决的核心问题是高效捕捉全局上下文关联。 它的两大核心设计完美兼顾了统计规律和顺序信息。 从 Ingram 到 Transformer，轻量级语义检索的实践感悟与技术脉络。 三、轻量级检索的定位，在够用与完美之间取舍。 梳理完技术脉络，再回头看我们的轻量级检索需求，答案就很清晰了。 Transformer 大模型确实能实现全局语义理解，能区分多义词，能处理复杂的句式。 但它的代价是巨大的，模型体积大、算力要求高、部署成本昂贵。 而且对于小说句段定位、知识库模糊检索这类场景，它的能力是过剩的。 我们需要的不是理解语言的人工智能。 而是高效匹配语义的工具。 这个工具不需要懂语法，不需要懂上下文的深层含义，只需要能判断两个文本的语义是否相近。 所以，CPP、结巴加 Bargram、Word to Vec、FastText 的组合才是这个场景下的最优解。 它处于传统关键字检索和 Transformer 大模型之间的中间地带。 比关键字检索聪明，能处理同义替换、模糊匹配。 比大模型轻便，纯 CPU 部署，毫秒级响应。 低依赖，易维护。 它放弃了全局语义理解的完美追求，却抓住了词频加局部关联的核心需求，足以覆盖80%的轻量及检索场景。 这就是工程化的智慧，不是技术越新越好，而是越贴合需求越 好。 四、实践感悟。 站在巨人的肩膀上，先得爬上去。 这一次实践让我深刻体会到，技术学习从来不是追星，而是溯源。 很多人一上来就啃 Transformer 啃大模型。 却不知道他们的底层逻辑来自于十几年前的统计模型。 不知道自注意力机制本质上是无限窗口的 Ingram，不知道位置编码是为了弥补磁带模型忽略顺序的缺陷。 就像盖房子，不打牢地基，就想建摩天大楼，最终只会摇摇欲坠。 NLP 领域的技术从来不是孤立的。 从语法规则到 Ingram，从词向量到 Transformer，每一步都是对前一步的继承和突破。 我们现在做的复古实践不是重新造轮子，而是为了真正理解巨人的肩膀在哪里。 最后想说，实践出真知。 听别人讲一百遍磁带模型、自注意力机制，不如自己动用手写一段分词代码，跑一次向量相似度计算。 只有亲手踩过坑，才能明白技术演进的痛点在哪里，才能明白每一个方案背后的工程取舍，这才是真正的理解。
修正脚本
从 InGram 到 Transformer，轻量级语义检索的实践感悟与技术脉络。 引言，理解的本质在于实践后的通透。 NLP 领域的技术演进从来不是空中楼阁式的理论创新，而是问题驱动、工程妥协、硬件迭代共同作用的结果。 最近在做轻量级中文文本模糊检索的概念验证时。 我重新梳理了从早期关键字匹配到 Transformer 大模型的技术脉络，才真正理解了一个道理，有些技术，听别人讲一千遍不如自己动手跑一遍。 就像学生时代听老师讲公式，总觉得听懂了，但到了考场才发现，真正的理解是能把知识拆解成解决问题的步骤。 NLP 技术的演进也是如此，从语法规则到统计模型，再到全局语义理解，每一步都踩在当时技术能落地的基石上，每一次突破都离不开硬件算力的支撑。 本文想分享的正是这次实践中关于轻量级检索定位、技术演进逻辑、工程化取舍的思考。 一、技术演进的核心矛盾，语法规则 vs 统计规律。 自然语言处理的起点是模仿编程语言的解析思路。 早期研究者们认为，既然 C 加加、Python 这类编程语言可以通过上下文无关文法 CFG 定义语法规则。 用 Parser 逐词解析，自然语言也能照此办理。 他们试图为中文、英文制定一套主谓宾的严格规则，用左匹配、右匹配的方式拆解句子，就像编译器解析代码一样。 但很快大家发现这条路走不通，自然语言的容错性和模糊性是编程语言的语法规则无法覆盖的。 比如中文里的倒装句，饭吃了没？它不符合主谓宾顺序，却不影响理解。 再比如你吃饭了没和你有没有吃饭，表达方式不同，语义却完全一致。 更不用说每天都在诞生的网络新词、方言俚语，规则库永远追不上语言的变化。 这个阶段的检索停留在关键字匹配加正则表达式的层面，差一个字就匹配失败，无法处理同义替换，规则维护成本高到离谱。 我之前尝试的 E L M 方案，本质上就属于这个阶段，依赖字符级的精准匹配，模糊检索能力极弱。 矛盾的转折点出现在统计模型的崛起。 研究者们发现，人类理解语言时，其实更关注关键词出现的频率，而非严格的语法顺序。 有个经典实验很能说明问题，把顺序写成序顺，人们快速扫描时依然能识别出正确的语义，这正是词袋模型 bag of words 的核心假设。 文本的语义主要由高频词的共现关系决定，于是，Ingram 家族应运而生。 Unigram 统计单词频率，Bigram 统计相邻两个词的共现频率，Trigram 扩展到三个词。 他们不再纠结语法是否正确，而是通过统计词与词相邻出现的概率，捕捉局部上下文关联。 比如，举头望明月这句话，Bigram 会关注相邻词的共现关系，哪怕句子顺序颠倒成明月望举头，模型也能通过高频词的关联判断出两者语义相近。 这个阶段的技术终于让检索从字符匹配升级到了语义模糊匹配。 而我们现在实践的 CPP 结巴加 Bigram 组合正是这个阶段的优化方案。 CPP 结巴解决中文分词的边界问题。 用词典加词频加动态规划的方式，把句子拆成符合人类习惯的词。 Bigram 则在分词结果上捕捉词与词的局部关联，让检索具备了模糊匹配的能力。 二、从局部关联到全局语义，词向量与 Transformer 的突破。 Ingram 模型解决了局部上下文的问题，但它的局限也很明显，上下文窗口是固定的。 Bigram 只能看相邻的两个词，无法处理长距离关联。 比如他说的那个词，就是昨天提到的 dico 里的 you，you 和 dico 的关联跨越了整句话，Ingram 模型根本捕捉不到。 于是分布式词向量模型登场了，以 Word RVec、Google 2013年开源和 FastText Facebook 开源为代表。 他们的核心思路是把每个词映射到一个低维向量空间，让语义相近的词向量距离更近。 比如举头和抬头，在向量空间里是邻近的点。 苹果手机和吃苹果里的苹果，虽然暂时无法区分，但已经能通过向量计算、余弦相似度实现精准的模糊检索。 Word R Vec 用 Skip-Gram 或 CBOW 模型学习词在上下文里的关联。 FastText 则引入了 subword 机制。 把词拆成更小的字符片段，解决了生僻词、未登录词的问题。 这一步让检索技术实现了第二次飞跃，从基于规则的局部匹配，升级到了基于统计的隐性语义匹配，但词向量模型依然没有跳出词袋模型的本质。 句子向量是词向量的加权平均，完全忽略了词的顺序和全局上下文。 举头望明月和明月望举头的向量，在模型看来几乎是一样的。 真正的革命性突破来自2017年那篇划时代的论文。 Attention is all you need Transformer 架构横空出世。 Transformer 解决的核心问题是高效捕捉全局上下文关联。 它的两大核心设计完美兼顾了统计规律和顺序信息。 三、轻量级检索的定位，在够用与完美之间取舍。 梳理完技术脉络，再回头看我们的轻量级检索需求，答案就很清晰了。 Transformer 大模型确实能实现全局语义理解，能区分多义词，能处理复杂的句式。 但它的代价是巨大的，模型体积大、算力要求高、部署成本昂贵。 而且对于小说句段定位、知识库模糊检索这类场景，它的能力是过剩的。 我们需要的不是理解语言的人工智能。 而是高效匹配语义的工具。 这个工具不需要懂语法，不需要懂上下文的深层含义，只需要能判断两个文本的语义是否相近。 所以，CPP、结巴加 Bigram、Word to Vec、FastText 的组合才是这个场景下的最优解。 它处于传统关键字检索和 Transformer 大模型之间的中间地带。 比关键字检索聪明，能处理同义替换、模糊匹配。 比大模型轻便，纯 CPU 部署，毫秒级响应。 低依赖，易维护。 它放弃了全局语义理解的完美追求，却抓住了词频加局部关联的核心需求，足以覆盖80%的轻量级检索场景。 这就是工程化的智慧，不是技术越新越好，而是越贴合需求越好。 四、实践感悟。 站在巨人的肩膀上，先得爬上去。 这一次实践让我深刻体会到，技术学习从来不是追星，而是溯源。 很多人一上来就啃 Transformer 啃大模型。 却不知道他们的底层逻辑来自于十几年前的统计模型。 不知道自注意力机制本质上是无限窗口的 Ingram，不知道位置编码是为了弥补词袋模型忽略顺序的缺陷。 就像盖房子，不打牢地基，就想建摩天大楼，最终只会摇摇欲坠。 NLP 领域的技术从来不是孤立的。 从语法规则到 Ingram，从词向量到 Transformer，每一步都是对前一步的继承和突破。 我们现在做的复古实践不是重新造轮子，而是为了真正理解巨人的肩膀在哪里。 最后想说，实践出真知。 听别人讲一百遍词袋模型、自注意力机制，不如自己动手写一段分词代码，跑一次向量相似度计算。 只有亲手踩过坑，才能明白技术演进的痛点在哪里，才能明白每一个方案背后的工程取舍，这才是真正的理解。
英文翻译
From InGram to Transformer: Practical Insights and Technical Evolution of Lightweight Semantic Retrieval. Introduction: The essence of understanding lies in clarity after hands-on practice. The technological evolution in NLP has never been an ivory-tower theoretical innovation, but rather the result of problem-driven approaches, engineering trade-offs, and hardware iteration. Recently, while working on a proof-of-concept for lightweight fuzzy retrieval of Chinese text, I revisited the technical progression from early keyword matching to Transformer large models, and truly understood a principle: some technologies are better grasped by running them yourself once than by hearing others explain them a thousand times. It's like when we were students: we thought we understood a formula after the teacher explained it, but only in the exam did we realize that true understanding means being able to break down knowledge into actionable steps. The evolution of NLP technology is the same—from grammatical rules to statistical models, and then to global semantic understanding. Each step was grounded in the technology that could be practically deployed at the time, and every breakthrough relied on the support of hardware computing power. This article aims to share the reflections from this practice on lightweight retrieval positioning, the logic of technological evolution, and engineering trade-offs. I. The Core Contradiction in Technological Evolution: Grammatical Rules vs. Statistical Patterns The starting point of natural language processing was to imitate the parsing approach of programming languages. Early researchers believed that since programming languages like C++ and Python can define grammatical rules using context-free grammars (CFG) and parse words one by one with a parser, natural language could be handled the same way. They attempted to establish strict subject-verb-object rules for Chinese and English, using left-matching and right-matching methods to break down sentences, just like a compiler parsing code. But they quickly discovered that this path was unviable. The fault tolerance and ambiguity of natural language cannot be covered by the grammatical rules of programming languages. For example, inverted sentences in Chinese, like "饭吃了没?" (Have you eaten?), do not follow the subject-verb-object order but still convey meaning. Similarly, "你吃饭了没" and "你有没有吃饭" have different expressions but the exact same meaning. Moreover, with new internet slang and dialect terms emerging every day, rule libraries can never keep up with language changes. Retrieval at this stage remained at the level of keyword matching plus regular expressions—failure to match if even one character differed, inability to handle synonym substitution, and prohibitively high maintenance costs for rules. The E L M solution I previously attempted essentially belonged to this stage, relying on character-level precise matching with extremely weak fuzzy retrieval capability. The turning point of the contradiction came with the rise of statistical models. Researchers found that when humans understand language, they actually focus more on the frequency of keywords rather than strict grammatical order. A classic experiment illustrates this well: when the word order is scrambled, such as "顺序" written as "序顺", people can still quickly identify the correct semantics. This is the core assumption of the bag-of-words model: the semantics of text are mainly determined by the co-occurrence of high-frequency words. Thus, the InGram family emerged. Unigram counts word frequency, Bigram counts the co-occurrence frequency of two adjacent words, and Trigram extends to three words. They no longer fixated on grammatical correctness, but instead captured local contextual associations by statistically modeling the probability of adjacent word pairs. For example, in the phrase "举头望明月" (Looking up, I see the moon), Bigram focuses on the co-occurrence of adjacent words. Even if the order is reversed to "明月望举头", the model can still determine semantic similarity through the association of high-frequency words. With this stage, retrieval finally evolved from character matching to semantic fuzzy matching. The combination of CPP Jieba plus Bigram that we are now practicing is precisely an optimized solution at this stage: CPP Jieba addresses the boundary issues in Chinese word segmentation, using a dictionary plus word frequency plus dynamic programming to split sentences into words that align with human habits; Bigram then captures local word associations on the segmented results, enabling fuzzy matching in retrieval. II. From Local Association to Global Semantics: Breakthroughs with Word Vectors and Transformer The InGram model solved the problem of local context, but its limitation was obvious: the context window was fixed. Bigram could only look at two adjacent words and could not handle long-distance associations. For instance, in the sentence "他说的那个词，就是昨天提到的dico里的you" (That word he mentioned is the you in the dico referred to yesterday), the association between "you" and "dico" spans the entire sentence, which the InGram model simply cannot capture. Then came distributed word vector models, represented by Word2Vec (Google, open-sourced in 2013) and FastText (Facebook, open-sourced). Their core idea is to map each word into a low-dimensional vector space, making vectors of semantically similar words closer in distance. For example, "举头" and "抬头" are neighboring points in the vector space. Meanwhile, "苹果" in "苹果手机" (iPhone) and "吃苹果" (eating apples) cannot be distinguished yet, but fuzzy retrieval with cosine similarity based on vector computation has already become precise. Word2Vec uses Skip-Gram or CBOW models to learn the association of a word within its context. FastText introduces subword mechanisms, breaking words into smaller character fragments, solving the problem of rare and out-of-vocabulary words. This step marked the second leap in retrieval technology: from rule-based local matching to statistics-based implicit semantic matching. However, word vector models still did not escape the essence of the bag-of-words model. Sentence vectors were weighted averages of word vectors, completely ignoring word order and global context. The vectors for "举头望明月" and "明月望举头" would appear almost identical to the model. The truly revolutionary breakthrough came from the landmark 2017 paper "Attention is All You Need", which introduced the Transformer architecture. The core problem Transformer solved was efficiently capturing global contextual associations. Its two key designs perfectly balanced statistical patterns and order information. III. Positioning Lightweight Retrieval: Trade-offs Between Sufficiency and Perfection After reviewing the technical evolution, the answer to our lightweight retrieval needs becomes clear. Transformer large models can indeed achieve global semantic understanding, distinguish polysemous words, and handle complex sentence structures. But their cost is enormous: large model size, high computational demands, and expensive deployment. Moreover, for scenarios like locating specific sentences in novels or fuzzy retrieval from knowledge bases, their capability is excessive. What we need is not an AI that understands language, but a tool that efficiently matches semantics. This tool does not need to understand grammar or deep contextual meaning—it only needs to determine whether two texts are semantically similar. Therefore, the combination of CPP Jieba plus Bigram plus Word2Vec plus FastText is the optimal solution for this scenario. It sits in the middle ground between traditional keyword retrieval and Transformer large models: smarter than keyword retrieval (handling synonym substitution and fuzzy matching), lighter than large models (pure CPU deployment, millisecond response, low dependencies, easy maintenance). It abandons the pursuit of perfect global semantic understanding but captures the core need of word frequency plus local association, covering 80% of lightweight retrieval scenarios. This is the wisdom of engineering: newer technology is not necessarily better; what fits the requirements is best. IV. Practical Insights: Standing on the Shoulders of Giants—First You Have to Climb This practice deeply impressed upon me that learning technology is never about chasing stars but tracing back to origins. Many people jump straight into Transformer and large models without knowing that their underlying logic comes from statistical models from over a decade ago—that the self-attention mechanism is essentially an InGram with an infinite window, and that positional encoding is meant to compensate for the bag-of-words model's neglect of order. It's like building a house: without a solid foundation, attempting a skyscraper will only end in instability. Technologies in NLP are never isolated. From grammatical rules to InGram, from word vectors to Transformer, each step is an inheritance and breakthrough of the previous one. The retro practice we are doing now is not about reinventing the wheel, but about truly understanding where the shoulders of giants lie. Finally, practice yields true knowledge. Hearing others explain bag-of-words models or self-attention mechanisms a hundred times is not as good as writing a piece of segmentation code yourself and running a vector similarity calculation. Only by stumbling into pitfalls firsthand can you understand the pain points of technological evolution and the engineering trade-offs behind each solution. That is true understanding.
back to top