我的征尘是星辰大海。。。
The dirt and dust from my pilgrimage forms oceans of stars...
-------当记忆的篇章变得零碎，当追忆的图片变得模糊，我们只能求助于数字存储的永恒的回忆
作者:黄教授
手机视频列表
从炼金术到精准工程
视频
音频
原始脚本
从炼金术到精准工程，MHC 架构学习心得备忘录。 一、引言。 被通俗化解释误导的困惑。 近期关注到 Deepseek 提出的 MHC 流行约束超连接架构相关内容，翻阅了不少自媒体分享后，却陷入了更深的困惑。 信息高速公路堵车限流这类比喻，对不求甚解的读者或许足够。 但对于想探究技术本质的人而言，反而模糊了核心逻辑。 大模型训练的真正痛点在哪里？MHC 架构的革命性价值究竟体现在何处？本文试图跳出通俗化陷阱，从大模型训练的炼金术式困境切入，结合反向传播梯度调控的底层逻辑，解释 MHC 架构如何将大模型训练从经验试错拉回数学约束的轨道。 为同好提供一份逻辑自洽的学习参考。 二、大模型训练的炼金术困境，在黑暗中摸索的调参之路。 大模型训练的核心是星星，正向传播预测加反向传播调参星星的循环。 但这个过程长期以来充斥着头痛医头、脚痛医脚的被动补救，本质就是一场靠经验和运气的炼金术。 一梯度消失与爆炸，链式求导的连城魔咒，反向传 波的核心是链式求导法则。 梯度从输出层往输入层传递时，是多层权重矩阵导数与激活函数导数的连成结果。 这就埋下了两个极端隐患，梯度消失。 若每层的导数、权重矩阵谱范数、激活函数导数都小于一，连成后梯度会指数级衰减，传到前层 层时趋近于0。 此时权重几乎无法更新，模型学不动了。 梯度爆炸，若每层导数都大于一，连成后梯度会指数级放大，权重更新幅度过大，直接飞出合理范围，模型预测完全失控。 更尴尬的是，这两个问题不会必然出现在训练初期。 初始权重接近随机值，梯度变化平缓。 随着训练推进，权重逐渐偏离初始值，在遇到某批分布极端的 mini batch 数据，如低频词占比极高的文本，才会突然爆发。 此时前期耗费的算力和时间全部作废，只能推倒重来。 二、被动补救工具，ReLU、JLU、残差、连接与 HC 的交往过正。 为了对抗梯度魔咒，工程师们发明了一系列补救工具。 但这些工具反而加剧了调参的复杂性。 ReLU、GELU 激活函数并非放大弱小信号，而是切断负数激活值的传递。 ReLU 直接将负数置零，GELU 平滑趋近于零，本质是扔掉可能引发梯度震荡的负数分支。 但这是一种一刀切的方案，会丢失部分特征信息。 残差连接给信号加一条直通路径，A 下划线 L 等于 Sigma Z L 加 X L。 主路径梯度消失时，残差路径的梯度约等于一，相当于给 梯度传递留后路，但它没有解决梯度失控的根源，只是保底手段。 超连接 HC 通过增加层间连接提升特征表达能力，但也让梯度传递的路径更复杂，进一步放大了连成魔咒的风险。 三，负数权重的由来。 反向调参的矫枉过正训练中，权重出现负数，是反向传播的必然结果。 当当前权重让信号过强时，梯度下降算法会朝着误差减小的方向调整权重，通过让部分权重变成负数，实现反向抵消过强的信号。 但这种调整是无约束的，负数权重过多会导致信号过弱，不得不靠 ReLU、GELU 将负数置零。 信号变弱后，梯度下降又会把权重调大，甚至再次引发梯度爆炸。 如此循环往复，调参变成了按下葫芦浮起瓢的拉锯战。 4、Mini Batch 分割，显存限制下的无奈之举，将训练数据切分成 Mini Batch。 根本原因是 GPU 显存不足，无法一次性加载全部数据，这让训练的不确定性进一步增加。 你永远不知道哪一批 mini batch 会成为压垮骆驼的最后一根稻草。 可能训练了几百批后，才因某批数据触发梯度爆炸，导致前功尽弃。 五、超参数设定，先画图后施工的尴尬模型的层数、学习率、正则化系数等超参数往往是预先设定的。 这就像先画好建筑图纸再施工，训练中发现力学结构不合理，梯度失控，只能销毁半成品，重新画图纸，调整超参数。 这种设计试错、推翻、再设计的循环，是大模型训练成本居高不下的核心原因。 3，MHC 架构的革命性价值，用数学约束终结炼金术。 MHC 架构的伟大之处，不是对现有工具的小修小补，而是从底层架构上用数学约束解决梯度失控的根源。 让大模型训练从炼金术变成精准工程。 一、核心抓手，双随机矩阵的稳定魔法 M H C 的核心是将超连接矩阵约束在双随机矩阵流行内。 双随机矩阵有三个关键特性，非负性，所有元素大于等于0，从根源上杜绝了负数权重 real 又至0的循环，无需激活函数。 擦屁股。 行和列合为一，矩阵乘法对信号的放大缩小倍数被严格锁定在一附近。 链式求导时，每层的梯度传递系数约等于一，谱范数约等于一，彻底打破连乘魔咒，梯度既不会指数级衰减，也不会指数级放大，始终稳定在可控区间。 二、动态维稳，SyncHercanuop 算法的纠偏能力。 MHC 不是初始化后就不管，而是在每一步训练后，通过 SyncHercanuop 算法将更新后的权重矩阵拉回双随机矩阵流行。 这意味着，哪怕梯度下降想把权重往极端值调整，MHC 都会强行将其拽回非负、行和列合为一的稳定区间。 最终实现权重无剧烈波动，梯度无失控风险，无需依赖被动补救工具的理想状态。 三、工程价值，从赌运气到稳扎稳打，MHC 架构带来的工程收益是颠覆性的。 训练稳定性提升，全程无梯度消失、爆炸风险。 无需频繁中断重训，算力和时间不再浪费。 调参复杂度降低，不再需要反复调整激活函数、残差连接参数。 超参数设定从经验试错变成有数学依据的设计。 内存消耗降低，双随机矩阵的约束减少了冗余参数，实验中可降低40%的训练内存占用，适配更大规模的模型。 四、总结。 技术创新的本质是化繁为简，大模型训练的发展史就是一部猩猩从被动补救到主动约束猩猩的进化史。 自媒体的通俗比喻或许能博眼球，但技术的真正魅力藏在解决了什么核心痛点，用什么逻辑解决的细节里。 MHC 价 购的价值不在于创造了什么全新的工具，而在于用严格的数学约束，剥离了大模型训练中层层叠叠的补救手段，回归到信号稳定传递、梯度稳定调控的本质。 这对于所有从事模型训练的开发者而言，都是一次从黑暗摸索到拨云见日的认知升级。 五、后记本文是个人学习 MHC 架构的心得，限于水平难免有疏漏之处。 希望能为同样被通俗化解释误导的同好提供一份参考，也期待更多人能跳出比喻陷阱，真正理解技术创新的底层逻辑。
修正脚本
从炼金术到精准工程，MHC 架构学习心得备忘录。 一、引言。 被通俗化解释误导的困惑。 近期关注到 Deepseek 提出的 MHC 流形约束超连接架构相关内容，翻阅了不少自媒体分享后，却陷入了更深的困惑。 信息高速公路堵车限流这类比喻，对不求甚解的读者或许足够。 但对于想探究技术本质的人而言，反而模糊了核心逻辑。 大模型训练的真正痛点在哪里？MHC 架构的革命性价值究竟体现在何处？本文试图跳出通俗化陷阱，从大模型训练的炼金术式困境切入，结合反向传播梯度调控的底层逻辑，解释 MHC 架构如何将大模型训练从经验试错拉回数学约束的轨道。 为同好提供一份逻辑自洽的学习参考。 二、大模型训练的炼金术困境，在黑暗中摸索的调参之路。 大模型训练的核心就是正向传播预测加反向传播调参的循环。 但这个过程长期以来充斥着头痛医头、脚痛医脚的被动补救，本质就是一场靠经验和运气的炼金术。 一梯度消失与爆炸，链式求导的连乘魔咒，反向传播的核心是链式求导法则。 梯度从输出层往输入层传递时，是多层权重矩阵导数与激活函数导数的连乘结果。 这就埋下了两个极端隐患，梯度消失。 若每层的导数、权重矩阵谱范数、激活函数导数都小于一，连乘后梯度会指数级衰减，传到前层时趋近于0。 此时权重几乎无法更新，模型学不动了。 梯度爆炸，若每层导数都大于一，连乘后梯度会指数级放大，权重更新幅度过大，直接飞出合理范围，模型预测完全失控。 更尴尬的是，这两个问题不会必然出现在训练初期。 初始权重接近随机值，梯度变化平缓。 随着训练推进，权重逐渐偏离初始值，在遇到某批分布极端的 mini batch 数据，如低频词占比极高的文本，才会突然爆发。 此时前期耗费的算力和时间全部作废，只能推倒重来。 二、被动补救工具，ReLU、GELU、残差、连接与 HC 的矫枉过正。 为了对抗梯度魔咒，工程师们发明了一系列补救工具。 但这些工具反而加剧了调参的复杂性。 ReLU、GELU 激活函数并非放大弱小信号，而是切断负数激活值的传递。 ReLU 直接将负数置零，GELU 平滑趋近于零，本质是扔掉可能引发梯度震荡的负数分支。 但这是一种一刀切的方案，会丢失部分特征信息。 残差连接给信号加一条直通路径，A 下划线 L 等于 Sigma Z L 加 X L。 主路径梯度消失时，残差路径的梯度约等于一，相当于给梯度传递留后路，但它没有解决梯度失控的根源，只是保底手段。 超连接 HC 通过增加层间连接提升特征表达能力，但也让梯度传递的路径更复杂，进一步放大了连乘魔咒的风险。 三，负数权重的由来。 反向调参的矫枉过正训练中，权重出现负数，是反向传播的必然结果。 当当前权重让信号过强时，梯度下降算法会朝着误差减小的方向调整权重，通过让部分权重变成负数，实现反向抵消过强的信号。 但这种调整是无约束的，负数权重过多会导致信号过弱，不得不靠 ReLU、GELU 将负数置零。 信号变弱后，梯度下降又会把权重调大，甚至再次引发梯度爆炸。 如此循环往复，调参变成了按下葫芦浮起瓢的拉锯战。 四、Mini Batch 分割，显存限制下的无奈之举，将训练数据切分成 Mini Batch。 根本原因是 GPU 显存不足，无法一次性加载全部数据，这让训练的不确定性进一步增加。 你永远不知道哪一批 mini batch 会成为压垮骆驼的最后一根稻草。 可能训练了几百批后，才因某批数据触发梯度爆炸，导致前功尽弃。 五、超参数设定，先画图后施工的尴尬模型的层数、学习率、正则化系数等超参数往往是预先设定的。 这就像先画好建筑图纸再施工，训练中发现力学结构不合理，梯度失控，只能销毁半成品，重新画图纸，调整超参数。 这种设计试错、推翻、再设计的循环，是大模型训练成本居高不下的核心原因。 三、MHC 架构的革命性价值，用数学约束终结炼金术。 MHC 架构的伟大之处，不是对现有工具的小修小补，而是从底层架构上用数学约束解决梯度失控的根源。 让大模型训练从炼金术变成精准工程。 一、核心抓手，双随机矩阵的稳定魔法 M H C 的核心是将超连接矩阵约束在双随机矩阵流形内。 双随机矩阵有三个关键特性，非负性，所有元素大于等于0，从根源上杜绝了负数权重反复置零的循环，无需激活函数擦屁股。 行和列和为一，矩阵乘法对信号的放大缩小倍数被严格锁定在一附近。 链式求导时，每层的梯度传递系数约等于一，谱范数约等于一，彻底打破连乘魔咒，梯度既不会指数级衰减，也不会指数级放大，始终稳定在可控区间。 二、动态维稳，SyncHercanuop 算法的纠偏能力。 MHC 不是初始化后就不管，而是在每一步训练后，通过 SyncHercanuop 算法将更新后的权重矩阵拉回双随机矩阵流形。 这意味着，哪怕梯度下降想把权重往极端值调整，MHC 都会强行将其拽回非负、行和列和为一的稳定区间。 最终实现权重无剧烈波动，梯度无失控风险，无需依赖被动补救工具的理想状态。 三、工程价值，从赌运气到稳扎稳打，MHC 架构带来的工程收益是颠覆性的。 训练稳定性提升，全程无梯度消失、爆炸风险。 无需频繁中断重训，算力和时间不再浪费。 调参复杂度降低，不再需要反复调整激活函数、残差连接参数。 超参数设定从经验试错变成有数学依据的设计。 内存消耗降低，双随机矩阵的约束减少了冗余参数，实验中可降低40%的训练内存占用，适配更大规模的模型。 四、总结。 技术创新的本质是化繁为简，大模型训练的发展史就是一部从被动补救到主动约束的进化史。 自媒体的通俗比喻或许能博眼球，但技术的真正魅力藏在解决了什么核心痛点，用什么逻辑解决的细节里。 MHC架构的价值不在于创造了什么全新的工具，而在于用严格的数学约束，剥离了大模型训练中层层叠叠的补救手段，回归到信号稳定传递、梯度稳定调控的本质。 这对于所有从事模型训练的开发者而言，都是一次从黑暗摸索到拨云见日的认知升级。 五、后记本文是个人学习 MHC 架构的心得，限于水平难免有疏漏之处。 希望能为同样被通俗化解释误导的同好提供一份参考，也期待更多人能跳出比喻陷阱，真正理解技术创新的底层逻辑。
英文翻译
From Alchemy to Precision Engineering: Notes on Learning the MHC Architecture. I. Introduction Confusion Caused by Oversimplified Explanations. Recently, I came across content related to Deepseek's MHC (Manifold Constrained Hyperconnection) architecture. After browsing through numerous self-media posts, I found myself even more perplexed. Metaphors like "traffic jams on the information highway" might suffice for readers who seek only a superficial understanding. But for those who wish to delve into the technical essence, these metaphors actually obscure the core logic. What are the real pain points in large model training? Where exactly lies the revolutionary value of the MHC architecture? This article attempts to step out of the trap of oversimplification, starting from the alchemy-like predicament of large model training, and combined with the underlying logic of backpropagation gradient regulation, explains how the MHC architecture shifts large model training from empirical trial-and-error onto a track of mathematical constraints. It aims to provide a logically consistent learning reference for fellow enthusiasts. II. The Alchemy Predicament of Large Model Training: Parameter Tuning in the Dark. The core of large model training is the cycle of forward propagation prediction plus backpropagation parameter adjustment. But this process has long been filled with reactive patches—treating symptoms without addressing root causes—essentially an alchemy relying on experience and luck. 1. Gradient Vanishing and Explosion: The Curse of Chained Multiplication in Chain Rule Derivation. The core of backpropagation is the chain rule of differentiation. When gradients propagate from the output layer to the input layer, they are the result of chained multiplication of weight matrix derivatives and activation function derivatives across multiple layers. This lays two extreme risks: - Gradient vanishing: If the derivative of each layer, the spectral norm of the weight matrix, and the activation function derivative are all less than 1, the chained multiplication causes the gradient to decay exponentially, approaching 0 when it reaches the front layers. In this case, weights can hardly update, and the model fails to learn. - Gradient explosion: If the derivative of each layer is greater than 1, the chained multiplication makes the gradient grow exponentially, causing weight updates to be too large and jump out of reasonable ranges, leading to completely uncontrollable model predictions. What's more awkward is that these two problems do not necessarily appear at the beginning of training. Initial weights are close to random values, and gradient changes are smooth. As training progresses, weights gradually deviate from initial values. When encountering a mini-batch with extreme distribution (e.g., text with a very high proportion of low-frequency words), the problems suddenly erupt. At that point, all previously spent computation and time are wasted, and one has to start over from scratch. 2. Passive Remedial Tools: ReLU, GELU, Residual Connections, and Hyperconnections—Overcorrection. To counter the gradient curse, engineers invented a series of remedial tools. But these tools have only increased the complexity of parameter tuning. - ReLU and GELU activation functions do not amplify weak signals; they cut off the transmission of negative activation values. ReLU directly sets negative values to zero; GELU smoothly approaches zero. Essentially, they discard the negative branches that may cause gradient oscillation. But this is a one-size-fits-all solution that loses some feature information. - Residual connections provide a direct path for signals: A_L = σ(Z_L) + X_L. When the main path's gradient vanishes, the residual path's gradient is approximately 1, offering a backup for gradient propagation. However, it doesn't solve the root cause of gradient instability; it's just a safety net. - Hyperconnections (HC) enhance feature representation by increasing inter-layer connections, but they also make gradient propagation paths more complex, further amplifying the risk of the chained multiplication curse. 3. The Origin of Negative Weights: Overcorrection in Backpropagation Adjustment. During training, negative weights appear as an inevitable result of backpropagation. When the current weights cause a signal to be too strong, the gradient descent algorithm adjusts weights in the direction of reducing error, turning some weights negative to offset the overly strong signal. But this adjustment is unconstrained. Too many negative weights cause the signal to become too weak, forcing the use of ReLU or GELU to zero out negative values. After the signal weakens, gradient descent pushes weights larger again, potentially triggering gradient explosion once more. This cyclical process turns parameter tuning into a seesaw battle. 4. Mini-Batch Splitting: A Necessary Evil Due to Memory Constraints. Splitting training data into mini-batches is fundamentally due to insufficient GPU memory to load all data at once. This further increases training uncertainty. You never know which mini-batch will be the straw that breaks the camel's back. It is possible that after hundreds of batches of normal training, a single batch triggers gradient explosion, ruining all previous work. 5. Hyperparameter Setting: The Awkwardness of Drawing Blueprints Before Construction. Hyperparameters such as layer count, learning rate, and regularization coefficients are often pre-determined. This is like drawing architectural blueprints before construction. If the structural mechanics (gradient control) are found unreasonable during training, the semi-finished product must be scrapped, and the blueprints (hyperparameters) redrawn. This cycle of design, trial-and-error, rejection, and redesign is the core reason for the high cost of large model training. III. The Revolutionary Value of the MHC Architecture: Ending Alchemy with Mathematical Constraints. The greatness of the MHC architecture lies not in minor tweaks to existing tools, but in using mathematical constraints at the fundamental architecture level to solve the root cause of gradient instability, transforming large model training from alchemy into precise engineering. 1. Core Grip: The Stabilizing Magic of Doubly Stochastic Matrices. The core of MHC is constraining the hyperconnection matrix within the manifold of doubly stochastic matrices. Doubly stochastic matrices have three key properties: - Non-negativity: All elements are ≥ 0. This fundamentally eliminates the cycle of negative weights being repeatedly zeroed out, removing the need for activation functions to "clean up." - Row and column sums equal to 1: The amplification or attenuation factor of matrix multiplication is strictly locked near 1. During chain-rule differentiation, the gradient propagation coefficient for each layer is approximately 1, and the spectral norm is approximately 1—completely breaking the chained multiplication curse. Gradients neither decay exponentially nor explode exponentially; they remain stable within a controllable range. 2. Dynamic Stabilization: The Correction Ability of the SyncHercanuop Algorithm. MHC does not just initialize and forget. After each training step, the SyncHercanuop algorithm pulls the updated weight matrix back onto the doubly stochastic matrix manifold. This means that even if gradient descent tries to push weights toward extreme values, MHC forcibly drags them back to the stable region of non-negativity and row/column sums equal to 1. The result is an ideal state with no drastic weight fluctuations, no risk of gradient runaway, and no need for passive remedial tools. 3. Engineering Value: From Gambling Luck to Steady Progress. The engineering benefits of the MHC architecture are revolutionary: - Training stability improved: No risk of gradient vanishing or explosion throughout the entire process. - No need for frequent interruptions and restarts: Computation and time are no longer wasted. - Reduced parameter tuning complexity: No need to repeatedly adjust activation functions or residual connection parameters. - Hyperparameter setting shifts from empirical trial-and-error to mathematically grounded design. - Reduced memory consumption: The doubly stochastic constraint cuts redundant parameters. Experiments show a 40% reduction in training memory usage, enabling larger-scale models. IV. Conclusion. The essence of technological innovation is to simplify complexity. The history of large model training is an evolution from passive remediation to active constraint. Self-media's popular metaphors may attract attention, but the true charm of technology lies in the details: what core pain point is solved, and with what logic. The value of the MHC architecture is not in creating some brand-new tool, but in using rigorous mathematical constraints to strip away the layers of remedial measures in large model training, returning to the essence of stable signal transmission and stable gradient regulation. For all developers engaged in model training, this represents a cognitive upgrade from groping in the dark to seeing the light. V. Postscript. This article is my personal notes on learning the MHC architecture. Due to my limited knowledge, there may inevitably be omissions or errors. I hope it can serve as a reference for fellow enthusiasts who have also been misled by oversimplified explanations, and I look forward to more people jumping out of the metaphor trap and truly understanding the underlying logic of technological innovation.
back to top