我的征尘是星辰大海。。。
The dirt and dust from my pilgrimage forms oceans of stars...
-------当记忆的篇章变得零碎，当追忆的图片变得模糊，我们只能求助于数字存储的永恒的回忆
作者:黄教授
手机视频列表
算力围城与架构困局3
视频
音频
原始脚本
第四章架构转型之难，MO 不是抄作业，而是全链条适配站。 一，MO 架构的核心门槛不止于专家拆分。 MO 架构的核心并非简单将 FFN 层拆分为专家模块，其真正的技术门槛在于三大关键问题，门控网络的精准调度。 专家负载的动态均衡、跨设备通信的高效协同。 这三大问题环环相扣，任何一个环节处理不当都会导致模型性能下降或算力开销飙升。 门控网络是 MODE 的大脑，需精准判断每个 token 所属的领域，将其分配给最优专家。 若门控调度不准，将数学 token 分配给文学专家，不仅无法提升性能，还会因参数不匹配导致输出错误。 为解决这一问题，谷歌 Deepseek 等机构需通过大量数据训练门控网络，甚至引入强化学习策略，让门控能够动态适应不同场景的需求。 专家负载均衡则是 MOE 规模化的关键。 若大量 token 肯集中分配给少数专家，会导致这些专家过载。 而其他专家闲置，不仅降低计算效率，还会因训练数据不均衡导致专家能力退化。 谷歌通过引入负载均衡损失函数，强制 Token 均匀分配给所有专家。 Deepseek 则采用动态专家容量机制，根据 Token 流量实时调整专家的处理能力，避免负载失衡。 跨设备通信的高效协同更是 MoE 的生死线。 在超大规模 MoE 模型中，专家模块分布在不同设备上，Token 调度需频繁进行跨设备数据传输。 若通信延迟过高会导致 GPU 等待数据，计算资源利用率大幅下降。 这也是 MOE 架构对硬件互联的要求远超稠密模型的核心原因。 二、底层通信的真功夫。 Deepseek 与字节的极致优化 MOE 架构的落地离不开底层通信库的定制化优化，这正是 Deepseek 、字节跳动等公司的核心竞争力。 也是普通团队难以抄作业的关键。 以 Deepseek 的 DPP 通信库为例，其并非简单调用英伟达的 NCCL 库，而是基于 NVSHMEM、GDR copy 等底层技术，针对模型专家并行需求进行了极致改造。 DPP 的核心优化包括三点，一是实现通信与计算的并行，通过互可机制让 GPU 在计算的同时进行数据传输，避免通信阻塞。 二是优化全对全通信内核，针对 Hopper 架构的 FP8低精度格式进行定制，将数据传输量降低一半，延迟压制163微秒。 三是采用虚拟通道隔离流量，避免不同专家的数据传输互相干扰，提升通信稳定性。 字节跳动的 Comet 通信库则更 进一步，通过易购通信调度策略，自动适配节点内 NVLink 和节点间 RDMA 的带宽差异，实现不同场景下的最优通信方案。 例如，在节点内传输时，优先使用 NVLink 实现低延迟通信。 在跨节点传输时，则切换至 RDMA 保证带宽。 这种精细化的优化需要团队对硬件架构、通信协议有深刻理解，绝非简单复制开源代码就能实现。 三、英伟达的缺位，通用方案与场景需求的错配，为何这些底层优化需要企业自行研发？核心原因在于英伟达的通用通信库，如 NCCL 与 Mode 的专属需求存在错配。 NCCL 是为稠密模型设计的，主要优化数据并行场景下的批量数据传输。 而 Mode 的专家并行需要频繁进行小批量、高频率的专家参数交换，这与 NCCL 的优化目标完全不同。 例如，NCCL 的全对全通信算法针对大规模批量数据优化，在小批量数据传输时效率极低。 而 Mode 的专家调度恰好需要大量小批量数据传输，导致 NCCL 无法满足需求。 此外，NCCL 缺乏对专家负载均衡的支持，无法动态调整通信策略，进一步限制了 MOE 的规模化部署。 对英伟达而言，这种缺位是商业理性的选择。 MOE 架构在早期属于小众场景，而稠密模型仍是市场主流，投入资源开发 MOE 专属通信库的投入产出比极低。 因此，英伟达选择开放 NVSHMEM、GDR Copy 等基础工具，将场景化优化的空间留给企业，这也导致 Moe 的落地门槛大幅提升。 四、头部玩家的转型困境，全链条适配的成本与风险，头部玩家向 Moe 转型的难度。 远超复制开源代码的想象，核心在于需要进行全链条的适配改造，涉及硬件、软件、业务等多个层面，成本与风险极高。 在硬件层面，现有 GPU 集群需改造网络拓扑、升级通信设备，甚至采购专用芯片。 例如阿里千问转向 Moe 时，不仅升级了 GPU 集群的 RDMA 网络，还引入了自研的高性能交换机，投入成本高达数亿元。 在软件层面，需重构训练框架、优化数据预处理流程、开发门控调度算法。 百度文心在测试 Moe 模型时。 发现现有训练框架无法高效处理专家负载均衡问题，不得不组建专项团队重构框架核心模块，耗时超过半年。 在业务层面，某模型需适配多场景的延迟精度需求。 例如，实时对话场景需控制激活专家数量以降低延迟，而科研计算场景则需增加专家协同以提升精度。 这要求企业针对不同业务场景进行定制化开发，而非直接套用开源模型。 此外，转型梦还可能面临训练不稳定、调试难度大等风险。 MO 的门控网络和专家模块存在大量离散参数，训练过程中易出现梯度消失、数值不稳定等问题，需要团队具备丰富的调试经验。 OpenAI 内部测试 MO 模型时，曾因专家负载失衡导致模型性能大幅波动。 最终不得不暂停转型计划，转而采用多模型场景拆分的妥协方案。 综上，MOE 架构的落地并非抄作业那么简单，而是一场涉及硬件改造、软件重构、业务适配的全链条适配战。 只有具备深厚的底层技术积累、充足的资源投入和强大的工程化能力。 才能真正驾驭 Mo 架构，这也是为何 Deepseek 开源 Mo 方案后，头部玩家仍需漫长时间才能实现规模化落地的核心原因。
修正脚本
第四章架构转型之难，MOE不是抄作业，而是全链条适配战。 一，MOE架构的核心门槛不止于专家拆分。 MOE架构的核心并非简单将 FFN 层拆分为专家模块，其真正的技术门槛在于三大关键问题，门控网络的精准调度。 专家负载的动态均衡、跨设备通信的高效协同。 这三大问题环环相扣，任何一个环节处理不当都会导致模型性能下降或算力开销飙升。 门控网络是 MOE 的大脑，需精准判断每个 token 所属的领域，将其分配给最优专家。 若门控调度不准，将数学 token 分配给文学专家，不仅无法提升性能，还会因参数不匹配导致输出错误。 为解决这一问题，谷歌 Deepseek 等机构需通过大量数据训练门控网络，甚至引入强化学习策略，让门控能够动态适应不同场景的需求。 专家负载均衡则是 MOE 规模化的关键。 若大量 token 会集中分配给少数专家，会导致这些专家过载。 而其他专家闲置，不仅降低计算效率，还会因训练数据不均衡导致专家能力退化。 谷歌通过引入负载均衡损失函数，强制 Token 均匀分配给所有专家。 Deepseek 则采用动态专家容量机制，根据 Token 流量实时调整专家的处理能力，避免负载失衡。 跨设备通信的高效协同更是 MoE 的生死线。 在超大规模 MoE 模型中，专家模块分布在不同设备上，Token 调度需频繁进行跨设备数据传输。 若通信延迟过高会导致 GPU 等待数据，计算资源利用率大幅下降。 这也是 MOE 架构对硬件互联的要求远超稠密模型的核心原因。 二、底层通信的真功夫。 Deepseek 与字节的极致优化 MOE 架构的落地离不开底层通信库的定制化优化，这正是 Deepseek 、字节跳动等公司的核心竞争力。 也是普通团队难以抄作业的关键。 以 Deepseek 的 DPP 通信库为例，其并非简单调用英伟达的 NCCL 库，而是基于 NVSHMEM、GDR copy 等底层技术，针对模型专家并行需求进行了极致改造。 DPP 的核心优化包括三点，一是实现通信与计算的并行，通过重叠机制让 GPU 在计算的同时进行数据传输，避免通信阻塞。 二是优化全对全通信内核，针对 Hopper 架构的 FP8低精度格式进行定制，将数据传输量降低一半，延迟压制163微秒。 三是采用虚拟通道隔离流量，避免不同专家的数据传输互相干扰，提升通信稳定性。 字节跳动的 Comet 通信库则更进一步，通过异构通信调度策略，自动适配节点内 NVLink 和节点间 RDMA 的带宽差异，实现不同场景下的最优通信方案。 例如，在节点内传输时，优先使用 NVLink 实现低延迟通信。 在跨节点传输时，则切换至 RDMA 保证带宽。 这种精细化的优化需要团队对硬件架构、通信协议有深刻理解，绝非简单复制开源代码就能实现。 三、英伟达的缺位，通用方案与场景需求的错配，为何这些底层优化需要企业自行研发？核心原因在于英伟达的通用通信库，如 NCCL 与 MOE 的专属需求存在错配。 NCCL 是为稠密模型设计的，主要优化数据并行场景下的批量数据传输。 而 MOE 的专家并行需要频繁进行小批量、高频率的专家参数交换，这与 NCCL 的优化目标完全不同。 例如，NCCL 的全对全通信算法针对大规模批量数据优化，在小批量数据传输时效率极低。 而 MOE 的专家调度恰好需要大量小批量数据传输，导致 NCCL 无法满足需求。 此外，NCCL 缺乏对专家负载均衡的支持，无法动态调整通信策略，进一步限制了 MOE 的规模化部署。 对英伟达而言，这种缺位是商业理性的选择。 MOE 架构在早期属于小众场景，而稠密模型仍是市场主流，投入资源开发 MOE 专属通信库的投入产出比极低。 因此，英伟达选择开放 NVSHMEM、GDR Copy 等基础工具，将场景化优化的空间留给企业，这也导致 Moe 的落地门槛大幅提升。 四、头部玩家的转型困境，全链条适配的成本与风险，头部玩家向 Moe 转型的难度，远超复制开源代码的想象，核心在于需要进行全链条的适配改造，涉及硬件、软件、业务等多个层面，成本与风险极高。 在硬件层面，现有 GPU 集群需改造网络拓扑、升级通信设备，甚至采购专用芯片。 例如阿里千问转向 Moe 时，不仅升级了 GPU 集群的 RDMA 网络，还引入了自研的高性能交换机，投入成本高达数亿元。 在软件层面，需重构训练框架、优化数据预处理流程、开发门控调度算法。 百度文心在测试 Moe 模型时，发现现有训练框架无法高效处理专家负载均衡问题，不得不组建专项团队重构框架核心模块，耗时超过半年。 在业务层面，某模型需适配多场景的延迟精度需求。 例如，实时对话场景需控制激活专家数量以降低延迟，而科研计算场景则需增加专家协同以提升精度。 这要求企业针对不同业务场景进行定制化开发，而非直接套用开源模型。 此外，转型中还可能面临训练不稳定、调试难度大等风险。 MOE 的门控网络和专家模块存在大量离散参数，训练过程中易出现梯度消失、数值不稳定等问题，需要团队具备丰富的调试经验。 OpenAI 内部测试 MOE 模型时，曾因专家负载失衡导致模型性能大幅波动。 最终不得不暂停转型计划，转而采用多模型场景拆分的妥协方案。 综上，MOE 架构的落地并非抄作业那么简单，而是一场涉及硬件改造、软件重构、业务适配的全链条适配战。 只有具备深厚的底层技术积累、充足的资源投入和强大的工程化能力，才能真正驾驭 MOE 架构，这也是为何 Deepseek 开源 MOE 方案后，头部玩家仍需漫长时间才能实现规模化落地的核心原因。
英文翻译
Chapter 4: The Difficulty of Architecture Transformation – MoE Is Not Copying Homework, but a Full-Chain Adaptation Battle. I. The Core Threshold of MoE Architecture Goes Beyond Splitting Experts. The core of MoE architecture is not simply splitting the FFN layer into expert modules. Its true technical barriers lie in three key issues: precise scheduling of the gating network, dynamic balancing of expert loads, and efficient coordination of cross-device communication. These three issues are interlinked; mishandling any one of them can lead to degraded model performance or soaring computational costs. The gating network is the brain of MoE, needing to accurately determine the domain of each token and assign it to the best expert. If the gating scheduling is inaccurate, assigning a mathematical token to a literature expert not only fails to improve performance but also causes output errors due to parameter mismatch. To solve this, institutions like Google and DeepSeek train the gating network with large amounts of data, even introducing reinforcement learning strategies to allow the gating to dynamically adapt to different scenario requirements. Expert load balancing is the key to MoE scaling. If a large number of tokens are concentrated on a few experts, those experts become overloaded while others remain idle, not only reducing computational efficiency but also causing expert capability degradation due to uneven training data. Google introduces a load balancing loss function to force tokens to be evenly distributed among all experts. DeepSeek adopts a dynamic expert capacity mechanism that adjusts experts' processing capacity in real time based on token traffic, avoiding load imbalance. Efficient coordination of cross-device communication is the lifeline of MoE. In ultra-large-scale MoE models, expert modules are distributed across different devices, and token scheduling requires frequent cross-device data transfers. If communication latency is too high, GPUs wait for data, leading to a significant drop in computing resource utilization. This is also the core reason why MoE architecture has much higher requirements for hardware interconnection than dense models. II. The Real Expertise in Underlying Communication. DeepSeek and ByteDance's Ultimate Optimization The implementation of MoE architecture relies on customized optimization of underlying communication libraries, which is the core competitiveness of companies like DeepSeek and ByteDance, and also the key reason why ordinary teams find it difficult to simply copy homework. Taking DeepSeek's DPP communication library as an example, it does not simply call NVIDIA's NCCL library, but is based on underlying technologies such as NVSHMEM and GDR copy, with extreme customization for the expert parallelism needs of the model. The core optimizations of DPP include three points: first, achieving parallelism of communication and computation through an overlap mechanism, allowing the GPU to perform data transfer while computing, avoiding communication blocking. Second, optimizing the all-to-all communication kernel, customizing it for the FP8 low-precision format of the Hopper architecture, halving the data transfer volume and suppressing latency to 163 microseconds. Third, using virtual channels to isolate traffic, preventing interference between data transfers of different experts and improving communication stability. ByteDance's Comet communication library goes a step further, using heterogeneous communication scheduling strategies to automatically adapt to bandwidth differences between NVLink within a node and RDMA between nodes, achieving optimal communication solutions in different scenarios. For example, when transferring within a node, it prioritizes NVLink for low-latency communication; when transferring across nodes, it switches to RDMA to guarantee bandwidth. Such fine-grained optimization requires the team to have a deep understanding of hardware architecture and communication protocols, which cannot be achieved by simply copying open-source code. III. NVIDIA's Absence: Mismatch Between General Solutions and Scenario Needs. Why do companies need to develop these underlying optimizations themselves? The core reason is the mismatch between NVIDIA's general communication library (such as NCCL) and the specific needs of MoE. NCCL is designed for dense models, mainly optimizing batch data transfer in data parallelism scenarios. However, MoE's expert parallelism requires frequent, small-batch, high-frequency exchange of expert parameters, which is completely different from NCCL's optimization goals. For example, NCCL's all-to-all communication algorithm is optimized for large-scale batch data, making it extremely inefficient for small-batch data transfers. MoE's expert scheduling precisely requires a large number of small-batch data transfers, causing NCCL to fail to meet the requirements. In addition, NCCL lacks support for expert load balancing and cannot dynamically adjust communication strategies, further limiting MoE's large-scale deployment. For NVIDIA, this absence is a rational business choice. In the early stage, MoE architecture was a niche scenario, while dense models remained the mainstream market. The return on investment for developing a dedicated MoE communication library was extremely low. Therefore, NVIDIA chose to open up basic tools like NVSHMEM and GDR Copy, leaving scenario-specific optimization space to enterprises, which has significantly raised the adoption threshold for MoE. IV. The Transformation Dilemma of Leading Players: Cost and Risk of Full-Chain Adaptation. The difficulty for leading players to transition to MoE far exceeds the imagination of simply copying open-source code. The core lies in the need for full-chain adaptation and modification, involving hardware, software, business, and other levels, with extremely high costs and risks. At the hardware level, existing GPU clusters need to modify network topology, upgrade communication equipment, and even purchase dedicated chips. For example, when Alibaba's Qwen turned to MoE, it not only upgraded the RDMA network of the GPU cluster but also introduced self-developed high-performance switches, with investment costs reaching hundreds of millions of yuan. At the software level, it is necessary to restructure training frameworks, optimize data preprocessing processes, and develop gating scheduling algorithms. When Baidu's ERNIE tested MoE models, it found that the existing training framework could not efficiently handle expert load balancing issues, forcing them to set up a dedicated team to reconstruct core framework modules, which took more than half a year. At the business level, models need to adapt to the latency-precision requirements of multiple scenarios. For example, real-time dialogue scenarios need to control the number of activated experts to reduce latency, while scientific computing scenarios need to increase expert collaboration to improve precision. This requires enterprises to carry out customized development for different business scenarios rather than directly applying open-source models. Additionally, during the transition, risks such as training instability and debugging difficulty may arise. The gating network and expert modules of MoE have a large number of discrete parameters, making them prone to gradient vanishing and numerical instability during training, requiring the team to have rich debugging experience. When OpenAI internally tested MoE models, expert load imbalance caused significant performance fluctuations, eventually forcing them to suspend the transition plan and adopt a compromise solution of splitting the model into multiple scenarios. In summary, the implementation of MoE architecture is not as simple as copying homework; it is a full-chain adaptation battle involving hardware modification, software reconstruction, and business adaptation. Only with deep underlying technical accumulation, sufficient resource investment, and strong engineering capabilities can one truly master the MoE architecture. This is also the core reason why, even after DeepSeek open-sourced its MoE solution, leading players still need a long time to achieve large-scale implementation.
back to top