
Core Announcement and Operational Scope
Tencent has released tencent/Hy4-flash on Hugging Face, introducing a cost-oriented multimodal Mixture-of-Experts (MoE) model designed for image-text-to-text workloads and long-context generation. Published in the Hunyuan family under the tencent-hy-license, the release pairs a 220-billion-parameter total footprint with roughly 12 billion active parameters per token and a context window scaling up to 1,048,576 tokens (1M).
Positioned as the most economical variant in Tencent's Hy4 lineup to date, Hy4 flash natively accepts combined text and image inputs to produce text responses across conversational, instruction-following, and developer tasks. While the release drew rapid attention on Hugging Face Trending, its custom license and substantial total weight footprint place it in the open-weights rather than permissive open-source category for engineering teams evaluating production adoption.
Hybrid Attention, LatentMoE, and Engram Architecture
Compared with the flagship Hy4 design, Tencent overhauled three core subsystems in Hy4 flash to reduce per-token compute and memory bandwidth pressure. Across its 42-layer language backbone—comprising one dense feed-forward layer (13,824 intermediate size) followed by 41 MoE layers with a hidden size of 3,072 and a 120,832-token vocabulary—the model uses identity Hyper-Connections (iHC) across four residual streams to support inter-layer information flow.
First, Hy4 flash adopts a 1:3 hybrid attention layout using Multi-head Latent Attention (MLA) with a query LoRA rank of 1,024 and a key-value LoRA rank of 512. Following two initial sparse attention layers, the backbone repeats ten blocks of three sliding-window attention (SWA) layers paired with one IndexPool sparse layer, yielding 12 sparse layers and 30 SWA layers. SWA restricts attention to a 512-token causal window, whereas sparse layers select up to 2,048 tokens via group-level selection using a 10,000,000 RoPE base and Interleaved MRoPE for multimodal positioning.
Second, its LatentMoE layers route experts inside a 1,536-dimensional latent space. Each token activates 16 of 512 routed experts alongside one shared expert, keeping active text compute at 11.3 billion parameters out of 155.5 billion core backbone weights. Third, an N-gram embedding mechanism called Engram adds 63.5 billion parameters (63.4 billion in embedding tables) at backbone layer indices 1, 11, 21, and 31, using eight hash heads per bigram and trigram order across 10.3 million hash slots per head.
Vision Pipeline, Speculative Decoding, and Base Benchmarks
Visual understanding relies on a 0.43-billion-parameter encoder adapted from SigLIP2-So400m-Patch16-Res512 and attached to the language backbone via an MLP projector. The vision stack spans 27 layers with a 1,152 hidden size, 16 attention heads, 16x16 patches, a 128x128 position embedding grid, and 2x2 visual feature merging. To accelerate long-context generation, Tencent integrates DFly2, a built-in 5-layer, 0.85-billion-parameter speculative drafter that reuses the target model's KV cache and sparse attention indices for low, constant-cost decoding.
In self-reported pretrained base evaluations on the model card, Hy4 flash Base posts 5-shot scores of 88.7 on MMLU, 71.1 on MMLU-Pro, 57.8 on SuperGPQA, 92.0 on C-Eval, 69.8 on U-MATH, and 93.8 on a five-language Belebele subset, alongside 73.7 on 3-shot MBPP-plus and 95.0 on 0-shot ARC-Challenge. As expected for a flash-tier model, it trails Hy4 Base (91.6 MMLU, 73.9 MMLU-Pro) across benchmarks while outscoring Tencent's reported figures for DeepSeek-V4-flash-Base on MMLU-Pro (68.6), SuperGPQA (52.5), Social IQa (80.1 versus 72.5), and U-MATH (57.4).
Developer Integration and Serving Workflows
For engineering teams building multimodal pipelines, the model card documents ready-to-run deployment paths across mainstream inference stacks. Python developers can invoke the weights directly through Hugging Face transformers using either the high-level image-text-to-text pipeline or AutoModelForSeq2SeqLM with automatic device mapping.
Production-oriented teams can launch OpenAI-compatible HTTP endpoints via vllm serve "tencent/Hy4-flash" or SGLang's sglang.launch_server and official lmsysorg/sglang:latest Docker container configured with GPU passthrough and 32 GB shared memory. Tencent also provides a Docker Model Runner command (docker model run hf.co/tencent/Hy4-flash) and points to GGUF quantization paths for llama.cpp, Ollama, and LM Studio, easing migration for self-hosted multimodal clients.
Licensing, Memory Footprint, and Verification Caveats
Despite its 12-billion active-parameter efficiency during inference, hosting full Hy4 flash weights still requires accommodating roughly 220 billion total parameters in memory—including the 63.5-billion-parameter Engram lookup tables—which constrains single-machine deployment without aggressive quantization or multi-GPU serving.
Developers should also weigh two key governance and evidentiary caveats. First, the model is distributed under tencent-hy-license rather than an OSI-approved license such as Apache 2.0 or MIT, making legal review essential before commercial redistribution or derivative integration. Second, the benchmark tables captured in the model card excerpt reflect self-reported pretrained base evaluations rather than instruction-tuned chat results or independent third-party audits, so teams should verify latency, DFly2 acceptance rates, and multimodal accuracy on their own workloads.
Automated Vertex AI (gemini-4-hydrogen) primary-source report; reviewed 2026-11-19.



