Zhang, Xingxuan, Ren, Gang, Yuan, Hao, Zou, Hao, Tan, Hongze, Wang, Hui, Song, Jianhao, Li, Jiansheng, Zhang, Jiayao, Zhang, Jinghan, Li, Kaifang, Mo, Lang, Mao, Li, Hao, Mingchao, Xu, Nuo, Ding, Rui, Zhang, Ruiji, Li, Shuyang, Mei, Siyu, Zhang, Tianyang, Mu, Weiyang, Dong, Yancheng, Wei, Yongxian, Xue, Yuan, Wang, Yuanrui, He, Yue, Yang, Zijia, Li, Ziyun, Li, Dongzhe, Wang, Fuqiang, Liu, Jiandong, Chen, Jiawei, Du, Jiaxin, Cheng, Kaijie, Li, Kehan, Sun, Lei, Zhou, Linjun, Dai, Ningbo, Wang, Qi, Xu, Renzhe, Du, Shaoxing, Yang, Shumeng, Lu, Wang, Chu, Wenjing, Huang, Xiannan, Lin, Xiaoyu, Ai, Xing, Han, Xinyan, Li, Xuanyue, Su, Xuanyue, Zhang, Xukun, Lu, Yan, Zhang, Yaxin, Qin, Yi, Huang, Yifei, Xu, Yihan, Lv, Yongle, Jiang, Yuanyuan, Han, Yushan, Cui, Peng
Abstract
We introduce LimiX-2, a new model in the LimiX family, developed through model and data scaling guided by our previously established scaling laws. LimiX-2 adopts the Contextual Mechanism Networks (CMNs) paradigm and is pretrained with Context-Conditional Masked Modeling (CCMM). CMNs shifts the organizing principle of in-context learning from target-centric prediction to mechanism-oriented joint modeling. Rather than centering the network on the $p(y \mid x, D_{\mathrm{context}})$ objective of conventional tabular PFNs, it is designed around learning $p(x, y \mid D_{\mathrm{context}})$, a context-dependent representation of the joint structure underlying data generation. Pretraining uses synthetic datasets generated by structural causal models (SCMs) spanning diverse graph structures, functional mechanisms, and observation processes. Evaluations on TabArena, TALENT, and BCCO show that LimiX-2 outperforms current dataset-specific models and tabular foundation models. Beyond predictive performance, the CMN paradigm also promotes causal awareness in LimiX-2: its feature attention encodes direct causal relationships, enabling accurate causal skeleton recovery.
Chinese Translation
我们介绍了LimiX-2，这是LimiX系列中的一个新模型，开发过程遵循我们之前建立的扩展法则进行模型和数据扩展。LimiX-2采用上下文机制网络（Contextual Mechanism Networks, CMNs）范式，并通过上下文条件掩蔽建模（Context-Conditional Masked Modeling, CCMM）进行预训练。CMNs将上下文学习的组织原则从以目标为中心的预测转变为以机制为导向的联合建模。与传统表格PFNs以$p(y ext{ | } x, D_{ ext{context}})$为目标的网络设计不同，它围绕学习$p(x, y ext{ | } D_{ ext{context}})$进行设计，这是数据生成背后联合结构的上下文依赖表示。预训练使用由结构因果模型（Structural Causal Models, SCMs）生成的合成数据集，这些数据集涵盖了多种图结构、功能机制和观察过程。在TabArena、TALENT和BCCO上的评估表明，LimiX-2在性能上超越了当前特定数据集模型和表格基础模型。除了预测性能，CMN范式还促进了LimiX-2的因果意识：其特征注意力编码了直接的因果关系，从而实现准确的因果骨架恢复。