跳到主要导航 跳到搜索 跳到主要内容

Hadoop-MTA: A system for Multi Data-center Trillion Concepts Auto-ML atop Hadoop

  • Keqian Li
  • , Yifan Hu
  • , Manisha Verma
  • , Fei Tan
  • , Changwei Hu
  • , Tejaswi Kasturi
  • , Kevin Yen
  • Yahoo Research Labs
  • Amazon.com, Inc.
  • XPeng Motors
  • RED

科研成果: 书/报告/会议事项章节 › 会议稿件 › 同行评审

摘要

The ever-growing computation capability distributed infrastructure brings tremendous opportunities for mining and analysis of data that was impossible otherwise. Meanwhile, the inherent computation model of distributed system also brings unique and non-trivial challenges for traditional Auto-ML, including the explosion of data dimensions, the expected absence of features, and the heterogeneity of information. This is especially the case in modern Internet enterprises, where data in the scale of trillions are stored in multiple data centers, and the discovery of subtle signals could incur significant impact in revenue and welfare. How can we best harness the large scale distributed machine learning, but without keeping engineers constantly in the loop? In this work, we present Hadoop-MTA, a system for Multi Data-center, Trillion Concepts, Auto-ML on top of the Hadoop distributed computation environment that leverages sparsity aware heterogeneous knowledge graph representation and dimensionality agnostic parallel learning. Through multiple large scale experiments, we find that Hadoop-MTA significantly output-performs competitive state of the art distributed learning algorithms and scales well to trillion scale data-sets. Our model is rolled out to Hadoop serving infrastructure in Yahoo covering billions of unique identities and shows improvements 129.5% accuracy and 106.5 % weighted F1-score (more than 2x) on key targeting use cases.

源语言英语
主期刊名Proceedings - 2021 IEEE International Conference on Big Data, Big Data 2021
编辑Yixin Chen, Heiko Ludwig, Yicheng Tu, Usama Fayyad, Xingquan Zhu, Xiaohua Tony Hu, Suren Byna, Xiong Liu, Jianping Zhang, Shirui Pan, Vagelis Papalexakis, Jianwu Wang, Alfredo Cuzzocrea, Carlos Ordonez
出版商Institute of Electrical and Electronics Engineers Inc.
页5953-5955
页数3
ISBN(电子版)9781665439022
DOI
出版状态已出版 - 2021
已对外发布是
活动2021 IEEE International Conference on Big Data, Big Data 2021 - Virtual, Online, 美国
期限: 15 12月 2021 → 18 12月 2021

丛书

姓名Proceedings - 2021 IEEE International Conference on Big Data, Big Data 2021

会议

会议2021 IEEE International Conference on Big Data, Big Data 2021
国家/地区美国
市Virtual, Online
时期15/12/21 → 18/12/21

学术指纹

探究 'Hadoop-MTA: A system for Multi Data-center Trillion Concepts Auto-ML atop Hadoop' 的科研主题。它们共同构成独一无二的学术指纹。

引用此