Data-intensive science and engineering: Requirements and challenges

Scientific exploration after experimental science, theoretical science and computational science phases, into data-intensive science phase, are accompanied by the arrival of the big data era. Generally, big data refers to a data set with a size of hundreds of TB, or several PB or even above, and it is often distributed, heterogeneous and in low-quality. It is critical to devise novel methods to manage big data since traditional database management techniques are unfeasible to manage big data efficiently and effectively, though such techniques, especially the commercial relational DBMSs, have achieved great success in the past decades. This paper discusses concrete requirements and realistic challenges of Data-Intensive Science and Engineering (DISE), ranging from data storage and organization, computational method, data analysis, to user interfaces. Meanwhile, data quality, data security and data curation should be paid more attentions. In this paper, we attempt to describe the architecture of DISE, review the recent progress, and discuss the challenges and future work briefly.