Parallel accessing massive NetCDF data based on MapReduce

Hui Zhao; Siyun Ai; Zhenhua Lv; Bo Li

doi:10.1007/978-3-642-16515-3_53

Parallel accessing massive NetCDF data based on MapReduce

Hui Zhao^*
, Siyun Ai
, Zhenhua Lv
, Bo Li

^*Corresponding author for this work

Software Engineering Institute

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

22 Scopus citations

Abstract

As a Network Common Data Format, NetCDF has been widely used in terrestrial, marine and atmospheric sciences. A new paralleling storage and access method for large scale NetCDF scientific data is implemented based on Hadoop. The retrieval method is implemented based on MapReduce. The Argo data is used to demonstrate our method. The performance is compared under a distributed environment based on PCs by using different data scale and different task numbers. The experiments result show that the parallel method can be used to store and access the large scale NetCDF efficiently.

Original language	English
Title of host publication	Web Information Systems and Mining - International Conference, WISM 2010, Proceedings
Pages	425-431
Number of pages	7
Edition	M4D
DOIs	https://doi.org/10.1007/978-3-642-16515-3_53
State	Published - 2010
Event	2010 International Conference on Web Information Systems and Mining, WISM 2010 - Sanya, China Duration: 23 Oct 2010 → 24 Oct 2010

Publication series

Name	Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)
Number	M4D
Volume	6318 LNCS
ISSN (Print)	0302-9743
ISSN (Electronic)	1611-3349

Conference

Conference	2010 International Conference on Web Information Systems and Mining, WISM 2010
Country/Territory	China
City	Sanya
Period	23/10/10 → 24/10/10

Keywords

Data intensive
MapReduce
NetCDF
Parallel access

UN SDGs

This output contributes to the following UN Sustainable Development Goals (SDGs)

Access to Document

10.1007/978-3-642-16515-3_53

Cite this

Zhao, H., Ai, S., Lv, Z., & Li, B. (2010). Parallel accessing massive NetCDF data based on MapReduce. In Web Information Systems and Mining - International Conference, WISM 2010, Proceedings (M4D ed., pp. 425-431). (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics); Vol. 6318 LNCS, No. M4D). https://doi.org/10.1007/978-3-642-16515-3_53

@inproceedings{0c3a58b828f448249e383f96d05446e6,

title = "Parallel accessing massive NetCDF data based on MapReduce",

abstract = "As a Network Common Data Format, NetCDF has been widely used in terrestrial, marine and atmospheric sciences. A new paralleling storage and access method for large scale NetCDF scientific data is implemented based on Hadoop. The retrieval method is implemented based on MapReduce. The Argo data is used to demonstrate our method. The performance is compared under a distributed environment based on PCs by using different data scale and different task numbers. The experiments result show that the parallel method can be used to store and access the large scale NetCDF efficiently.",

keywords = "Data intensive, MapReduce, NetCDF, Parallel access",

author = "Hui Zhao and Siyun Ai and Zhenhua Lv and Bo Li",

year = "2010",

doi = "10.1007/978-3-642-16515-3\_53",

language = "英语",

isbn = "3642165141",

series = "Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)",

number = "M4D",

pages = "425--431",

booktitle = "Web Information Systems and Mining - International Conference, WISM 2010, Proceedings",

edition = "M4D",

note = "2010 International Conference on Web Information Systems and Mining, WISM 2010 ; Conference date: 23-10-2010 Through 24-10-2010",

}

Zhao, H, Ai, S, Lv, Z & Li, B 2010, Parallel accessing massive NetCDF data based on MapReduce. in Web Information Systems and Mining - International Conference, WISM 2010, Proceedings. M4D edn, Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), no. M4D, vol. 6318 LNCS, pp. 425-431, 2010 International Conference on Web Information Systems and Mining, WISM 2010, Sanya, China, 23/10/10. https://doi.org/10.1007/978-3-642-16515-3_53

Parallel accessing massive NetCDF data based on MapReduce. / Zhao, Hui; Ai, Siyun; Lv, Zhenhua et al.
Web Information Systems and Mining - International Conference, WISM 2010, Proceedings. M4D. ed. 2010. p. 425-431 (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics); Vol. 6318 LNCS, No. M4D).

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

TY - GEN

T1 - Parallel accessing massive NetCDF data based on MapReduce

AU - Zhao, Hui

AU - Ai, Siyun

AU - Lv, Zhenhua

AU - Li, Bo

PY - 2010

Y1 - 2010

N2 - As a Network Common Data Format, NetCDF has been widely used in terrestrial, marine and atmospheric sciences. A new paralleling storage and access method for large scale NetCDF scientific data is implemented based on Hadoop. The retrieval method is implemented based on MapReduce. The Argo data is used to demonstrate our method. The performance is compared under a distributed environment based on PCs by using different data scale and different task numbers. The experiments result show that the parallel method can be used to store and access the large scale NetCDF efficiently.

AB - As a Network Common Data Format, NetCDF has been widely used in terrestrial, marine and atmospheric sciences. A new paralleling storage and access method for large scale NetCDF scientific data is implemented based on Hadoop. The retrieval method is implemented based on MapReduce. The Argo data is used to demonstrate our method. The performance is compared under a distributed environment based on PCs by using different data scale and different task numbers. The experiments result show that the parallel method can be used to store and access the large scale NetCDF efficiently.

KW - Data intensive

KW - MapReduce

KW - NetCDF

KW - Parallel access

UR - https://www.scopus.com/pages/publications/78649508618

U2 - 10.1007/978-3-642-16515-3_53

DO - 10.1007/978-3-642-16515-3_53

M3 - 会议稿件

AN - SCOPUS:78649508618

SN - 3642165141

SN - 9783642165146

T3 - Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)

SP - 425

EP - 431

BT - Web Information Systems and Mining - International Conference, WISM 2010, Proceedings

T2 - 2010 International Conference on Web Information Systems and Mining, WISM 2010

Y2 - 23 October 2010 through 24 October 2010

ER -

Zhao H, Ai S, Lv Z, Li B. Parallel accessing massive NetCDF data based on MapReduce. In Web Information Systems and Mining - International Conference, WISM 2010, Proceedings. M4D ed. 2010. p. 425-431. (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics); M4D). doi: 10.1007/978-3-642-16515-3_53

Parallel accessing massive NetCDF data based on MapReduce

Abstract

Publication series

Conference

Keywords

UN SDGs

Access to Document

Other files and links

Fingerprint

Cite this