TY - GEN
T1 - Indexing and parallel query processing support for visualizing climate datasets
AU - Su, Yu
AU - Agrawal, Gagan
AU - Woodring, Jonathan
PY - 2012
Y1 - 2012
N2 - With increasing emphasis on analysis of large-scale scientific data, and with growing dataset sizes, a number of new challenges are arising. Particularly, novel data management solutions are needed, which can work together with the existing tools. This paper examines indexing support for supporting high-level queries (primarily those for sub setting) on array-based scientific datasets. This work is motivated by the limitations arising in visualizing climate datasets (stored in Net CDF), using tools like Para View. We have developed a new indexing strategy, which can help support a variety of sub setting queries over these datasets, including those requiring sub setting over dimensions/coordinates and those involving variable values. Our approach is based on bitmaps, but involves use of two-level indices and careful partitioning, based on query profiles. We also show how our indexing support can be used for sub setting operations executed in parallel. We compare our solutions against a number of other solutions, and demonstrate that our method is more effective.
AB - With increasing emphasis on analysis of large-scale scientific data, and with growing dataset sizes, a number of new challenges are arising. Particularly, novel data management solutions are needed, which can work together with the existing tools. This paper examines indexing support for supporting high-level queries (primarily those for sub setting) on array-based scientific datasets. This work is motivated by the limitations arising in visualizing climate datasets (stored in Net CDF), using tools like Para View. We have developed a new indexing strategy, which can help support a variety of sub setting queries over these datasets, including those requiring sub setting over dimensions/coordinates and those involving variable values. Our approach is based on bitmaps, but involves use of two-level indices and careful partitioning, based on query profiles. We also show how our indexing support can be used for sub setting operations executed in parallel. We compare our solutions against a number of other solutions, and demonstrate that our method is more effective.
UR - http://www.scopus.com/inward/record.url?scp=84871117589&partnerID=8YFLogxK
UR - http://www.scopus.com/inward/citedby.url?scp=84871117589&partnerID=8YFLogxK
U2 - 10.1109/ICPP.2012.33
DO - 10.1109/ICPP.2012.33
M3 - Conference contribution
AN - SCOPUS:84871117589
SN - 9780769547961
T3 - Proceedings of the International Conference on Parallel Processing
SP - 249
EP - 258
BT - Proceedings - 41st International Conference on Parallel Processing, ICPP 2012
T2 - 41st International Conference on Parallel Processing, ICPP 2012
Y2 - 10 September 2012 through 13 September 2012
ER -