关键词: 气象大数据, 存储, 索引, Avro, HBase, ElasticSearch

Abstract: Referring to the mature meteorological data storage ideas of traditional storage technology and the research results of meteorological big data storage technology of meteorological departments at all levels at present, a set of provincial meteorological big data storage model based on big data related technology from the storage and index of structured and unstructured meteorological data was designedthe results showed that unstructured meteorological small files use Avro to design data mode, According to the storage rules,are merged and packaged as the Avro format files and stored on HDFS to reduce the number of files and save the consumption of memory resources; Based on HBase, put forward the design idea of wide table storage for regular structured meteorological data, including tables, column fields and row keys; Elasticsearch is used to design a unified provincial meteorological big data storage index library to unify the retrieval entrance and improve the retrieval efficiency. The design ideas and the use of related technologies in this paper can provide reference for provincial meteorological departments to carry out the storage planning and scheme formulation of meteorological big data.

Key words: meteorological big data, storage, index, Avro, HBase, ElasticSearch