Abstract：As more and more 3D seismic data are acquired and rapid data updating for the oilfield development is repeatedly achieved,one of the toughest challenges for the software system architecture is efficient mass data storage and analysis in the nowadays big data era.Inspired by the distributed system framework of the Google File System (GFS) and Map-Reduce shared by Google,we design PetroV distributed system framework and develop some key techniques for mass data storage and analysis.First,3D seismic or digital rock data cubes are splited with the spatial octree encoding algorithm into multi-scale partition structure and data blocks.Then,differentiated from SEGY sequential access,the octree data block access framework is implemented,which embodies parallel read and write,multi-scale stream render,and two-step memory cache.After that,intuitive and heuristic distributed file storage solutions upon commodity computers are transparently deployed and respectively named by master nodes and data chunk nodes.Distributed files have the similar API as a stand-alone file except for shedding massive 3D data blocks in commodity machines with redundant backup.Finally,thanks to the octree data block,the Map-Reduce parallel analytic framework is developed on plenty of task nodes.
Sheng Xiujie,Jin Zhijun,Peng Cheng et al.Some ideas in PetroV architecture design and development.OGP,2015,50(4):766-774.
[4]
王珊,萨师煊.数据库系统概论(第4版).北京:高等教育出版社,2005.
[5]
孙家广.计算机图形学(第3版).北京:清华大学出版社,1998.
[6]
白鑫.基于八叉树结构的大规模地震数据的快速加载机制[学位论文].吉林长春:吉林大学,2013.
Bai Xin.Fast Loading Mechanism of Large Seismic Data Based on Octree Structure[D].Jinlin University,Changchun,Jilin,2013.
[7]
熊翥.我国物探技术的进步及展望.石油地球物理勘探,2003,38(5):565-578.
Xiong Zhu.Progress and prospect of geophysical prospecting technology in China.OGP,2003,38(5):565-578.
[8]
McKusick K,Quinlan S.GFS:Evolution on fast-for-ward.Communication of the ACM,2010,53(3):42-49.
[9]
Brewer E A.Towards robust distributed systems//Proceedings of the Nineteenth Annual ACM Symposium on Principles of Distributed Computing.New York,2000.
[10]
Dean J and Ghemawat S."MapReduce:simplified data processing on large clusters".Communication of the ACM,2008,51(3):107-113.
