Big data analysis helps identify patterns, trends, and correlations that facilitate informed decision making and effective strategies. Successful analysis requires using algorithms across high-quality data from disparate repositories.

The concept of dealing with large datasets has been around for decades. The emergence of “big data” was fueled by advancements in various technologies and computer sciences, including the proliferation of the internet and the increasing digitization of information and systems. These developments resulted in an exponential increase in the amount of data being generated, collected, and stored by various systems and instruments. As governmental or international organizations and industries realized the potential value of this data for analysis and decision-making, the term “big data” came into widespread use to describe the challenges and opportunities associated with managing and extracting insights from large and complex datasets.

Successful analysis of big data requires algorithms and training sets. In an article, Mining Conditional Functional Dependency Rules on Big Data^[1], the authors present a fault-tolerant rule discovery and conflict-resolution algorithm to address the low-quality issue of big data.

In climate modeling, big data plays a large role by enabling the collection, analysis, and interpretation of information. Data is collected by sensors from around the globe and instruments on satellites that are used to assess the Earth’s conditions and help to predict future events. Analytics are then used to process climate-related data, such as temperature, precipitation, atmospheric conditions, and oceanic patterns. By analyzing historical and real-time data, scientists can develop sophisticated climate models that simulate future scenarios. It helps identify patterns, trends, and correlations that facilitate informed decision-making and effective strategies.

IEEE Senior member Jean-Philippe Montillet is one of those scientists who relies on big data for his work at the World Physikalisch-Meteorologisches Observatorium/World Radiation Center in Davos, Switzerland.

Dr. Montillet is a geoscientist and data scientist working in environmental geodesy (i.e., the science of accurately measuring and understanding three fundamental properties of the Earth), geophysics, Earth’s energy budget, and other applications related to climate change. 

His research projects focused on crustal deformation, sea-level rise, and the estimation of total solar irradiance, that is, the spectrally integrated solar flux at Earth. He has been involved in the analysis of Earth and space observations for various missions, including the Gravity Recovery and Climate Experiment (GRACE), Satellite altimetry, Variability of Irradiance and Gravity Oscillations (VIRGO) on the mission Solar and Heliospheric Observatory (SOHO). Additionally, he has produced several models and algorithms to detect and estimate transient signals and select optimally stochastic and functional models in time series analysis. He has worked with colleagues at PMOD to correct observations based on machine learning and data fusion.

“Since the turn of the 21st century, we have more and more instruments and research projects that are all generating data,” says Dr. Montillet. “We can use that data to build algorithms to better understand correlations and build models for analysis. This is very relevant when we look at climate modeling.” 

One example is forecasting the recent landslide in the Swiss village of Brienz in early June 2023. Using data collected from sensors on the ground as well as satellite imagery, for years scientists have been actively tracking the movement of the mountain surrounding the idyllic village. With enough warning based on continuous data analysis, the village authorities were able to evacuate the town in advance of the landslide. Within weeks, the mountain did in fact slide, narrowly missing the village. Analyzing the data, and acting based on that data, was critical in keeping people safe. 

​​Big data has the inherent challenge of not enabling integration or mashup among heterogeneous datasets from diversified domain repositories. The IEEE IC Big Data Governance and Metadata Management: Standards Roadmap^[2] addresses the challenge that would allow data to be discoverable, accessible, and reusable through a machine-readable and actionable standard data infrastructure.

Quality data is also critical in ensuring quality analysis. In the article Big Data Challenges in Climate Science: Improving the Next-Generation Cyberinfrastructure^[3], the authors represent the need for an improved cyberinfrastructure to process the large amount of critical scientific data. 

Understanding the impacts and mitigation of climate change requires the collaboration of an interdisciplinary set of researchers, scientists, and engineers. These specialists design the systems to be measured and interpret the results to drive conclusions. To determine courses of action and decision-making, a great deal of research and analysis must happen first. 

Dr. Montillet’s background exemplifies this interdisciplinary approach. He studied mobile telecommunications at the Aalborg University in Denmark, started his work in geodesy and geodynamics of the Earth after his Ph.D. studies at the University of Nottingham in the United Kingdom, and now works in space science on the accurate estimation of the Earth energy budget. All along, he has been analyzing data to understand the Earth’s environment for making a better world. He is also a member of IEEE’s Geoscience and Remote Sensing Society (GRSS), the home for scientists and engineers working at the intersection of big data, analytics, and climate modeling.