The Change of Monitoring Storage Models in the Era of Big Data

In today's highly developed Internet, the era of big data is coming. This article focuses on analyzing the era of big data, changes in monitoring storage models, and the new look of smart cities brought by big data.

The arrival of the Internet in the era of big data, especially the development of mobile Internet, has accelerated the infiltration of information technology into all aspects of society and economy and the daily life of the general public. According to statistics, in 1998, the average monthly usage of Internet users around the world was 1MB (megabytes), in 2000 it was 10MB, in 2003 it was 100MB, in 2008 it was 1GB (1GB equals 1024MB), and in 2014 it will be 10GB. The cumulative time for all network traffic to reach 1 EB (ie, 1 billion GB or 1000 PB) is one year in 2001, one month in 2004, and one week in 2007, and 2013 only takes one day, ie, the amount of information generated per day Engraved with 188 million DVD discs. The number of Internet users in China ranks first in the world, and the amount of data generated each day is also among the highest in the world. Taobao.com has more than tens of millions of transactions per day, with more than 50TB of data generated per day (1TB equals 1000GB) and 40PB of storage (1PB equals 1000TB). Baidu's current total data volume is close to 1,000PB, and the number of stored web pages is close to 1 trillion pages. It takes about 6 billion search requests and dozens of PB data per day. An 8Mbps (megabits per second) camera can generate 3.6GB of data in one hour. If a city is equipped with hundreds of thousands of traffic and security cameras, the amount of data generated each month will reach several tens of PB. Hospitals are also where data is concentrated. Nowadays, the CT image data of a patient amount to several tens of gigabytes, while the number of outpatient clinics in the country is billions, and their information needs to be stored for a long time. In short, big data exists in all walks of life, and an era of big data is coming.

The information explosion did not start today, but in recent years, people have even more experienced the rapid development of big data. On the one hand, the number of Internet users continues to increase. On the other hand, the number of connected devices represented by the Internet of Things and home appliances has grown faster. In 2007, there were 500 million devices connected to the world and 0.1 per capita. In 2013, there will be 50 billion devices connected to the world and 70 per capita. With the development of broadband, per capita network access bandwidth and traffic have also rapidly increased. The newly generated global data has increased by 40% annually, that is, the total amount of information can be doubled every two years. This trend will continue. At present, it is not uncommon for a single data set to have a capacity of more than tens of terabytes or even a few petabytes, and its size is so large that it cannot be captured, managed, and processed with conventional software tools within the allowable time.

The larger the data size, the more difficult it is to handle, but the more valuable it is to mine, which is why big data is hot. First, big data reflects public sentiment and public opinion. The massive data generated by netizens on the Internet records their thoughts, behaviors and even emotions. This is the result of the deep integration of the actual society and cyber space in the information age, and contains rich connotations and a lot of regularity information. According to statistics from the China Internet Network Information Center, the number of Internet users in China at the end of 2012 was 564 million, and the number of mobile Internet users was 420 million. Through analysis of relevant data, public demand, appeals, and opinions can be understood. Second, the information systems of enterprises and governments are continuously generating large amounts of data every day. According to Symantec's research report, the total information storage of global companies has reached 2.2ZB (1ZB equals 1000EB), an increase of 67% year-on-year. Hospitals, schools and banks also collect and store large amounts of information. The government can deploy sensing units such as sensors to collect information needed for environmental and social management. In 2011, a special issue published by Nature magazine in the United Kingdom pointed out that if we can organize and use big data more effectively, human beings will have more opportunities to use science and technology to greatly promote social development.

Big data: Separating storage systems Big data is not a specific type of data. Each unstructured data can be considered big data. This includes data on social networking sites, online financial transaction data, company records, weather monitoring data, satellite data and other monitoring, research and development data. The amount of big data is huge and unstructured.

1. Manage Big Data Storage Through Quarantine If you have multiple storage boxes in your enterprise, it is definitely a good idea to apply databases, line transaction processing (OLTP), and Microsoft Exchange to specific storage systems. Other specialized storage systems are used for big data applications such as portals, online streaming media applications, and so on.

If your business can't afford to separate storage systems, put specific front-end storage ports into databases, OLTP, and so on; commit to big data applications to other ports. The underlying rationale is to use dedicated ports, while large data traffic is measured in kilobytes or megabytes. OLTP application traffic is measured in input/output operations per second (IOPS) because the size of the data block is larger than big data. Larger and smaller than OLTP applications. OLTP applications are CPU intensive, while big data applications use more front-end ports. Therefore, more ports can focus on big data applications.

2. Big Data Analysis Today, many companies provide data management compatible storage systems. You should evaluate these companies when looking for your big data storage management solution. EMC Isilon's clustered storage systems are a better choice for big data storage management because big data can grow to multibyte data in a single file system. In addition to storage, another big challenge for big data management is data analysis. General data analysis applications do not handle big data well. After all, it involves a lot of data.

At present, companies such as EMC Greenplum are using tools that specialize in the management and analysis of big data. These applications run on clustered storage systems to ease the management of big data. It is recommended that the application be selected to work on the cluster storage system at the same time and analyze the data quickly and efficiently. Fast indexing ensures that metadata always resides on the Solid State Drive (SSD) if the storage box gives you such a choice.

Another major concern in managing big data is the future of data growth. Your big data storage management system should be scalable enough to meet future storage needs.

Big Data: Wisdom Engine of Smart City Big data is applied to smart cities to promote the process of urbanization in China. Smart cities are the necessary stages for the development of urban informationization and smart development. At the same time, the development of smart cities will also drive the overall development of related industries and become a new engine for economic transformation, industrial upgrading, and city improvement.

From the perspective of the smart city's architecture, since the basis of a smart city is the Internet of Things technology, the smart city architecture and the architecture of the Internet of Things are similar and can be divided into four layers: the perception layer, the transmission layer, and the platform layer. Application layer. The biggest difference between the smart city and the previous concept of the digital city lies in the intelligent processing of the information acquired by the sensory layer. Therefore, the smart city can also be considered as an upgraded version of the digital city. From city digitization to city intelligence, the key is to realize intelligent processing of digital information. The core of this is the introduction of big data processing technology.

Big data technology is the inevitable choice for processing data in the sensing layer The sensing layer is the basic physical network that the smart city system senses, identifies, and collects information in the real world. Massive data is generated in the sensing layer. Taking video surveillance as an example, there are 500,000 cameras currently used for video surveillance in Beijing. The amount of data per hour for one camera is a few G. The amount of data collected in Beijing every day is around 3 PB, and the annual video surveillance in a medium-sized city produces The data is about 300PB. These cameras return information in real time. The requirements for massive data storage and concurrent processing of data are nearly exacting. At the same time, the processing of massive data is not only an indispensable requirement for intelligence, but also a protection for IT investment, otherwise it will not only be able to fully tap the value of data, but will also be burdensome for massive amounts of data.

The construction of smart cities brings explosive growth in data volume. Big data is like blood everywhere in all aspects of smart cities such as smart transportation, smart medical care, and smart living. Urban management is changing from “experienced governance” to “scientific governance.” .

Big data provides powerful decision support for all areas of smart cities. In terms of urban planning, through the exploration of urban geography, meteorological and other natural information and economic, social, cultural, and other humanities and social information, it can provide strong decision support for urban planning and strengthen the scientific and forward-looking nature of urban management services. In terms of traffic management, real-time excavation of road traffic information can effectively alleviate traffic congestion and quickly respond to unexpected situations, providing scientific decision-making basis for the healthy operation of urban traffic. In terms of public opinion monitoring, through keyword search and semantic intelligence analysis, it can improve the timeliness and comprehensiveness of public opinion analysis, fully grasp social conditions and public opinions, improve public service capabilities, cope with public events in the network, and crack down on illegal crimes. In the field of security and disaster prevention, through the mining of big data, man-made or natural disasters and terrorist incidents can be discovered in a timely manner, and emergency response capabilities and security prevention capabilities can be improved.