3.1

Topology data management framework

Based on the traditional grid topology data management method, this paper integrates the time dimension data of the grid topology and constructs a multi-temporal grid topology data management framework, including grid topology data model management, grid topology data import, multi-temporal grid topology data storage and multi-temporal grid topology data access as shown in Figure 1.

Figure 1.

Multitemporal grid topology data management framework

3.2

Topology data model

In view of the fact that the original grid topology data model is difficult to support the management of multi-temporal grid topology data, this paper designs a grid topology data model that integrates time dimensions based on the basic theory of graph computing and graph data model, and realizes the model support of multi-temporal grid topology data management.

(1)Multi-temporal grid topology data model definition. Sort out the key equipment and components in the power grid topology, analyze the typical characteristics of the power grid topology, abstract the interconnected equipment and components such as power points, busbars, substations, circuit breakers, and load points in the power grid into vertices, and abstract the power lines connecting these vertices into edges to form vertex entities and edge entities, and define and describe them through entity definitions, entity properties, time labels, etc. respectively. Entity definitions mainly include entity numbers, entity names, entity space characteristics, etc., entity attributes mainly include entity types, entity information, power attributes, management information, entity descriptions, etc., and time labels describe the time version characteristics of entities in the form of timestamps. Table 1 shows the detailed model of vertex solid models, including vertex definitions, vertex properties, and timestamps. Table 2 describes the detailed model of edge entity models, including edge definitions, edge attributes, and timestamps.

Table 1.

Vertex entities

Table 2.

Edge entities

(2)Design of multi-temporal grid topology data model based on GraphX. GraphX defines a Property Graph data model that creates graphs in vertices and edges. GraphX stores graph data in the form of Resilient Distributed Datasets(RDD) distributed on the nodes of the cluster, where the vertex RDD name corresponding to Vertics is VertexRDD, and the attributes include vertex ID and vertex attribute for storing vertex collections. The RDD name corresponding to Edges is EdgeRDD, and the attributes include source vertex ID, target vertex ID, and edge attribute to store edge collections. This paper designs a multi-temporal grid topology data model based on GraphX to support the management of multi-temporal grid topology data. The specific steps are as follows:

4.1

Multitemporal data storage implementation

(1)The underlying storage system. Hbase divides the storage of data by column family, and sets several columns under the column family to achieve flexible data access, and uniquely distinguishes the data of a row by introducing Rowkey as the primary key. HBase uses regions for partitioned data storage, allocating the data of a large table to different regions based on different ranges of Rowkey, and each region is responsible for a certain range of data access and storage, providing access speed and reducing data latency. HBase’s time version data storage is achieved through timestamps, by using different timestamps to identify the same Rowkey row corresponding to the different versions of the data, the same Rowkey data according to timestamp reverse order, users can read the old version of the data with the specified timestamp value. According to HBase, Key-Value(KV) mode is adopted, and the data storage mode of the underlying storage system of the multitemporal data storage module is (Rowkey, Family, Column, Timestamp) -> Value, where (Rowkey, Family, Column, Timestamp) is the key, which is used to identify the storage and reading of data, as shown in Figure 2.

Figure 2.

Key-value model of the multitemporal data storage module

(2)Data storage and access methods. The way to store and access is to combine Spark with HBase, and use the HBase-Spark connection tool class provided by HBase to support Spark to read and write HBase in batches in RDD by creating HBaseContext objects. The HBaseContext creation process is as follows:

The HBaseContext object provides an interface for batch processing of HBase data, where bulkPut method is used to send put operations to HBase massively parallel, bulkDelete method is used to send delete operations to HBase massively parallel, and bulkGet method is used to send get operations to HBase massively parallel. Based on the interface provided by HBaseContext, secondary development and encapsulation are carried out to form a batch processing method that supports temporal data storage and reading, as shown in Table 3.

Table 3.

Temporal data processing methods

4.2

Power grid topology data management implementation

(1)GraphX-based data storage and reading. Storage methods include storage for vertex collections and storage for edge collections. Vertex storage is done by creating a vertex RDD (vertexTSRDD[VertexUnit]) containing a timestamp and calling the timeVersionDataBulkPut method to store the collection of vertices in the HBase data table. Edge storage is done by creating an edge RDD (edgeTSRDD[EdgeUnit]) containing timestamps, and calling the timeVersionDataBulkPut(rdd:RDD, rdd_table:String) method of the underlying storage to store the collection of edges in the specified table (rdd_table) in HBase. For different graph objects, vertex data RDDs and edge data RDDs are stored in different HBase tables, vertex data RDDs with different temporals of the same graph object will be stored in the same HBase table (such as vertex_table), and edge data RDDs with different temporal states will also be stored in the same HBase table (such as edge_table), and the data time version will be defined by timestamp. The process of data access is to obtain the RDD of vertex data and RDD of edge data by calling the underlying storage method, and then build a graph object (Graph) after conversion to provide access to the graph. The specific steps are: first call the timeVersionDataBulkGet(rdd_table:String, timestamp: Long) method to get vertexTSRDD[VertexUnit] and edgeTSRDD[EdgeUnit], respectively, and then perform Map conversion to vertexRDD[keys: Long, values:String] respectively. and edgeRDD[src: Long, dst: Long, values: String], and finally create a graph object (Graph) via Graph (VertexRDD, EdgeRDD).

(2)Data import. The data import module includes a data connection part, a data preprocessing part and a Data transformation storage part, as shown in Figure 3.

Figure 3.

Flow chart of the data import module

Data connection. Based on Spark components, data connection modules for different data storage forms such as relational databases, non-relational databases, text data, and data streams have been developed, JDBC is used for database connection, text data is directly read, and SparkStreaming is used for data streaming. The main methods of JDBC-based database connections are as follows:

Data preprocessing. Based on the DataFrame formed after data connection, using the rich DataFrame operator provided by Spark, retrieve duplicate data and deduplicate them, find missing data and fill them according to the average value of neighboring data or set default values, analyze abnormal data and replace outliers according to the change trend of neighboring data. Then, the DataFrame.rdd method is used to convert the DataFrame into RDD, and finally the vertex RDD dataset and edge RDD dataset that meet the data quality requirements are formed.

Data transformation storage. After obtaining the preprocessed RDD dataset, the multitemporal grid topology data model is applied to add timestamp information to the RDD dataset based on GraphX, in which the structure of the vertex RDD is converted from (keys: Long, attribute:String) to (keys: Long, attribute:String, timestamp: Long), and the structure of the edge RDD is converted by (src: Long, dst: Long, attribute: string) is converted to (src: Long, dst: Long, attribute: String, timestamp: Long) to form vertex datasets and edge datasets containing temporal attributes, and then stored into the multitemporal data storage system using the data storage interface provided by the data model management module.

(3)Data access. Based on GraphX and multi-temporal grid topology data model, develop a data reader, according to the given power topology object and time version, read the specified temporal grid topology data through the data access interface provided by the data model management module, and provide it in the form of grid topology diagram (Graph), and the attribute data of vertices and edges of the grid topology map is restored and read by parsing the attribute strings of vertices and edges.