TOWARDS a FEDERATIVE POLYGLOT ARCHITECTURE FOR MANAGING SMART GRID DATA

TOWARDS A FEDERATIVE POLYGLOT ARCHITECTURE FOR MANAGING SMART GRID DATA Presented by: NHU QUYNH NGUYEN 20112648 – SIC PFIEV 56 Supervisor: Professor Christine Collet Professor Christophe Bobineau Professor Binh Minh Nguyen Postdoctoral Researcher Houssem Chihoub Performed at: Grenoble Computer Science Laboratory (LIG) HADAS Team Outline I Introduction II Polyglot Solution Overview of architecture Data layer III Architecture Implementation et Preliminary Performance Evaluation Data layer implementation Benchmarking queries IV Conclusion and Future work I Introduction: What is the Smart Grid? Many Definitions – But One VISION - Engaging Consumers - Enhancing Efficiency - Ensuring Reliability - Enabling Renewables & Electric Transportation I Introduction: Data Management Challenge in Smart Grid (1) The Electricity Consumption Information Collection System of the Grid Collector RDBMS • In France ERDF is planning to deploy 35 million smart meters by the year 2021 • Smart meter readings every 15 minutes => 96 million reads per day for every million meters => 3000-fold increase in data I Introduction: Data Management Challenge in Smart Grid (2) • Five separate classes of smart grid data, each with its own unique characteristics I Introduction: Data Management Challenge in Smart Grid (3) • Taking long time to perform a data analysis • Mismatch between the database model and the programming model • Difficult to modify a relational schema • Increasing the amount of data II Polyglot Solution: Concept   Using multiple data storage technologies for an individual application Chosen based upon the best way data can be stored and retrieved for the application Application II Polyglot Solution: Architecture of proposed system II Polyglot Solution: Data layer (1) Meter Data Weather Data Geographic Data II Polyglot Solution: Data layer (2)  Why using PostgreSQL for Client data? • Client data stores contact details of customers such as first name, last name, address • Client data requires concurrency control strategies, data uniqueness, data security and read-only access • A relational database provides more control and guarantees over data • PostgreSQL is powerful, a open source objectrelational database system • PostgreSQL supports CSV file, more reliability, can run in Linux, BSD, Windows… 10 II Polyglot Solution: Data layer (3)  Why using Cassandra for Meter data • Meter data stores measurements of customers are recorded by smart meters, time series data • Data arrives from many locations, requires read and write scalability • Cassandra is an excellent fit for handling data in sequence regardless of datatype or size • Cassandra is highly performant with tables that have thousands of columns 11 II Polyglot Solution: Data layer (4)  Why using MongoDB for Weather and Geographic data • Geographic data: stores location of smart meters, geospatial queries, simple model • Weather data: stores weather conditions such as wind speed, dry bulb temperature… • MongoDB: provides scale-out capabilities along with smoother and faster data access • Indexing is needed in this use case in order to efficiently perform geospatial queries 12 III Architecture Implementation: Technologies    Spring framework: Java platform that provides comprehensive infrastructure support for developing Java applications Apache Maven: a software project management and comprehension tool based on the concept of a project object model (POM) RESTful API: REpresentational State Transfer architecture based web services and uses HTTP Protocol for data communication 13 III Architecture Implementation: Data layer (1)  Client data modeling and loading 14 III Architecture Implementation: Data layer (2)  Configuration PostgreSQL 15 Architecture Implementation: Data layer (3)  Meter data modeling and loading 16 III Architecture Implementation: Data layer (4)  Configuration Cassandra 17 III Architecture Implementation: Data layer (5)  Geographic and Weather data modeling and loading 18 III Architecture Implementation: Data layer (6)  Configuring MongoDB 19 III Architecture Implementation: Benchmarking Queries (1)  Query 1: Search highest electricity consuming measured by smart meter of client that have registered a specific address in the city of Lyon 20 III Architecture Implementation: Benchmarking Queries (2) Query 2: Calculate the total amount of electricity consumption by clients in Lyon at minimum temperature below than an input temperature  21 III Architecture Implementation: Benchmarking Queries (3) Query 3: Calculate the electric bill using during one month of electricity consumption based on a specific meter ID of client  22 IV Conclusion and Future work  • • Conclusion: Research on a new solution designed for management and storage data with high-performance – Polyglot solution Deployment an architecture platform and a data model simulation for Smart Grid management system  Future mission: • Performance comparison with traditional architecture • Increasing complexity of the query 23 24 [...]... Protocol for data communication 13 III Architecture Implementation: Data layer (1)  Client data modeling and loading 14 III Architecture Implementation: Data layer (2)  Configuration PostgreSQL 15 Architecture Implementation: Data layer (3)  Meter data modeling and loading 16 III Architecture Implementation: Data layer (4)  Configuration Cassandra 17 III Architecture Implementation: Data layer (5)...II Polyglot Solution: Data layer (3)  Why using Cassandra for Meter data • Meter data stores measurements of customers are recorded by smart meters, time series data • Data arrives from many locations, requires read and write scalability • Cassandra is an excellent fit for handling data in sequence regardless of datatype or size • Cassandra is highly performant with tables that have thousands of... columns 11 II Polyglot Solution: Data layer (4)  Why using MongoDB for Weather and Geographic data • Geographic data: stores location of smart meters, geospatial queries, simple model • Weather data: stores weather conditions such as wind speed, dry bulb temperature… • MongoDB: provides scale-out capabilities along with smoother and faster data access • Indexing is needed in this use case in order... efficiently perform geospatial queries 12 III Architecture Implementation: Technologies    Spring framework: Java platform that provides comprehensive infrastructure support for developing Java applications Apache Maven: a software project management and comprehension tool based on the concept of a project object model (POM) RESTful API: REpresentational State Transfer architecture based web services and uses... Geographic and Weather data modeling and loading 18 III Architecture Implementation: Data layer (6)  Configuring MongoDB 19 III Architecture Implementation: Benchmarking Queries (1)  Query 1: Search highest electricity consuming measured by smart meter of client that have registered a specific address in the city of Lyon 20 III Architecture Implementation: Benchmarking Queries (2) Query 2: Calculate... and Future work  • • Conclusion: Research on a new solution designed for management and storage data with high-performance – Polyglot solution Deployment an architecture platform and a data model simulation for Smart Grid management system  Future mission: • Performance comparison with traditional architecture • Increasing complexity of the query 23 24 ... total amount of electricity consumption by clients in Lyon at minimum temperature below than an input temperature  21 III Architecture Implementation: Benchmarking Queries (3) Query 3: Calculate the electric bill using during one month of electricity consumption based on a specific meter ID of client  22 IV Conclusion and Future work  • • Conclusion: Research on a new solution designed for management