Software architecture with python

Software Architecture with Python Table of Contents Software Architecture with Python Credits About the Author About the Reviewer www.PacktPub.com Customer Feedback Preface What this book covers What you need for this book Who this book is for Conventions Reader feedback Customer support Downloading the example code Downloading the color images of this book Errata Piracy Questions Principles of Software Architecture Defining Software Architecture Software Architecture versus design Aspects of Software Architecture Characteristics of Software Architecture An architecture defines a structure An architecture picks a core set of elements An architecture captures early design decisions An architecture manages stakeholder requirements An architecture influences the organizational structure An architecture is influenced by its environment An architecture documents the system An architecture often conforms to a pattern Importance of Software Architecture System versus enterprise architecture Architectural quality attributes Modifiability Testability Scalability Performance Availability Security Deployability Summary Writing Modifiable and Readable Code What is modifiability? Aspects related to Modifiability Understanding readability Python and readability Readability - antipatterns Techniques for readability Document your code Follow coding and style guidelines Review and refactor code Commenting the code Fundamentals of Modifiability – Cohesion & Coupling Measuring cohesion and coupling Measuring cohesion and coupling – string and text processing Exploring strategies for modifiability Providing explicit interfaces Reducing two-way dependencies Abstract common services Using inheritance techniques Using late binding techniques Metrics – tools for static analysis What are code smells? Cyclomatic complexity – the McCabe metric Testing for metrics Running Static Checkers Refactoring Code Refactoring code – fixing complexity Refactoring code - fixing code smells Refactoring code - fixing styling and coding issues Summary Testability – Writing Testable Code Understanding testability Software testability and related attributes Testability – architectural aspects Testability – strategies Reduce system complexity Improving predictability Control and isolate external dependencies White-box testing principles Unit testing Unit testing in action Extending our unit test case Nosing around with nose2 Testing with py.test Code coverage Measuring coverage using coverage.py Measuring coverage using nose2 Measuring coverage using py.test Mocking things up Tests inline in documentation – doctests Integration tests Test automation Test automation using Selenium Web Driver Test-Driven Development TDD with palindromes Summary Good Performance is Rewarding! What is performance? Software performance engineering Performance testing and measurement tools Performance complexity Measuring performance Measuring time using a context manager Timing code using the timeit module Measuring the performance of our code using timeit Finding out time complexity – graphs Measuring CPU time with timeit Profiling Deterministic profiling Profiling with cProfile and profile Prime number iterator class – performance tweaks Profiling – collecting and reporting statistics Third-party profilers Line profiler Memory profiler Substring (subsequence) problem Other tools Objgraph Pympler Programming for performance – data structures Mutable containers – lists, dictionaries, and sets Lists Dictionaries Sets Immutable containers – tuples High performance containers – the collections module deque defaultdict OrderedDict Dropping duplicates from a container without losing the order Implementing a Least Recently Used (LRU) cache dictionary Counter ChainMap namedtuple Probabilistic data structures – bloom filters Summary Writing Applications That Scale Scalability and performance Concurrency Concurrency versus parallelism Concurrency in Python – multithreading Thumbnail generator Thumbnail generator – producer/consumer architecture Thumbnail generator – resource constraint using locks Thumbnail generator – resource constraint using semaphores Resource constraint – semaphore versus lock Thumbnail generator – URL rate controller using conditions Multithreading – Python and GIL Concurrency in Python – multiprocessing A primality checker Sorting disk files Sorting disk files – using a counter Sorting disk files – using multiprocessing Multithreading versus multiprocessing Concurrecy in Python - Asynchronous Execution Pre-emptive versus cooperative multitasking The asyncio module in Python Waiting for a future – async and await Concurrent futures – high-level concurrent processing Disk thumbnail generator Concurrency options – how to choose? Parallel processing libraries Joblib PyMP Fractals – the Mandelbrot set Fractals – Scaling the Mandelbrot set implementation Scaling for the Web Scaling workflows – message queues and task queues Celery – a distributed task queue The Mandelbrot set using Celery Serving with Python on the Web—WSGI uWSGI – WSGI middleware on steroids Gunicorn – unicorn for WSGI Gunicorn versus uWSGI Scalability architectures Vertical scalability architectures Horizontal scalability architectures Summary Security – Writing Secure Code Information Security architecture Secure coding Common security vulnerabilities Is Python secure? Reading input Evaluating arbitrary input Overflow errors Serializing objects Security issues with web applications Server Side Template Injection Server-Side Template Injection – mitigation Denial of Service Cross-Site Scripting(XSS) Mitigation – DoS and XSS Strategies for security – Python Secure coding strategies Summary Design Patterns in Python Design patterns - Elements Categories of design patterns Pluggable hashing algorithms Summing up pluggable hashing algorithm Patterns in Python – Creational The Singleton pattern The Singleton – we need a Singleton? State sharing – Borg versus Singleton The Factory pattern The Prototype pattern Prototype – deep versus shallow copy Prototype using metaclasses Combining patterns using metaclasses The Prototype factory The Builder pattern Patterns in Python – Structural The Adapter pattern The Facade pattern Facades in Python The Proxy pattern An instance-counting proxy Patterns in Python – Behavioral The Iterator pattern The Observer pattern The State pattern Summary Python – Architectural Patterns Introducing MVC Model Template View (MTV) – Django Django admin – automated model-centric views Flexible Microframework – Flask Event-driven programming Chat server and client using I/O multiplexing with the select module Event-driven programming versus Concurrent programming Twisted Twisted – a simple web client Chat Server using Twisted Eventlet Greenlets and Gevent Microservice architecture Microservice frameworks in Python Microservices example – restaurant reservation Microservices – advantages Pipe and Filter architectures Pipe and Filter in Python Summary Deploying Python Applications Deployability Factors affecting Deployability Tiers of software deployment architecture Software deployment in Python Packaging Python code Pip Virtualenv Virtualenv and pip Relocatable virtual environments PyPI Packaging and submission of an application The init .py files The setup.py file Installing the package Submitting the package to PyPI PyPA Remote deployments using Fabric Remote deployments using Ansible Managing remote daemons using Supervisor Deployment – patterns and best practices Summary 10 Techniques for Debugging Maximum subarray problem The power of "print" Analysis and rewrite Timing and optimizing the code Simple debugging tricks and techniques Word searcher program Word searcher program—debugging step Word searcher program—debugging step Word searcher program—final code Skipping blocks of code Stopping execution External dependencies—using wrappers Replacing functions with their return value/data (Mocking) Saving to / loading data from files as cache Saving to / loading data from memory as cache Returning random/mock data dictionaries / Dictionaries sets / Sets immutable containers / Immutable containers – tuples high performance containers / High performance containers – the collections module Probabilistic data structures / Probabilistic data structures – bloom filters bloom filters / Probabilistic data structures – bloom filters Performance Engineering Life Cycle (PGLC) / Software performance engineering performance testing about / Testability – architectural aspects load testing / Testability – architectural aspects stress testing / Testability – architectural aspects scalability testing / Testability – architectural aspects Perl Webserver Gateway Interface (PSGI) about / uWSGI – WSGI middleware on steroids Pip / Pip Pipe and Filter architecture about / Pipe and Filter architectures in Python / Pipe and Filter in Python pluggable hashing algorithms / Pluggable hashing algorithms summing up / Summing up pluggable hashing algorithm pre-emptive multitasking versus cooperative multitasking / Pre-emptive versus cooperative multitasking about / Pre-emptive versus cooperative multitasking primality checker about / A primality checker profiling about / Profiling deterministic profiling / Deterministic profiling with cProfile and profile / Profiling with cProfile and profile prime number iterator class / Prime number iterator class – performance tweaks statistics, collecting / Profiling – collecting and reporting statistics statistics, reporting / Profiling – collecting and reporting statistics third-party profilers / Third-party profilers Prototype pattern about / The Prototype pattern Prototype factory / The Prototype factory Builder pattern / The Builder pattern Prototype patterns / Categories of design patterns Proxy pattern / Categories of design patterns Publish-Subscribe / The Observer pattern py.test used, for testing / Testing with py.test Pycodestyle / Metrics – tools for static analysis Pyflakes / Metrics – tools for static analysis Pylint / Metrics – tools for static analysis running / Running Static Checkers PyMP about / PyMP Pympler about / Pympler Python concurrency, with multithreading / Concurrency in Python – multithreading and GIL, used for multithreading / Multithreading – Python and GIL asyncio module / The asyncio module in Python security / Is Python secure? reading input / Reading input Python Enhancement Proposal (PEP) / Follow coding and style guidelines about / Serving with Python on the Web—WSGI Python Imaging Library (PIL) about / Thumbnail generator, Fractals – the Mandelbrot set Python Open Web Application Security Project (OWASP) project / Strategies for security – Python Python Package Index (PyPI) about / PyPI Python Packaging Authority (PyPA) about / PyPA URL / PyPA Q quality attributes about / Architectural quality attributes modifiability / Modifiability testability / Testability scalability / Scalability performance / Performance availability / Availability security / Security deployability / Deployability Quality of Service (QoS) about / Horizontal scalability architectures R RabbitMQ about / Scaling workflows – message queues and task queues readability about / Aspects related to Modifiability, Understanding readability and Python / Python and readability antipatterns / Readability - antipatterns techniques / Techniques for readability code, documenting / Document your code coding and style guidelines, following / Follow coding and style guidelines code, refactor and review / Review and refactor code Receiving Applications about / Scaling workflows – message queues and task queues Relational Database Systems (RDBMs) about / Horizontal scalability architectures resource constraint(s) implementing, with locks / Thumbnail generator – resource constraint using locks implementing, with semaphores / Thumbnail generator – resource constraint using semaphores Response For Class (RFC) / Reduce system complexity reusability about / Aspects related to Modifiability RLock about / Concurrency in Python – multithreading root-mean-squared (RMS) / Commenting the code S scalability measuring / Scalability and performance scalability, quality attributes about / Scalability horizontal scalability / Scalability vertical scalability / Scalability scalability architectures about / Scalability architectures vertical scalability architectures / Vertical scalability architectures horizontal scalability architectures / Horizontal scalability architectures scaling on web / Scaling for the Web workflows / Scaling workflows – message queues and task queues message queues / Scaling workflows – message queues and task queues task queues / Scaling workflows – message queues and task queues secure architecture about / Information Security architecture secure coding about / Secure coding strategies / Secure coding security, quality attributes about / Security integrity / Security origin / Security authenticity / Security security,strategies about / Strategies for security – Python reading input / Strategies for security – Python expressions, evaluating / Strategies for security – Python serialization / Strategies for security – Python overflow errors / Strategies for security – Python string formatting / Strategies for security – Python files / Strategies for security – Python passwords and sensitive information, handling / Strategies for security – Python local data / Strategies for security – Python race conditions / Strategies for security – Python security updates / Strategies for security – Python security issues with web applications / Security issues with web applications Server Side Template Injection (SSTI) / Server Side Template Injection Server Side Template Injection (SSTI), mitigation / Server-Side Template Injection – mitigation Denial of Service (DOS) / Denial of Service Cross-Site Scripting (XSS) / Cross-Site Scripting(XSS) Cross-Site Scripting (XSS), mitigation / Mitigation – DoS and XSS Denial of Service (DOS), mitigation / Mitigation – DoS and XSS security testing / Testability – architectural aspects security vulnerabilities about / Common security vulnerabilities overflow errors / Common security vulnerabilities unvalidated input / Common security vulnerabilities improper access control / Common security vulnerabilities cryptography issues / Common security vulnerabilities information leak / Common security vulnerabilities race conditions / Common security vulnerabilities insecure file operations / Common security vulnerabilities Selenium Web Driver used, for test automation / Test automation using Selenium Web Driver semaphores used, for implementing resource constraint(s) / Thumbnail generator – resource constraint using semaphores versus locks / Resource constraint – semaphore versus lock Sending Applications about / Scaling workflows – message queues and task queues Server Side Template Injection (SSTI) about / Server Side Template Injection Service Oriented Architectures (SOA) / Microservice architecture sets / Sets setup.py file / The setup.py file Singleton pattern about / The Singleton pattern Software Architecture defining / Defining Software Architecture versus design / Software Architecture versus design aspects / Aspects of Software Architecture characteristics / An architecture defines a structure, An architecture picks a core set of elements, An architecture manages stakeholder requirements, An architecture influences the organizational structure, An architecture often conforms to a pattern importance / Importance of Software Architecture software deployment about / Software deployment in Python Python code, packaging / Packaging Python code Pip / Pip Virtualenv / Virtualenv Virtualenv and pip / Virtualenv and pip virtual environments, relocatable / Relocatable virtual environments PyPI / PyPI application, packaging / Packaging and submission of an application application, submission / Packaging and submission of an application init .py files / The init .py files setup.py file / The setup.py file package, installing / Installing the package package, submitting to PyPI / Submitting the package to PyPI Fabric, using for remote deployments / Remote deployments using Fabric Supervisor, used for managing remote daemons / Managing remote daemons using Supervisor software deployment, patterns continuous deployment / Deployment – patterns and best practices BlueGreen deployment / Deployment – patterns and best practices Canary releases / Deployment – patterns and best practices Bucket testing (A/B testing) / Deployment – patterns and best practices induced chaos / Deployment – patterns and best practices Chaos Monkey / Deployment – patterns and best practices software deployment architecture, tiers about / Tiers of software deployment architecture Development/Test/Stage/Production / Tiers of software deployment architecture Development and Test/Stage/Production / Tiers of software deployment architecture Development and Test/ Stage and Production / Tiers of software deployment architecture Software Development Life Cycle (SDLC) / Software performance engineering Software Performance Engineering (SPE) / Software performance engineering State pattern / The State pattern strace used, for system call tracing / System call tracing using strace stress testing tools / Performance testing and measurement tools structural pattern about / Categories of design patterns, Patterns in Python – Structural proxy pattern / Categories of design patterns Composite pattern / Categories of design patterns Adapter pattern / The Adapter pattern Facade pattern / The Facade pattern Facades / Facades in Python Proxy pattern / The Proxy pattern instance-counting proxy / An instance-counting proxy subarray issues / Maximum subarray problem print, power / The power of "print" analysis and rewrite / Analysis and rewrite code, timing / Timing and optimizing the code code, optimizing / Timing and optimizing the code Supervisor used, for managing remote daemons / Managing remote daemons using Supervisor Supervisord about / Horizontal scalability architectures symmetric multiprocessing (SMP) about / Parallel processing libraries System Architecture versus, Enterprise Architecture / System versus enterprise architecture T task about / Celery – a distributed task queue task queues about / Scaling workflows – message queues and task queues Test-Driven Development (TDD) about / Test-Driven Development with palindromes / TDD with palindromes testability about / Understanding testability software testability / Software testability and related attributes architectural, aspects / Testability – architectural aspects strategies / Testability – strategies reduce system complexity / Reduce system complexity predictability, improving / Improving predictability control and isolate external dependencies / Control and isolate external dependencies testability, quality attributes about / Testability test automation about / Test automation Selenium Web Driver used / Test automation using Selenium Web Driver third-party profilers line profiler / Line profiler memory profiler / Memory profiler substring (subsequence) problem / Substring (subsequence) problem threading module about / Concurrency, Concurrency in Python – multithreading thumbnail generator about / Thumbnail generator producer/consumer architecture / Thumbnail generator – producer/consumer architecture resource constraint(s), implementing with locks / Thumbnail generator – resource constraint using locks resource constraint(s), implementing with semaphores / Thumbnail generator – resource constraint using semaphores URL rate controller, implementing with conditions / Thumbnail generator – URL rate controller using conditions timeit module used, for code timing / Timing code using the timeit module used, for measuring performance / Measuring the performance of our code using timeit used, for measuring CPU time / Measuring CPU time with timeit trace module / The trace module tuples / Immutable containers – tuples twisted about / Twisted two-way dependencies reducing / Reducing two-way dependencies twsited chat server, using for / Chat Server using Twisted U unit testing about / Testability – architectural aspects, Unit testing URL rate controller implementing, with conditions / Thumbnail generator – URL rate controller using conditions usability testing / Testability – architectural aspects uWSGI about / uWSGI – WSGI middleware on steroids versus Gunicorn / Gunicorn versus uWSGI V vertical scalability architectures about / Vertical scalability architectures resources, adding to existing system / Vertical scalability architectures existing resources, using in system / Vertical scalability architectures Virtualenv about / Virtualenv and pip / Virtualenv and pip W web scaling on / Scaling for the Web Web Content Accessibility Guidelines (WCAG) / Testability – architectural aspects Web Server Gateway Interface (WSGI) about / Serving with Python on the Web—WSGI White-box testing about / Testability – architectural aspects principles / White-box testing principles unit testing / Unit testing unit testing, in action / Unit testing in action unit test case / Extending our unit test case nose2 / Nosing around with nose2 py.test, testing with / Testing with py.test code coverage / Code coverage mocking / Mocking things up integration tests / Integration tests test automation / Test automation workers about / Celery – a distributed task queue ... Principles of Software Architecture Defining Software Architecture Software Architecture versus design Aspects of Software Architecture Characteristics of Software Architecture An architecture. . .Software Architecture with Python Table of Contents Software Architecture with Python Credits About the Author About the Reviewer www.PacktPub.com... An architecture is influenced by its environment An architecture documents the system An architecture often conforms to a pattern Importance of Software Architecture System versus enterprise architecture