Musée d’Orsay, Paris, France Located on the Seine’s left bank, the Musée d’Orsay is housed in a breathtaking Beaux-Arts building originally designed as the world’s first electrified urban railway station The original “Gare d’Orsay” was built on the site of the old Palais d’Orsay, which had lain empty since it burned in 1871 during the Paris Commune The building opened on Bastille Day, July 14, 1900, to help celebrate Paris’s Fifth Universal Exhibition Designated a Historical Monument in 1978, it was then recreated as a museum by Renaud Bardon, Pierre Colboc, and Jean-Paul Philippon of the ACT architecture group Per the museum’s official history, the new architects “highlighted the great hall, using it as the main artery of the visit, and transformed the magnificent glass awning into the museum’s entrance.” Inside, Gae Aulenti adapted the enormous station into museum spaces, unified via consistent stone wall and floor surfaces Opened in 1986, the new museum brought together three major art collections from the era 1848-1914 More than three million visitors now come every year to see works from artists including Cézanne, Courbet, Degas, Gauguin, Manet, Monet, and Renoir The Python Standard Library by Example Developer’s Library Visit informit.com/devlibrary for a complete list of available publications The Developer’s Library series from Addison-Wesley provides practicing programmers with unique, high-quality references and tutorials on the latest programming languages and technologies they use in their daily work All books in the Developer’s Library are written by expert technology practioners who are exceptionally skilled at organizing and presenting information in a way that is useful for other programmers Developer’s Library titles cover a wide range of topics, from open source programming languages and technologies, mobile application development, and web development to Java programming and more The Py thon Standard Library by Example Doug Hellmann Boston • Columbus • Indianapolis • New York • San Francisco • Amsterdam • Cape Town Dubai • London • Madrid • Milan • Munich • Paris • Montreal • Toronto • Delhi • Mexico City São Paulo • Sydney • Hong Kong • Seoul • Singapore • Taipei • Tokyo Many of the designations used by manufacturers and sellers to distinguish their products are claimed as trademarks Where those designations appear in this book, and the publisher was aware of a trademark claim, the designations have been printed with initial capital letters or in all capitals The author and publisher have taken care in the preparation of this book, but make no expressed or implied warranty of any kind and assume no responsibility for errors or omissions No liability is assumed for incidental or consequential damages in connection with or arising out of the use of the information or programs contained herein For information about buying this title in bulk quantities, or for special sales opportunities (which may include electronic versions; custom cover designs; and content particular to your business, training goals, marketing focus, or branding interests), please contact our corporate sales department at corpsales@pearsoned.com or (800) 382-3419 For government sales inquiries, please contact governmentsales@pearsoned.com For questions about sales outside the U.S., please contact intlcs@pearson.com Visit us on the Web: informit.com/aw Library of Congress Control Number: 2017932317 Copyright © 2017 Pearson Education, Inc All rights reserved Printed in the United States of America This publication is protected by copyright, and permission must be obtained from the publisher prior to any prohibited reproduction, storage in a retrieval system, or transmission in any form or by any means, electronic, mechanical, photocopying, recording, or likewise For information regarding permissions, request forms and the appropriate contacts within the Pearson Education Global Rights & Permissions Department, please visit www.pearsoned.com/permissions/ ISBN-13: 978-0-13-429105-5 ISBN-10: 0-13-429105-0 17 For Theresa, my one true love This page intentionally left blank Contents at a Glance Introduction xxxi Acknowledgments xxxiii About the Author xxxv Chapter Text Chapter Data Structures 65 Chapter Algorithms 143 Chapter Dates and Times 211 Chapter Mathematics 239 Chapter The File System 295 Chapter Data Persistence and Exchange 395 Chapter Data Compression and Archiving 477 Chapter Cryptography 523 Chapter 10 Concurrency with Processes, Threads, and Coroutines 535 Chapter 11 Networking 687 Chapter 12 The Internet 753 Chapter 13 Email 841 Chapter 14 Application Building Blocks 887 Chapter 15 Internationalization and Localization 1003 Chapter 16 Developer Tools 1023 Chapter 17 Runtime Features 1169 Chapter 18 Language Tools 1279 vii viii Contents at a Glance Chapter 19 Modules and Packages 1329 Appendix A Porting Notes 1351 Appendix B Outside of the Standard Library 1367 Index of Python Modules 1373 Index 1375 1402 re module, search options (continued) input with multiple lines, 37–38 overview of, 14 Unicode, 38–39 verbose expression syntax, 39–43 read() method access data from archive, 514 asynchronous I/O, 651, 654 calling subprocesses, 664–666 customizing protocol handlers, 770–773 I/O multiplexing abstractions, 724–725 reading configuration files, 961–963 readable sockets, 730–731, 738–739 Reading compressed data in gzip, 489–490 compressed files in bz2, 497–498 configuration files, 961 files in pathlib, 312 GNU zip files, gzip, 486 from Maildir mailbox, 858 mbox mailbox, 853–854 memory-map files, 361–362 metadata from archive, tarfile, 504–505 metadata from archive, zipfile, 512–514 read text efficiently See linecache module Unicode data, 498–499 readline module accessing completion buffer, 927–931 auto-completion for commands, 942–944 cmd module using, 938 completing text, 924–927 configuring readline, 923–924 GNU readline library with, 922–923 hooks, 934–935 input history, 931–934 overview of, 887 readmodule(), 1161–1163 Receiving signals, 554–555 Recursion controlling, 1190–1191 deep copy, 133–136 Index examining file system contents, 1228–1229 pprint, 139 in substitution errors during interpolation, 977 Redirecting output streams, contextlib, 201 Reducing data sets, functools, 158–160 ref class, reference callbacks, 122–123 Reference counts, memory management, 1187–1188 ReferenceError exception, weakref proxies, 127 References circular, 402–404 finding to uncollectable objects, 1259–1261 impermanent object See weakref module tracing, 1255–1257 register() attribute, of generic functions, 161–163 Registering alternate API names, xmlrpc.server, 831–832 arbitrary API names, xmlrpc.server, 831–832 concrete classes, abc, 1287–1288 custom encoding, codecs, 386–389 data channels using poll(), 737–738 decorator syntax, atexit, 994 dispatching calls, xmlrpc.server, 835–836 dotted API names, xmlrpc.server, 830–831 exit callbacks, atexit, 993–994 introspection API, xmlrpc.server, 837–839 new column types, sqlite3, 424 Registry, shutil-managed archive, 348–349 Regular expressions See re module Relational database, embedded See sqlite3 module Reload modules, 1213–1214 Removing directory contents, 344–345 empty directory, 318 messages from Maildir mailbox, 858–860 messages from mbox mailbox, 854 sections/options from ConfigParser, 971 Reordering keys in OrderedDict, 96–97 mappings with ChainMap, 76–77 repeat() function, values in itertools, 170–172 Repetition regular expression character sets, 23 regular expression escape codes, 24–26 regular expression patterns, 18–20 of warnings, 1283–1284 replace() method, dates, 224–225 replace mode, decoding errors, 375 Request handlers creating network servers, socketserver, 742–744, 748 implementing web servers, http.server, 781–783 requests package, Internet, 1369 Resolution, datetime, 221–222, 224 resolve() method, normalizing path, 306, 313 resource module, 1169, 1251–1254 Resources controlling access to, multiprocessing, 603–604 controlling access to, threading, 572–576 controlling concurrent access to, multiprocessing, 605–607 limiting access to concurrent, threading, 581–583 thread specific data, 583–585 ResourceWarning warnings, debugging with asyncio, 673 Restricting access to data, sqlite3, 442–444 Index Results gathering from coroutines, 635–636 of tasks, concurrent.futures, 679–680, 683 Retrieving data, sqlite3, 415 return command, stepping through program in pdb, 1116–1117 Reverse lookups, socket, 697, 698–699 rfc822 module, removed from Python 3, 1354 rglob() method, directory contents in pathlib, 311–312 Rich comparison methods, functools, 151–153 Rich comparison operators, 184 RLock, re-entrant locks, 576–577 RobotFileParser.can_fetch(), testing access permissions, 774–775 robots.txt, urllib robotparser, 773–776 rollback() method, preserve changes in transactions, 429–430 rollover() method, temporary spooled files, 334 Rotation double-ended queue, 87–88 of log files, 981–982 Rounding, decimal, 245–247 Row objects, sqlite3, 417–419 RSS feed, converting M3U files to, 987–989 Rules, sharing parser (in argparse), 901–902 Runtime accessing operating system features See os module accessing system version information See platform module determining interpreter settings See sysconfig module finding message catalogs at, 1007–1008 garbage collection See gc module load and activate message catalog in gettext, 1004 managing system resources See Resources 1403 overview of, 1169 system-specific configuration See sys module rx argument, matching files with regular expression, 1156–1157 S safe_substitute() method, string.Template, Sampling, random, 260–261 Saving configuration files, 972 state, random, 255–256 scandir, examining file system contents, 1229–1230 sched module canceling events, 1001–1002 event priorities, 1001 implementing timed event scheduler, 998–999 overlapping events, 1000 running events with delay, 999–1000 timed event scheduler using, 998–999 Scheduling callbacks in asyncio, 622–623 implementing timed event scheduler See sched module individual tasks in concurrent.futures, 678–679 Schema, database in sqlite3, 413 Search See also re module, search options constraints in regular expressions, 28–29 criteria in imaplib, 872–874 files in shutil, 345–346 messages in imaplib, 872 options in configparser, 972–975 Sections, configparser configuration file format, 961 modifying settings, 970–971 option search path, 972–975 testing if present, 965 Secure password prompt See getpass module Seeding, in random, 255 seek() method in-memory streams, io, 391 reading compressed data, gzip, 489–490 reading compressed files, bz2, 497–498 reading/writing Unicode data, bz2, 499 temporary files, tempfile, 332 select module non-blocking I/O with timeouts, 734–737 platform-specific options, 742 using poll(), 737–741 using select(), 729–734 wait for I/O efficiently with, 728–729 select statements, query parameters in sqlite3, 421 select()function, calling programs in selectors, 724 Selecting mailbox, imaplib, 871 selectors module Echo client, 726–727 Echo server, 724–725 I/O multiplexing abstractions with, 724 new in Python 3, 1352 operating model, 724 server and client together, 727–728 Self-referencing regular expressions, 48–53 Semaphore multiprocessing, 605–607 threading, 581–583 Sequence operators, functional interface for, 186–187 SequenceMatcher class, difflib, 61–64 Sequences comparing See difflib module Counter initialization via data, 80 deques as type of container for, 84 efficiently merging sorted, 108–109 of fixed type data See array module reducing data set in functools, 160 Serialization object See also pickle module, 587 XML to stream, 464–466 1404 ServerProxy object, connecting to XML-RPC server, 817–819 Server(s) connecting to IMAP, 864–866 connecting to XML-RPC, 817–819 creating network See socketserver module I/O multiplexing abstractions, selectors, 724–728 implementing web See http.server module looking up addresses in network, 700–702 sample mail See smtpd module sending binary data to, 722 TCP/IP, 704–711 types, socketserver, 742–743 UDP, 711–713 UDS, 714–716 Service information, socket, 697 Sessions, signaling between processes, 550–553 set() method modifying settings in ConfigParser, 970–971 signaling between threads, 571–572 sets module, removed from Python 3, 1355 Settings accessing configuration, 963–970 modifying configparser, 970–971 setuptools-git, 1371 setuptools module, 1371 sha module, removed from Python 3, 1355 SHA1 algorithm in hashlib, 525 in hmac, 529 UUID and 5, name-based values, 800–801 shallow argument, comparing files in filecmp, 354 Shallow copies, duplicate objects, 130–131 shallow module, 1340 Shared namespaces, multiprocessing, 608–610 Index Shared state, multiprocessing, 608 Sharing parser rules, argparse, 901–902 shell argument, subprocess, 537 Shell commands, running, 947–948 Shell-style syntaxes, parsing See shlex module shelve module, 405–408, 1362 ShelveLoader custom package importing, 1211–1213 importing modules from shelf, 1205–1211 reloading modules, 1213–1214 shlex module controlling parser, 956–957 embedded comments, 954 error handling, 957–958 including other sources of tokens, 955–956 making safe strings for shells, 953 parse-style syntaxes with, 951 parsing quoted strings, 951–953 POSIX vs non-POSIX parsing, 959–960 splitting strings into tokens, 954 Short-form options, getopt, 917 shorten() function, truncating long text, 12–13 show_code, printing summary of function, 1299 showwarning(), warning message delivery, 1284 shuffle() function, random, 258–260 shutdown() method, concurrent.futures, 683–684 shutil module archives, 346–350 copying file metadata, 340–342 copying files, 337–340 file system space, 350–351 finding files, 345–346 as high level file operations, 337 working with directory trees, 342–345 SIGINT See signal module Sign flag, creating Decimals from, 240 Signal handlers mixing signals and threads, 558–560 receiving signals via, 554–555 retrieving registered, 555–556 signal module alarms, 556–557 asynchronous system events via, 553 changes in Python 3, 1362 ignoring signals, 557–558 receiving signals, 554–555 retrieving registered handlers, 555–556 signaling between processes, subprocess, 548–553 signals and threads, 558–560 Signals mixing threads and, 558–560 between processes, 602–603 between threads, 571–572 Signals, receiving Unix, 668–670 signature(), method or function arguments, 1319–1322 Signing cryptographic messages, hmac, 528 Simple Mail Transport Protocol (SMTP) email See smtplib module sample mail servers See smtpd module SimpleCompleter class, autocompletion of text in readline, 925–927 SimpleObject, reconstructing objects in pickle, 400 SimpleXMLRPCServer, 828–829, 835–836 Sine function, math, 284, 288 Single-character wildcards, 321–322 singledispatch() decorator, generic functions, 161–163 site module customizing site configuration, 1175–1176 customizing user configuration, 1176–1177 import path, 1169–1170 managing user directories, 1171–1172 overview of, 1169, 1177–1178 Index scanning for path configuration files, 1172–1175 site-wide configuration, 1169 Size caching in functools, 156–158 constraining queue, 88–89, 111 struct buffer See weakref module skipkeys argument, encoding dictionaries, 807 skipping unittests for unmet conditions, 1067 sleep() function alarms in signal, 557 mixing signals and threads, 559–560 processor clock time, 216 sched using, 999 using map() with basic thread pool, 677–678 SMTP (Simple Mail Transport Protocol) email See smtplib module sample mail servers See smtpd module smtpd module create custom mail server, 847 debugging server, 850 mail server base class, 847–850 proxy server, 851 smtplib module authentication and encryption, 843–846 create client to send data to test server, 848–849 sending email message, 841–843 verifying email address, 846–847 SMTP_SSL, email encryption in smtplib, 843 Sniffer class, detecting CSV dialects, 472–474 SOCK_DGRAM sockets, 694 socket module changes in Python 3, 1362 finding service information, 697–700 IP address representations, 702–703 looking up hosts on network, 694–697 1405 looking up server addresses, 700–702 multicast, 717–721 network communication with, 693 non-blocking communication and timeouts, 723 sending binary data, 721–723 TCP/IP client and server, 704–711 user datagram client and server, 711–717 socket.error exception, 695, 723 socketpair() function, UDS sockets, 717 Sockets enabling SSL in asyncio on, 656–658 monitoring using select()function, 729–734 using poll() in select, 737–741 socketserver module changes in Python 3, 1363 compressing network data in bz2, 499–500 creating network servers with, 742 Echo example, 744–749 implementing server, 743 request handlers, 743 server objects, 743 server types, 742–743 threading and forking, 749–751 socketserver.TCPServer, 786 SOCK_STREAM sockets, 694 Sort algorithm See heapq module Sorted order queue See queue module use bisect to maintain lists in, 109–111 Source code creating message catalogs from, 1004–1007 disassembling, 1301–1302 retrieving for class or method, 1318–1319 retrieving from ZIP archive, 1346–1347 Source files compiling to byte-code, 1155–1159 reading Python, 329–330 Sources of arguments, argparse, 895–897 spawning new processes, 1244–1245 Special constants, math, 265 Special values, decimal, 243–244 Spider access control, Internet, 773–776 Splitting iterators, itertools, 164–167 parsing paths, os.path, 296–298 with patterns, re, 55–57 strings into tokens, shlex, 954 Spooled files, tempfile, 333–334 SQL-injection attacks, 419 sqlalchemy ORM, 1368 sqlite3 module bulk loading, 421–422 creating database, 412–415 custom aggregation, 440–441 defining new column types, 422–425 determining types for columns, 426–427 embedded relational database via, 412 exporting contents of database, 435–436 isolation levels, 431–434 in-memory databases, 434–435 query metadata, 417 querying with regular expressions, 439–440 restricting access to data, 442–444 retrieving data, 415–417 row objects, 417–419 threading and connection sharing, 441–442 transactions, 428–430 using Python functions in SQL, 436–438 using variables with queries, 419–421 Square root, computing in math, 278–279 sre module, removed from Python 3, 1355 SSL communication, enabling on sockets, 656–658 SSLContext, encryption, 656 1406 Stack controlling recursion, 1190–1191 examination of variables in, 1107–1111 examining, 1079–1081 inspecting frames and, 1324–1327 stack level in warnings, 1286 traces See traceback module watching with trace hooks, 1224–1225 Stacking behavior, ChainMap, 77 Stacks, dynamic context manager, 202–209 Standard deviation, statistics variance, 292–294 starmap() function, converting inputs in itertools, 169 start argument, 170, 257 start() method, search for patterns in text, 14 Start-up hook, readline, 934 start_server() method, 652, 656 STARTTLS extension, email authentication/encryption, 844–845 stat() method managing file system permissions, 1230–1231 pathlib file properties, 315–316 Static analysis tool, Python Code Quality Authority, 1372 Static methods, marking as abstract, 1295–1296 statistics module averages, 290–292 new in Python 3, 1352 variance, 292–294 Statistics, profiling and, 1140, 1144–1148 Status conditions, IMAP4 mailbox, 869–871 status(), mailbox in imaplib, 869–871 statvfs module, removed from Python 3, 1355 stddev() function, statistics variance, 293–294 Index stderr attribute, subprocess capturing error output, 543–544 capturing output, 538–541 interaction with another command, 546–548 stdin attribute, subprocess, 542–543, 546–548 stdin attribute, sys, 1186–1187 stdout attribute, subprocess capturing error output, 543–544 capturing output, 538–541 connecting segments of pipe, 545–546 interaction with another command, 546–548 working with pipes, 542–543 stdout attribute, sys, 1186–1187 step argument new values, itertools, 170 random integers, math, 257 stepping through program, pdb, 1111–1114 step() method, custom aggregation in sqlite3, 440–441 stop argument, random integers in math, 257 Storage of objects, persistent See shelve module store action, arguments, 891 str class modify strings with patterns, 53–54 as text processing tool, Stream-oriented transport, SOCK_STREAM sockets, 694 StreamReader instance, asynchronous I/O, 651, 653 Streams asynchronous IO, in asyncio, 650–655 calling subprocesses with, in asyncio, 664–666 mixed-content, in bz2, 495 mixed content, in zlib, 480–481 serializing XML to, 464–466 working with file-like, in pickle, 397–399 working with, in gzip, 490–491 working with, in json, 813–814 wrapping byte, in io, 392–393 StreamWriter instance, asynchronous I/O, 651, 653 strftime() function, 219–220, 230–231 strict mode handling decoding errors, 375 handling encoding errors, 373 string module advanced templates, 4–6 changes in Python 3, 1363 constants, 1, 6–7 formatter, functions, 1–2 template tool, 1, 2–4, 1367 StringIO buffers, in-memory streams, 390–391 Strings configuration settings, 964 converting in argparse, 910–912 creating fractions, 251 dis accepting string arguments, 1301–1302 encoding and decoding See codecs module encoding/decoding data in, 396–397 making safe for shells, 953 multiliner, 970 packing/unpacking data into, 117–118 parsing command line arguments with, 889–890 parsing in XML, 455–457 parsing quoted, 951–953 retrieving docstring for object, 1316–1317 splitting into tokens, 954 Unicode primer, 365–366 value types, 966–969 string.Template, 1–6 strptime() function datetime, 230–231 time, 218–219 Struct class, 117–121 struct module as binary data structures, 117 buffers, 120–121 changes in Python 3, 1363 defined, 65 endianness, 118–120 functions vs Struct class, 117 packing and unpacking, 117–118 sending binary data, 721–723 Index struct_time, 217–220 sub(), modify strings with patterns, 53–54 Subclasses from abstract base class, 1289–1291 from base class, 1288–1289 creating Process, multiprocessing, 597–598 creating Thread, threading, 568–569 Subdirectory, listing files in, 321 Submodules, within packages, 1333 subn(), modify strings with patterns, 54 subprocess module changes in Python 3, 1363 connecting segments of pipe, 545–546 interacting with another command, 546–548 running external command, 536–542 signaling between processes, 548–553 spawning additional processes, 535–536 working with pipes directly, 542–545 subprocesses, asyncio calling with coroutines/streams, 664–666 sending data to, 666–668 using protocol abstraction with, 661–664 SubprocessProtocol, asyncio, 661–664 Substitution errors, interpolation in configparser, 977–978 subTest(), repeating unitests with different inputs, 1065–1066 suffix property, parsing paths in pathlib, 308 Switch interval, controlling/debugging threads, 1197–1198 Switching translations, gettext, 1012 Symbolic links creating in pathlib, 313 functions for, 1234 1407 removing in pathlib, 318–319 Synchronization primitives, asyncio Condition, 640–641 Event, 639–640 Lock, 637–639 overview of, 637 Queue, 642–644 Synchronizing operations, multiprocessing, 604–605 Synchronizing threads, threading, 578–581 sys module applied to build-time version information, 1178–1179 built-in modules, 1201–1202 byte ordering, 1194 changes in Python 3, 1363–1364 command-line arguments captured by interpreter, 1185–1186 CPython command-line flags, 1180–1181 custom importers, 1203–1205 custom package importing, 1211–1213 debugging threads, 1198–1200 defining maximum values in memory, 1191 determining object size, 1188–1190 exception handling, 1194–1197 exception monitoring, 1225–1227 floating-point values, 1192–1193 handling of import errors, 1214–1215 imported modules, 1200–1201 importer cache, 1217–1218 importing from shelf, 1205–1211 input/output streams (stdin/stdout), 1186–1187 integer values in memory management, 1193 interactive prompts, 1182–1183 interpreter install location, 1184–1185 interpreter display hook, 1183–1184 memory management features, 1187 meta-path, 1218–1221 module import path, 1202–1203 module reload, 1213–1214 overview of, 1169 recursion, 1190–1191 reference counts and garbage collection, 1187–1188 retrieval of import package data, 1215–1217 returning exit code (exit()), 1187 switch interval for threads, 1197–1198 for system-specific configuration, 1178 thread support, 1197 tracing function calls, 1221–1222 tracing inside functions, 1222–1224 Unicode defaults, 1181–1182 watching the stack, 1224–1225 sys.argv, commands from, 950–951 sysconfig module configuration variables, 1270–1272 getting version and platform information, 1276–1277 management of installation paths, 1272–1276 overview of, 1169, 1270 sys.exit(), invoking registered callbacks, 997 sys.path, compiling, 1157 sys.stderr, getpass(), 936–937 sys.stdout, getpass(), 936 sys.stdout.buffer, 464–466 System clock, 214 system(), running external commands, 1239–1240 SystemRandom class, random, 262–263 T Tab completion GNU readline library, 923 text in readline, 924–927 tabnanny module indentation validator, 1153–1154 1408 tabnanny module (continued) overview of, 1023–1024 supplements to standard library, 1371 Tabs vs windows, webbrowser, 796 Tangent hyperbolic functions, 288 trigonemetric functions, 284 tar archive access See tarfile module tarfile module appending to archives, 510 creating new archives, 508 extracting files from archive, 506–507 reading metadata from archive, 504–505 tar archive access via, 503 testing tar files, 503–504 using alternative archive member names, 508–509 working with compressed archives, 510–512 writing data from sources other than files, 509 Tasks duration modeling, 264 executing concurrently, asyncio, 628–632 managing pools of concurrent See concurrent.futures module as subclass of Future, asyncio, 618 timed event scheduler, sched, 998–1002 TCP/IP (Transmission Control Protocol/Internet Protocol) socket choosing address for listening, 708–711 as client, 705–706 client and server together, 706–707 easy client connections, 707–708 as server, 704–705 UDS vs., 714 TCP (Transmission Control Protocol), 694 TCPServer class, socketserver, 742–743 tempfile module named files, 333 predicting names, 335–336 spooled files, 333–334 Index temporary directories, 335 temporary file location, 336–337 as temporary file system objects, 330 temporary files, 331–332 Templates advanced string, 4–6 string, 1–2 using ChainMap instances as, 77 Temporary directories, tempfile, 335 Temporary files, tempfile, 331–332, 336–337 Terminal, using getpass() without, 937–938 Terminating processes, multiprocessing, 564–567 Testing access permissions, urllib.robotparser, 774–775 with doctest See doctest module exceptional values, math, 265–267 file types, pathlib, 313–314 files, os.path, 303–304 in-memory databases, sqlite3, 434–435 tar files, 503–504 with unittest See unittest module ZIP files, 512 test_patterns() function, regular expressions, 17, 33–34 Text command-line filter framework for, 987–992 comparing See difflib module completing in readline, 924–927 constants and templates See string module content of nodes, 449–450 formatting paragraphs See textwrap module getting help with plain, 1024–1025 overview of, reading efficiently See linecache module regular expressions See re module supplements to standard library, 1367 wrapping byte streams for, 392–393 TextIOWrapper reading compressed data in gzip, 489–490 reading compressed files in bz2, 495 reading/writing Unicode data in bz2, 498–499 wrapping byte streams for text data, 392–393 textwrap module combining dedent and fill(), 9–10 defined, example data, filling paragraphs, as formatting text paragraphs, hanging indents, 12 indenting blocks, 10–11 removing existing indentation, 8–9 truncating long text, 12–13 Thread context, decimal, 248–249 thread module, removed from Python 3, 1355 Thread objects, 560–561 Thread-safe FIFO implementation See queue module Thread-specific data, threading module, 583–585 Threading adding in http.server, 786–787 adding in socketserver, 749–751 combining coroutines with, asyncio, 670–673 and connection sharing, sqlite3, 441–442 debugging, 1198–1200 managing processes like See multiprocessing module mixing signals and, 558–560 supplements to standard library, 1369 switch interval for, 1197–1198 sys support for, 1197 threading module changes in Python 3, 1364 Index controlling access to resources, 572–576 daemon versus non-daemon threads, 564–567 determining current thread, 562–564 enumerating all threads, 567–568 isolation levels, sqlite3, 432 limiting concurrent access to resources, 581–583 managing concurrent operations within process via, 560 re-entrant locks, 576–578 signaling between threads, 571–572 subclassing Thread, 568–569 synchronizing threads, 578–581 Thread objects, 560–562 thread-specific data, 583–585 Timer threads, 570 ThreadingMixIn, 749–750, 787 ThreadPoolExecutor, 670–673, 677–678 Thresholds, garbage collection, 1261–1265 Time clock time See time module components, 217–218 datetime See datetime module event scheduler to run tasks at specific, 998–1002 execution, for small bits of Python code, 1148–1153 formatting in locale, 1022 overview, 211 scheduling callback for specific, 624–625 supplements to standard library, 1368 zones, 218–219, 231–232 time class, datetime_time, 221–222 time module changes in Python 3, 1364 as clock time, 211 comparing clocks, 211–213 defined, 211 monotonic clocks, 214 parsing and formatting times, 219–220 performance counter, 216–217 processor clock time, 214–216 1409 supplements to standard library, 1368 time components, 217–218 wall clock time, 213–214 working with time zones, 218–219 time-to-live value (TTL), sending multicast messages, 720 timedelta objects, datetime, 225–227 time()function, clock time, 211, 213–214 timeit module contents of, 1148 example, 1148–1149 overview of, 1023 values stored in dictionary, 1149–1152 Timeouts non-blocking I/O with, 734–735 socket operations, 723 using poll(), 737 Timer class, timeit, 1148–1151 TLS (transport layer security) encryption, email in smtplib, 843–846 today() class method, datetime_date, 222–223 tofile() method, arrays and files, 101 Tokens including other sources of, shlex, 955–956 splitting strings into, shlex, 954 Trace hooks overview of, 1221 tracing inside functions, 1222–1224 watching stack, 1224–1225 trace module applied to execution, 1069–1070 code coverage report, 1070–1073 example program, 1069 function relationship report, 1073–1074 options, 1077–1078 overview of, 1023 programming interface, 1074–1075 saving data, 1076–1077 Trace object, 1074–1075 traceback module applied to low-level exception APIs, 1082–1086 applied to low-level stack APIs, 1086–1089 examining stack (FrameSummary), 1080–1081 examining stack (StackSummary), 1079–1080 example, 1079 exceptions, 1081–1082 overview of, 1023, 1078 stack summary, 1079–1081 TracebackException class, 1081–1082 Tracebacks See also traceback module applying dis to, 1302–1303 detailed tracebacks, cgitb, 1090–1092 doctest for, 1032–1033 examining local variables, cgitb, 1093–1096 logging tracebacks, cgitb, 1098–1101 traceback dump, cgitb, 1089–1090 Tracing exception monitoring, 1225–1227 function calls, 1221–1222 inside functions, 1222–1224 references, 1255–1257 trackcalls, function calls report, 1073–1074 Transactions database, sqlite3, 428 discarding changes, sqlite3, 429–430 preserving changes, sqlite3, 428–429 Translations codec used for many data, 377–378 encoding, 376–377 message catalog See gettext module pattern, 325–326 Transmission Control Protocol/Internet Protocol See TCP/IP (Transmission Control Protocol/Internet Protocol) socket Transmission Control Protocol (TCP), 694 1410 Transport argument, asyncio, 662 Transport classes, asynchronous I/O, 644–645 Transport layer security (TLS) encryption, email in smtplib, 843–846 Transport protocol, retrieve port number to, 699–700 Tree structure, logging, 984–985 TreeBuilder, customizing, 453–455 Triangular distribution, random, 263 Trigonometry, math, 284–288 truediv()function, operator, 185–186 trunc() function, math, 270–271 Truncating long text, 12–13 Truth-checking, unittest for, 1054 truth() function, logical operations, 183 try:except statement, ignoring exceptions in contextlib, 199–200 try:finally block, context manager API, 192 TTL (time-to-live value), sending multicast messages, 720 tty, using getpass() without terminal, 937–938 Tuples as convenient containers for simple uses, 89 encoding/decoding simple data types in json, 804–805 Enum non-integer member values, 72–75 row objects in sqlite3, 417–419 subclass with named fields See namedtuple module Twisted library, 1369 TypeError exceptions caching in functools, 158 comparing enums, 68 dbm module, 411 reducing data set in functools, 160 Types, specific shelve, 408 TZ environment variable, time zones, 218–219 Index tzinfo abstract base class, datetime_timezone, 231–232 tzset() function, time zones, 218–219 U UDP (user datagram protocol) datagram sockets associated with, 694 sending multicast messages via, 718–720 UDPServer class, socketserver, 743 UDS (Unix Domain Sockets), 694, 714–717 Unicode data and network communication, codecs, 380–383 decoding errors, codecs, 374–376 defaults, 1181–1182 encoding errors, codecs, 373–374 primer, codecs, 365–366 reading and writing , bz2, 498–499 reading configuration files, config, 962–963 regular expression search, re, 38–39 Unique values, Enum, 69–71 unittest module addCleanup(), 1371 basic test structure, 1051 changes in Python 3, 1364 containers, 1056–1061 doctests compared with, 1026 equality-checking, 1054–1056 exceptions, 1061–1062 fixtures tests, 1062–1065 ignoring test failure, 1068–1069 integrating doctests with, 1047–1048 outcomes, 1052–1054 overview of, 1023 repeating tests with different inputs, 1065–1066 running, 1051–1052 skipping tests for unmet conditions, 1067 truth-checking, 1054 Universally Unique Identifiers See uuid module Unix key-value databases See dbm module receiving signals with asyncio, 668–670 Unix Domain Sockets (UDS), 694, 714–717 UnixDatagramServer class, socketserver, 743 UnixStreamServer class, socketserver, 743 unlink() method, removing file system object in pathlib, 318–319 unlock() method, remove messages from mbox mailbox, 854 Unpacking archive files, shutil, 349–350 data into strings, struct, 117–118 pre-allocated buffers, struct, 120–121 Unparsing URLs, urllib.parse, 756–758 Unpicklable objects, 400–402 Unregister, canceling exit callbacks in atexit, 994–995 until command, stepping through program in pdb, 1115–1116 unused_data attribute, mixed-content streams, 481, 495 Updates incremental, hashlib, 526–527 initialization, Counter, 80 query parameters, sqlite3, 421 wrapper, functools, 145–147 Upload files for encoding, urllib.request, 767–770 messages, imaplib, 881–883 urllib.parse module encoding query arguments, 759–761 joining, 758 parsing, 754–756 split URLs into components with, 753 unparsing, 756–758 urllib.request module Index adding outgoing headers, 765–766 creating custom protocol handlers, 770–773 encoding arguments, 763–764 HTTP GET, 761–763 HTTP POST, 764–765 Internet-related supplements to standard library, 1369 network resource access with, 761 posting form data from request, 766–767 uploading files, 767–770 urllib.robotparser Internet spider access control with, 773 long-lived spiders, 775–776 robots.txt, 773–774 testing access permissions, 774–775 urlopen() function, urllib.request, 761–766 urlparse() function, urllib.parse, 754–758 URLs building threaded podcast client, 114–117 finding nodes in XML document, 447–448 network resource access See urllib.request module safe variations for Base64 encoding, 778–780 splitting into components See urllib.parse module traversing parsed tree, 447 User datagram protocol See UDP (user datagram protocol) user module, removed from Python 3, 1355 use_rawinput, cmd, 949 USER_BASE directory, 1171–1172 UserDict module, changes in Python 3, 1365 UserList module, changes in Python 3, 1365 Users customizing configuration of, 1176–1177 managing directories, 1171–1172 UserString module, changes in Python 3, 1365 1411 UTC, time zones, datetime, 231–232 UTF-16 encoding byte order in codecs, 370–372 understanding encodings, 366 Western languages, 366 UTF-32 encoding, codecs, 370–372 UTF-8 encoding Base64 encoding, 777 loading standard codecs from, 387 reading Unicode configuration files, 962–963 understanding encodings, 367 Western languages, 366 uuid module changes in Python 3, 1365 implementing UUIDs, 797–798 UUID 1, IEEE 802 MAC Address, 798–800 UUID and 5, name-based values, 800–801 UUID 4, random values, 802 working with UUID objects, 802–803 uvloop library, 1369 V value property, enumerations, 66, 71 Value types, configparser, 966–969 ValueError exception Enum, 70 handling in shlex, 957–958 namedtuple invalid field names, 91–92 square root of negative value, 279 Values accessing existing, ChainMap, 75–76 approximating, fraction, 253 combining with interpolation, configparser, 975–979 duplicate, bisect, 110–111 encoding, http.cookies, 793–794 floating-point, math, 271–272 floating-point, memory management, 1192–1193 integer, memory management, 1193 manipulating date and time, datetime, 221–222 maximum, memory management, 1191 missing keys returning default, Counter, 82–84 new iterator, itertools, 169–172 retrieving, Counter, 81 returning from coroutines, asyncio, 619–620 special, decimal, 243–244 testing for exceptional, math, 265–267 unique enumeration, enum, 69–75 updating, ChainMap, 77–78 Variable argument lists, argparse, 908–910 Variables compiler optimizations, 1309–1311 configuration, 1270–1272 environment, managing process environment, 1237–1238 examining local, in tracebacks, 1093–1096 examining stack, pdb, 1107–1111 parsing paths in os.path, 296 using with queries, sqlite3, 419 variable_stack() method, contextlib, 204, 209 Variance, statistics, 292–294 variations, IMAP4 client library, 864 venv module contents of virtual environment, 1164–1165 creating virtual environments, 1163–1164 managing package versions during development, 1336–1338 new in Python 3, 1352 supplements to standard library, 1371 using virtual environments, 1165–1167 1412 Verbosity levels, logging API, 982–984 regular expression syntax, 39–43 verbose option, connecting to XML-RPC server, 818 verbose_copy(), directory trees in shutil, 342–343 Verify email address, smtplib, 846–847 version action, arguments, 891–892 Versions build-time information, 1178–1179 getting platform information and, 1276–1277 site disabled for backward compatibility, 1177–1178 Virtual environments creating, 1163–1167 venv module See venv module virtualenv module, 1336–1338, 1371 virtualenv module, 1336–1338, 1371 von Mises, or circular normal, distribution, random, 264 vonmisesvariate() function, angular distribution, 264 W wait for I/O efficiently See select module wait() method child processes, 1242–1244 multiple coroutines, 632–635 signaling between threads, 571–572 synchronizing threads, 580–581 walk(), examine file system contents, 1228–1229 Wall clock time See also time module, 213–214 warn(), generating warnings, 1280–1281 warnings module alternatives for message delivery, 1284–1285 categories and filtering, 1280 filtering with patterns, 1281–1283 formatting, 1285 generating, 1280–1281 Index logging integration with, 985–986 overview of, 1279 repeated, 1283–1284 stack level in, 1286 wc() function, statistics variance, 293 WeakKeyDictionary class, caching objects, 127 weakref module caching objects, 127–130 finalizing objects, 123–126 as impermanent references to objects, 121 memory management with, 65 proxies, 126–127 reference callbacks, 122–123 references, 122 WeakValueDictionary class, caching objects, 127–130 Web Robots Page, robots.txt, 774 webbrowser module, displaying Web pages, 796–797 weekheader() method, calendar, 234 Weibull distribution, random, 264 weibullvariate() function, Weibull distribution, 264 What’s New, Python documentation, 1351 which() function, finding files in shutil, 345–346 whichdb module, 408–411, 1365 Whitespace difference-based reporting (REPORT_NDIFF), 1036–1037 managing blank lines in doctest, 1034–1036 NORMALIZE_WHITESPACE, 1037–1039 width argument, pprint, 140–142 Wildcards, filename pattern matching in glob, 320–322 Windows OS obtaining operating information, 1249–1250 vs tabs, webbrowser, 796 with statement context manager enabled by, 191–194 converting generator function into context manager, 196–198 decimal_context_manager, 247–248 stacking context managers, 202–206 WRAPPER_ASSIGNMENTS, functools, 146 WRAPPER_UPDATES, functools, 146–147 wrapt package, 1367–1368 writable sockets, 730–732, 738–739 write() method alternative archive member names, zipfile, 516 creating new archive, zipfile, 514–516 saving configuration files in config, 972 serializing XML to stream, 464–466 Writeback, shelves, 406–408 writepy() method, Python ZIP archives, 519–521 writer() function, sqlite3, 432 Writing compressed files in bz2, 495–497 compressed files in gzip, 486–488 data from sources other than files, tarfile, 509 data from sources other than files, zipfile, 517–518 files in pathlib, 312 GNU zip files, gzip, 486 Unicode data, 498–499 X xgettext, message catalogs, 1004–1005 XML manipulation API See xml.etree.ElementTree module XML-RPC client library for See xmlrpc.client module server See xmlrpc.server module xmlcharrefreplace, lossless error handling, 374 Index xml.dom.minidom, pretty-printing XML, 458–459 xml.etree.ElementTree module building documents with element nodes, 457–458 building trees from lists of nodes, 461–464 changes in Python 3, 1365 creating custom TreeBuilder, 453–455 data supplements to standard library, 1368 ElementTree library, 445 finding nodes in document, 447–448 parsed node attributes, 449–450 parsing strings, 455–457 parsing XML document, 445–446 pretty-printing XML, 458–459 serializing XML to stream, 464–466 setting element properties, 459–461 traversing parsed tree, 446–447 watching events while parsing, 451–453 XML manipulation API via, 445 xmlrpc.client module binary data, 823–825 combining calls into one message, 826–827 connecting to server, 817–819 data types, 819–822 exception handling, 825–826 passing objects, 822–823 1413 XML-RPC client library, 816–817 xmlrpc.server module alternate API names, 829–830 arbitrary API names, 831–832 dispatching calls, 835–836 dotted API names, 830–831 exposing methods of objects, 832–834 introspection API, 837–839 simple server, 828–829 XML-RPC server, 827 Y YAML, application configuration, 1371 Z ZIP archive See also zipimport module accessing code from, 1345–1346 example, 1344–1345 finding modules in, 1345 retrieving source code from, 1346–1347 Zip files, read and write GNU See gzip module zip() function, itertools, 165, 171 zipfile module appending to files, 518–519 creating new archives, 514–516 extracting archived files from archive, 514 limitations, 521 Python ZIP archives, 519–521 reading metadata from archive, 512–514 testing ZIP files, 512 using alternative archive member names, 516 writing data from sources other than files, 517 writing with ZipInfo instance, 517–518 ZIP archive access with, 511 zipimport module accessing code from ZIP archive, 1345–1346 changes in Python 3, 1365 determining if package or regular module, 1348 example, 1344–1345 finding modules inside ZIP archive, 1345 finding non-code data, 1348–1350 overview of, 1329, 1344 retrieving source code from ZIP archive, 1346–1347 ZipInfo instance, zipfile, 517–518 zip_longest() function, itertools, 165 zlib module checksums, 481–482 compressing network data, 482–486 GNU zlib compression via, 477 GZipFile vs., 490 incremental compression/ decompression, 479–480 mixed content streams, 480–481 working with data in memory, 477–479 This page intentionally left blank REGISTER YOUR PRODUCT at informit.com/register Access Additional Benefits and SAVE 35% on Your Next Purchase • Download available product updates • Access bonus material when applicable • Receive exclusive offers on new editions and related products (Just check the box to hear from us when setting up your account.) • Get a coupon for 35% for your next purchase, valid for 30 days Your code will be available in your InformIT cart (You will also find it in the Manage Codes section of your account page.) Registration benefits vary by product Benefits will be listed on your account page under Registered Products InformIT.com–The Trusted Technology Learning Source InformIT is the online home of information technology brands at Pearson, the world’s foremost education company At InformIT.com you can • Shop our books, eBooks, software, and video training • Take advantage of our special offers and promotions (informit.com/promotions) • Sign up for special offers and content newsletters (informit.com/newsletters) • Read free articles and blogs by information technology experts • Access thousands of free chapters and video lessons Connect with InformIT–Visit informit.com/community Learn about InformIT community events and programs Addison-Wesley • Cisco Press • IBM Press • Microsoft Press • Pearson IT Certification • Prentice Hall • Que • Sams • VMware Press Developer’s Library ESSENTIAL REFERENCES FOR PROGRAMMING PROFESSIONALS Node.js, MongoDB and AngularJS Web Development Brad Dayley Python in Practice Linux for Developers Mark Summerfield William Rothwell ISBN-13: 978-0-13-429105-5 ISBN-13: 978-0-13-465728-8 ISBN-13: 978-0-321-99578-0 Other Developer’s Library Titles TITLE AUTHOR ISBN-13 PHP and MySQL Phrasebook Christian Wenz 978-0-321-83463-8 Programming in CoffeeScript Mark Bates 978-0-321-82010-5 Cloud Native Go Kevin Hoffman Dan Nemeth 978-0-672-33779-6 PHP and MySQL Web Development, Luke Welling 5th Edition Laura Thomson Developer’s Library books are available at most retail and online bookstores For more information or to order direct, visit our online bookstore at informit.com/store Online editions of all Developer’s Library titles are available by subscription from Safari Books Online at safari.informit.com 978-0-321-83389-1 Developer’s Library informit.com/devlibrary ... Wrapping Byte Streams for Text Data 32 6 32 6 32 7 32 8 32 8 32 9 33 0 33 1 33 3 33 3 33 5 33 5 33 6 33 7 33 7 34 0 34 2 34 5 34 6 35 0 35 1 35 1 35 3 35 5 35 7 36 1 36 1 36 2 36 4 36 5 36 5 36 8 37 0 37 2 37 6 37 7 37 8 38 0 38 3 39 0 39 0... Source 19 .3. 5 Packages 19 .3. 6 Data 132 9 132 9 132 9 133 0 133 1 133 2 133 4 133 4 133 6 133 8 134 0 134 1 134 4 134 4 134 5 134 5 134 6 134 8 134 8 Appendix A Porting Notes A.1 References A.2 New Modules A .3 Renamed... asynchat A.5.2 formatter 135 1 135 1 135 2 135 2 135 4 135 4 135 4 135 4 135 4 135 4 135 4 135 4 135 4 135 4 135 4 135 5 135 5 135 5 135 5 135 5 135 5 135 5 135 5 135 5 135 5 Contents A.6 A.5 .3 imp A.5.4 optparse Summary