1. Trang chủ
  2. » Công Nghệ Thông Tin

Tài liệu ORACLE8i- P21 pdf

40 366 0

Đang tải... (xem toàn văn)

Tài liệu hạn chế xem trước, để xem đầy đủ mời bạn chọn Tải xuống

THÔNG TIN TÀI LIỆU

Thông tin cơ bản

Định dạng
Số trang 40
Dung lượng 361,08 KB

Nội dung

T his is the last piece of the tuning trilogy that began with Chapter 15. In this chapter we will look at latches, locks, and waits—mechanisms in Oracle database processing that can all cause significant execution delays. Users’ sessions can start to “pile up,” waiting for an available resource. If you can proactively monitor the database and know when and where a problem is occurring, you’ll have an opportunity to tune these mechanisms before users even know any- thing is amiss. We’ll start with a discussion of locking, latching, and waiting and how they affect the database. When you understand the purpose and effects of these events, you can more easily identify the problems that are most closely associated with them. From there, we’ll cover monitoring these mechanisms and tuning them based on the results. Latches and Locks Much of the memory that Oracle allocates to the database process is shared mem- ory—shared among all of Oracle’s processes for both reading and writing. In many cases, Oracle needs to limit access to a particular area of memory to just one process. Limiting access to blocks and other memory structures helps ensure that other processes do not interfere in the ongoing operations of other sessions. Also, because of the concurrency and consistency requirements of the database, Oracle must be able to restrict access to user objects such as tables, clusters, partitions, and the data in the rows in those objects. Two Oracle mechanisms facilitate some form of restricted access to memory blocks: the latch and the lock (also known as an enqueue). Although very different in their implementation of access control, latches and locks have something else in common: They both can be symptoms of database performance problems. Much as fever and coughing are symptoms of the flu, the behavior of latches and locks can help you diagnose underlying problems in database operations. By monitoring various statis- tics related to locks and latches, you can improve the effectiveness of your tuning efforts on the database and on SQL statements. What’s a Latch? A latch is an elemental locking mechanism used by Oracle to serialize access to vari- ous memory structures. Oracle uses latches on structures that are expected to remain unlocked for a long period of time. Latches protect memory structures and processes such as redo-log buffer copy operations, the structures that manage the buffers in the database buffer cache, and other operations that interact with the SGA. C opyright ©2002 SYBEX, Inc., Alameda, CA www.sybex.com Please purchase PDF Split-Merge on www.verypdf.com to remove this watermark. Oracle uses many different kinds of latches, and a given process may use more than one latch. In fact, more than one latch may be used for a single operation of a process. For example, when an Oracle process needs to write to the redo log buffer, one or two latches are used to control that access, depending on the size of the redo to be written. Latches are low-level locking mechanisms and are more restrictive than most of the locks used by Oracle. Latching is an atomic operation. In fact, locks use latches as part of the overall locking mechanism. With a few exceptions, latches restrict any kind of access (read or write) to the memory area to the process that is holding the latch. Latch Contention So, what happens during the life of a process waiting to acquire a latch? Getting a latch is like winning the Cannonball Run—the first one to the latch wins. More specifically, the first process to request the latch acquires the latch. Once a latch is acquired by a process, other processes wait based on the mode in which the latch was requested. These modes are willing-to-wait mode and no-wait mode. Willing-to-Wait Latches If a process acquires a latch in willing-to-wait mode, other processes requesting that latch with a willing-to-wait request will stack up waiting for the latch to become available to them. These waiting processes go through two distinct operations while waiting: a spin operation, which involves continual attempts to reacquire the latch over and over again. If the subsequent latch get attempts are not successful after a fixed period of spinning, the process enters a sleep operation. The waiting process’s spin operation is essentially a timed loop. The length of spin time is controlled by the hidden parameter _SPIN_COUNT. Through each iteration of the spin process Oracle checks again to see if the latch is available. Spin operations are CPU intensive because the process is active, looping and doing very little of worth. Still, spinning is much preferred to sleeping in most cases. Spinning represents intense use of CPU resources, but sleeping generally has more negative ramifications on over- all process performance. The reasons for this are discussed throughout this chapter. Sleeping occurs when the requesting process fails to get the latch after coming out of the spin operation (or, in some cases, after awakening from another sleep opera- tion). The sleep operation is a suspension of the thread or process for a given amount of time or until the sleeping process is awakened. Putting a process in sleep mode works the CPU fairly hard because of context switching (or the cost of moving the process and its related structures in and out of memory). When Oracle puts the process to sleep, it also indicates how long the process should sleep. After this sleep operation expires, the process awakens and attempts to acquire the latch. If that fails, the process enters the sleep cycle again, and so on until the latch get is successful—or the power 797 LATCHES AND LOCKS Beyond Simple Database Management PART III C opyright ©2002 SYBEX, Inc., Alameda, CA www.sybex.com Please purchase PDF Split-Merge on www.verypdf.com to remove this watermark. CHAPTER 17 • MONITORING AND TUNING LATCHES, LOCKS, AND WAITS 798 company runs out of coal! As a waiting process moves from one sleep operation to the next, the overall sleep time grows exponentially as Oracle increases the sleep cycle duration. All this can have an adverse effect on the performance of the process because latches are not taken in any designated order. So when a process wakes, the latch it wants may have already been acquired by another process. The process sleeps yet again, unable to acquire the latch it wants. And with each sleep operation, the sleep duration increases, sometimes taking far longer than it should have to get the latch. One way to reduce the delay to a minimum is to raise the system’s _SPIN_COUNT parameter so that processes spin for a longer time before sleeping. We’ll get into this later in the sections on monitoring and tuning. Latch Wait Posting Another measure for controlling sleep operations is to use latch wait posting. When latch wait posting is used, the process holding the latch will wake the sleeping process that is waiting for the latch. This is a more expensive arrangement, but it can decrease the overall time waiting to acquire a latch. By default, only requests for the library cache and the shared pool latches use latch wait posting. If you want to take advantage of latch wait posting for all latches, set the hidden parameter _LATCH_WAIT_POSTING to 2. You can also disable this feature completely by setting the parameter to 0. TIP You can tell how many times a process has slept by looking for the Latch Free Wait statistics in the V$SESSION_EVENT view, explained in the later section about Oracle waits. Monitor latching very carefully if you choose to change the _LATCH_WAIT_ POSTING or the _SPIN_COUNT parameters, to make sure that the changes have a positive impact. In particular, watch the V$LATCH and the V$SYSTEM_EVENT data dictionary views for excessive latch contention. Look for significant wait times in the latch free waits event in V$SYSTEM_EVENT. In both views, look for increased wait times for latches in general. (You can see why having TIMED_STATISTICS turned on is so important!) When a willing-to-wait process uses latch wait posting, the process is still put to sleep because it cannot acquire the required latch. Beforehand, however, the process posts a request for the latch in the latch wait list, which notifies processes that cur- rently own the latch that other processes are waiting. Based on the latch wait list, the process acquiring the latch wakes up the sleeping process when the latch is freed. C opyright ©2002 SYBEX, Inc., Alameda, CA www.sybex.com Please purchase PDF Split-Merge on www.verypdf.com to remove this watermark. 799 No-Wait Mode Latches When a process takes a latch in no-wait mode, the process will not sleep or spin in order to wait for a given latch. The process immediately tries to obtain the latch again, without waiting, until it succeeds. Generally, no-wait latches are used in cases where latch acquisition deadlocks between two or more processes might occur, and other latches have already been acquired at the same or lower level. In this case, if Oracle cannot service the process’s request, the latches already allocated are released and the entire operation must be repeated. Very few latches use no-wait mode. Parent/Child Latches Oracle allows parent latches and child latches. Structures that require multiple latches, such as the shared pool, will have a parent latch (shared pool latch) and one or more child latches. Some parent latches have only one child (for example, the cache buffers LRU chain, which has one child latch in the database we use) and some have many children (for example, cache buffer chains, which has 512 in the database we use). The use and number of parent/child latches vary among releases of Oracle. Latch Levels The V$LATCH view (discussed shortly) includes a LEVEL# column. Oracle assigns level numbers to latches in order to avoid deadlocks. Frequently, an operation requires acquisition of multiple latches, and multiple processes may attempt to acquire the same latch. Deadlock problems are prevented by Oracle’s assignment of an order for the acquisition of latches. The level numbers range from 0 to 15, and Oracle acquires latches beginning at a lower level and proceeding to higher levels. When a process attempts to get a latch, Oracle checks to make sure that a latch at the same or higher level is not already being held. Say that Processes 1 and 2 need Latches A and B. If Process 1 acquires Latch B first and then tries to allocate Latch A, and Process 2 grabs A first and then tries to acquire B, you’d have a deadlock condition. The two processes are each trying to acquire a latch already acquired by the other process. Oracle prevents this by means of its latch- level architecture. Processes 1 and 2 will always try to grab a latch first, because it’s a lower-level latch. If Process 1 grabs Latch A, Process 2 will spin or sleep until Process 1 releases latch A. Process 2 never tries to grab Latch B until it has acquired Latch A, thus preventing a potential deadlock situation. Important Latches for Monitoring Oracle’s V$LATCH view comprises 152 different latch types (in Oracle 8.1.7). Which ones are the significant latches that you should be keeping an eye on? The following list of important latches is organized based on the parts of the SGA that are protected by a particular latch group. LATCHES AND LOCKS Beyond Simple Database Management PART III C opyright ©2002 SYBEX, Inc., Alameda, CA www.sybex.com Please purchase PDF Split-Merge on www.verypdf.com to remove this watermark. CHAPTER 17 • MONITORING AND TUNING LATCHES, LOCKS, AND WAITS 800 Database Buffer Cache Latches The cache buffers LRU chain is respon- sible for protecting the blocks of the database buffer cache. This important latch is used to allow the addition of a new block into the database buffer cache. The cache buffers chain is requested when a process needs access to a block in the SGA. Library Cache Latches The library cache latch must be obtained by a process in order to add a new statement to the library cache. The latch is not used if the SQL statement is already in the library cache. The library cache pin latch must be obtained when a statement in the library cache is to be reexecuted. The row cache objects latch must be acquired by a user process in order to access the data dictionary. Redo Buffer Latches The well-known redo allocation latch is required so that space can be allocated in the redo log buffer for redo log entries. The redo copy latch is uniquely handled by Oracle. First of all, it’s not one latch but several. The number of redo copy latches is defined by the setting of the init.ora parameter _LOG_SIMULTANEOUS_COPIES, which is calculated by Oracle automat- ically, generally based on the CPU count of the system. If the request for a latch fails, Oracle goes down the list of redo copy latches until it reaches the last one. Oracle tries to acquire this last latch in willing-to-wait mode, incurring the spin- ning and sleeping operations described earlier. Shared Pool Latch The shared pool latch is required when space in the shared pool needs to be allocated or deallocated. Where to Find Latch Information The primary views for information on latches are V$LATCH, V$LATCH_PARENT, and V$LATCH_CHILDREN. V$LATCH This view contains statistics for every type of latch used in Oracle and is critical to latch monitoring. We will use the V$LATCH view later in this chapter as part of our proactive database monitoring and tuning strategy. Following are the important columns in this view: • NAME and LEVEL#: These two columns identify the latch being monitored and its level. • GETS: The number of times a willing-to-wait request for the latch was suc- cessful. This is not the number of requests, but the number of successes; the count in this column does not increase until the latch get is successful. C opyright ©2002 SYBEX, Inc., Alameda, CA www.sybex.com Please purchase PDF Split-Merge on www.verypdf.com to remove this watermark. 801 • MISSES: The number of times a willing-to-wait request for a latch failed. Each miss results in a spin operation. This is a cumulative total of all failed attempts to get a latch. So if a process fails four times before a successful latch get, this column will show a 4 for those misses. • SLEEPS: The number of times a willing-to-wait request for this latch missed and the process ended up sleeping. • SPIN_GETS: The number of times a latch was acquired after the requesting process went into spin mode. • IMMEDIATE_GETS: The number of successful no-wait requests for the latch. • IMMEDIATE_MISSES: This column displays the number of failed no-wait requests for the latch. • SLEEP1 through SLEEP4: The number of times the latch went through sleep operations. The SLEEP1 column lists the number of times a latch slept once, the SLEEP2 column lists the number of times that the latch swept twice, and so on. Columns SLEEP5 through SLEEP11 in V$LATCH are not used in Oracle8i. V$LATCH_PARENT and V$LATCH_CHILDREN These views provide a more detailed look at the statistics for parent and child latches. V$LATCH_ PARENT includes parent latches as well as latches with no associated child latches. Its columns are the same as for V$LATCH. In V$LATCH_CHILDREN you get an additional column, CHILD#. You can get a feel for the relationship of parent and child latches by using the fol- lowing query (which only works in Oracle8i): SELECT b.name, b.num_parent, c.num_child FROM (select name, count(*) num_parent from V$LATCH_parent group by name) b, (select name, count(*) num_child from V$LATCH_CHILDREN group by name) c WHERE b.name=c.name (+); NOTE If you need details about the columns of V$LATCH, V$LOCK, and the other Oracle views mentioned in this chapter, refer to the Oracle8i Reference Manual. LATCHES AND LOCKS Beyond Simple Database Management PART III C opyright ©2002 SYBEX, Inc., Alameda, CA www.sybex.com Please purchase PDF Split-Merge on www.verypdf.com to remove this watermark. CHAPTER 17 • MONITORING AND TUNING LATCHES, LOCKS, AND WAITS 802 What’s a Lock? Multiple statement processing means an object may be needed by multiple transactions at the same time. A lock (also known as an enqueue) is an automatic internal mechanism that protects access to Oracle data structures, similarly to the way latches protect access to memory. For example, when a process has issued a DML statement (INSERT, UPDATE, or DELETE) that changes a row in a table, Oracle locks that row in order to prevent other processes from changing it. The lock remains in place until the related process commits the transaction or issues a rollback that negates the DML action. Rollbacks can take a long time, which means the object remains locked for a long time. NOTE An Oracle oddity: If you ROLLBACK to a SAVEPOINT, the locks taken after SAVE- POINT will be released—but they are not made available to any waiting transaction. New transactions will be able to access rows affected by the locks, however. Fortunately, Oracle handles locks automatically, so application developers don’t have to worry about them. Also, Oracle always uses the lowest level of locking possible, so you don’t end up with locking that’s more restrictive than required. Waiting sessions are held in a queue until the lock is released. Unlike latches, this queue is managed on a first-in, first-out (FIFO) basis. Therefore, the first process to request and wait for a lock is the first process serviced after the lock is released. Subsequent requesting processes will be handled in the order of the requests. NOTE Though Oracle handles locking automatically, it also provides tools needed by application developers to preemptively lock objects when the application design calls for custom locking. Unless otherwise noted, in this section we address issues related to auto- matic locking. Native Oracle locks do not prevent read access to the data in a table (but you can opt to use restrictive locks that do so). Keep in mind that when you query a table, the data you see is the value of the data before application of any uncommitted transac- tions from other sessions. Thus, if you UPDATE in a SQL*Plus session but do not COMMIT, you won’t see that data in a different SQL*Plus session. Your UPDATE will be visible in your session, however, even before you COMMIT it. That’s Oracle data concurrency and consistency in action! C opyright ©2002 SYBEX, Inc., Alameda, CA www.sybex.com Please purchase PDF Split-Merge on www.verypdf.com to remove this watermark. 803 Also, Oracle’s enqueue locking system provides a queuing system for lock requests. Thus, if a process is blocked from acquiring a lock, that request for the lock is queued up and will be processed in the order received. Note that this differs from latching, where the fastest process gets the latch whether it was first or not. Locking, on the other hand, provides a first-in first-out (FIFO) queueing system, that provides access to resources to requesting processes in order. Working with the V$LOCK View At the risk of belaboring the point, locking can cause serious performance problems and is often the cause of very slow running queries. To get information about who is locking what objects in your database, a primary tool is the V$LOCK view. It tells you about locks on tables and rows, and who is taking those locks. If users performing INSERTS, UPDATES, and DELETES are complaining that their systems aren’t moving smoothly, check V$LOCK—you’ll probably find the session that is causing the problem. Lock Types and Locking Modes Locks come in different shapes and sizes. Oracle uses the following: Data or DML Locks Used to protect data, DML locks can lock an entire table or specific rows of a table. While it’s locked, your data is protected from other users’ actions; and if you are making a change, other users’ actions will not conflict with that change. Data Dictionary or DDL Locks DDL locks are used for data dictionary protection (duh!). These locks generally are used to prevent the removal of objects from the database, and to prevent changes in the structure of objects in the data- base while those objects are being used. Dictionary locks are maintained during both the parse and execute stages of SQL statement processing and are held for the lifetime of the transaction. Internal Locks These locks protect the internal structure of the database datafiles. Internal locks are used in association with latches to protect various internal database structures. For each lock listed in the V$LOCK view, you’ll encounter codes representing the types of lock and graduated methods of locking objects. Together, the codes tell you what type of locking is occurring (a DML row lock, for instance) and the extent of the locking (exclusive or shared, for instance). Lock Type Codes The lock type indicates what type of lock is being acquired. This might be a row-level lock, a table lock, or a lock created by a user. In the TYPE column you’ll see the associ- ated codes for each lock type, as defined in Tables 17.1, 17.2, and 17.3. LATCHES AND LOCKS Beyond Simple Database Management PART III C opyright ©2002 SYBEX, Inc., Alameda, CA www.sybex.com Please purchase PDF Split-Merge on www.verypdf.com to remove this watermark. CHAPTER 17 • MONITORING AND TUNING LATCHES, LOCKS, AND WAITS 804 NOTE You can find more information on DDL locks by looking at the V$LOCKED_OBJECT view. TABLE 17.1: DML LOCK TYPES DML Lock Type Code Description Row lock TX This is a row-level lock, the most granular lock in Oracle. Row-level locks are always exclusive. Table lock TM Acquired with various DML operations including INSERT, DELETE, UPDATE, SELECT FOR UPDATE, and when the LOCK TABLE clause is used. User lock UL A lock created explicitly by the user via use of the DBMS_LOCK package. TABLE 17.2: DDL LOCK TYPES DDL Lock Type Code Description Exclusive DDL lock N/A A lock on a resource being modified. Prevents simultane- ous DDL operations on the same object. For example, you won’t be able to drop a partition and move the same parti- tion at the same time. Share DDL lock N/A Prevents exclusive DDL locks from being taken against the object locked. Thus, this prevents a table from being dropped while you are updating a procedure that refer- ences the table. Table 17.3 describes the internal lock codes that appear in the TYPE column of V$LOCK. In general, you don’t need to worry about them, but you may need to handle them in your monitoring scripts. C opyright ©2002 SYBEX, Inc., Alameda, CA www.sybex.com Please purchase PDF Split-Merge on www.verypdf.com to remove this watermark. 805 TABLE 17.3: INTERNAL LOCK TYPES Lock code Description Lock Code Description BL Buffer hash table instance NA to NZ Library cache pin instance (A Z = namespace) CF Control file schema PF Password file global enqueue CI Cross-instance function PI and PS Parallel operation invocation instance CU Cursor bind PR Process startup DF Data file instance QA QZ Row cache instance (A Z = cache) DL Direct loader parallel RT Redo thread global enqueue index create DM Mount/startup db SC System COMMIT number primary/secondary instance instance DR Distributed recovery process SM SMON DX Distributed transaction entry SN Sequence number instance FS File set SQ Sequence number enqueue HW Space management SS Sort segment operations on a specific segment IN Instance number ST Space transaction enqueue IR Instance recovery SV Sequence number value serialization global enqueue IS Instance state TA Generic enqueue IV Library cache invalidation TS Temporary segment enqueue instance (ID2=0) JQ Job queue TS New block allocation enqueue (ID2=1) KK Thread kick TT Temporary table enqueue LA to LP Library cache lock instance UN User name lock (A P = namespace) MM Mount definition US Undo segment DDL global enqueue MR Media recovery WL Being-written redo log instance LATCHES AND LOCKS Beyond Simple Database Management PART III C opyright ©2002 SYBEX, Inc., Alameda, CA www.sybex.com Please purchase PDF Split-Merge on www.verypdf.com to remove this watermark. [...]... change to a table row, via INSERT, UPDATE, or DELETE operations Allows DML activity to continue on the table Exclusive table access can be enforced via the LOCK TABLE command Please purchase PDF Split-Merge on www.verypdf.com to remove this watermark www.sybex.com Copyright ©2002 SYBEX, Inc., Alameda, CA LATCHES AND LOCKS 807 TABLE 17.4: ORACLE LOCKING MODES (CONTINUED) Lock Mode Lock Mode Lock Type Number... access to a table by any other transaction Lock Conversions NOTE SELECT FOR UPDATE statements vis-à-vis locking are covered in more detail later in this chapter, so read on Please purchase PDF Split-Merge on www.verypdf.com to remove this watermark www.sybex.com Copyright ©2002 SYBEX, Inc., Alameda, CA PA R T III Beyond Simple Database Management Oracle converts locks rather than escalates them What’s... SQL*Plus session has updated a table record, but the transaction has not yet committed the record A second SQL*Plus session is trying to update the same record, without success Please purchase PDF Split-Merge on www.verypdf.com to remove this watermark www.sybex.com Copyright ©2002 SYBEX, Inc., Alameda, CA LATCHES AND LOCKS 809 Having read this far, you know the problem is that the first session is blocking... -17 131075 674 6 0 1 2 1 0 4 0 0 3 1 0 6 0 0 2 2 0 4 0 0 2 3 0 4 0 0 2 4 0 4 0 0 2 5 0 4 0 0 15 2890 0 3 0 0 17 2890 0 3 0 0 15 131075 674 0 6 0 Please purchase PDF Split-Merge on www.verypdf.com to remove this watermark www.sybex.com Copyright ©2002 SYBEX, Inc., Alameda, CA III Beyond Simple Database Management SID 810 CHAPTER 17 • MONITORING AND TUNING LATCHES, LOCKS, AND... is locked Listing 17.3: Finding the Blocked Table/Row SELECT a.sid, a.serial#, b.object_name, b.object_type, c.file_name FROM v$session a, dba_objects b, dba_data_files c Please purchase PDF Split-Merge on www.verypdf.com to remove this watermark www.sybex.com Copyright ©2002 SYBEX, Inc., Alameda, CA LATCHES AND LOCKS 811 WHERE a.row_wait_obj#=b.object_id and a.row_wait_file#=c.file_id and a.sid=15;... the foreign key columns of the child table If you don’t, every time you delete a record in the parent table you’ll get a full table scan of the child table to ensure that Please purchase PDF Split-Merge on www.verypdf.com to remove this watermark www.sybex.com Copyright ©2002 SYBEX, Inc., Alameda, CA Beyond Simple Database Management EMPNO ENAME 812 CHAPTER 17 • MONITORING AND TUNING LATCHES, LOCKS,... because the locks that are acquired might affect users come Monday morning Always make sure users are aware of the impacts that SELECT FOR UPDATE statements have on a system Please purchase PDF Split-Merge on www.verypdf.com to remove this watermark www.sybex.com Copyright ©2002 SYBEX, Inc., Alameda, CA WAIT MANAGEMENT 813 Wait Management During database operations, various events occur, such as the database... particular buffer The V$LOCK view, described in the preceding section on locks, can be a significant aide in determining database locking problems With this view you can locate the Please purchase PDF Split-Merge on www.verypdf.com to remove this watermark www.sybex.com Copyright ©2002 SYBEX, Inc., Alameda, CA PA R T III Beyond Simple Database Management NOTE 814 CHAPTER 17 • MONITORING AND TUNING LATCHES, LOCKS,... sync 657 1 0 0 db file sequential read 538 0 0 0 db file scattered read 145 0 0 0 5 0 0 0 db file parallel write 53 0 0 0 db file parallel read 2 0 0 0 db file single write Please purchase PDF Split-Merge on www.verypdf.com to remove this watermark www.sybex.com Copyright ©2002 SYBEX, Inc., Alameda, CA 815 WAIT MANAGEMENT direct path read 30 0 0 0 direct path write 20 0 0 0 instance state change 2 0... AVERAGE_WAIT MAX_WAIT - - - -1 pmon timer 768 762 0 0 0 2 latch free 4 4 0 0 0 2 rdbms ipc message 712 701 0 0 0 2 direct path read 10 0 0 0 0 Please purchase PDF Split-Merge on www.verypdf.com to remove this watermark www.sybex.com Copyright ©2002 SYBEX, Inc., Alameda, CA Beyond Simple Database Management III Here’s an example of a query against V$SESSION_EVENT: 816 CHAPTER . SGA. C opyright ©2002 SYBEX, Inc., Alameda, CA www.sybex.com Please purchase PDF Split-Merge on www.verypdf.com to remove this watermark. Oracle uses many different kinds. Management PART III C opyright ©2002 SYBEX, Inc., Alameda, CA www.sybex.com Please purchase PDF Split-Merge on www.verypdf.com to remove this watermark. CHAPTER 17 • MONITORING AND

Ngày đăng: 26/01/2014, 19:20

TỪ KHÓA LIÊN QUAN

w