PUG Challenge EMEA. Promon for Dummies & Savants. Click to edit Master title style. Presented by: Dan Foreman

Size: px

Start display at page:

Download "PUG Challenge EMEA. Promon for Dummies & Savants. Click to edit Master title style. Presented by: Dan Foreman"

Melinda Gloria Bradford
5 years ago
Views:

1 PUG Challenge EMEA Click to edit Master title style Promon for Dummies & Savants And how it may help in troubleshooting certain DB problems Presented by: Dan Foreman 1

2 Dan Foreman Progress User since 1984 Recent Employment History is erratic Polar Cryogenics BravePoint Progress 11/2014-7/2016 Strategic Information Group 7/ /2016 White Star Software 2017 till death? 2

3 Dan Foreman Author of several Progress related Publications Progress Performance Tuning Guide This year is the 25 th Anniversary of the Tuning Guide! 1992 : 2017: 140 pages in 18 point font 475 pages in 13 point font Progress Database Administration Guide Progress System Table Guide Available in book (paper) or electronic media Basketball Fanatic which sometimes leads to unexpected trips to the ER 3

4 4 Dan Foreman

5 Promon basics promon first appeared in Progress V5 Totally overhauled in V6.3 when the new Shared Memory architecture was introduced Basic concept: Take a snapshot of a particular section of DB Shared Memory Therefore there is no promon option for single user mode Limitations: Single database Fixed Character Interface No Warnings or Alerts 5

6 Promon versus Virtual System Tables Virtual System Tables were introduced in V8 Initially VSTs just replicated the promon menu structure Eventually there was functionality added that didn t exist in promon Examples: _tablestat Table Level I/O _indexstat Index Level I/O Tools that use the VSTs ProTop ProMonitor OpenEdge Management 6

7 Important Metrics Top Level 3. Block Access DB Requests the number of times a shared memory client requested a DB Block DB Reads the number of times the block requested was not in the DB Buffer Cache (-B/-B2) and had to be retrieved from storage Data displayed for the entire DB and individual Client processes (if they are connected at the time of the snapshot) Type Usr:Ten Name Domain DB Requests DB Reads BI Reads AI Reads \Writes \Writes \Writes Acc TOTAL Acc 0 Dan

8 Important Metrics Top Level 4. Lock Table I recommend using the _Lock VST for monitoring & debugging record locking issues The promon view of the Lock Table doesn t have filters so it can be difficult to locate locking conflicts The performance issues of accessing _Lock were fixed in V11.4 8

9 Important Metrics Top Level 5. Activity Activity - Sampled at 04/20/17 10:41 for 0:04:09. Event Total Per Sec Event Total Per Sec Commits Undos Record Updates Record Reads Record Creates Record Deletes DB Writes DB Reads BI Writes BI Reads AI Writes Record Locks Record Waits Checkpoints Buffs Flushed Rec Lock Waits 0 % BI Buf Waits 1 % AI Buf Waits 0 % Writes by APW 13 % Writes by BIW 26 % Writes by AIW 0 % Buffer Hits 100 % Primary Hits 100 % Alternate Hits 0 % DB Size 633 MB BI Size 2168 K AI Size 0 K FR chain 59 blocks RM chain 3 blocks Shared Memory 17669K Segments 1 0 Servers, 2 Users (2 Local, 0 Remote, 0 Batch),1 Apws 9

10 Important Metrics Top Level 5. Activity DB Uptime because many of the metrics are based upon how long the DB has been running Record Reads Record Waits Checkpoints CPs will be discussed in more detail later in this presentation Buffers Flushed Zero is a perfect score but not always possible Buffer Hits % - MISLEADING METRIC The Buffer Hit Ratio is a much better measurement of DB Cache effectiveness 10

11 Important Metrics Top Menu 6. Shared Resources Locking table entries in use Locking table high water mark 11

12 Important Settings Top Level 7. Database Status Database block size Before Image Block Size After Image Block Size Before Image Cluster Size Database Status: Database version number: 4269 Database state: Open (1) Database damaged flags: None (0) Integrity flags: None (1536) Database block size (bytes): 4096 Total number of database blocks: Database blocks high water mark: Free blocks below highwater mark: 59 Record blocks with free space: 3 Before image block size (bytes): 8192 Before image cluster size (kb): 512 After image block size (bytes):

13 R&D Section History Originally undocumented First public documentation appeared in my Progress Performance Tuning Guide but was not approved by Progress Software Eventually (sort of) documented by Progress Full of stuff that is useless to a DBA But also full of equally important stuff 13

14 R&D Section 04/20/17 OpenEdge Release 11 Monitor (R&D) 10:49:34 Main (Top) Menu 1. Status Displays Activity Displays Other Displays Administrative Functions Adjust Monitor Options 14

15 R&D Section 1. Status 04/20/17 OpenEdge Release 11 Monitor (R&D) 10:50:41 Status Displays Menu 1. Database 2. Backup 3. Servers 4. Processes/Clients Files 6. Lock Table 7. Buffer Cache 8. Logging Summary 9. BI Log 10. AI Log 11. Two-Phase Commit 12. Startup Parameters 13. Shared Resources 14. Shared Memory Segments 15. AI Extents 16. Database Service Manager 17. Servers By Broker 18. Client Database-Request Statement Cache Schema Locks & Wait Queue 20. Broker Startup Parameters 15

16 Important Metrics R&D 1. Status 4. Processes/Clients > 2. Blocked Clients Shows record lock conflicts and some (but not all!!) shared memory contention More info on this later in the session 16

17 Important Metrics R&D 1. Status 7. Buffer Cache Empty Buffers indicate potentially wasted memory Total buffers: 3002 Hash table size: 887 Used buffers: 40 Empty buffers:

18 Important Metrics R&D 1. Status 16. Database Service Manager Very important for OpenEdge Replication users Related to the pica Cache Communication Area Size : KB Total Message Entries : 595 Free Message Entries : 595 Used Message Entries : 0 Used HighWater Mark : 0 Area Filled Count : 0 Service Latch Holder : -1 Access Count : 0 Access Collisions : 0 18

19 Important Metrics R&D 1. Status 9. BI Log The only place that you can actually experience the duration of a Checkpoint; the same duration that users feel Before-image cluster age time: 0 seconds Before-image block size: 8192 bytes Before-image cluster size: 512 kb ( bytes) Number of before-image extents: 1 Before-image log size (kb): 2168 Bytes free in current cluster: (20 %) Last checkpoint was at: 04/19/17 10:58 Number of BI buffers: 20 Full buffers: 0 19

20 Important Metrics R&D 2. Activity 04/20/17 OpenEdge Release 11 Monitor (R&D) 11:16:33 Activity Displays Menu 1. Summary 2. Servers 3. Buffer Cache 4. Page Writers 5. BI Log 6. AI Log 7. Lock Table 8. I/O Operations by Type 9. I/O Operations by File 10. Space Allocation 11. Index 12. Record 13. Other 20

21 Important Metrics R&D 2. Activity 5. BI Log Total Per Min Per Sec Per Tx Total BI writes BIW BI writes Records written Bytes written Total BI Reads Records read Bytes read Clusters closed Busy buffer waits Empty buffer waits Log force waits Log force writes Partial writes Input buffer hits Output buffer hits Mod buffer hits BO buffer hits

22 Important Metrics R&D 2. Activity 13. Other See Rich Banville s session on Shared Memory Contention from 2008 Exchange Total Per Min Per Sec Per Tx Commit Undo Wait on semaphore Non-blocking waits Semaphore latch waits Flush master block

23 Brief History of the debghb option in promon Added to promon V6.3 by Gus Purpose: The shared memory architecture introduced in V6.3 was new and very different from earlier versions and a way to monitor shared memory activity at a detailed level was needed The debghb option was not a formally endorsed enhancement but written by Gus in his spare time deb = DEBug ghb = Gus s initials.the middle initial is a Top Secret fanatically guarded by the Finnish government 23

24 Warning The debghb option is not formally documented by Progress DO NOT call or Gus or PTS for help in using this option Many of the screens and/or metrics have no value to a DBA The view of the data is not transactionally consistent sometimes even on the same screen; example to follow Some of the data is not accurate (data overflow, rounding errors, etc) Some of the screens are broken (don t display any data) The debghb option can be altered, removed, or hidden by Progress any time they want to 24

25 Warning #2 DB activities in shared memory can be slowed down if certain options are enabled 05/08/13 OpenEdge Release 10 Monitor (R&D) 19:10:44 Adjust Latch Options 1. Spins before timeout: Enable latch activity data collection 3. Enable latch timing data collection 4. Initial latch sleep time: 10 milliseconds 5. Maximum latch sleep time: 5000 milliseconds 6. Record Free Chain Search Depth Factor: 5 25

26 How do I access debghb? Start promon Enter: R&D (also works in lower case starting in V10) Enter: debghb You have now entered the debghb zone Two main differences in the world of debghb Extensions to some existing R&D screens Enables access to This menu is not here Menu Enter: 6 even though there is no visible option 6 See next slide 26

27 This menu is not here Menu 06/06/13 OpenEdge Release 10 Monitor (R&D) 15:23:01 This menu is not here Menu 1. Cache Entries 2. Hash Chain 3. Page Writer Queue 4. Lru Chains 5. Locked Buffers 6. Buffer Locks 7. Buffer Use Counts 8. Resource Queues 9. TXE Lock Activity 10. Adjust TXE Options 11. Latch Counts 12. Latch Times 13. I/O Wait Time by Type 14. I/O by File 15. Buffer Lock Queue 16. Semaphores 17. Shutdown 27

28 Operating Hints Allow at least lines of screen data Allow at least columns of screen width Zero out the stats ( z ) to get a clean starting place This zeroing does not wipe out the actual shared memory counters but only affects the current promon session Update the stats periodically ( u ) to get snapshots All the above can be scripted 28

29 Operating Hints - User# -1 Usecnt = # of concurrent processes accessing the block When initially examining the BLQ there were 5 Clients accessing the same DBKEY But before all 5 could be displayed: One Client left, i.e. released the Lock, before it was displayed Another one of the 5 is partially displayed; i.e. the -1 User# That s the nice reason; the probable reason is it s a bug 02/06/13 Status: Buffer Lock Queue 00:37:21 User DBKEY Area T Status Type Usect I LOCKED SHARE I LOCKED SHARE I LOCKED SHARE I LOCKED SHARE 3 29

30 Useful Screens - Checkpoints Extensions to the normal Checkpoint screen added in V10.2B SP5 Columns of interest Duration: the amount of time required to complete a Checkpoint; the entire Database is transactionally frozen during this time _CheckPoint._CheckPoint-Duration (V10.2B SP5) Sync Time: a subset of the Duration column; the amount of time required to execute fdatasync() system call _CheckPoint._CheckPoint.Synctime (V10.2B SP5) See for an excellent description of fdatasync (don t confuse it with fsync). Sample data on the next slide 30

31 Useful Screens - Checkpoints The Duration of the Checkpoints (i.e. the total freeze time) is very high for most of the CPs displayed A Duration of less than 1 second is a good goal The 10 sec Duration is approximately 1/3 of the CP Freq. In other words, a CP is occurring approximately every seconds and for up to 10 seconds of that period, NO Client transaction activity can take place. Ckpt Database Writes ---- No. Time Len Freq Dirty CPT Q Scan APW Q Flushes Duration Sync Time :56: :52: :51: :51: :50: :49: :49:

32 Useful Screens - Checkpoints In V11.6 the Checkpoint Duration is broken down even further. Thank you Rich Banville!! (except that I have to use a tiny font to fit everything on one line) Sync Time DB Write BI Write 10/05/16 Checkpoints 12:57:08 Ckpt Database Writes Performance Timings No. Time Len Freq Dirty CPT Q Scan APW Q DB Writes Bi Writes Duration Sync Time DB Write Bi Write NumChkpt 6 12:57: :57: :56: :56: :56: :55: :54:

33 Useful Screens Resource Queues NHM (Not Here Menu) - Option #8 Do not confuse Resources with Latches; Rich Banville gave a good talk on this topic at Exchange 2008 In general the busiest locks will be: DB Buf S Lock (S = Share) DB Buf X Lock (X= Exclusive) Record Lock Waits that can be problematic: DB Buf I Lock (I = Intent but these are for Index blocks) Sample on the next slide 33

34 Useful Screens Resource Queues 01/31/13 Activity: Resource Queues 00:31:57 01/31/13 00:26 to 01/31/13 00:31 (5 min) Queue - Requests Waits Total /Sec Total /Sec Pct Record Lock Trans Commit DB Buf I Lock Record Get DB Buf Read DB Buf Write DB Buf S Lock DB Buf X Lock DB Buf S Lock LRU DB Buf X Lock LRU DB Buf Write LRU BI Buf Read BI Buf Write TXE Share Lock TXE Update Lock TXE Commit Lock

35 Useful Screens Latch Counts NHM (Not Here Menu) - Option #11 The R&D Blocked Clients screen doesn t show Latch contention so debghb is the only place in promon where detailed Latch activity is visible Definition of Naps: When spin is used up by a Progress Client, the process Naps (i.e. does no useful work) for a while and tries again General Principle: Napping is bad (unless it s my wife) Samples on the next few slides 35

36 Latch Counts OM Latch OM (Object Cache) Latch activity can be totally eliminated by setting the -omsize parameter equal to or greater than the number of _StorageObject records 04/24/13 Activity: Latch Counts 00:59:28 04/24/13 00:54 to 04/24/13 00:59 (5 min 1 sec) Locks Busy --- Naps Spins Nap Max - Owner Total /Sec /Sec Pct /Sec /Sec /Lock /Busy Total HWM MTX USR OM

37 Latch Counts USR Latch The contention on the USR (DB Connection Table) Latch is because Statement Caching is enabled 04/25/13 Activity: Latch Counts 00:33:17 04/25/13 00:28 to 04/25/13 00:33 (5 min 0 sec) Locks Busy --- Naps Spins Nap Max Owner Total /Sec /Sec Pct /Sec /Sec /Lock /Busy Total HWM MTX USR OM BIB SCH LKP GST TXT SEQ AIB TXQ EC LKF BFP BHT PWQ CPQ LRU LRU BUF

38 Latch Counts LRU Chain The total number of Locks for LRU is the second highest of all the Resources shown (BHT Buffer Hash Table is #1) The # of Naps per Second is the highest of all latches (Zero is ideal) 01/31/13 Activity: Latch Counts 00:05:38 01/31/13 00:00 to 01/31/13 00:05 (5 min 0 sec) Locks Busy Naps Owner Total /Sec /Sec Pct /Sec MTX OM BHT CPQ LRU LRU BUF BUF BUF BUF

39 Latch Counts LRU Chain The # of locks on the second LRU (Alternate Buffer Cache, ABC) is zero because all the ABC blocks completely fit in the amount of B2 memory allocated 01/31/13 Activity: Latch Counts 00:05:38 01/31/13 00:00 to 01/31/13 00:05 (5 min 0 sec) Locks Busy Naps Owner Total /Sec /Sec Pct /Sec BHT CPQ LRU LRU BUF BUF

40 Latch Counts LRU Chain Owner column: if the User# doesn t change (in value or frequency) that can be a problem indicator because Latches should be held for only a fraction of a second 01/31/13 Activity: Latch Counts 00:05:38 01/31/13 00:00 to 01/31/13 00:05 (5 min 0 sec) Locks Busy Naps Owner Total /Sec /Sec Pct /Sec BHT CPQ LRU LRU BUF BUF

41 Using Latch Counts to set -spin Short answer Forget It! If it was easy Progress would/should have done it already Past attempts have not been successful Also the optimal value of spin is not going to be the same for each individual Latch General guidelines: Greater than 1,000 Less than 50,000 Current Default: 6,000 * (# of CPU Cores) Default is not advised if you have more than 16 Cores Dan s Formula (Patent Pending): (DBA-Birthday-Year * ) Gus s formula : 5,000 41

42 Useful Screens Buffer Lock Queue NHM (Not Here Menu) - Option #15 The normal R&D Blocked Clients screen does not show the Area that the DBKEY belongs to The Buffer Lock Queue (BLQ) Screen shows the Area as well as the Block Type Examples on the next two slides 42

43 R&D Blocked Clients The R&D Blocked Clients screen doesn t show enough information to identify the Object involved in this contention storm for DBKEY There were 29 Clients all blocked on the same DBKEY 01/31/13 Status: Blocked Clients 00:26:41 Usr Name Type Wait Wait Info Trans id Login time 730 _AUTO-B SELF/ABL BKSH /30/13 23: _AUTO-B SELF/ABL BKSH /30/13 23: _AUTO-B SELF/ABL BKSH /30/13 23: _AUTO-B SELF/ABL BKSH /30/13 23: _AUTO-B SELF/ABL BKSH /30/13 23: _AUTO-B SELF/ABL BKSH /30/13 23: _AUTO-B SELF/ABL BKSH /30/13 23:22 43

44 Buffer Lock Queue IF there is a matching DBKEY on the the BLQ screen, we can get the Area# and the Block Type (I = Index) There were 29 processes on the Blocked Clients screen with this DBKEY and only 4 on the BLQ screen with the same DBKEY (remember, not transactionally consistent) 01/31/13 Status: Buffer Lock Queue 00:26:41 User DBKEY Area T Status Type Usect I LOCKED SHARE I LOCKED SHARE I LOCKED SHARE I LOCKED SHARE I LOCKED SHARE 4 <lines unrelated to DBKEY snipped> 44

Thank You! Questions? dforeman@bravepoint.

45 Thank You! Questions? (I own the domain) 45

FACTORS AFFECTING CONCURRENT TRUNCATE

T E C H N I C A L N O T E FACTORS AFFECTING CONCURRENT TRUNCATE DURING BATCH PROCESSES Prepared By David Kurtz, Go-Faster Consultancy Ltd. Technical Note Version 1.00 Thursday 2 April 2009 (E-mail: david.kurtz@go-faster.co.uk,