[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

20001218: LDM: Meaning of nfree in pqmon and if values <10 indicate a need to change something

>From: Gregory Grosshans <address@hidden>
>Organization: UCAR/Unidata
>Keywords: 200012181729.eBIHTno07428

>Can you tell me what nfree and maxfree are and if the LDM configuration
>or queue size should be changed when nfree is less than 10 most of the
>time?  I've searched for pqmon on the web based email archive and didn't
>see any discussion in regards to this issue.

Gregg,

NFREE is the number of free space regions. It is good for this number to
be as small as possible since that means that the available space is in
contiguous areas. MAXFREE is the greatest value that nfree ever reaches.
When the LDM has to delete products in order to make room for a new
product, it creates regions of free space. Deletion occurs until
there is a region that is large enough to hold the incoming data.
Fewer NFREE regions means that searching for available regions is
faster. This searching is order LOG(NFREE), so MAXFREE is something
to watch, but can be pretty large before slow down would occur.
I have heavilly loaded systems with MAXFREE ~5000. Our fast systems
usually only show MAXFREE ~100. If you had a heavily IO loaded system,
then you might see this number get big.


>
>In particular, a system, ldmalice,  is ingesting NOAAPORT NWSTG and
>GOES-EAST channels, as well as a UNISYS radar feed containing single
>site radar data and national mosaics.  The queue size is 750 MB and the
>version of LDM is 5.1.2.  The output of pqmon over the past 12 hours
>results in nfree usually hovering around 1-3 and one time it was as high
>as 21.  Maxfree was 26 and increased to 28 during the last few hours.

These numbers are very good.


>
>On another system, ldmpooh, LDM 5.1.2 system with a 750 MB queue,
>ingesting NOAAPORT NWSTG and the UNISYS radar data the pqmon output over
>the past 12 hours results in nfree in the 10-30s, with a high of 130 and
>a value of 1 being logged less than 10 times.  Maxfree started out at 91
>and increased to 135.

Probably not much to wory about. NFREE generally will grow if a lot
of data comes in very fast and a lot of deletion must occur. The algorithm
used allows the data to decrement the free regions as it will fit. When
given the debug flag, pqmon will show the size of each of the free regions.
If you are trying to insert a 26MB GOESE 1km visible, and deletion is removing
lots of 1K nwstg products, the number of regions will grow some, but Russ
Rew has spent considerable time ensuring that this is much faster
than the previous pqexpire which ran every 5 minutes to try to
keep up with the amount of free space needed.





>
>Below I've included several lines of output from pqmon from both
>systems.
>
>Do the low values of nfree on ldmalice indicate something should be
>changed in the LDM configuration?
>
>Thanks,
>Gregg
>
>output from pqmon on ldmalice:  Time is UTC.
>Dec 18 16:00:17 pqmon:  41261     2  141842   715164048     71473
>28    111631  25365360 6534
>Dec 18 16:01:17 pqmon:  41611     2  141492   721272800     71473
>28    111631  25365360 6594
>Dec 18 16:02:17 pqmon:  41911     1  141193   726136640     71473
>28    111631  23865536 6654
>Dec 18 16:03:17 pqmon:  42155     1  140949   728213880     71473
>28    111631  21788296 6714
>Dec 18 16:04:17 pqmon:  42404     1  140700   729933704     71473
>28    111631  20068472 6774
>Dec 18 16:05:17 pqmon:  42628     1  140476   732010896     71473
>28    111631  17991280 6834
>Dec 18 16:06:17 pqmon:  42986     1  140118   734549720     71473
>28    111631  15452456 6894
>Dec 18 16:07:17 pqmon:  43378     1  139726   737208824     71473
>28    111631  12793352 6954
>Dec 18 16:08:17 pqmon:  43688     1  139416   739417344     71473
>28    111631  10584832 7014
>Dec 18 16:09:17 pqmon:  44127     1  138977   742004192     71473
>28    111631   7997984 7074
>Dec 18 16:10:17 pqmon:  44581     1  138523   748933552     71473
>28    111631   1068624 7135
>Dec 18 16:11:17 pqmon:  43252     2  139851   749978064     71473
>28    111631     23888 7064
>Dec 18 16:12:17 pqmon:  43429     1  139675   749966144     71473
>28    111631     36032 7101
>Dec 18 16:13:17 pqmon:  43643     1  139461   749995512     71473
>28    111631      6664 7135
>Dec 18 16:14:17 pqmon:  43581     1  139523   749995768     71473
>28    111631      6408 7165
>Dec 18 16:15:17 pqmon:  43593     1  139511   749967592     71473
>28    111631     34584 7183
>Dec 18 16:16:17 pqmon:  43744     1  139360   749996048     71473
>28    111631      6128 7185
>Dec 18 16:17:17 pqmon:  43660     2  139443   749998024     71473
>28    111631      3920 7182
>Dec 18 16:18:18 pqmon:  43918     1  139186   749988240     71473
>28    111631     13936 7219
>Dec 18 16:19:18 pqmon:  44087     1  139017   749999720     71473
>28    111631      2456 7255
>Dec 18 16:20:18 pqmon:  44376     2  138727   749989048     71473
>28    111631     12832 7287
>Dec 18 16:21:18 pqmon:  44558     1  138546   749998672     71473
>28    111631      3504 7322
>Dec 18 16:22:18 pqmon:  44765     1  138339   749974176     71473
>28    111631     28000 7373
>Dec 18 16:23:18 pqmon:  44952     2  138151   749979328     71473
>28    111631     21976 7423
>Dec 18 16:24:18 pqmon:  43581    11  139513   724076144     71473
>28    111631   5612992 7242
>Dec 18 16:25:18 pqmon:  43979     8  139118   731879848     71473
>28    111631   5612992 7302
>Dec 18 16:26:18 pqmon:  44301     8  138796   735153616     71473
>28    111631   5612992 7362
>Dec 18 16:27:18 pqmon:  44714     6  138385   738839000     71473
>28    111631   5612992 7422
>Dec 18 16:28:18 pqmon:  45244     2  137859   741934136     71473
>28    111631   5612992 7482
>Dec 18 16:29:18 pqmon:  45728     1  137376   745185200     71473
>28    111631   4816976 7542
>Dec 18 16:30:18 pqmon:  46171     1  136933   747740328     71473
>28    111631   2261848 7602
>Dec 18 16:31:18 pqmon:  46307     7  136791   748690120     71473
>28    111631    496304 7626
>Dec 18 16:32:18 pqmon:  46364     2  136739   749794232     71473
>28    111631    207440 7648
>Dec 18 16:33:18 pqmon:  46248     1  136856   749995440     71473
>28    111631      6736 7650
>Dec 18 16:34:18 pqmon:  46184     1  136920   749986936     71473
>28    111631     15240 7677
>Dec 18 16:35:18 pqmon:  46063     1  137041   749999400     71473
>28    111631      2776 7698
>Dec 18 16:36:18 pqmon:  45760     2  137343   749999328     71473
>28    111631      1976 7673
>Dec 18 16:37:18 pqmon:  45409     1  137695   750001904     71473
>28    111631       272 7652
>Dec 18 16:38:18 pqmon:  45491     2  137612   749665640     71473
>28    111631    336352 7656
>Dec 18 16:39:18 pqmon:  45822     1  137282   749615240     71473
>28    111631    386936 7689
>Dec 18 16:40:18 pqmon:  43478    16  139611   702894136     71473
>28    111631  11237896 7223
>Dec 18 16:41:18 pqmon:  43831     7  139267   714306664     71473
>28    111631  11237896 7283
>Dec 18 16:42:18 pqmon:  44270     4  138831   719661056     71473
>28    111631  11237896 7343
>Dec 18 16:43:19 pqmon:  44704     3  138398   723602152     71473
>28    111631  11237896 7403
>Dec 18 16:44:19 pqmon:  45159     3  137943   727980744     71473
>28    111631  11237896 7463
>Dec 18 16:45:19 pqmon:  45573     2  137530   731232536     71473
>28    111631  11237896 7523
>Dec 18 16:46:19 pqmon:  46100     2  137003   734536184     71473
>28    111631  11237896 7583
>Dec 18 16:47:19 pqmon:  46734     1  136370   739679328     71473
>28    111631  10322848 7643
>Dec 18 16:48:19 pqmon:  47296     1  135808   744846840     71473
>28    111631   5155336 7703
>Dec 18 16:49:19 pqmon:  47736     3  135366   749072368     71473
>28    111631    683632 7756
>Dec 18 16:50:19 pqmon:  47864     2  135239   749872784     71473
>28    111631    128792 7789
>Dec 18 16:51:19 pqmon:  47930     1  135174   746948680     71473
>28    111631   3053496 7826
>Dec 18 16:52:19 pqmon:  48124     1  134980   748769176     71473
>28    111631   1233000 7886
>Dec 18 16:53:19 pqmon:  48365     1  134739   749994984     71473
>28    111631      7192 7940
>Dec 18 16:54:19 pqmon:  44524     2  138579   740785528     71473
>28    111631   8453360 7547
>Dec 18 16:55:19 pqmon:  44812     1  138292   747609672     71473
>28    111631   2392504 7607
>Dec 18 16:56:19 pqmon:  45234     1  137870   749986808     71473
>28    111631     15368 7665
>Dec 18 16:57:19 pqmon:  45504     1  137600   749991440     71473
>28    111631     10736 7699
>Dec 18 16:58:19 pqmon:  45618     1  137486   749997992     71473
>28    111631      4184 7733
>Dec 18 16:59:19 pqmon:  45787     2  137316   750000048     71473
>28    111631      1232 7768
>Dec 18 17:00:19 pqmon:  45918     1  137186   749954936     71473
>28    111631     47240 7809
>Dec 18 17:01:19 pqmon:  46036     1  137068   749944928     71473
>28    111631     57248 7809
>
>
>output from pqmon on ldmpooh.  Time is in UTC.
>Dec 18 16:00:52 pqmon:  73157    33  109915   748922760     89724
>135     93374    361944 12019
>Dec 18 16:01:52 pqmon:  73323    14  109768   749979104     89724
>135     93374     18928 12040
>Dec 18 16:02:52 pqmon:  73460    13  109632   749952944     89724
>135     93374     30144 12083
>Dec 18 16:03:52 pqmon:  73457    29  109619   749979008     89724
>135     93374     12544 12102
>Dec 18 16:04:52 pqmon:  73285    39  109781   749978000     89724
>135     93374      2984 12115
>Dec 18 16:05:52 pqmon:  73104    37  109964   749983816     89724
>135     93374      6008 12096
>Dec 18 16:06:52 pqmon:  73095    35  109975   749995936     89724
>135     93374       568 12066
>Dec 18 16:07:52 pqmon:  73136    35  109934   749994368     89724
>135     93374      1312 12064
>Dec 18 16:08:52 pqmon:  73240    42  109823   749969832     89724
>135     93374      5232 12043
>Dec 18 16:09:52 pqmon:  73224    62  109819   749959032     89724
>135     93374     11520 12019
>Dec 18 16:10:52 pqmon:  73202    63  109840   749766608     89724
>135     93374    209224 12004
>Dec 18 16:11:52 pqmon:  73312    67  109726   749969464     89724
>135     93374      7824 11988
>Dec 18 16:12:52 pqmon:  73345    74  109686   749956808     89724
>135     93374     16920 11963
>Dec 18 16:13:52 pqmon:  73404    69  109632   749972016     89724
>135     93374      1768 11958
>Dec 18 16:14:52 pqmon:  73182    73  109850   749960600     89724
>135     93374      5840 11914
>Dec 18 16:15:52 pqmon:  73029    80  109996   749952632     89724
>135     93374      5040 11868
>Dec 18 16:16:52 pqmon:  73013    74  110018   749969040     89724
>135     93374      2808 11848
>Dec 18 16:17:52 pqmon:  73040    43  110022   749975776     89724
>135     93374     15992 11850
>Dec 18 16:18:52 pqmon:  73092    12  110001   749979936     89724
>135     93374     15056 11831
>Dec 18 16:19:52 pqmon:  72944    12  110149   749585776     89724
>135     93374    338416 11818
>Dec 18 16:20:52 pqmon:  72989    20  110096   749963376     89724
>135     93374     12984 11826
>Dec 18 16:21:52 pqmon:  73022    22  110061   749987824     89724
>135     93374      2896 11826
>Dec 18 16:22:52 pqmon:  73111    22  109972   749963192     89724
>135     93374     27400 11846
>Dec 18 16:23:52 pqmon:  73011    23  110071   749974632     89724
>135     93374      8152 11865
>Dec 18 16:24:52 pqmon:  72732    25  110348   749966384     89724
>135     93374     20872 11849
>Dec 18 16:25:52 pqmon:  72686    29  110390   749941216     89724
>135     93374     13776 11815
>Dec 18 16:26:52 pqmon:  72675    16  110414   749997512     89724
>135     93374      2360 11798
>Dec 18 16:27:52 pqmon:  72682    15  110408   749983296     89724
>135     93374      7504 11796
>Dec 18 16:28:52 pqmon:  72493    17  110595   749974512     89724
>135     93374     14528 11788
>Dec 18 16:29:52 pqmon:  72239    42  110824   749945920     89724
>135     93374      6504 11748
>Dec 18 16:30:52 pqmon:  72145    31  110929   749806232     89724
>135     93374     99944 11762
>Dec 18 16:31:52 pqmon:  72055    14  111036   749989688     89724
>135     93374      9272 11751
>Dec 18 16:32:52 pqmon:  71929    16  111160   749989216     89724
>135     93374      8816 11711
>Dec 18 16:33:52 pqmon:  71862    26  111217   749918408     89724
>135     93374     53832 11681
>Dec 18 16:34:52 pqmon:  71742    22  111341   749978728     89724
>135     93374     16488 11675
>Dec 18 16:35:52 pqmon:  71653    30  111422   749983712     89724
>135     93374      6200 11650
>Dec 18 16:36:52 pqmon:  71652    45  111408   749957928     89724
>135     93374      7968 11616
>Dec 18 16:37:52 pqmon:  71803    21  111281   749877600     89724
>135     93374    121352 11626
>Dec 18 16:38:52 pqmon:  71496    25  111584   749135720     89724
>135     93374    363560 11563
>Dec 18 16:39:52 pqmon:  71494    32  111579   749974168     89724
>135     93374     11552 11576
>Dec 18 16:40:52 pqmon:  71522    26  111557   749988464     89724
>135     93374      5744 11558
>Dec 18 16:41:52 pqmon:  71673    10  111422   749934912     89724
>135     93374     55792 11537
>Dec 18 16:42:52 pqmon:  71720    28  111357   749917136     89724
>135     93374     43328 11528
>Dec 18 16:43:52 pqmon:  71690    18  111397   749968048     89724
>135     93374     13368 11521
>Dec 18 16:44:52 pqmon:  71578    15  111512   749965464     89724
>135     93374     16360 11504
>Dec 18 16:45:52 pqmon:  71463    15  111627   749983112     89724
>135     93374      6952 11505
>Dec 18 16:46:52 pqmon:  71374    15  111716   749990552     89724
>135     93374      6792 11487
>Dec 18 16:47:52 pqmon:  71352    28  111725   749977256     89724
>135     93374      5808 11496
>Dec 18 16:48:52 pqmon:  71422    20  111663   749981448     89724
>135     93374      9048 11478
>Dec 18 16:49:52 pqmon:  71244    30  111831   749925456     89724
>135     93374     12472 11462
>Dec 18 16:50:52 pqmon:  71185    31  111889   749948744     89724
>135     93374     17920 11489
>Dec 18 16:51:52 pqmon:  71205    29  111871   749961536     89724
>135     93374     27320 11494
>Dec 18 16:52:52 pqmon:  71303    15  111787   749980696     89724
>135     93374      9848 11502
>Dec 18 16:53:52 pqmon:  71069    18  112018   749988240     89724
>135     93374      2144 11484
>Dec 18 16:54:52 pqmon:  70970    27  112108   749952664     89724
>135     93374      8792 11495
>Dec 18 16:55:52 pqmon:  71148    16  111941   749957600     89724
>135     93374     15584 11478
>Dec 18 16:56:52 pqmon:  71166    14  111925   749991120     89724
>135     93374      2424 11462
>Dec 18 16:57:52 pqmon:  71212    15  111878   749979016     89724
>135     93374     12256 11447
>Dec 18 16:58:52 pqmon:  71349     2  111754   749989744     89724
>135     93374     12240 11452
>Dec 18 16:59:52 pqmon:  71278    31  111796   749937672     89724
>135     93374     19576 11441
>Dec 18 17:00:52 pqmon:  71254    26  111825   749969384     89724
>135     93374      7688 11439
>Dec 18 17:01:52 pqmon:  71346    17  111742   749975720     89724
>135     93374     13976 11432
>
>


I have several queues running at 2GB (and we have tested 64 bit versions 
with extended queues). The new LDM design ensures that the searching for
space does not rapidly become too exhaustive with large queues. Your 750MB
queue seems to be functioning very well. You can use the AGE field to
tune the size of your queue. If you only want 1-2 hours worth of data in the 
queue, then you can increase or decrease the queue size until AGE shows that
the oldest product is staying within your expected range.

Steve Chiswell