I've resurrected my ucnids.c program to provide short term relief for the
zlib compression. Since the compression seems to only be on the first 4000
bytes of the N?B products, it's not so straightforward to decompress. So
I'm back using my ucnids program. I tweaked it to work with the N?B
products.
Here is what Pete Pokrandt implemented in the pqact file:
NEXRAD ^SDUS5. .... (..)(....).*/p(N0B)(...) PIPE
-close /usr/local/ldm/util/ucnids -c -
/data/NIDS/\4/N0B/N0B_(\1:yy)(\1:mm)\1_\2
*NOTE: *Since this is tweaked for the N?B products, it won't work with the
N?S products. I'm working on a fix for that.
Dan.
On Mon, Mar 7, 2022 at 1:18 PM Mike Zuranski <zuranski.wx@xxxxxxxxx> wrote:
> FWIW it's not just N.B products doing this...
>
> 20220307T191633.718220Z notifyme[70117]
> notifyme.c:notifymeprog_5:212 INFO 18370 20220307191632.821513
> NEXRAD3 73348475 SDUS26 KHNX 071911 /pNBBHNX !nids/
> 20220307T191633.718326Z notifyme[70117]
> notifyme.c:notifymeprog_5:212 INFO 6152 20220307191632.867833
> NEXRAD3 73348476 SDUS58 PAFC 071914 /pN0SAIH
> 20220307T191633.766274Z notifyme[70117]
> notifyme.c:notifymeprog_5:212 INFO 19778 20220307191632.870118
> NEXRAD3 73348477 SDUS28 PAFC 071910 /pN3BAKC !nids/
> 20220307T191633.766402Z notifyme[70117]
> notifyme.c:notifymeprog_5:212 INFO 2178 20220307191632.870344
> NEXRAD3 73348478 SDUS52 KMLB 071913 /pDPAMLB
> 20220307T191633.810195Z notifyme[70117]
> notifyme.c:notifymeprog_5:212 INFO 98306 20220307191632.886132
> NEXRAD3 73348479 SDUS21 KCTP 071913 /pN2BCCX !nids/
> 20220307T191633.810287Z notifyme[70117]
> notifyme.c:notifymeprog_5:212 INFO 1552 20220307191632.888075
> NEXRAD3 73348480 SDUS55 KSLC 071910 /pNCRSLC
> 20220307T191633.854166Z notifyme[70117]
> notifyme.c:notifymeprog_5:212 INFO 67908 20220307191633.029729
> NEXRAD3 73348481 SDUS52 KFFC 071915 /pTV0ATL !nids/
> 20220307T191633.854309Z notifyme[70117]
> notifyme.c:notifymeprog_5:212 INFO 4380 20220307191633.030216
> NEXRAD3 73348482 SDUS52 KMLB 071913 /pNTPMLB
> 20220307T191633.898156Z notifyme[70117]
> notifyme.c:notifymeprog_5:212 INFO 233 20220307191633.030298
> NEXRAD3 73348483 SDUS55 KSLC 071910 /pNVLSLC
> 20220307T191633.898265Z notifyme[70117]
> notifyme.c:notifymeprog_5:212 INFO 4050 20220307191633.032406
> NEXRAD3 73348484 SDUS52 KMLB 071913 /pDSPMLB !nids/
> 20220307T191633.946219Z notifyme[70117]
> notifyme.c:notifymeprog_5:212 INFO 4484 20220307191633.035764
> NEXRAD3 73348485 SDUS32 KMLB 071913 /pN1PMLB
> 20220307T191633.946316Z notifyme[70117]
> notifyme.c:notifymeprog_5:212 INFO 73239 20220307191633.047202
> NEXRAD3 73348486 SDUS55 KSLC 071915 /pNYGICX !nids/
> 20220307T191633.990205Z notifyme[70117]
> notifyme.c:notifymeprog_5:212 INFO 15915 20220307191633.049020
> NEXRAD3 73348487 SDUS26 KHNX 071911 /pNBUHNX !nids/
>
> I have not dug any deeper on those files however.
>
> -Mike
>
> ======================
> Mike Zuranski
> Meteorology Support Analyst
> College of DuPage - Nexlab
> Weather.cod.edu <http://weather.cod.edu/>
> ======================
>
>
> On Mon, Mar 7, 2022 at 1:11 PM Mike Zuranski <zuranski.wx@xxxxxxxxx>
> wrote:
>
>> I'm gonna go out on a limb and say it's a bug...
>>
>> An observation I made that correlates with yours: Shortly after
>> RAX began issuing N.B products there was a period where they were coming
>> in... badly somehow. The detail I noticed was the missing "!nids/" at the
>> end of nexrad products while watching the nexrad3 feed. And like you
>> noticed that was temporary.
>>
>> I see the same thing this morning from a number of sites (LOT grabbed my
>> attention first). I also see the number of sites doing this are
>> decreasing, so I'm wondering if this is being worked on somewhere. But any
>> nexrad products that I see in LDM that do not contain "!nids/" at the end
>> do not jive with McIDAS nor Gempak while all others do.
>>
>> Brief example:
>> 20220307T185809.320700Z notifyme[10618]
>> notifyme.c:notifymeprog_5:212 INFO 123297 20220307185809.170361
>> NEXRAD3 73287741 SDUS58 PAFC 071856 /pN0BAKC !nids/
>> 20220307T185811.881051Z notifyme[10618]
>> notifyme.c:notifymeprog_5:212 INFO 314053 20220307185811.690988
>> NEXRAD3 73287963 SDUS51 KRLX 071857 /pN0BRLX
>> 20220307T185813.488004Z notifyme[10618]
>> notifyme.c:notifymeprog_5:212 INFO 191116 20220307185813.222289
>> NEXRAD3 73287985 SDUS53 KGRR 071856 /pN0BGRR !nids/
>>
>> Daryl: I'm not sure there's a correlation to exengine4.fox.com, last
>> time it was data straight out of SBN doing this. My Noaaport's been down a
>> while so I can't confirm that right now though.
>>
>> -Mike
>>
>> ======================
>> Mike Zuranski
>> Meteorology Support Analyst
>> College of DuPage - Nexlab
>> Weather.cod.edu <http://weather.cod.edu/>
>> ======================
>>
>>
>> On Mon, Mar 7, 2022 at 1:04 PM Daniel Vietor - NOAA Affiliate via
>> ldm-users <ldm-users@xxxxxxxxxxxxxxxx> wrote:
>>
>>> Is it just me or are people seeing a strange zlib compression of the new
>>> N0B products. When RAX first came out, it was OK and then for a couple of
>>> days I saw the front end of the product was zlib compressed like the old
>>> days of the nexrad products. It went away so I didn't think much about it.
>>>
>>> But then yesterday it came back. Several sites were intermittently zlib
>>> compressing the N0B product. I noted Topeka, Kansas City, Springfield, St
>>> Louis, Chicago, Evansville, Indianapolis and N Indiana were doing it. But
>>> it was intermittent. One scan would be zlib compressed and the next
>>> wouldn't be. It was like something in the product, like size, was
>>> triggering the zlib compression.
>>>
>>> I took a look at some of the data. Only the first 4000 bytes are zlib
>>> compressed. The radial payload is still bzip2 compressed. Since only the
>>> header, which is about 100 bytes, is uncompressed, the zlib compression is
>>> compressing over 3800 bytes of the radial payload. This seems strange. Why
>>> compress data that are already compressed?
>>>
>>> After last night, it seems like this is a feature of the new N0B radar
>>> data and not a bug. So I resurrected the old ucnid.c program I wrote 24
>>> years ago to remove the zlib compression on the N0B products.
>>>
>>> So is this a bug or a feature?
>>>
>>> Dan.
>>>
>>> On Fri, Mar 4, 2022 at 10:36 AM Gilbert Sebenste <
>>> gilbert@xxxxxxxxxxxxxxxx> wrote:
>>>
>>>> Nothing beats a good Daryl LDM-users rant in the morning. He never lets
>>>> me down!
>>>>
>>>> Imagine my utter surprise when I did what Daryl said and I found
>>>> products I didn't know were being sent over NOAAport. Then, I got curious
>>>> and instead of using the "NEXRAD" feedtype, I used "ANY". That's when I was
>>>> shocked to find several products being sent under the HRS feed that were
>>>> completely undocumented by the NWS, and were needed! I alerted Steve
>>>> Emmerson, who put out a fix for it. 6.13.16 has it all sorted out now.
>>>>
>>>> I'll point out that this thinking goes for any product. If there's a
>>>> product you think should be there that isn't coming in, and you know you're
>>>> getting the entire feed, use a request of "ANY" on that header. Rarely,
>>>> it's an LDM bug, but like the example above, it can happen where a product
>>>> shows up on an unexpected feedtype.
>>>>
>>>> Gilbert
>>>>
>>>> > On Mar 4, 2022, at 10:17 AM, Herzmann, Daryl E [AGRON] <
>>>> akrherz@xxxxxxxxxxx> wrote:
>>>> >
>>>> > Good morning,
>>>> >
>>>> > I'd like to chime on the topic of "best practice for pqact entries".
>>>> For the WMO header found in products, there's a general form for the
>>>> entries found:
>>>> >
>>>> > TTAAII CCCC DDHHMI
>>>> >
>>>> > When there's a string found in the line below the WMO header that is
>>>> six or so characters, this "PIL" is appended to LDM product name in the
>>>> form like so:
>>>> >
>>>> > TTAAII CCCC DDHHMI /pPILXXX
>>>> >
>>>> > Summary: I *do not* trust the TTAAII value and only sparingly use it
>>>> within a pqact entry, but instead fully trust the /pPILXXX section.
>>>> >
>>>> > In December 2015, I meet with the NWS Data Management folks
>>>> expressing interest in quality controlling the CCCC portion of the WMO
>>>> header. Can you imagine that it is sometimes wrong? Well, it has been 6+
>>>> years now, hundreds of emails, and I'm still not done getting them to
>>>> correct issues with it. As an example of how painful the rabbit hole is,
>>>> NWS Cheyenne KCYS was issuing TAFs for 3 sites using KOAX as the CCCC.
>>>> This took 8 months to fix and I am still unsure if it is fully fixed.
>>>> >
>>>> > I have not even attempted automated quality control of the TTAAII,
>>>> nor bugging NWS to fix it. Another story. I actually did attempt to get
>>>> NWS to fix the TTAAII in the case of a product that wasn't using 6
>>>> characters for the TTAAII, can you imagine that it is sometimes only 5?
>>>> That request was rejected. Anyway, this value used to be more rigorously
>>>> set, but is now more arbitrary than ever. So in the original example
>>>> below.
>>>> >
>>>> > NEXRAD ^SDUS[2357]. ....
>>>> ([0-3][0-9])([0-2][0-9])([0-6][0-9]).*/p(...)(...)
>>>> >
>>>> > I would only trust the SDUS and drop any checks on the other TTAAII
>>>> fields. Instead, put your limiters into the /p section.
>>>> >
>>>> > NEXRAD ^SDUS.. ....
>>>> ([0-3][0-9])([0-2][0-9])([0-6][0-9]).*/p(N0Q|N0B|N0G|Nwhatever)(...)
>>>> >
>>>> > Oh, there are examples in the TTAAII, where the AA is wrong, but I
>>>> have ranted/whined enough already.
>>>> >
>>>> > daryl
>>>> >
>>>> > ________________________________________
>>>> > From: ldm-users <ldm-users-bounces@xxxxxxxxxxxxxxxx> on behalf of
>>>> Gilbert Sebenste <gilbert@xxxxxxxxxxxxxxxx>
>>>> > Sent: Friday, March 4, 2022 9:54 AM
>>>> > To: Daniel Vietor - NOAA Affiliate
>>>> > Cc: LDM-Users; Mike Voss
>>>> > Subject: Re: [ldm-users] NEXRAD ID changes??
>>>> >
>>>> > If you have LDM version 6.13.16, this shouldn't be happening. A bug
>>>> fix was applied to ensure that everything NEXRAD3 is supposed to be there.
>>>> >
>>>> > Gilbert
>>>> >
>>>> > On Mar 4, 2022, at 9:15 AM, Daniel Vietor - NOAA Affiliate via
>>>> ldm-users <ldm-users@xxxxxxxxxxxxxxxx> wrote:
>>>> >
>>>> >
>>>> > I see a lot of them on the HDS feedtype. So I request both NEXRAD and
>>>> HDS.
>>>> >
>>>> > Dan.
>>>> >
>>>> > On Fri, Mar 4, 2022 at 9:04 AM Mike Voss <mike.voss@xxxxxxxx<mailto:
>>>> mike.voss@xxxxxxxx>> wrote:
>>>> > Hi All,
>>>> > For me at least, some of my NEXRAD products stopped filing correctly
>>>> yesterday at 18Z using this generic pattern match:
>>>> >
>>>> > NEXRAD ^SDUS[2357]. ....
>>>> ([0-3][0-9])([0-2][0-9])([0-6][0-9]).*/p(...)(...)
>>>> > FILE -overwrite -close
>>>> nexrad/NIDS/\5/\4/\4_(\1:yyyy)(\1:mm)\1_\2\3
>>>> >
>>>> > as a test, when I changed to this:
>>>> > NEXRAD ^SDUS66 ....
>>>> >
>>>> > I started getting my "N0Q" products for MUX station.
>>>> >
>>>> > Also, I stopped receiving the FNEXRAD NEXRCOMP products. Is it
>>>> possible these things are related such that the NEXRCOMP depends on the
>>>> NIDS product ID's and somehow a recent change (yesterday) cause this to
>>>> break?
>>>> >
>>>> > thanks for any insights,
>>>> > -MIke
>>>> > _______________________________________________
>>>> > NOTE: All exchanges posted to Unidata maintained email lists are
>>>> > recorded in the Unidata inquiry tracking system and made publicly
>>>> > available through the web. Users who post to any of the lists we
>>>> > maintain are reminded to remove any personal information that they
>>>> > do not want to be made public.
>>>> >
>>>> >
>>>> > ldm-users mailing list
>>>> > ldm-users@xxxxxxxxxxxxxxxx<mailto:ldm-users@xxxxxxxxxxxxxxxx>
>>>> > For list information or to unsubscribe, visit:
>>>> https://www.unidata.ucar.edu/mailing_lists/
>>>> >
>>>> >
>>>> > --
>>>> > Dan Vietor
>>>> > Senior Research Meteorologist
>>>> > CIRA, Colorado State Univ
>>>> > Aviation Weather Center
>>>> > Kansas City, MO
>>>> > 816.584.7211
>>>> >
>>>> > _______________________________________________
>>>> > NOTE: All exchanges posted to Unidata maintained email lists are
>>>> > recorded in the Unidata inquiry tracking system and made publicly
>>>> > available through the web. Users who post to any of the lists we
>>>> > maintain are reminded to remove any personal information that they
>>>> > do not want to be made public.
>>>> >
>>>> >
>>>> > ldm-users mailing list
>>>> > ldm-users@xxxxxxxxxxxxxxxx
>>>> > For list information or to unsubscribe, visit:
>>>> https://www.unidata.ucar.edu/mailing_lists/
>>>>
>>>
>>>
>>> --
>>> *Dan Vietor*
>>> *Senior Research Meteorologist*
>>> CIRA, Colorado State Univ
>>> Aviation Weather Center
>>> Kansas City, MO
>>> 816.584.7211
>>>
>>> _______________________________________________
>>> NOTE: All exchanges posted to Unidata maintained email lists are
>>> recorded in the Unidata inquiry tracking system and made publicly
>>> available through the web. Users who post to any of the lists we
>>> maintain are reminded to remove any personal information that they
>>> do not want to be made public.
>>>
>>>
>>> ldm-users mailing list
>>> ldm-users@xxxxxxxxxxxxxxxx
>>> For list information or to unsubscribe, visit:
>>> https://www.unidata.ucar.edu/mailing_lists/
>>>
>>
--
*Dan Vietor*
*Senior Research Meteorologist*
CIRA, Colorado State Univ
Aviation Weather Center
Kansas City, MO
816.584.7211
/********************************************************************
* ucnids - NIDS decompression utility for data compressed with zlib
*
* syntax: ucnids [options] ifile ofile
* ifile and ofile can be "-" to use standard input and output
* options:
* none - output uncompressed product with NOAAPORT CCB
* -c - output with standard WMO CCB
* -r - output with strippped WMO CCB (RPS output)
* -w - output with WXP-like WMO header
* -n - output stripping header and leaving raw NIDS data
********************************************************************/
#include <stdlib.h>
#include <stdio.h>
#include <string.h>
#include <zlib.h>
#include <sys/stat.h>
void mkdirs( char *path ){
int len;
int i;
len = strlen( path );
for( i = 1; i < len-1; i++ ){
if( path[i] == '/' ){
path[i] = 0;
mkdir( path, 0755 );
path[i] = '/';
}
}
}
int main( int argc, char **argv ){
FILE *ifile;
FILE *ofile;
unsigned char soh[101];
unsigned char seq[101];
unsigned char wmo[101];
unsigned char awip[101];
unsigned char inbuf[1000];
unsigned int insize;
unsigned char outbuf[10000];
z_stream zs;
int out;
int len;
int iret;
int off;
int i;
int verb;
int wxp;
int inbytes;
int outbytes;
int check;
int floc;
verb = 0;
off = 0;
while( 1 ){
if( argc > 1 && !strcmp( argv[1+off], "-v" )){
verb = 1;
off++;
}
else if( argc > 1+off && !strcmp( argv[1+off], "-c" )){
out = 1;
off++;
}
else if( argc > 1+off && !strcmp( argv[1+off], "-w" )){
out = 2;
off++;
}
else if( argc > 1+off && !strcmp( argv[1+off], "-n" )){
out = 3;
off++;
}
else if( argc > 1+off && !strcmp( argv[1+off], "-r" )){
out = 4;
off++;
}
else
break;
}
if( argc == 1+off || argv[1+off][0] == '-' )
ifile = stdin;
else
ifile = fopen( argv[1+off], "r" );
if( ifile == NULL ) exit( 1 );
if( argc < 3+off || argv[2+off][0] == '-' )
ofile = stdin;
else {
mkdirs( argv[2+off] );
ofile = fopen( argv[2+off], "w" );
if( !ofile ) exit( 1 );
}
fgets( soh, 100, ifile );
/*
Account for both raw input (0x01) and WXP input (**)
*/
wxp = 0;
off = 0;
check = ((soh[0] << 8 + soh[1]) % 31);
if( check == 0 ){
off = strlen( soh );
memcpy( inbuf, soh, off );
}
else if( soh[0] == '\n' ){
fgets( soh, 100, ifile );
len = strlen( soh )-7;
memcpy( wmo, soh+3, len );
wmo[len] = 0;
fgets( awip, 100, ifile );
wxp = 1;
}
else if( soh[0] == '*' ){
len = strlen( soh )-7;
memcpy( wmo, soh+3, len );
wmo[len] = 0;
fgets( awip, 100, ifile );
wxp = 1;
}
else if( soh[0] == 0x01 ){
fgets( seq, 100, ifile );
fgets( wmo, 100, ifile );
fgets( awip, 100, ifile );
}
iret = 0;
zs.total_out = 4000;
for( i = 0; i < 1000 && (iret != Z_STREAM_END || zs.total_out == 4000); i++
){
/*
Read in a block of data
*/
floc = ftell( ifile );
insize = fread( inbuf+off, 1, 1000-off, ifile );
if( verb ) fprintf( stderr, "Read: %X %d\n", floc, insize );
if( off == 0 && insize == 0 ) break;
inbytes+=insize;
len = insize + off;
/*
Check for 789C byte sequence denoting zlib compression
If data are not compressed, pass through raw data
*/
check = (((inbuf[0] << 8) + inbuf[1]) % 31);
//printf( "check %X %X %d\n", inbuf[0], inbuf[1], check );
if( i == 0 && check != 0 ){
if( wxp ){
if( out == 1 ){
fwrite( "\001\r\r\n001\r\r\n", 1, 10, ofile );
fprintf( ofile, "%18.18s\r\r\n", wmo );
}
else if( out == 2 ){
fprintf( ofile, soh );
fprintf( ofile, awip );
}
}
else {
if( out == 1 ){
fprintf( ofile, soh );
fprintf( ofile, seq );
fprintf( ofile, wmo );
fprintf( ofile, awip );
}
else if( out == 2 ){
fprintf( ofile, "** %18.18s ***\n%s", wmo, awip );
}
}
fwrite( inbuf, 1, insize, ofile );
while(( insize = fread( inbuf, 1, 1000, ifile )) > 0 ){
fwrite( inbuf, 1, insize, ofile );
}
exit( 0 );
}
if(( check == 0 && inbuf[0] == 0x78 && i < 5 )|| iret != Z_STREAM_END ){
zs.avail_in = len;
zs.avail_out = 10000;
zs.next_in = inbuf;
zs.next_out = outbuf;
/*
Check to see if 4000 byte block has been read and reinitialize
*/
if( i == 0 || iret == Z_STREAM_END ){
zs.zalloc = NULL;
zs.zfree = NULL;
inflateInit( &zs );
}
/*
Inflate NIDS data
*/
iret = inflate( &zs, Z_STREAM_END );
if( verb ) fprintf( stderr, "Inf: %d -- %d %d %d -- 10000 %d %d -- %2X
%2X -- %2X %2X\n",
iret, len, zs.avail_in, zs.total_in, zs.avail_out, zs.total_out,
inbuf[0], inbuf[1], outbuf[0], outbuf[1] );
off = zs.avail_in;
}
else {
memcpy( outbuf, inbuf, len );
zs.avail_out = 10000-len;
off = 0;
if( verb ) fprintf( stderr, "Cpy: %d - %2X %2X\n", len, inbuf[0],
inbuf[1] );
}
/*
Process header data for first block
WMO CCB output
*/
if( i == 0 && out == 1 ){
fwrite( "\001\r\r\n001\r\r\n", 1, 10, ofile );
fwrite( outbuf+24, 1, 10000-zs.avail_out-24, ofile );
}
/*
WXP header output
*/
else if( i == 0 && out == 2 ){
fprintf( ofile, "** %18.18s ***\n%s", wmo, awip );
fwrite( outbuf+54, 1, 10000-zs.avail_out-54, ofile );
}
/*
Raw NIDS output
*/
else if( i == 0 && out == 3 ){
fwrite( outbuf+54, 1, 10000-zs.avail_out-54, ofile );
outbytes+=10000-zs.avail_out-54;
}
/*
Stripped WMO CCB output
*/
else if( i == 0 && out == 4 ){
fwrite( outbuf+24, 1, 10000-zs.avail_out-24, ofile );
}
/*
Raw output with NOAAPORT CCB
*/
else {
fwrite( outbuf, 1, 10000-zs.avail_out, ofile );
outbytes+=10000-zs.avail_out;
}
/*
Move remaining data that still is compressed and prepared
for next inflate
*/
memcpy( inbuf, inbuf+len-off, off );
if( iret < 0 ) break;
}
exit( 0 );
}
