Have you tried Matts command on this data?
On Jan 28, 2021, at 2:35 PM, McIDAS Help Desk
<mug@xxxxxxxxxxxxx<mailto:mug@xxxxxxxxxxxxx>> wrote:
His 1080 files from his NOAA S3 source are in:
/nas/home/jayh/mcidas/data/doug/*.nc
-Jay
On 1/28/21 1:42 PM, Russ Dengel wrote:
Jay,
There are a few things that could be happening.
The files vary in length could indicate that Matts files are including
extra fields per record
The Point ADDE servers do include Fortran code which could involve
static arrays that are being exceeded
There might be resource issues/differences between Matts machine and
yours.
Do we have some of Matts files?
Russ
On Jan 28, 2021, at 1:36 PM, McIDAS Help Desk
<mug@xxxxxxxxxxxxx<mailto:mug@xxxxxxxxxxxxx>> wrote:
Hi Russ-
I've attached Doug's trce file.
I don't know if he's tried SSEC servers, I'll ask as I think they do use our
data at times.
He has gotten these commands to work when he changes the INC= from INC=300 300
to INC=60 60 and does each hour individually. So, that seems to disprove the
idea that there is a corrupt file or reading. I was wondering if the server is
timing out or maybe it's something particular in his files. When he gets the
error, it plots some data before failing. But I'm at a loss as to why I can't
recreate the error with his files.
Thanks, Jay
On 1/28/21 10:30 AM, Russ Dengel wrote:
Jay,
The error is occurring in the ADDE Client/Server interface. I would need a
trce file generated from the GLMN server running on Matt’s server to diagnose
the problem past this point.
Has Matt tried to use our (SSEC) server to try the same command?
This —> "Our files look similar, but Doug's are all 15000 bytes larger.” is
suspicious. Has Matt used GLMDISP to plot other data from these NOAA S3
datasets?
Russ
On Jan 27, 2021, at 7:23 PM, McIDAS Help Desk
<mug@xxxxxxxxxxxxx<mailto:mug@xxxxxxxxxxxxx>> wrote:
Hi Russ-
I've been working with Doug Spangenberg from NASA Langley on a GLMDISP problem
he is having. I'm at a loss. He has 1080 files, representing 6 hours of data.
We've simplified the example to displaying over a WORL map, all files in one
directory, and he sent me all his files.
DSSERVE ADD DOUG2/GLM-GROUP GLMN TYPE=POINT
INFO=/home/mcidas/data/GLM_GROUP.cfg
DIRFILE=/nas/home/jayh/mcidas/data/doug/*.nc
MAKFRM 1 800 1000
SF new frame
GLMDISP WORL X DATASET=DOUG2/GLM-GROUP DAY=2019066 TIME=21:12 INC=300 300
TYPE=GROUP POS=GROUND COLOR=5 TOP='+ G16 LIGHTNING' 5 788 720 8 DEV=CCC TRACE=1
ECHO=YES
This GLMDISP command takes a while for me, but does work. Doug's plots some
data and fails with "PTDISP: Unable to read data record from server, PTDISP:
Failed during record fetch."
The last few lines of Doug's DEV=CCC show this:
PTDISP* GOTO NVINI
PTDISP* BACK FROM NVINI
PTDISP* Read of server has failed. Wanted 4
PTDISP* Did
PTDISP: Unable to read data record from server
PTDISP: Failed during record fetch
PTDISP*
PTDISP* Number of records received = 1367248
PTDISP* Number of records plotted = 1367248
PTDISP*
PTDISP* m0pttot totbytsent: 10938008
PTDISP - done
GLMDISP - done
It looks like Doug's gets about 1/3 done then is failing. Here is my DEV=CCC:
PTDISP* GOTO NVINI
PTDISP* BACK FROM NVINI
PTDISP*
PTDISP* Number of records received = 4168105
PTDISP* Number of records plotted = 4168105
PTDISP*
PTDISP* m0pttot totbytsent: 33344864
PTDISP - done
GLMDISP - done
I'm out of ideas on what to suggest. He's using McIDAS-X 2020.1 on a Linux 7
machine, and I'm testing on the same. Our files look similar, but Doug's are
all 15000 bytes larger. He got them from a NOAA S3 archive, I got mine from
/arcdata. There are subtle differences in the header when checking with ncdump
-h, but they didn't seem relevant.
What does the "Unable to read data record from server" error mean? And do you
have any other thoughts for Doug?
Thanks, Jay
-------- Forwarded Message --------
Subject: Re: [EXTERNAL] Re: G16 GLM
Date: Tue, 26 Jan 2021 23:54:07 +0000
From: Spangenberg, Douglas A. (LARC-E302)[Science Systems & Applications,
Inc.] <douglas.a.spangenberg@xxxxxxxx><mailto:douglas.a.spangenberg@xxxxxxxx>
To: McIDAS Help Desk <mug@xxxxxxxxxxxxx><mailto:mug@xxxxxxxxxxxxx>
Hi Jay,
I have 1080 files as well. The files I’ve been using were downloaded from a
NOAA S3 archive. I also tried running the command with INC=60 60 5X through
day 066, hours 16-21, and it worked for all the files in the dsserve command
DIRFILE= below. It looks like the INC=300 300 is the part that didn’t work for
me with unable to read from server error. I’m attaching the log of my run with
DEV= ECHO= added.
I also put a copy of the files (tar file) I’m using in this location if you
wanted to download and try:
https://satcorps.larc.nasa.gov/prod/NOAA-G16-GLM/2019/066/
DIRFILE=/gsaas_fsx/SatCORPS-Ancilliary/NOAA-G16-GLM/2019/066/*/*.nc
Thanks,
Doug
From: McIDAS Help Desk <mug@xxxxxxxxxxxxx><mailto:mug@xxxxxxxxxxxxx>
Date: Tuesday, January 26, 2021 at 2:35 PM
To: "Spangenberg, Douglas A. (LARC-E302)[Science Systems & Applications,
Inc.]"<douglas.a.spangenberg@xxxxxxxx><mailto:douglas.a.spangenberg@xxxxxxxx>
Subject: Re: [EXTERNAL] Re: G16 GLM
Hi Doug-
Thanks for working through this with me. I think I'm working with the same
dataset as you in the same directory structure now. I obtained my files from
SDS and have 180 files in each directory named
/home/jayh/mcidas/data/glm/2019/066/16/*, ../2019/066/17/*, through
../2019/066/21/*. So, it's searching through 1080 files, which doesn't seem
like too many. Do you have the same file count?
Can you add DEV=CCC and ECHO=YES to your last glmdisp.k over the WORL map and
send me the output? My command is receiving and plotting 4168274 records with
no errors. I'm using a DIRFILE= with /home/jayh/mcidas/data/glm/2019/066/*/*.nc
Also, did you get your files from CLASS, SDS or another source?
Thanks, Jay
On 1/22/21 8:01 PM, Spangenberg, Douglas A. (LARC-E302)[Science Systems &
Applications, Inc.] wrote:
Hi Jay,
\A0
I ran it with WORL map (see below) and it still gave the error about \201C
Unable to read data record from server\201D .\A0 However, it did make a map
with lightning displayed although likely less than normal depending on what
file the program stopped on.\A0 I attached the trace log. Perhaps it would be
good to add to the program more information about the file it was parsing, the
date and time of the file when the message happens since there are so many
files, it can be hard to track.\A0 I did see the files were all same
permissions and similar sizes so at least that was consistent.
\A0
For individual hours in the dsserve command, it worked on all of them without a
read server error, so there might be a problem going across the hours or for
more than a certain number of files?\A0 I also put all hours in one directory
and used no * in the path but that didn\2019 t work either.
\A0
glmdisp.k WORL 1 DATASET=G16/GLM-GROUP2 DAY=2019066 TIME=21:12 INC=300 300
TYPE=GROUP POS=GROUND COLOR=5 TRACE=1 TOP='+ G16 LIGHTNING' 5 488 520 8
\A0
Thanks,
Doug
\A0
\A0
From: McIDAS Help Desk <mug@xxxxxxxxxxxxx><mailto:mug@xxxxxxxxxxxxx>
Date: Thursday, January 21, 2021 at 5:39 PM
To: "Spangenberg, Douglas A. (LARC-E302)[Science Systems & Applications,
Inc.]"<douglas.a.spangenberg@xxxxxxxx><mailto:douglas.a.spangenberg@xxxxxxxx>
Subject: Re: [EXTERNAL] Re: G16 GLM
\A0
Hi Doug-
I put my GLM files in a similar directory structure.\A0 However, I only have
16-21Z for day 2019064 and 2019064.\A0 I've only been able to replicate the No
NetCDF file data satisfies error.\A0 A guess would be there is a corrupt file
or non-GLM file getting lumped in when searching which might lead to the Unable
to read data record error.
Thoughts:
1.\A0 Instead of overlaying over the satellite image, what happens if you use
MAP WORL for the domain (with the command that gives the Unable to read data
record error)?
2.\A0 For the command that gives the Unable to read data record error, can you
add TRACE=1 to the command?\A0 Then a "trce" file should be written into your
working directory.\A0 Please send me that file.
Thanks, Jay
On 1/20/21 10:41 AM, Spangenberg, Douglas A. (LARC-E302)[Science Systems &
Applications, Inc.] wrote:
Hi Jay,
\A0
I\2019 m getting the PTDISP: No NetCDF file data satisfies the request when I
use no * in the directory path and run the same command one hour at a time.\A0
When I put one star in the path for the hour on day 066, I get the other
message PTDISP: Unable to read data record from server, PTDISP: Failed during
record fetch. Also, on day 2019066, I have GLM data for hours 16-21 only in
there, not the entire day.
\A0
Thanks,
\A0
Doug
\A0
\A0
From: McIDAS Help Desk <mug@xxxxxxxxxxxxx><mailto:mug@xxxxxxxxxxxxx>
Date: Friday, January 15, 2021 at 1:42 PM
To: "Spangenberg, Douglas A. (LARC-E302)[Science Systems & Applications, Inc.]"
<douglas.a.spangenberg@xxxxxxxx><mailto:douglas.a.spangenberg@xxxxxxxx>
Subject: Re: [EXTERNAL] Re: G16 GLM
\A0
Hi Doug-
With the data from 2019066 displayed over that domain, I'm getting the "PTDISP:
No NetCDF file data satisfies the request" message, which makes sense since
there is no lightning data in the frame.
I'm still unsure why you are getting the "PTDISP: Unable to read data record
from server" message.
Could you try defining your DIRFILE= without the "*" wildcards (*.nc is fine,
just change the others to point to the 2019066 data directly)?\A0 I'm curious
to see what error you get.
Thanks, Jay
On 1/14/21 6:10 PM, Spangenberg, Douglas A. (LARC-E302)[Science Systems &
Applications, Inc.] wrote:
Hi Jay,
\A0
1. From the dsserve command:
G16/GLM-GROUP\A0 POINT GLMN Info=/home/mcidas/data/GLM_GROUP.cfg
DIRFILE=/gsaas_fsx/SatCORPS-Ancilliary/NOAA-G16-GLM/2019/0*/*/*.nc
It\2019 s local and not on a remote server.
\A0
2.\A0 It\2019 s from McIDAS-X 2020.1 version with Linux 7.9
\A0
3. Navigation is from these commands with 500x700 frame size
imgcopy.k AGOES16/CONUS A.7000 SIZE=SAME BAND=13 MAG=-1 DAY=2019066 TIME=18:22
18:26
imgremap.k A.7000 A.7001 PRO=RECT LATLON=41.4 91.4 RES=1.3 SIZE=500 700
imgdisp.k A.7001 2 LATLON=41.4 91.4 \A0 MAG=-1
\A0
thanks,
\A0
Doug
\A0
\A0
From: McIDAS Help Desk <mug@xxxxxxxxxxxxx><mailto:mug@xxxxxxxxxxxxx>
Date: Thursday, January 14, 2021 at 1:27 PM
To: "Spangenberg, Douglas A. (LARC-E302)[Science Systems & Applications, Inc.]"
<douglas.a.spangenberg@xxxxxxxx><mailto:douglas.a.spangenberg@xxxxxxxx>
Subject: [EXTERNAL] Re: G16 GLM
\A0
Hi Doug-
Regarding the 2nd command, you are correct, that is the error received when
there is no lightning in the frame.
For the 1st command, I downloaded the 2019066 GOES 16 GLM files and put them in
a local directory to do some testing.\A0 I wasn't able to recreate the
error.\A0 My GLMDISP completed with no errors and showed some lightning data
over Colorado and Utah.\A0 So, I have a few questions:
1. I noted the error you are getting is the same when there is a period "."
somewhere in the DIRFILE= (inq.
17059<https://gcc02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fmcidas.ssec.wisc.edu%2Finquiry-x%2Findex.php%3Finquiry%3D17059&data=04%7C01%7Cdouglas.a.spangenberg%40nasa.gov%7Cba0080e19c07490fcdde08d8c23188a2%7C7005d45845be48ae8140d43da96dd17b%7C0%7C0%7C637472865286472700%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=A6SRxoyMcRjvahIT8oI7muPFSsE7bIRNvvJrHOE5feg%3D&reserved=0>
-\A0 which hasn't been released yet).\A0 I was curious to see what the DSSERVE
command was for the G16/GLM-GROUP dataset (and whether it's local or
remote)?\A0 However, I would expect your other GLMDISP command to fail the same
way, so I'm not confident this is the answer.
2. If #1 didn't solve the problem, what version of -X are you using, what
operating system?
3. What is the IMGDISP command you are using as navigation, and what size frame
are you using?
Thanks, Jay
On 1/13/21 5:33 PM, Spangenberg, Douglas A. (LARC-E302)[Science Systems &
Applications, Inc.] wrote:
McIDAS Help,
\A0
I\2019 ve been running some glmdisp commands with historical GOES-16 lightning
netcdf files and have seen the following messages.\A0 Not on every time period
just for some. Do you know what is happening?
\A0
If I run this as separate commands with 20 minutes duration of lightning for
each, it doesn\2019 t give any error messages:
glmdisp.k X 2 DATASET=G16/GLM-GROUP DAY=2019066 TIME=21:12 INC=300 300
TYPE=GROUP NAV=2 POS=GROUND COLOR=5 TOP='+ G16 LIGHTNING' 5 488 520 8
Plotting GLM data from G16/GLM-GROUP.ALL
PTDISP: Unable to read data record from server
: Failed during record fetch\A0 \A0 \A0 \A0 \A0 \A0 \A0 \A0
\A0
I also get this message. I\2019 m assuming for those periods when no lightning
occurred in the frame:
glmdisp.k X 2 DATASET=G16/GLM-GROUP DAY=2019064 TIME=16:32 INC=280 280
TYPE=GROUP NAV=2 POS=GROUND COLOR=5 TOP='+ G16 LIGHTNING' 5 488 520 8
Plotting GLM data from G16/GLM-GROUP.ALL
PTDISP: No NetCDF file data satisfies the request
GLMDISP - done
\A0
All glmdisp commands were run with G16 IR images on frames showing the upper
Midwest USA.
\A0
Thanks,
\A0
Doug
-->
-->
-->
--><g16glm.2019066.output.log>
<trce_doug.txt>
