[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

20050110: CONDUIT feed timing

Jerry,

Regarding queue size, If you are only receiving GFS,
hen you should expect the following:

.5 degree files: 39MB * 29 forecast hours = 1.1GB
1 degree files: 26MB * 61 forecast hours = 1.5GB
2.5 degree files: 4MB * 68 forecast hours = 272MB

Since the GFS takes over an hour to post to the NWS servers,
your queue doesnot see all that instantaneously of course, but
if you have less than a 2GB queue, you will likely have periods of
tme when you have less than 1 hour's worth of data as you can follow
using the ~ldm/bin/pqmon command.

If you were getting the ull conduit feed, you could see up to 2.7 GB in
an hours, which is the maximum I observed in the past 2 weeks, but generally 
2.2GB per
hour has been the 4x per day maximum. 

When you have redundant feed requests, and one is substantially faster then the 
other,
the less preferred route generally contributes little, but if your
queue holds less than 1 hour's worth of data, then as new data arrive from your 
faster route and scour the old data, the less preffered can
occaisionally send you a really of product that you had previously received.
The 6.1.0 ldm release does a better job at limiting the "how old" to the
default of the rpc.ldmd "-m" time which defaults to 3600 seconds.

Your previous 400MB queue would definitely be a problem. a 1GB queue I suspect 
will
besmall, unless you are selectively feeding parts of the data stream.

Thanks for the input you have provided, it is the best source of information
for helping us balance the data stream throughput versus usability.

Steve Chiswell
Unidata User Support


>From: Jerrold Robaidek <address@hidden>
>Organization: UCAR/Unidata
>Keywords: 200501101942.j0AJgWv2004366

>
>
>Hi Steve,
>
>Thanks again for all your efforts on this.
>
>I agree with Pete regarding the timing of the data.  This has been negatively 
> impacting a 
>few of our users.
>
>Also,  I am still seeing duplicate data. (I've increased my queue size to 1GB 
> (was 400 MB) 
>and I don't see an appreciable difference.)
>
>Also I am running LDM 6.0.14  do you think 6.1.0 will help any of my problems?
>
>Thanks,
>
>Jerry
>
>
>Steve Chiswell wrote:
>> Pete,
>> 
>> In pushing the queing lag time from the NWS computer queueing the data
>> to the NWS computer doing the data insertion, some of the latency is
>> now in the LDM queue as the volume you are receiving is increased
>> into a shorted time window- whereas before it was all in the insertion.
>> 
>> I agree that the parallel insertion is at some expense of the early
>> files since the slug of insertion starts slowing that down and
>> we'll have to see if we can optimize the insertion.
>> 
>> Steve Chiswell
>> Unidata User Support
>> 
>> 
>> 
>> 
>> 
>> On Fri, 2005-01-07 at 13:53, Pete Pokrandt wrote:
>> 
>>>In a previous message to me, you wrote: 
>>>
>>> >
>>> >Jerry,
>>> >
>>> >Yesterday at 1830Z we implemented a parallel queueing scheme at the
>>> >NWS that we hope will improve the timeliness of data being ingected into
>>> >the CONDUIT data stream. Any feedback you can provide on how
>>> >this affects your reception would be greatly appreciated.
>>> >
>>> >Since data will be inserted in parallel, you will notice that multiple 
>>> >model runds and forecast times will probably be interspersed where
>>> >previously they had been serialized.
>>> >
>>> >I watched the 00Z GFS last night, and the posting gap between f084 and
>>> >f132 was matched on the FTP server at 0422Z and later at 0509Z, the
>>> >other grids were posted to the NWS servers, so all appears to be
>>> >behaving correctly on this end.
>>> >
>>> >Steve Chiswell
>>> >Unidata User Support
>>>
>>>Steve,
>>>
>>>Since the parallel queueing scheme was implemented, I have noticed two
>>>things, both are problematic.
>>>
>>>1) The lag goes way up since so much more data is being inserted into
>>>the queue in a shorter amount of time. I was getting lag times of 3000-4000
>>>seconds in peak gfs times.  I moved my ldm queue on f5 to a faster disk, and
>>>that helped, but it's still getting up to 300-400 seconds.
>>>
>>>2) The biggest problem I see, is that now it takes MUCH longer for the
>>>early hours of a forecast run to complete.  Most of my model plotting
>>>scripts and all of our real-time mesoscale modelling efforts here
>>>take advantage of the fact that an entire model run doesn't need to be
>>>complete before we can start.  Since the parallel queueing was enabled,
>>>the 00 hour forecast of the eta model for example, takes over an hour
>>>to get here, where previously it was taking maybe 5 minutes.
>>>
>>>If this is how it's going to be, I'd much prefer the old serial ingest,
>>>or maybe some kind of blend, so the first 12 hours or so (or 48 or 
>>>whatever) of a model run can get priority. 
>>>
>>>It really hurts us to have the 00 hour forecast complete around the 
>>>same time as the later forecasts, even if the entire model run gets 
>>>in faster as a result.
>>>
>>>For what it's worth.
>>>
>>>Pete
>>>
>>>
>>>--
>>>+>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>+<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<+
>>>^ Pete Pokrandt                    V 1447  AOSS Bldg  1225 W Dayton St^
>>>^ Systems Programmer               V Madison,         WI     53706    ^
>>>^                                  V      address@hidden       ^
>>>^ Dept of Atmos & Oceanic Sciences V (608) 262-3086 (Phone/voicemail) ^
>>>^ University of Wisconsin-Madison  V       262-0166 (Fax)             ^
>>><<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<+>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>+
>
>
>-- 
>Jerrold Robaidek                       Email:  address@hidden
>SSEC Data Center                       Phone: (608) 262-6025
>University of Wisconsin                Fax: (608) 263-6738
>Madison, Wisconsin
>
--
NOTE: All email exchanges with Unidata User Support are recorded in the
Unidata inquiry tracking system and then made publicly available
through the web.  If you do not want to have your interactions made
available in this way, you must let us know in each email you send to us.