[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

[CONDUIT #SSL-833782]: LDM - conduit files >16MB being blocked?

John,
Now I see your latencies look much better since last night!
http://www.unidata.ucar.edu/cgi-bin/rtstats/iddstats_nc?CONDUIT+adiabat.rwic.und.edu

Now if you aren't getting the data out to disk, then the next thing would
be to examing the pqcat processing to see if the processing can be sped up.

Steve Chiswell
Unidata User Support



> Steve,
> 
> Thanks for the info, that clears up a lot of stuff.  I guess
> it's all relative, you should have seen our lantencies a couple weeks
> ago when our state I2 link was suffering lots of lost packets.
> 
> I appreciate the help.
> 
> On Thu, 12 Oct 2006, Unidata CONDUIT Support wrote:
> 
> > > Greetings.
> > >
> > > We're seeing a rather strange issue here at UND:
> > > any files being sent via CONDUIT that are over 16 megabytes
> > > in length are not showing up.  As a workaround we're fetching
> > > them via the telecom gateway, which is how we know they exist
> > > and are > 16MB.
> >
> > John,
> >
> > The data on the CONDUIT feed are the individual grib messages that exist 
> > within
> > the large files, so no product is bigger than a few hundred kbytes (eg not
> > a 16MB product such as the aggregate file on the ftp server). Therefore, 
> > nothing
> > can be blocked by the original file size of the NWS disk as all LDM 
> > products in
> > CONDUIT are individual grib products regardless.
> >
> >
> > >
> > > Our primary ingestor is running LDM 6.3.0 under FreeBSD
> > > 4.9.  This machine feeds from bigbird.tamu.edu via I2 and most
> > > of the time runs little latency:
> > >
> > > http://www.unidata.ucar.edu/cgi-bin/rtstats/iddstats_nc?CONDUIT+adiabat.rwic.und.edu
> > >
> > > Any idea why this only seems to affect CONDUIT files over
> > > 16MB?
> >
> > To the contrary, you feed shows large latencies during each 6 hourly data 
> > cycle.
> > Since the data is being transmitted as individual grib products, that is 
> > independent of
> > the size of the original data file. By comparison, the latency at bigbird 
> > is almost
> > always less than 60 seconds:
> > http://www.unidata.ucar.edu/cgi-bin/rtstats/iddstats_nc?CONDUIT+bigbird.tamu.edu
> > so, the holdup here is definitely attributable to your feed.
> >
> > Your latency plot shows a consitent latency of 2500 seconds at the peak 
> > (with one
> > over 4000s). If Bigbird's queue holds less than 2500 seconds of data at 
> > those times,
> > it will get overwritten in their queue by newer data before you receive it.
> > The other possibility is that it takes your own system so long to process 
> > the data out of
> > your local queue that it would get overwritten locally by new data before 
> > your pqact
> > process gets around to fileing or piping it.
> >
> > To get the data from bigbird to your machine faster, you may be able to use 
> > multiple
> > request lines. If you are currently feeding with a single CONDUIT pattern 
> > of ".*",
> > you can try instead using 5 request lines each getting 20% of the data such 
> > as:
> > request CONDUIT "[09]$" bigbird.tamu.edu
> > request CONDUIT "[18]$" bigbird.tamu.edu
> > request CONDUIT "[27]$" bigbird.tamu.edu
> > request CONDUIT "[36]$" bigbird.tamu.edu
> > request CONDUIT "[45]$" bigbird.tamu.edu
> >
> >
> > If on the other hand, you are seeing adta appear on your disk with 
> > significant time
> > bewteen when it arrives in the data queue, then you need to split up your
> > pqact processing so that the pqact processes can keep up with the rate of 
> > data coming in to your system.
> >
> > The last option is to cut down on the data you are receiving if you are not
> > utilizing all of the data in the CONDUIT feed. This would not solve your
> > underlying problem with keeping up with the volume however and that may
> > still become an issue in the future as more data are added to CONDUIT.
> >
> > Steve Chiswell
> > Unidata User Support
> >
> >
> > >
> > > Thanks.
> > >
> > > =========================================================================
> > > John Nordlie                    |     Regional Weather Information Center
> > > |              University of North Dakota
> > > =========================================================================
> > >
> > >
> > > Greetings.
> > >
> > > We're seeing a rather strange issue here at UND:
> > > any files being sent via CONDUIT that are over 16 megabytes
> > > in length are not showing up.  As a workaround we're fetching
> > > them via the telecom gateway, which is how we know they exist
> > > and are > 16MB.
> > >
> > > Our primary ingestor is running LDM 6.3.0 under FreeBSD
> > > 4.9.  This machine feeds from bigbird.tamu.edu via I2 and most
> > > of the time runs little latency:
> > >
> > > http://www.unidata.ucar.edu/cgi-bin/rtstats/iddstats_nc?CONDUIT+adiabat.rwic.und.edu
> > >
> > > Any idea why this only seems to affect CONDUIT files over
> > > 16MB?
> > >
> > > Thanks.
> > >
> > > =========================================================================
> > > John Nordlie                    |     Regional Weather Information Center
> > > |              University of North Dakota
> > > =========================================================================
> > >
> > >
> >
> >
> > Ticket Details
> > ===================
> > Ticket ID: SSL-833782
> > Department: Support CONDUIT
> > Priority: Normal
> > Status: Closed
> >
> 
> 


Ticket Details
===================
Ticket ID: SSL-833782
Department: Support CONDUIT
Priority: Normal
Status: Closed