Re: [thredds] Read-Only GribIndex cache conflict

  • To: John Caron <caron@xxxxxxxx>
  • Subject: Re: [thredds] Read-Only GribIndex cache conflict
  • From: Dan Swank - NOAA Affiliate <dan.swank@xxxxxxxx>
  • Date: Mon, 10 Aug 2015 10:46:04 -0400
Correct,
I have a DatasetScan (that gives the R/O file system error) along
side featureCollections using the read-only pool of indecies - those work
OK.
The datasetScans want to put .ncx3's into the GribIndex configured path.

On Fri, Aug 7, 2015 at 3:45 PM, John Caron <caron@xxxxxxxx> wrote:

> You only want one TDM for each set of files that need indexing. Multiple
> TDMs would step on each other. So that part is right.
>
> Your problem is that you are serving the files out individually also.
> Presumably you want those individual indices to be made only on demand?
> Cached for a bit then discarded? Are these in a datasetScan?
>
> John
>
> On Fri, Aug 7, 2015 at 1:23 PM, Dan Swank - NOAA Affiliate <
> dan.swank@xxxxxxxx> wrote:
>
>> I have a 3 tier server environment.   And the production
>> will be load balenced.   So the sense is that we cannot
>> afford to have 4+ TDM servers all indexing 32+ Million file units
>> even semi-regularly.   Too much I/O.   So I am trying to make
>> a master TDM server that upkeeps the GribIndex, while the
>> TDS server(s) run from a read-only copy of that.
>> The TDM master will signal all TDS instances when it finishes a pass.
>>
>> Perhaps I am overlooking something obvious
>> about how to set this up better.
>>
>> On Thu, Aug 6, 2015 at 9:07 PM, John Caron <caron@xxxxxxxx> wrote:
>>
>>> Hi Dan:
>>>
>>> A a read-only GribIndex Cache doesnt make too much sense. Why read-only?
>>>
>>> John
>>>
>>> On Thu, Aug 6, 2015 at 8:47 AM, Dan Swank - NOAA Affiliate <
>>> dan.swank@xxxxxxxx> wrote:
>>>
>>>> (v.4.6.2)  For a few THREDDS instances I need to run
>>>> the server from a read-only GribIndex Cache that TDM may update
>>>> on other servers ~ and this file system is shared with various
>>>>  virtual machines for static use.
>>>>
>>>> I've run into an issue where the featureCollections will
>>>> work just fine under the above arrangement, but if attempting to use
>>>> Services [OpenDap/Ncss] on a directory listed file units - outside
>>>> the featureCollections, would attempt to create a .ncx3 into this
>>>>  GribIndex cache location - and fail.
>>>>
>>>> Is there currently a way a workaround?
>>>>
>>>> Being able to configure a fail-over location to wirte
>>>> on-demand non-FC, gribIndex files may be needed.
>>>>
>>>> ---
>>>> Dan Swank
>>>> NOMADS System Support, STG Inc. - Federal Government Contractor
>>>> NOAA's National Centers for Environmental Information (NCEI)
>>>> Data Stewardship Devision (DSD)
>>>> 151 Patton Ave
>>>> Asheville, NC 28801
>>>> dan.swank@xxxxxxxx
>>>> Phone: 828-271-4007
>>>> Customer Support:  828-271-4800 or http://www.ncdc.noaa.gov/contact
>>>>
>>>> The newly formed NCEI merges the National Climatic Data Center (NCDC),
>>>> the National Geophysical Data Center (NGDC), and the National Oceanographic
>>>> Data Center (NODC).
>>>>
>>>> Facebook: http://www.facebook.com/NOAANationalClimaticDataCenter
>>>> Twitter: @NOAANCDC @NOAAOceanData
>>>>
>>>> ======================= Disclaimer ===========
>>>> I am not a Federal Employee.
>>>> Any opinions expressed in this message are mine personally and
>>>> do not represent official positions from STG Inc, ERT, NCEI, or NOAA
>>>> ============================================
>>>>
>>>> _______________________________________________
>>>> thredds mailing list
>>>> thredds@xxxxxxxxxxxxxxxx
>>>> For list information or to unsubscribe,  visit:
>>>> http://www.unidata.ucar.edu/mailing_lists/
>>>>
>>>
>>>
>>
>>
>> --
>>
>> Dan Swank
>> NOMADS System Support, STG Inc. - Federal Government Contractor
>> NOAA's National Centers for Environmental Information (NCEI)
>> Data Stewardship Devision (DSD)
>> 151 Patton Ave
>> Asheville, NC 28801
>> dan.swank@xxxxxxxx
>> Phone: 828-271-4007
>> Customer Support:  828-271-4800 or http://www.ncdc.noaa.gov/contact
>>
>> The newly formed NCEI merges the National Climatic Data Center (NCDC),
>> the National Geophysical Data Center (NGDC), and the National Oceanographic
>> Data Center (NODC).
>>
>> Facebook: http://www.facebook.com/NOAANationalClimaticDataCenter
>> Twitter: @NOAANCDC @NOAAOceanData
>>
>> ======================= Disclaimer ===========
>> I am not a Federal Employee.
>> Any opinions expressed in this message are mine personally and
>> do not represent official positions from STG Inc, ERT, NCEI, or NOAA
>> ============================================
>>
>
>


-- 

Dan Swank
NOMADS System Support, STG Inc. - Federal Government Contractor
NOAA's National Centers for Environmental Information (NCEI)
Data Stewardship Devision (DSD)
151 Patton Ave
Asheville, NC 28801
dan.swank@xxxxxxxx
Phone: 828-271-4007
Customer Support:  828-271-4800 or http://www.ncdc.noaa.gov/contact

The newly formed NCEI merges the National Climatic Data Center (NCDC), the
National Geophysical Data Center (NGDC), and the National Oceanographic
Data Center (NODC).

Facebook: http://www.facebook.com/NOAANationalClimaticDataCenter Twitter:
@NOAANCDC @NOAAOceanData

======================= Disclaimer ===========
I am not a Federal Employee.
Any opinions expressed in this message are mine personally and
do not represent official positions from STG Inc, ERT, NCEI, or NOAA
============================================
  • 2015 messages navigation, sorted by:
    1. Thread
    2. Subject
    3. Author
    4. Date
    5. ↑ Table Of Contents
  • Search the thredds archives: