Hi John:
Thanks for the info. Right now the sample files I can get are different from
the files we aggregate, so I don’t have a good way of testing this. When NASA
makes the general switch over I should be able to set up a small test system.
I did open a L3 test file in toolsUI, maybe I am misreading the toolsUI info
but it seem to read the file okay but as you said, not recognize the coordinate
system.
-Roy
> On Jun 2, 2015, at 9:08 AM, John Caron <caron@xxxxxxxx> wrote:
>
> The SMIs look ok. In principle you can aggregate over file types is they are
> truly homogenous (same variable names, dimensions, coordinate systems, etc).
> You really have to look hard at the result and test. You might be better off
> starting a new aggregation where the new files start.
>
> From CDM POV, groups are ok, but they complicate the coordinate system
> discovery, unless you are careful. In practice, the dataset writers are often
> not careful.
>
> On Mon, Jun 1, 2015 at 8:36 PM, Roy Mendelssohn - NOAA Federal
> <roy.mendelssohn@xxxxxxxx> wrote:
> Hi John:
>
> Thanks.
>
> The L2 files we process and make our own composite files so they aren’t an
> issue It is mainly the L3’s (SMI) that we are concerned about. As I said,
> the two issues are just can they be served by TDS, forget any other problems
> (and we can do what we have been doing is add time as a coordinate in an NcML
> file which is what we then aggregate).
>
> But the bigger issue is whether we can aggregate over the different file
> types. If we are actually aggregating NcML files (which is usually the
> case), will TDS care that some of the files being pointed to in the NcML are
> HDF4 and some are netcdf4? If TDS doesn’t care that the filetypes are
> different, we probably can kludge a solution to a lot of the issues I
> raised. Otherwise, I may have to figure out how to re-download a ton of
> files over a bandwidth that is already often over-saturated.
>
> And can I open up the can of worms again? I am curious what people feel
> about the use of groups in netcdf4 files? It is no longer a theoretical
> question. If you are getting ocean color data from NASA, that is what you
> are getting. If people are interested I can post the file headers (or
> maybe more appropriately move this discussion to the netcdf mail-list) so you
> can see how they are being used. If you are familiar with the old HDF4
> format, it is mostly a mapping of that structure.
>
> -Roy
>
>
> > On Jun 1, 2015, at 7:20 PM, John Caron <caron@xxxxxxxx> wrote:
> >
> > Hi Roy:
> >
> > The grids look ok (L3m), although they didnt put the time values into a
> > coordinate, but in global attributes.
> >
> > The swaths (L2) can probably be fixed up with a little NcML. Its not clear
> > if aggregation is possible or useful.
> >
> > The points (L3b) are incomprehensible to me. Someone expert would have to
> > explain whats in there.
> >
> > Not a lot of CF compliance that I can see.
> >
> > John
> >
> > On Mon, Jun 1, 2015 at 5:24 PM, Roy Mendelssohn - NOAA Federal
> > <roy.mendelssohn@xxxxxxxx> wrote:
> > HI All:
> >
> > I don’t know how many of you get notices like this:
> >
> > > MODIS Ocean Color and SST Data Users:
> > >
> > > Within the next week, the Ocean Biology Processing Group will begin the
> > > ocean color (OC) reprocessing of MODIS on Aqua (MODISA). At that time, we
> > > will also make available the fully reprocessed SST from MODIS on Aqua and
> > > Terra, and we will start producing forward-stream products in the
> > > reprocessing configuration for all four data streams (MODIS SST and OC
> > > from both satellite platforms). General information on the reprocessing
> > > changes for OC and SST can be found here:
> >
> > The new format, if you don’t know, are netcdf-4 files, but they make heavy
> > use of groups. As some of you may remember, a proposal to develop a
> > standard for use of groups in CF was put forward by Charlie Zender and Ted
> > Haberman, but got shot down. Be that as it may, the NASA product is what
> > it is, and this raised some questions for our data services. To wit:
> >
> > 1. Does anyone know if TDS can handle these files correctly (because of
> > the groups and not being strictly CF)? toolsUI seems to be able to read
> > them okay, but I am not certain that it is picking up the data types
> > properly (so recognizes that something is an array but not necessarily a
> > grid)? Sample files are at:
> >
> > http://oceancolor.gsfc.nasa.gov/DOCS/FormatChange.html
> >
> >
> > 2. For some of the data we do some further processing and write our own
> > files, but some we have in either netcdf3 or HDF4 files in TDS? Do
> > aggregations work over different file types, as long as the variable name
> > and the variable to aggregated over are the same?
> >
> > 3. In past incarnations, the chlorophyll files did not contain the two
> > carbon variables POC, PIC, these were in separate files. Now they are
> > included in the chlorophyll file. Is there any way that anyone can think
> > of to continue that PIC and POC aggregations with the new files (remember
> > I have use the same files for the chlorophyll aggregations) or I am pretty
> > much stuck having to redownload the re-processed files?
> >
> > Thanks,
> >
> > -Roy
> >
> > **********************
> > "The contents of this message do not reflect any position of the U.S.
> > Government or NOAA."
> > **********************
> > Roy Mendelssohn
> > Supervisory Operations Research Analyst
> > NOAA/NMFS
> > Environmental Research Division
> > Southwest Fisheries Science Center
> > ***Note new address and phone***
> > 110 Shaffer Road
> > Santa Cruz, CA 95060
> > Phone: (831)-420-3666
> > Fax: (831) 420-3980
> > e-mail: Roy.Mendelssohn@xxxxxxxx www: http://www.pfeg.noaa.gov/
> >
> > "Old age and treachery will overcome youth and skill."
> > "From those who have been given much, much will be expected"
> > "the arc of the moral universe is long, but it bends toward justice" -MLK
> > Jr.
> >
> > _______________________________________________
> > thredds mailing list
> > thredds@xxxxxxxxxxxxxxxx
> > For list information or to unsubscribe, visit:
> > http://www.unidata.ucar.edu/mailing_lists/
> >
>
> **********************
> "The contents of this message do not reflect any position of the U.S.
> Government or NOAA."
> **********************
> Roy Mendelssohn
> Supervisory Operations Research Analyst
> NOAA/NMFS
> Environmental Research Division
> Southwest Fisheries Science Center
> ***Note new address and phone***
> 110 Shaffer Road
> Santa Cruz, CA 95060
> Phone: (831)-420-3666
> Fax: (831) 420-3980
> e-mail: Roy.Mendelssohn@xxxxxxxx www: http://www.pfeg.noaa.gov/
>
> "Old age and treachery will overcome youth and skill."
> "From those who have been given much, much will be expected"
> "the arc of the moral universe is long, but it bends toward justice" -MLK Jr.
>
>
**********************
"The contents of this message do not reflect any position of the U.S.
Government or NOAA."
**********************
Roy Mendelssohn
Supervisory Operations Research Analyst
NOAA/NMFS
Environmental Research Division
Southwest Fisheries Science Center
***Note new address and phone***
110 Shaffer Road
Santa Cruz, CA 95060
Phone: (831)-420-3666
Fax: (831) 420-3980
e-mail: Roy.Mendelssohn@xxxxxxxx www: http://www.pfeg.noaa.gov/
"Old age and treachery will overcome youth and skill."
"From those who have been given much, much will be expected"
"the arc of the moral universe is long, but it bends toward justice" -MLK Jr.
