Sean,
Yes, the lat & lon change (are different) between each day/granule as the
swath track shifts slightly between each day. So we need to have some way
to reference a different/unique lat/lon for each time=T, which I thought
would happen with a joinNew and including this in the variableAgg listing.
Can you elaborate more on this "swath feature type" that is in development?
We are very interested in testing and helping with the development of this.
You can see (below) that using your suggested joinNew method of assuming
that the lat & lon do not change (i.e., excluding from the variableAgg
list) has the last aggregation in the array looking correct (as this is the
lat/lon adopted for all other days/granules) and then the penultimate value
has garbled swath data (along with all other days).
* last day (T=324) in the aggregation (GOOD: this is what we expect to see)
[image: Screenshot 2020-05-12 16.12.21.png]
* penultimate day (T=323) in the aggregation (this is NOT what we expect
to see)
[image: Screenshot 2020-05-12 16.12.32.png]
http://tds.coaps.fsu.edu/thredds/dodsC/coaps/qscat/nc_L2B_v4p0_12/nc_L2B_12_daily_TESTjoinNew-lon-lat-invariant.html
On Tue, May 12, 2020 at 9:12 AM Sean Arms <sarms@xxxxxxxx> wrote:
> Greetings Michael,
>
> Do lon and lat change between each granule of the aggregation in the
> daily files? If so, there will be issues with tools using netCDF-Java
> going through the part of the stack that needs to understand things at
> a feature type level, such as GRID, as netCDF-Java does not deal with
> time varying spatial coordinate systems (this includes WMS and the
> netCDF Subset Service in the TDS, for example)...there is some code
> that makes me think it might partially deal with it, but I'm guessing
> it's nowhere near complete (not sure how much swath feature type was
> developed).
>
> That said, I was able to get the following to read in toolsUI as a
> GRID by assuming lat/lon to not vary in time (likely an incorrect
> assumption):
>
> <?xml version="1.0" encoding="UTF-8"?>
> <netcdf xmlns="http://www.unidata.ucar.edu/namespaces/netcdf/ncml-2.2";>
> <aggregation dimName="aggTime" type="joinNew">
> <variableAgg name="wind_speed"/>
> <scan location="C:/Users/sarms/Desktop/qsat/qscat_data/"
> suffix=".nc" dateFormatMark="QSCAT_12_#yyyyMMdd" />
> </aggregation>
> <variable name="wind_speed">
> <attribute name="coordinates" value="aggTime lon lat" />
> </variable>
> </netcdf>
>
> again, this assumes that lat/lon are not time dependent across the
> aggregation files. Also note that I've updated the coordinates
> attribute to reference the new aggTime coordinate variable that's
> automatically created as part of the joinNew aggregation.
>
> Sean
>
> On Mon, May 11, 2020 at 4:15 PM Michael McDonald <mcdonald@xxxxxxxxxxxxx>
> wrote:
> >
> > Antonio,
> > While the "along_track" is the UNLIMITED dimension in all of these
> > swath files. Our attempts to just use this as the dimension for the
> > joinExisting aggregation does not throw any issues wrt THREDDS at
> > catalog init time, but when you actually try to load/use this dataset
> > it does not work and fails with the message in Panoply,
> >
> > "There was an error preparing the data: Failed creating the data
> > handler: Could not read lon axis array."
> >
> >
> >
> > e.g., joinExisting aggregation on along_track (fails with: Could not
> > read lon axis array.)
> >
> http://tds.coaps.fsu.edu/thredds/dodsC/coaps/qscat/nc_L2B_v4p0_12/nc_L2B_12_daily_TESTjoinExisting-along_track.html
> >
> > <netcdf xmlns="http://www.unidata.ucar.edu/namespaces/netcdf/ncml-2.2";>
> > <aggregation type="joinExisting" dimName="along_track">
> > <scan
> location="/Net/data/qscat/data/nc_L2B_v4p0_12/nc_L2B_12_daily/2009"
> > suffix=".nc" subdirs="false" />
> > </aggregation>
> > </netcdf>
> >
> >
> > This *simplified* product is a collection of multiple quikscat swath
> > passes in a 24 hour period to assemble a full coverage of the globe.
> >
> > orbit_file_source:
> > https://podaac-opendap.jpl.nasa.gov/opendap/allData/quikscat/L2B12/v4.0/
> >
> > Which looks like this for a single day.
> >
> http://tds.coaps.fsu.edu/thredds/wms/coaps/qscat/nc_L2B_v4p0_12/nc_L2B_12_daily/2009/QSCAT_12_2009325.nc?STYLES=&SERVICE=WMS&FORMAT=image%2Fpng&REQUEST=GetMap&HEIGHT=512&WIDTH=1024&VERSION=1.1.1&BBOX=-180.0%2C-80.0%2C180.0%2C90.0&LAYERS=wind&SRS=EPSG%3A4326&TRANSPARENT=TRUE
> >
> >
> > Our *end goal* is to have a virtually aggregated collection of all
> > this simplified daily swath data from 1999 to 2009 for ingestion as an
> > OPeNDAP/WMS dataset into a web-based data visualizer like OWGIS
> > (https://owgis.org/), so that users can pick the date(s) they want,
> > and view this in a web GUI.
> >
> >
> > We tried to use joinNew to aggregate all these like files on a new
> > dimension "T" and populate the time info using the filename
> > (dateFormatMark="QSCAT_#HH_yyyyDDD") and get the same error (fails
> > with: Could not read lon axis array.)
> >
> >
> > e.g., joinNew aggregation on new dimension T
> >
> http://tds.coaps.fsu.edu/thredds/dodsC/coaps/qscat/nc_L2B_v4p0_12/nc_L2B_12_daily_TESTjoinNew-time-lon-lat.html
> >
> > <netcdf xmlns="http://www.unidata.ucar.edu/namespaces/netcdf/ncml-2.2";>
> > <aggregation dimName="T" type="joinNew">
> > <variableAgg name="time"/>
> > <variableAgg name="lon"/>
> > <variableAgg name="lat"/>
> > <variableAgg name="eastward_wind"/>
> > <variableAgg name="northward_wind"/>
> > <variableAgg name="wind_speed"/>
> > <variableAgg name="wind_to_direction"/>
> > <variableAgg name="simplified_wvc_quality_flag"/>
> > <variableAgg name="rain_impact"/>
> > <variableAgg name="distance_from_coast"/>
> > <scan dateFormatMark="QSCAT_#HH_yyyyDDD"
> > location="/Net/data/qscat/data/nc_L2B_v4p0_12/nc_L2B_12_daily/2009"
> > suffix=".nc" subdirs="false" />
> > </aggregation>
> > </netcdf>
> >
> >
> > Any ideas what we are doing wrong in THREDDS, or is the issue in the
> > data files we simplified?
> >
> >
> > >
> > > Ian,,
> > >
> > > What aggregation are you expecting? just on the 'along_track'
> dimension?
> > >
> > > Antonio
> > > On 5/5/20 19:46, Ian Dimitri wrote:
> > >
> > > Hello All,
> > >
> > > Adding onto Michael's existing thread, I am working closely with him
> attempting to figure out the issue we are having with the aggregation of
> swath data that has a two-dimensional grid with track and cross-track
> coordinates. We have tried using FMRC, joinExisting, and joinNew file
> aggregation methods within THREDDS, all with no success.
> > >
> > > We tried using many different aggregations methods
> > > that exist within the ncWMS2 interface. We have tried doing the simple
> wildcard aggregation attempt by using the wildcard character (e.g.
> /ASCAT/nc_L2B_25_daily/2020/*.nc).
> > >
> > > We have also tried to use the "Dynamic Services" that is also
> available through ncWMS2, but that also did not seem to work with the
> multi-dimensional swath data that we are
> > > trying to aggregate.
> > >
> > > Finally another method we tried was through geoserver, where we
> attempted to use the image mosaic option that it has but that option also
> proved to be a dead end.
> > >
> > > None of the methods mentioned above worked properly, and we are
> wondering if anyone has had any success aggregating swath data with the
> methods, and would like to share any
> > > special steps that had to be taken to make said method work. Are there
> any other methods I did not mention above that could possibly be able to
> aggregate the multi-demensional swath data?
> > >
> > > Any insight or help would be appreciated
> > >
> > > unaggregated data
> > >
> http://tds.coaps.fsu.edu/thredds/catalog/coaps/qscat/nc_L2B_v4p0_12/nc_L2B_12_daily/catalog.html
> > >
> > > attempted aggregation with joinNew
> > >
> http://tds.coaps.fsu.edu/thredds/satellite_testing.html?dataset=joinnew-TEST-coaps-qscat-nc_L2B_v4p0_12-nc_L2B_12_daily
> >
> > _______________________________________________
> > NOTE: All exchanges posted to Unidata maintained email lists are
> > recorded in the Unidata inquiry tracking system and made publicly
> > available through the web. Users who post to any of the lists we
> > maintain are reminded to remove any personal information that they
> > do not want to be made public.
> >
> >
> > thredds mailing list
> > thredds@xxxxxxxxxxxxxxxx
> > For list information or to unsubscribe, visit:
> https://www.unidata.ucar.edu/mailing_lists/
>
--
Michael McDonald
Florida State University
