Proposed Thredds Architecture Changes for OSGI/JigSaw

This post provides some preliminary ideas on the consequences of moving TDS to use OSGI or JigSaw

Assumptions:

  1. OSGI and Jigsaw will be sufficiently similar so that this proposal with work with either with some tweeks.
  2. Initial target is Thredds server
  3. We will want to dynamically load at least the following kinds of things on the server.
    • OSPs (e.g. netcdf4, grib, etc)
    • RAFs (e.g. S3 and HDFS)
    • Services (e.g. DAP4) I will refer to these all generically as "bundles" (OSGI terminology)

The loading process could be either:

  1. lazy - load only when actually requested
  2. eager- load at startup to provide a specifically configured TDS starting with a skeleton TDS.

For the eager case, we can assume that some config file (e.g. ThreddsConfig.xml) contains the information needed to dynamically extend the tds to make various bundles available.

For the lazy case, it must be possible to create a "signal" that some bundle is needed and must be preloaded. I can see two obvious ways to do this.

  1. Stubs -- we provide stub classes for all the bundles so that calling the stub API the first time causes the bundle to be loaded and then used from then on.
  2. Explicit -- any user of a bundle must explicitly invoke some code to load the required bundle.

My current inclination is to use the eager approach since it is simpler and still allows us to keep a small footprint .war file.

Another question is: where are the bundles stored? I assume they are not kept in the .war file since that would defeat one of the purposes of using dynamic loading. I presume there would be some default repository(s) plus a configurable set of additional repositories from which bundles can be pulled. It may be that NEXUS is usable for this purpose.

A note on IOSPs. Currently the IOSP to use is determined by calling a method that looks at a RAF wrapping a file. This method decides if itcan process that associated file. If we were to use lazy loading, it is probable that for IOSP's we would need to divide the IOSP into two parts: one for testing applicability and one for processing. This is an argument for using eager loading.

Plotting AWIPS Map Resources with Python

The python-awips package provides access to the entire AWIPS Maps Database for use in Python GIS applications. Map objects are returned as Shapely geometries (Polygon, Point, MultiLineString, etc.) and can be easily plotted by Matplotlib, Cartopy, MetPy, and other packages.

[Read More]

Unit Testing in the netCDF-c library using cmake

It appears to be the case that all current tests in the netcdf-c source tree are integration tests. That is they only access the library using calls to the externally visible API.

The DAP4 code testing is different in that it includes (for the first time) unit tests. These tests reference code that is considered internal to the library. For the library generated under autoconf, this is not a problem because those symbols are visible in the library: nothing is hidden.

[Read More]

AWIPS NEXRAD Level 3 Rendered with Matplotlib

Shown here are plots for Base Reflectivity (N0Q, 94) and Base Velocity (N0U, 99) using AWIPS data rendered with Matplotlib, Cartopy, and MetPy. This example improves upon existing Level 3 Python rendering by doing the following:

  • Display scaled and labeled colorbar below each figure.
  • Plot radar radial images as coordinate maps in Cartopy and label with lat/lon.
  • 8 bit Z and V colormap and data scaling added to MetPy from operational AWIPS.
  • Level 3 data are retrieved from the Unidata EDEX Cloud server (edex-cloud.unidata.ucar.edu)
  • Raw HDF5 byte data are converted to product values.
[Read More]

New TDS Cloud Architectures: Proposal 1

The Thredds Data server (TDS) was designed to operate in a client-server architecture. Recently, Unidata has moved TDS into the cloud using its existing architecture.

There seems to be agreement inside Unidata that we need to begin rethinking that architecture to adapt to the realities of the cloud.

[Read More]
Unidata Developer's Blog
A weblog about software development by Unidata developers*
Unidata Developer's Blog
A weblog about software development by Unidata developers*

Welcome

FAQs

News@Unidata blog

Take a poll!

What if we had an ongoing user poll in here?

Browse By Topic
Browse by Topic
« May 2017
SunMonTueWedThuFriSat
 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
26
27
28
29
30
31
   
       
Today