[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

[netCDF #JSG-595652]: RE: NetCDF streaming

Hi Malcolm,

> A couple of questions for you based on your responses below.  If I
> wanted to replace your existing ncstream transport with a different
> transport, would that be a lot of work?

I'm Cc:ing John Caron, the ncstream developer, to respond to that question.

> The second question concerns szip.  I understand the issues with writing
> szip content, and why you don't ship software that does this.  Can we do
> our own szip integration?  How much trouble would that be?  The szip
> folks told me that the product is free for use on our system, so I'd
> like to be able to swap out the zlib compression if szip works better.

 Yes, I think that should be pretty easy.  Remember to build HDF5 with
szip, and configure netCDF with the --with-szlib=... option to allow
reading szip-compressed files, but you'll have to do the modifications
to add the capability to write szip files, just following the pattern
for zlib.

--Russ

> -----Original Message-----
> From: Russ Rew [mailto:address@hidden]
> Sent: Monday, August 23, 2010 4:18 PM
> To: support-netcdf
> Cc: Horne, Randy (Contractor); address@hidden
> Subject: Re: NetCDF streaming
> 
> 
> Hi Malcolm,
> 
> Unidata makes available a different technology for low latency data
> distribution, the LDM client-server software used for the Internet Data
> Distribution project:
> 
> http://www.unidata.ucar.edu/software/ldm/
> 
> The LDM distributes any kinds of data products over a network, using an
> event-oriented architecture that pushes the data products to downstream
> sites in a way that aims to minimize latency.
> 
> We have also implemented an experimental version of an "ncstream"
> on-the-wire protocol for streaming data that conforms to the Unidata
> Common Data Model:
> 
> http://www.unidata.ucar.edu/software/netcdf-java/stream/NcStream.html
> 
> The goals of that implementation include "write optimized" data
> encoding, unlike netCDF which can be characterized as better for the
> reader than the writer.
> 
> The Web service version of ncstream is "cdmremote":
> 
> http://www.unidata.ucar.edu/software/netcdf-java/stream/CdmRemote.html
> 
> As far as transport goes, ncstream is built on Google Protocol Buffers,
> so I think those are independent of the transport layer used:
> 
> http://code.google.com/apis/protocolbuffers/
> 
> We don't support writing with szip compression in our software because
> of licensing issues with the SZIP compression algorithm.
> 
> I'm also Cc:ing John Caron, the creator of ncstream and cdmremote, in
> case he has anything to add or correct in what I've written above.
> 
> --Russ
> 
> > This is a multi-part message in MIME format.
> >
> > ------_=_NextPart_001_01CB42EC.124D694B
> > Content-Type: text/plain;
> >     charset="us-ascii"
> > Content-Transfer-Encoding: quoted-printable
> >
> > Hello,
> >
> > =20
> >
> > I'm an engineer working on the GOES-R Ground Segment.  I work with
> Randy
> > Horne, and I saw an earlier e-mail thread where you discussed the
> NetCDF
> > roadmap.  I'm currently looking at mechanisms to transfer large
> amounts
> > of NetCDF format data between sites with low latency.  Your research
> in
> > streaming may apply to this problem, but the information on the web
> site
> > is very sketchy.  I'd like to know more about what you are doing, and
> > see if it would benefit my application.  If so maybe we could be part
> of
> > a beta test.
> >
> > =20
> >
> > I'm very interested in what sort of transport you plan to provide.
> > We're currently looking at both TCP and UDP multicast based options.
> > We're also very interested in how this would combine with the
> integrated
> > szip compression feature. =20
> >
> > =20
> >
> > I look forward to hearing from you.
> >
> > =20
> >
> > Malcolm McRoberts
> >
> > Harris Corporation
> >
> > (321) 984 6109
> >
> > =20
> >
> >
> > ------_=_NextPart_001_01CB42EC.124D694B
> > Content-Type: text/html;
> >     charset="us-ascii"
> > Content-Transfer-Encoding: quoted-printable
> >
> > <html xmlns:v=3D"urn:schemas-microsoft-com:vml" =
> > xmlns:o=3D"urn:schemas-microsoft-com:office:office" =
> > xmlns:w=3D"urn:schemas-microsoft-com:office:word" =
> > xmlns:x=3D"urn:schemas-microsoft-com:office:excel" =
> > xmlns:p=3D"urn:schemas-microsoft-com:office:powerpoint" =
> > xmlns:a=3D"urn:schemas-microsoft-com:office:access" =
> > xmlns:dt=3D"uuid:C2F41010-65B3-11d1-A29F-00AA00C14882" =
> > xmlns:s=3D"uuid:BDC6E3F0-6DA3-11d1-A2A3-00AA00C14882" =
> > xmlns:rs=3D"urn:schemas-microsoft-com:rowset"
> xmlns:z=3D"#RowsetSchema" =
> > xmlns:b=3D"urn:schemas-microsoft-com:office:publisher" =
> > xmlns:ss=3D"urn:schemas-microsoft-com:office:spreadsheet" =
> > xmlns:c=3D"urn:schemas-microsoft-com:office:component:spreadsheet" =
> > xmlns:odc=3D"urn:schemas-microsoft-com:office:odc" =
> > xmlns:oa=3D"urn:schemas-microsoft-com:office:activation" =
> > xmlns:html=3D"http://www.w3.org/TR/REC-html40"; =
> > xmlns:q=3D"http://schemas.xmlsoap.org/soap/envelope/"; =
> > xmlns:rtc=3D"http://microsoft.com/officenet/conferencing"; =
> > xmlns:D=3D"DAV:" xmlns:Repl=3D"http://schemas.microsoft.com/repl/"; =
> > xmlns:mt=3D"http://schemas.microsoft.com/sharepoint/soap/meetings/"; =
> > xmlns:x2=3D"http://schemas.microsoft.com/office/excel/2003/xml"; =
> > xmlns:ppda=3D"http://www.passport.com/NameSpace.xsd"; =
> > xmlns:ois=3D"http://schemas.microsoft.com/sharepoint/soap/ois/"; =
> > xmlns:dir=3D"http://schemas.microsoft.com/sharepoint/soap/directory/";
> =
> > xmlns:ds=3D"http://www.w3.org/2000/09/xmldsig#"; =
> > xmlns:dsp=3D"http://schemas.microsoft.com/sharepoint/dsp"; =
> > xmlns:udc=3D"http://schemas.microsoft.com/data/udc"; =
> > xmlns:xsd=3D"http://www.w3.org/2001/XMLSchema"; =
> >
> xmlns:sub=3D"http://schemas.microsoft.com/sharepoint/soap/2002/1/alerts/
> "=
> >  xmlns:ec=3D"http://www.w3.org/2001/04/xmlenc#"; =
> > xmlns:sp=3D"http://schemas.microsoft.com/sharepoint/"; =
> > xmlns:sps=3D"http://schemas.microsoft.com/sharepoint/soap/"; =
> > xmlns:xsi=3D"http://www.w3.org/2001/XMLSchema-instance"; =
> > xmlns:udcs=3D"http://schemas.microsoft.com/data/udc/soap"; =
> > xmlns:udcxf=3D"http://schemas.microsoft.com/data/udc/xmlfile"; =
> > xmlns:udcp2p=3D"http://schemas.microsoft.com/data/udc/parttopart"; =
> > xmlns:wf=3D"http://schemas.microsoft.com/sharepoint/soap/workflow/"; =
> > xmlns:dsss=3D"http://schemas.microsoft.com/office/2006/digsig-setup"; =
> > xmlns:dssi=3D"http://schemas.microsoft.com/office/2006/digsig"; =
> >
> xmlns:mdssi=3D"http://schemas.openxmlformats.org/package/2006/digital-si
> g=
> > nature" =
> >
> xmlns:mver=3D"http://schemas.openxmlformats.org/markup-compatibility/200
> 6=
> > " xmlns:m=3D"http://schemas.microsoft.com/office/2004/12/omml"; =
> >
> xmlns:mrels=3D"http://schemas.openxmlformats.org/package/2006/relationsh
> i=
> > ps" xmlns:spwp=3D"http://microsoft.com/sharepoint/webpartpages"; =
> >
> xmlns:ex12t=3D"http://schemas.microsoft.com/exchange/services/2006/types
> "=
> >  =
> >
> xmlns:ex12m=3D"http://schemas.microsoft.com/exchange/services/2006/messa
> g=
> > es" =
> >
> xmlns:pptsl=3D"http://schemas.microsoft.com/sharepoint/soap/SlideLibrary
> /=
> > " =
> >
> xmlns:spsl=3D"http://microsoft.com/webservices/SharePointPortalServer/Pu
> b=
> > lishedLinksService" xmlns:Z=3D"urn:schemas-microsoft-com:" =
> > xmlns:st=3D"&#1;" xmlns=3D"http://www.w3.org/TR/REC-html40";>
> >
> > <head>
> > <meta http-equiv=3DContent-Type content=3D"text/html; =
> > charset=3Dus-ascii">
> > <meta name=3DGenerator content=3D"Microsoft Word 12 (filtered
> medium)">
> > <style>
> > <!--
> >  /* Font Definitions */
> >  @font-face
> >     {font-family:Calibri;
> >     panose-1:2 15 5 2 2 2 4 3 2 4;}
> >  /* Style Definitions */
> >  p.MsoNormal, li.MsoNormal, div.MsoNormal
> >     {margin:0in;
> >     margin-bottom:.0001pt;
> >     font-size:11.0pt;
> >     font-family:"Calibri","sans-serif";}
> > a:link, span.MsoHyperlink
> >     {mso-style-priority:99;
> >     color:blue;
> >     text-decoration:underline;}
> > a:visited, span.MsoHyperlinkFollowed
> >     {mso-style-priority:99;
> >     color:purple;
> >     text-decoration:underline;}
> > span.EmailStyle17
> >     {mso-style-type:personal-compose;
> >     font-family:"Calibri","sans-serif";
> >     color:windowtext;}
> > .MsoChpDefault
> >     {mso-style-type:export-only;}
> > @page Section1
> >     {size:8.5in 11.0in;
> >     margin:1.0in 1.0in 1.0in 1.0in;}
> > div.Section1
> >     {page:Section1;}
> > -->
> > </style>
> > <!--[if gte mso 9]><xml>
> >  <o:shapedefaults v:ext=3D"edit" spidmax=3D"1026" />
> > </xml><![endif]--><!--[if gte mso 9]><xml>
> >  <o:shapelayout v:ext=3D"edit">
> >   <o:idmap v:ext=3D"edit" data=3D"1" />
> >  </o:shapelayout></xml><![endif]-->
> > </head>
> >
> > <body lang=3DEN-US link=3Dblue vlink=3Dpurple>
> >
> > <div class=3DSection1>
> >
> > <p class=3DMsoNormal>Hello,<o:p></o:p></p>
> >
> > <p class=3DMsoNormal><o:p>&nbsp;</o:p></p>
> >
> > <p class=3DMsoNormal>I&#8217;m an engineer working on the GOES-R
> Ground
> > Segment.&nbsp; I work with Randy Horne, and I saw an earlier e-mail =
> > thread
> > where you discussed the NetCDF roadmap.&nbsp; I&#8217;m currently =
> > looking at mechanisms
> > to transfer large amounts of NetCDF format data between sites with low
> > latency.&nbsp; Your research in streaming may apply to this problem,
> but =
> > the
> > information on the web site is very sketchy.&nbsp; I&#8217;d like to =
> > know more
> > about what you are doing, and see if it would benefit my =
> > application.&nbsp; If
> > so maybe we could be part of a beta test.<o:p></o:p></p>
> >
> > <p class=3DMsoNormal><o:p>&nbsp;</o:p></p>
> >
> > <p class=3DMsoNormal>I&#8217;m very interested in what sort of
> transport =
> > you plan
> > to provide.&nbsp; We&#8217;re currently looking at both TCP and UDP =
> > multicast
> > based options.&nbsp; We&#8217;re also very interested in how this
> would =
> > combine
> > with the integrated szip compression feature.&nbsp; <o:p></o:p></p>
> >
> > <p class=3DMsoNormal><o:p>&nbsp;</o:p></p>
> >
> > <p class=3DMsoNormal>I look forward to hearing from
> you.<o:p></o:p></p>
> >
> > <p class=3DMsoNormal><o:p>&nbsp;</o:p></p>
> >
> > <p class=3DMsoNormal>Malcolm McRoberts<o:p></o:p></p>
> >
> > <p class=3DMsoNormal>Harris Corporation<o:p></o:p></p>
> >
> > <p class=3DMsoNormal>(321) 984 6109<o:p></o:p></p>
> >
> > <p class=3DMsoNormal><o:p>&nbsp;</o:p></p>
> >
> > </div>
> >
> > </body>
> >
> > </html>
> >
> > ------_=_NextPart_001_01CB42EC.124D694B--
> 
> 

Russ Rew                                         UCAR Unidata Program
address@hidden                      http://www.unidata.ucar.edu



Ticket Details
===================
Ticket ID: JSG-595652
Department: Support netCDF
Priority: Normal
Status: Closed