Hi Don, John, Thanks for your quick fixes! All of my current examples now work in IDV (nightly build Nov 16).I see what you mean with some data ending up in structs, also the attached temp_*.bufr files suffer from that problem. The data in those structs are not visible in either IDV or toolsUI which is a problem.
I also noticed that I can not mix BUFR types in one file, IDV and toolsUI will only select one (attached mixed_ship_synop.bufr). What are the requirements for putting messages together, do all messages need to have identical data descriptors (expanded/unexpanded)?
ReinoudPS: please also send messages directly to my email, sometimes I only accidentally see a message in the idvusers digest emails.
Message: 1 Date: Fri, 12 Nov 2010 07:46:56 -0700 From: Don Murray <Don.Murray@xxxxxxxx> To: idvusers@xxxxxxxxxxxxxxxx Subject: Re: [idvusers] idvusers Digest, Vol 21, Issue 6 Message-ID: <4CDD5360.5050101@xxxxxxxx> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Hi Reinoud-John and I discussed this offline and the problem that the IDV (and maybe ToolsUI) is having is that some of the fields end up as netCDF DataTypes (Sequences and Structures (IDV variables seq1 and seq2) that the IDV was assuming were just plain numbers. John's going to look at whether this is the correct way to handle these, but in the mean time, the nightly build of the IDV should handle these BUFR files now. I just checked in another fix to make it even more robust, so you might want to wait until tomorrow's nightly build. Let us know if you find more problems with these and/or other BURF files that are problematic.Don On 11/12/10 1:54 AM, Reinoud Bokhorst wrote:Hi John, Tell me about the complexity of BUFR... The last two files were meteorological SYNOP files coded according to the WMO standard tables for FM94 BUFR Edition 4: http://www.wmo.int/pages/prog/www/WMOCodes/TDCFtables.html It uses the common sequence 3-07-080 (= Sequence for representation of synoptic reports from a fixed land station suitable for SYNOP data) from table D. I also use a ECMWF bufr decoder package (http://www.ecmwf.int/products/data/software/bufr.html). Attached is 'decode_bufr.txt' which is a dump of test1.bufr using the decode_bufr program in this package. The output compares well with what I see in toolsUI 4.2. However I now discovered when I press the 'Detail Info' button in toolsUI, I get a similar conversion error as reported for IDV (see attached). The reason why I am interested in this data is that the WMO GTS system (for SYNOP, TEMP, SHIP, BUOY, etc) is migrating from the alphanumeric code formats to the binary formats (http://www.wmo.int/pages/prog/www/WMOCodes.html). The deadline was due this month but I believe it has been postponed a while. Regards, Reinoud idvusers-request@xxxxxxxxxxxxxxxx wrote:---------------------------------------------------------------------- Message: 1 Date: Wed, 10 Nov 2010 16:27:09 -0700 From: John Caron <caron@xxxxxxxxxxxxxxxx> To: idvusers@xxxxxxxxxxxxxxxx Subject: Re: [idvusers] BUFR support in IDV Message-ID: <4CDB2A4D.7050808@xxxxxxxxxxxxxxxx> Content-Type: text/plain; charset=ISO-8859-1; format=flowed HI all: Apparently the BUFR reading is ok, and it reads ok into the FeatureType adapters, buts its not clear if IDV is using the new adapter. Reinoud: We can read well-formed BUFR messages into the CDM, but figuring out what they mean in order to get them geolocated and visualized in the IDV is a hard problem. BUFR is a particularly complex format where providers have too many choices on how to do the encoding, with little or no guidance or best practices AFAICT. The original example you sent (MSSARG_00217064.bufr) is good. The latest two are, um, difficult, and the possibility of us being able to understand arbitrary BUFR messages like this is not good. Perhaps if you could track down any documentation on that "template", we will see what we can do. John On 11/9/2010 8:36 AM, Don Murray wrote:Hi Reinoud- Thanks for trying this out. The BUFR support in the IDV has not really been tested. It relies on the netCDF-Java package to do the reading, so by default, some things "just work", while others (as you found) don't. Please submit this to support-idv@xxxxxxxxxxxxxxxx and the developers can take a look. It'll probably be passed on to the netCDF-Java support people for action. Don On 11/9/10 2:16 AM, Reinoud Bokhorst wrote:Hi Don, After a short break I have started looking into this again. Attached two surface SYNOP files in BUFR that load fine in toolsUI 4.2 but not in IDV 2.9u2. I can open the files in IDV as "Netcdf /Gempak point data files" but when I try to view the data I get the following error: ucar.ma2.ForbiddenConversionException at ucar.ma2.ArrayStructure.getFloat(ArrayStructure.java:1022) at ucar.ma2.StructureDataW.getScalarFloat(StructureDataW.java:139) at ucar.ma2.StructureDataW.convertScalarFloat(StructureDataW.java:96) at ucar.unidata.data.point.PointObFactory.makePointObs(PointObFactory.java:1910) at ucar.unidata.data.point.NetcdfPointDataSource.makeObs(NetcdfPointDataSource.java:360) at ucar.unidata.data.point.NetcdfPointDataSource.makeObs(NetcdfPointDataSource.java:322) at ucar.unidata.data.point.PointDataSource.getDataInner(PointDataSource.java:1266) at ucar.unidata.data.DataSourceImpl.getData(DataSourceImpl.java:2238) at ucar.unidata.data.DirectDataChoice.getData(DirectDataChoice.java:332) at ucar.unidata.data.DataChoice.getData(DataChoice.java:637) at ucar.unidata.data.DataInstance.getData(DataInstance.java:243) at ucar.unidata.data.DataInstance.getData(DataInstance.java:207) at ucar.unidata.data.DataInstance.dataOk(DataInstance.java:295) at ucar.unidata.data.point.PointDataInstance.init(PointDataInstance.java:83) at ucar.unidata.data.point.PointDataInstance.<init>(PointDataInstance.java:69) at ucar.unidata.idv.control.ObsDisplayControl.doMakeDataInstance(ObsDisplayControl.java:795) at ucar.unidata.idv.control.DisplayControlImpl.initializeDataInstance(DisplayControlImpl.java:3084) at ucar.unidata.idv.control.DisplayControlImpl.setData(DisplayControlImpl.java:3066) at ucar.unidata.idv.control.StationModelControl.setData(StationModelControl.java:1459) at ucar.unidata.idv.control.StationModelControl.init(StationModelControl.java:463) at ucar.unidata.idv.control.DisplayControlImpl.init(DisplayControlImpl.java:1333) at ucar.unidata.idv.control.DisplayControlImpl.init(DisplayControlImpl.java:1034) at ucar.unidata.idv.ControlDescriptor.initControl(ControlDescriptor.java:986) at ucar.unidata.idv.ControlDescriptor$1.run(ControlDescriptor.java:911) at ucar.unidata.util.Misc$3.run(Misc.java:1090) Regards, Reinoud Don Murray wrote:Hi Reinoud- On 10/19/10 7:43 AM, Reinoud Bokhorst wrote:Thank you for your fast response and fix.No problem. It was my bug, I figured I should fix it and I had a few spare cycles. ;-)This particular example now works. But when I try loading other samples from GTS (Global Telecom System) data it doesn't.. I am a bit confused now about the level of support of BUFR in IDV (after your fix that made it work) but as I understand it, it is only partly because the CDM supports it (correct me if I am wrong ;). Do you want more sample data at this point? Or rather wait until BUFR is officially going to be supported?Can you load all these other files in the ToolsUI? If they conform to what the CDM thinks are point or station obs, then they should be readable in the IDV. Although, the CDM implementation might not be correct. The IDV won't be able to handle soundings (profiles) or trajectories which the CDM might handle. Sample files would help.HP, yes I'll probably send a note to the User's Committee, although by no means I am an expert on this field.Cheers, Don------------------------------ _______________________________________________ idvusers mailing list idvusers@xxxxxxxxxxxxxxxx For list information, to unsubscribe, visit: http://www.unidata.ucar.edu/mailing_lists/ End of idvusers Digest, Vol 21, Issue 6 ***************************************_______________________________________________ idvusers mailing list idvusers@xxxxxxxxxxxxxxxx For list information, to unsubscribe, visit: http://www.unidata.ucar.edu/mailing_lists/
Attachment:
mixed_ship_synop.bufr
Description: Binary data
Attachment:
ship.bufr
Description: Binary data
Attachment:
temp_land.bufr
Description: Binary data
Attachment:
temp_ship.bufr
Description: Binary data
idvusers archives: