NOTE: The bufrtables mailing list is no longer active. The list archives are made available for historical reasons.
Hi Chris:Very good points about data models/formats becoming complex. There are a number of features that appear to complicate things without obvious corresponding benefit. Perhaps it would be helpful if we start to enumerate exactly what are the basic features and which not?
As I mentioned in a previous post, the checksum idea was to verify which table was used, rather than message integrity, which can now be assumed to be handled at a lower layer. Whether that's an idea whose complexity is worth it is another issue! Some bufr complexity appears to be motivated by keeping the record small. I wonder whether a simpler data layout with dictionary based compression (eg LZW) might not work better, now that CPUs are so much faster than IO. I will be doing some testing with that idea, once my decoder is reliably working. Regards, John Little, Chris wrote:
Hi Milan and John, Perhaps I can usefully contribute here. We did originally have a checksum for each of the main sections of BUFR, but the justification was mainly to do with unreliable bit orientated telecoms, and limits on block sizes. We addressed those by saying that the telecoms protocol should handle them. We wanted a clean, orthogonal separation of transport protocol and content. We never considered correcting errors between tables, messages and applications. We didn't even envisage more than one Master Table - multiple master tables were a bit of an afterthought, to accommodate the oceanographers. We never really thought through in detail how things would be administered. It was left as a trivial exercise for the readers!From the outside, I can see you all adding bits here and there to fix things. I would say Bufr is now in its 'baroque' or 'rococo' phase, and I would recommend some serious pruning/deprecation/simplification/re-factoring to get back to a more 'classical' structure, perhaps a 'BUFR- lite'. I think complexity, even if it is necessary, seems to be a barrier to wider take up of BUFR. I think the success of NetCDF and HDF came from simplicity and elegance first (and supplied software and APIs), and then complexity came after it had been taken up and the community recognised that the extras were needed.I hope you can come up with a straightforward table maintenance process.Best wishes, Chris Chris Little International Telecommunications Met Office FitzRoy Road Exeter Devon EX1 3PB United Kingdom Tel: +44(0)1392 886278 Fax: +44(0)1392 885681 Mobile: +44(0)7753 880514 E-mail: chris.little@xxxxxxxxxxxxxxxx Met Office climate change predictions can be viewed on Google Earth http://www.metoffice.gov.uk/research/hadleycentre/google/
bufrtables archives: