[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

20030109: data precision



Dorothy,

You didn't mention how you created the field, but if you
are inserting into a grid file, you need to determine the number of
bits/precision your data has.

For example, when using gddiag, you will see the parameter GPACK.
If you are using your own routine calling the gempak library,
then you have to specify the packing method and number of bits.

When you pack data, you are specifying the resolution that is
necessary to represent your data, and if you don't specify enough bits,
then you will see truncation.

If you can tell me how you are calculating and storing the data,
I may be able to tell where you are losing your precision. Typically, if using
GRIB packing (MDGGRB) in a call to gd_wpgd, the number of bits
necessary would be
        rdif = maxval - minval
        rbits = abs ( log ( rdif ) ) / log ( 2.0 )
        nbits = int(rbits) + 1
where 2 <= nbits <= 32.

Steve Chiswell

On Thu, 9 Jan 2003, Dorothy Durnford wrote:

> Hi Steve,
>
>     This one really puzzles me.  I'm trying to plot PV at 200 hPa, in
> PVU.  When I print out some of the grid point values at the end of the
> calculating program, just before the values are written to a Gempak
> grid, I get something like 6 decimal points, and everything looks fine.
> When I use gdmap, with a scale of 2, every integer value is followed by
> exactly the same two numbers.  e.g., 1.56 PVU and 1.99 PVU both appear
> in gdmap as 104, and 5.66 PVU and 5.32 PVU both appear as the same
> number.  I realized there were problems when I was in gdcntr.  I usually
> set "contur = 5/3".  With this field, I get these horrible, jagged
> contour shapes.  The same thing happens with "contur = 3/3".  At "contur
> = 0, 0/3, or 0/6", the contours look perfect.  When I didn't think the
> contur setting should have so much effect, I started to look at gdmap.
> Do I really have a problem, or is this normal?  If I have a problem, can
> you suggest where it might be?
>
> Yours for ever grateful, Dorothy Durnford.
>
>