NOTE: The decoders
mailing list is no longer active. The list archives are made available for historical reasons.
============================================================================== Robb Kambic Unidata Program Center Software Engineer III Univ. Corp for Atmospheric Research rkambic@xxxxxxxxxxxxxxxx WWW: http://www.unidata.ucar.edu/ ============================================================================== ---------- Forwarded message ---------- Date: Fri, 26 Sep 2003 21:25:00 -0600 (MDT) From: ncdigest <owner-ncdigest@xxxxxxxxxxxxxxxx> To: ncdigest@xxxxxxxxxxxxxxxx Subject: ncdigest V1 #724 ncdigest Friday, September 26 2003 Volume 01 : Number 724 Today's Topics: Re: perl interface problem, plus ncdump oddity ---------------------------------------------------------------------- Date: Fri, 26 Sep 2003 15:54:55 -0400 From: Jason Thaxter <thaxter@xxxxxxxxxx> Subject: Re: perl interface problem, plus ncdump oddity - --SLDf9lqlvOQaIe6s Content-Type: text/plain; charset=us-ascii Content-Disposition: inline On Tue, Sep 16, 2003 at 11:20:13AM -0400, Jason Thaxter wrote:
I've found two odd problems with NetCDF tools recently. I suspect they are related to compiler and perl versions, but I don't really know. 1) ---------------------------------------------------- The Perl interface issue involves values retrieved by NetCDF::recget. The variable is a scalar of type BYTE. Perl always thinks it has a zero (0), even
I figured out this particular problem; the perl-netcdf library does not appear to be misbehaving at all, though it does things that might trap an unwary - and at least in this case, fairly wary - perl programmer. Variables of the NetCDF::BYTE type cannot be immediately used as perl variables. They will end up looking like "^A" or "^@" or something else when you try to use them as strings; worse, if you look at them as numbers, they will always appear to be zero. Suppose that $value is an actual single value retrieved from attget/varget/recget. To use a NetCDF::BYTE properly, do this $value = unpack('C',$value); I put this trick into a wrapper module I've written around NetCDF, called GoMOOS::NetCDF. There are a number of other tricks there, too - initializing arrays before passing them to NetCDF functions, retrieving variables by names instead of index number, etc. I wouldn't say it's complete or flawless; it's a little inconsistent about return values, and it's probably more useful for taking a record-oriented view than a variable-oriented view, and it's totally useless for writing NetCDF files. Nevertheless, anyone using perl-netcdf may find it useful and so I've attached it to this message. If the attachment doesn't make it through majordomo onto the list, feel free to e-mail me for a copy. Thanks, Jason - -- - ---------------------------------------------- Jason Thaxter GoMOOS, P.O. Box 4919, Portland, ME 04112-4919 Office Location: 1 Canal Plaza, 7th Floor Office: 207.773.0423 Fax: 207.773.8672 Email: thaxter@xxxxxxxxxx - ------------www.gomoos.org-------------------- - --SLDf9lqlvOQaIe6s Content-Type: application/x-perl Content-Disposition: attachment; filename="NetCDF.pm" Content-Transfer-Encoding: quoted-printable package GoMOOS::NetCDF;=0A=0A=3Dhead1 NAME=0A=0AGoMOOS::NetCDF - a library of routines for handling NetCDF files=0A=0A=3Dhead1 SYNOPSIS=0A=0AWraps bas ic NetCDF functions for use raw by scripts, or by other classes=0A(e.g. GoM OOS::Buoys::NetCDF). Note that many of the functions in this library=0Amay have a tendency to die suddenly. We try to check them, and we mark them =0Aif possible.=0A=0ANote that this is primarily a record-oriented library. =0A=0A=3Dcut=0Amy $version =3D '$Revision: 1.29 $';=0A=0Ause strict;=0Ause Carp 'cluck';=0A=0Ause NetCDF;=0A=0Amy @types;=0A$types[NetCDF::FLOAT] =3D 'float';=0A$types[NetCDF::DOUBLE] =3D 'double';=0A$types[NetCDF::LONG] =3D 'long';=0A$types[NetCDF::SHORT] =3D 'short';=0A$types[NetCDF::BYTE] =3D 'by te';=0A$types[NetCDF::CHAR] =3D 'char';=0Amy $ncGLOBAL =3D NetCDF::GLOBAL; =0A=0Amy $_MODE_RO =3D NetCDF::NOWRITE;=0A=0Asub new {=0A my $proto =3D shi ft;=0A my $class =3D ref($proto) || $proto;=0A my $self =3D {};=0A=0A if ( my $file =3D shift){=0A $self->{_FILE} =3D $file;=0A }=0A=0A bless($self,$ class);=0A return $self;=0A}=0A=0Asub open {=0A my ($self,$file) =3D @_;=0A =0A # may pass argument to open=0A if ($file){ $self->{_FILE} =3D $file; } =0A=0A unless(-e $self->{_FILE}){=0A cluck "No such file: " . $self->{_FIL E};=0A return; # NetCDF::open will terminate!! if file can't be opened=0A }=0A=0A # TODO: check to see if it looks like a netcdf file...=0A # it shou ld be a binary file:=0A # file tmp/test.nc =3D tmp/test.nc: data=0A # and the first three chars are 'CDF'=0A=0A # this next line will crash on fa il, can't eval!!=0A my $ncid =3D NetCDF::open($self->{_FILE},$_MODE_RO); # read-only mode should be default=0A $self->{_NCID} =3D $ncid;=0A=0A # Store basic info=0A my ($ndims,$nvars,$ngatts,$recdim);=0A NetCDF::inquire($ncid ,$ndims,$nvars,$ngatts,$recdim);=0A #warn "ndims=3D$ndims,nvars=3D$nvars,na tt=3D$ngatts,recdim=3D$recdim\n";=0A $self->{_NDIMS} =3D $ndims; #=0A $se lf->{_NVARS} =3D $nvars;=0A $self->{_NGATTS} =3D $ngatts;=0A $self->{_RECD IM} =3D $recdim;=0A=0A my ($numrecs,$dim);=0A=0A # STORE OUR DIMENSION SIZ ES=0A for (my $i=3D0;$i<$ndims;$i++){=0A my ($dname,$dsize);=0A if (NetCD F::diminq($ncid,$i,\$dname,\$dsize) =3D=3D 0){=0A # STORE FOR REFERENCE =0A $self->{_DIM}{$dname} =3D $dsize;=0A # STORE OUR RECORD DIMENSION: TELLS HOW MANY RECORDS WE HAVE=0A if ($i =3D=3D $recdim){=0A $numrecs =3D $dsize;=0A $self->{_NUMRECS} =3D $numrecs;=0A }=0A }=0A else { =0A cluck "NetCDF::diminq returned non-zero";=0A }=0A }=0A # TODO: this could cause other problems... maybe we should yell=0A return $self->{_NCID} unless ($self->{_NUMRECS});=0A=0A # CREATE LIST OF ALL VARIABLES=0A my @al l_vars =3D ();=0A for my $i (0 ... ($self->{_NVARS} - 1)){=0A my ($var,$ty pe,$ndims,@dimids,$natts);=0A NetCDF::varinq($ncid,$i,\$var,\$type,\$ndims ,\@dimids,\$natts);=0A push @all_vars,$var;=0A # also remember the type: used by get_rec_val=0A $self->{_VAR_TYPES}{$var} =3D $type;=0A }=0A $self-
{_ALLVARS} =3D \@all_vars;=0A=0A # CREATE LIST OF RECORD VARIABLES=0A # cu
rrently only works if =0A if (NetCDF::diminq($ncid,$self->{_RECDIM},\$dim,\ $numrecs) =3D=3D 0){=0A # Get record zero and store var names=0A my @recv arids =3D ();=0A my @recsizes =3D ();=0A my $nrecvars;=0A NetCDF::recinq ($self->{_NCID}, $nrecvars, \@recvarids, \@recsizes) =3D=3D 0 ||=0A cluck "Couldn't inquire about record variables\n";=0A=0A my @varnames =3D (); =0A if ($numrecs > 0){=0A my @record =3D ();=0A NetCDF::recget($self-> {_NCID}, 0, \@record) =3D=3D 0 || cluck "Couldn't get record zero\n";=0A # So we can immediately call get_next_rec()=0A $self->{_CURRENT_RECORD_NU M} =3D -1;=0A for my $k (0 ... $#record){=0A my $varid =3D $recvarids[ $k];=0A my ($var,$type,$ndims,@dimids,$natts);=0A NetCDF::varinq($nci d,$varid,\$var,\$type,\$ndims,\@dimids,\$natts);=0A push @varnames, $var ;=0A }=0A }=0A $self->{_RECVARS} =3D \@varnames;=0A }=0A=0A=0A return $self->{_NCID}; # return 'true' (!=3D0) value if succeeded=0A}=0A=0Asub clo se {=0A my $self =3D shift;=0A=0A unless($self->{_NCID}){=0A cluck "don't know how we can close NetCDF with NCID=3D" . $self->{_NCID};=0A return;=0A }=0A=0A my $rs =3D NetCDF::close($self->{_NCID});=0A=0A return !$rs;=0A} =0A=0Asub numrecs { # return number of records=0A my $self =3D shift;=0A r eturn $self->{_NUMRECS};=0A}=0A=0A=3Dhead2 get_rec_vars=0A=0AGet the list o f record variables for this file.=0A=0A=3Dcut=0Asub get_rec_vars {=0A my $s elf =3D shift;=0A my @vars =3D @{$self->{_RECVARS}};=0A=0A return @vars;=0A }=0A=0A=3Dhead2 get_all_vars=0A=0AGet the list of all variables for this fi le.=0A=0A=3Dcut=0A=0Asub get_all_vars {=0A my $self =3D shift;=0A my @vars =3D @{$self->{_ALLVARS}};=0A=0A return @vars;=0A}=0A=0A=0A=3Dhead2 get_gatt =0A=0AGet the values of a global attribute and return as a list. This is u seful=0Awhen you just want to extract the value of a given attribute. NOTE : it returns=0Aa list, even if there is only one value.=0A=0A=3Dcut=0Asub g et_gatt {=0A my ($self,$attr) =3D @_;=0A my ($type,$len);=0A=0A # First get the type of the global attribute so we know how to handle the result=0A my $ret =3D NetCDF::attinq($self->{_NCID}, NetCDF::GLOBAL, $attr, $type, $len );=0A=0A my @values =3D (); # NetCDF::attget will core if this is an undef =0A NetCDF::attget($self->{_NCID},NetCDF::GLOBAL,$attr,\@values);=0A @value s =3D perlify_values($type,\@values);=0A=0A #print "gvalues(" . scalar @val ues . ")=3D(" . join(",",@values) . ")\n";=0A return @values;=0A}=0A=0A=3Dh ead2 get_all_gatts=0A=0AGet all global attributes and return as a hash.=0A =0ASample:=0A my %gatts =3D $nc->get_all_gatts();=0A=0A foreach my $gatt ( sort keys %gatts){=0A my @vals =3D @{$gatts{$gatt}};=0A print "global att $gatt =3D " . join(",",@vals) . "\n";=0A }=0A=0A=3Dcut=0Asub get_all_gatts {=0A my $self =3D shift;=0A=0A my %gatts =3D ();=0A=0A for my $i (0 ... ($ self->{_NGATTS} - 1)){=0A=0A my ($name,$value,$type,$len);=0A NetCDF::att name($self->{_NCID}, NetCDF::GLOBAL, $i, \$name);=0A=0A # First get the ty pe of the global attribute so we know how to present=0A # the result=0A m y $ret =3D NetCDF::attinq($self->{_NCID}, NetCDF::GLOBAL, $name, $type, $le n);=0A=0A my @values =3D (); # NetCDF::attget will core if this is an unde f=0A NetCDF::attget($self->{_NCID},NetCDF::GLOBAL,$name,\@values);=0A @va lues =3D perlify_values($type,\@values);=0A $gatts{$name} =3D \@values;=0A }=0A=0A return %gatts;=0A}=0A=0A=3Dhead2 get_var_att=0A=0APass a name of a variable and the name of its attribute, and get back the value=0Aof the at tribute. This also checks to see if there is such an attribute, since=0Ath e charming NetCDF library will DIE if you try to look at a non-existent=0Aa ttribute. NOTE: it returns a list, even if there is only one value.=0A=0A =3Dcut=0Asub get_var_att {=0A=0A my ($self,$var,$attr) =3D @_;=0A my @value s =3D (); # NetCDF::attget will core if this is an undef=0A=0A # GET VARIAB LE ID=0A # which will kill you dead unless you have a valid variable=0A if ( !( grep { $_ eq $var; } @{$self->{_ALLVARS}}) ){=0A return undef;=0A } =0A my $varid =3D NetCDF::varid($self->{_NCID}, $var);=0A=0A # Now get the number of attributes it has=0A my ($var,$vtype,$ndims,@dimids,$natts);=0A N etCDF::varinq($self->{_NCID},$varid,\$var,\$vtype,\$ndims,\@dimids,\$natts) ;=0A #warn "varinq =3D $var,$vtype,$ndims,(" . join(",",@dimids) . "),$natt s" if $GoMOOS::DEBUG;=0A=0A # Find the correct attribute id... you'll see w hy in a minute=0A my $attid =3D -1;=0A for my $j (0 ... ($natts - 1)){=0A my $name;=0A NetCDF::attname($self->{_NCID}, $varid, $j, \$name);=0A if ( $name eq $attr){=0A $attid =3D $j;=0A #warn "$j:name =3D $name\n";=0A last;=0A }=0A }=0A # Return an empty list if there is no such attribute. =0A if ($attid =3D=3D -1){ return (); }=0A #warn "attid =3D $attr\n" if $Go MOOS::DEBUG;=0A=0A # Now go after the attribute itself.=0A my ($type,$len); =0A # This one dies with an untrappable error if $attr is bogus... (eval ju st dies)=0A # That's what all the above work is for!=0A NetCDF::attinq($sel f->{_NCID}, $varid, $attr, $type, $len);=0A # Now get the actual values=0A NetCDF::attget($self->{_NCID},$varid,$attr,\@values);=0A @values =3D perlif y_values($type,\@values);=0A=0A # Return single value as scalar for conveni ence=0A if (@values =3D=3D 1){=0A my $retval =3D $values[0];=0A return $r etval;=0A }=0A=0A #warn "attr values(" . scalar @values . ")=3D" . join("," ,@values) if $GoMOOS::DEBUG;=0A return @values;=0A}=0A=0A=3Dhead2 get_rec =0A=0APassed a record number: 0 to numrecs - 1 get the record and store it in _CURRENT_RECORD_NUM.=0ASee get_rec_val().=0A=0A=3Dcut=0A=0A=0Asub get_re c {=0A my ($self,$recnum, %opts) =3D @_;=0A=0A if( $self->{_NUMRECS} =3D=3D 0){=0A cluck "NO RECORDS";=0A return 0;=0A }=0A=0A if ($recnum < 0 || $r ecnum >=3D $self->{_NUMRECS}){=0A if (defined($self->{_NCID})){=0A unles s( $opts{INTERNAL}){=0A cluck "Invalid record number: $recnum";=0A } =0A cluck "Invalid record number: $recnum\n";=0A $self->{_CURRENT_RECOR D_NUM} =3D -1;=0A }=0A else {=0A #warn "No NCID: file not open!! (recnu m =3D $recnum)\n";=0A cluck "No NCID: file not open!! (recnum =3D $recnum )\n";=0A $self->{_CURRENT_RECORD_NUM} =3D -1;=0A }=0A return 0;=0A }=0A =0A # GET RECORD=0A my @current_record =3D ();=0A my $status =3D NetCDF::re cget($self->{_NCID}, $recnum, \@current_record);=0A if($status){=0A cluck "Could not get record $recnum\n";=0A $self->{_CURRENT_RECORD_NUM} =3D -1; =0A return 0;=0A }=0A=0A $self->{_CURRENT_RECORD_NUM} =3D $recnum;=0A $sel f->{_CURRENT_RECORD} =3D \@current_record;=0A=0A return 1;=0A}=0A=0A=3Dhead 2 get_rec_val=0A=0AReturn the values from current record returned by the la st call to get_rec().=0APass a variable name and get a scalar if the variab le reference is a scalar or=0Aif it is an array with only one value, also r eturn a scalar. If it is a reference to an=0Aarray just return the array re ference.=0A Sample:=0A my $sclr =3D $nc->get_rec_val('temperature_qc');=0A # time is an array but with only one value.=0A my $sclr_time =3D $nc->get_r ec_val('time');=0A Note: @arry =3D $nc->get_rec_val('time') will also work, arry will have one value.=0A my @arry =3D @{$nc->get_rec_val('current_u')} ;=0A=0A=3Dcut=0A=0Asub get_rec_val {=0A my ($self,$var) =3D @_;=0A my @recv ars =3D @{$self->{_RECVARS}};=0A my @current_record =3D @{$self->{_CURRENT_ RECORD}};=0A my $type =3D $self->{_VAR_TYPES}{$var};=0A die "WHAT? NO RECOR D VARIABLES in $self->{_FILE}: (@recvars)" unless @recvars;=0A=0A my $retva l =3D undef;=0A for my $k (0 ... $#recvars){=0A if ($recvars[$k] eq $var){ =0A $retval =3D perlify_values($type,$current_record[$k]);=0A last;=0A }=0A }=0A #if (!defined($retval)){ cluck "did not find $var\n"; }=0A=0A re turn $retval;=0A}=0A=0Asub get_next_rec {=0A my $self =3D shift;=0A=0A if( $self->{_NUMRECS} =3D=3D 0){ return 0; }=0A=0A $self->{_CURRENT_RECORD_NUM} ++;=0A=0A if ($self->{_CURRENT_RECORD_NUM} >=3D $self->{_NUMRECS}){ return 0; }=0A=0A return $self->get_rec($self->{_CURRENT_RECORD_NUM}, INTERNAL=3D> 1);=0A}=0A=0Asub get_prev_rec { =0A my $self =3D shift;=0A=0A if( $self->{_ NUMRECS} =3D=3D 0){ return 0; }=0A=0A $self->{_CURRENT_RECORD_NUM}--;=0A=0A if ($self->{_CURRENT_RECORD_NUM} < 0){ return 0; }=0A=0A return $self->get _rec($self->{_CURRENT_RECORD_NUM}, INTERNAL=3D>1);=0A}=0A=0Asub get_rewind { # reset get next record pointer=0A my ($self,%opts) =3D @_;=0A=0A if (def ined($opts{REVERSE})){=0A $self->{_CURRENT_RECORD_NUM} =3D $self->{_NUMREC S};=0A }=0A else {=0A $self->{_CURRENT_RECORD_NUM} =3D 0;=0A }=0A=0A retur n;=0A=0A}=0A=0A=3Dhead2 get_var ($var,[START=3D>$start,END=3D>$end,COUNT=3D
])=0A=0AReturn value(s) corresponding to a NetCDF variable. This accesses
variables in=0Aan array format, as opposed to the record format.=0A=0AThe parameters are: START, which tells which value (record) to start at (0 is t he=0Afirst); TO, which gives the index of the last value to return; and COU NT, which=0Atells a number of records to return. If none are specified, th e first record is=0Areturned. If both COUNT and TO are specified, COUNT ta kes precedence. If TO=0Ais -1, then values up to the last one are returned .=0A=0AIf this is called in a scalar context, then the first value is retur ned.=0AOtherwise, an array is returned. Caller beware.=0A=0A$start and $co unt are naturally an offset and a number; they default to 0 and 1,=0Awhich gives the first value only, which is useful when there is only one, which =0Ais the only time you wouldn't want to speficy it anyway. If $count is - 1 then it=0Aretrieves the entire array.=0A=0A=3Dcut=0Asub get_var {=0A my ( $self,$var,%range) =3D @_;=0A my @values =3D (); # NetCDF::varget might cor e if this is an undef=0A=0A # careful... NetCDF::varget will CORE if start or count are undef...=0A my ($start,$count);=0A $start =3D 0 if not defined ($range{START});=0A if ($range{COUNT}){=0A $count =3D $range{COUNT};=0A } =0A elsif ($range{TO}){=0A $count =3D $range{TO} - $range{START};=0A }=0A $count =3D 1 if not $count;=0A=0A # GET VARIABLE ID=0A # which will kill yo u dead unless you have a valid variable, so we check that=0A # it's valid b efore we do anything with it=0A if ( !( grep { $_ eq $var; } @{$self->{_ALL VARS}}) ){=0A return undef;=0A }=0A my $varid =3D NetCDF::varid($self->{_N CID}, $var);=0A=0A # GET NUMBER OF RECORDS IF POSSIBLE AND ADJUST COUNT IF NECESSARY=0A if ($self->{_DIM}{$var}){=0A if ($self->{_DIM}{$var} < ($star t + $count) or=0A ($range{TO} and not $range{COUNT})){=0A $count =3D $ self->{_DIM}{$var} - $start;=0A }=0A }=0A=0A # NOW CHECK START, COUNT, ETC .=0A # NOW GET THE VALUES=0A NetCDF::varget($self->{_NCID},$varid,\$start,\ $count,\@values);=0A # And don't forget to perlify them=0A @values =3D perl ify_values($self->{_VAR_TYPES}{$var},\@values);=0A=0A # Return single value as scalar for convenience=0A if (not wantarray){=0A my $retval =3D $value s[0];=0A # see get_rec_val: here it could also be a string, so we vary the test...=0A if (not $retval =3D~ /\w/ and $retval =3D=3D 0){ return 0; } =0A return $retval;=0A }=0A=0A return @values;=0A}=0A=0A=3Dhead2 perlify_v alues=0A=0AThe purpose of this function is to prettify the values extracted from NetCDF=0Afiles into something more like what a Perl programmer expect s - strings or=0Anumbers, not arrays of numbers representing char values or bytes that don't look=0Alike numbers.=0A=0AThis expects two arguments: the value's NetCDF type, and a reference to the=0Avalues. The values can be e ither an array OR scalar, depending on whether we've=0Ajust gotten the cont ents of a NetCDF variable - which itself can be either - or=0Athe content o f a variable's attribute.=0A=0AThis function returns either an array or a r eference to an array, e.g.=0A=0A @get_array =3D perlify_values($type,\@valu es);=0A $get_ref =3D perlify_values($type,\@values);=0A @use_array =3D @$ get_ref;=0A=0A=3Dcut=0Asub perlify_values {=0A my ($type,$input) =3D @_;=0A my @vals;=0A my $result;=0A=0A # the values can be an array reference, a s calar reference, or a scalar.=0A # just don't pass arrays.=0A if (ref($i nput) eq 'ARRAY') { @vals =3D @$input }=0A elsif (ref($input) eq 'SCALAR' ){ @vals =3D ($$input) }=0A else { @vals =3D ($input) }=0A=0A if ($type =3D=3D NetCDF::CHAR){=0A # weed out nulls, which can sneak in because NetC DF stores attribute strings=0A # as arrays of chars, not as null-terminate d strings=0A @vals =3D grep { $_ !=3D 0 } @vals;=0A # we also split it ba ck into an array of lines, which is the perl-ish way to=0A # look at strin gs=0A $result =3D [split "\n",(pack( 'c' x @vals, @vals))];=0A }=0A # the numeric types are easy... the input is the same as the output=0A elsif ($t ype =3D=3D NetCDF::FLOAT){ $result =3D $input }=0A elsif ($type =3D=3D NetC DF::DOUBLE){ $result =3D $input }=0A elsif ($type =3D=3D NetCDF::SHORT){ $r esult =3D $input }=0A elsif ($type =3D=3D NetCDF::LONG){ $result =3D $inpu t }=0A # except this one doesn't do what you'd expect in perl=0A elsif ($ty pe =3D=3D NetCDF::BYTE){=0A $result =3D [map { unpack('C',$_) } @vals];=0A }=0A else {=0A die "UNKNOWN NetCDF type=3D'$type'";=0A }=0A=0A return (wa ntarray ? @$result : (@$result =3D=3D 1 ? $result->[0] : $result));=0A}=0A =0A=3Dhead1 VERSION=0A=0ARevsion: $Revision: 1.29 $=0A=0A=3Dcut=0A=0A1;=0A =0A - --SLDf9lqlvOQaIe6s-- ------------------------------ End of ncdigest V1 #724 ***********************
decoders
archives: