[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

[netCDF #FRX-290710]: Novice help with cxx4

Hi Tom,

Sorry to have taken so long to respond to your question ...
> My name is Tom Niedzielski.  We're evaluating NetCDF as a storage
> format for our raster data.  So far, we're impressed with its
> efficiency, ease of use, and versatility compared to our in-house
> solution.  We've achieved these results through the C library.  We
> would like to do the same with the cxx4 library, but I'm having
> trouble.
> 
> Before I show you the code, let me explain what I'm trying to do.  I'm
> getting random integers from test_fstream, loading them into a 2D
> array of integers, then writing compressed data to a file, row by row.
> After writing, I read and discard the data from the file I wrote to.
> 
> The issue:
> When I try to read the data back, I get an error in the HDF5 layer.
> This does not happen if I'm not compressing.  Looking at the file
> itself, I see that nothing but fill values were written to the
> variable "value".  I had this problem while working with the C code,
> but that time I quickly found that it was caused by a type mismatch
> between the variable and the data I was trying to write to it.  Here,
> I'm having trouble seeing the mismatch, if that's even the cause.

The netcdf-cxx4 library was developed and contributed to Unidata by a
third-party developer (Lynton Appel of the Culham Centre for Fusion Energy 
(CCFE) in Oxfordshire, UK).  It is still somewhat experimental, and we
don't have the resources at Unidata to provide good support for the
package yet.

I forwarded your question to Lynton, and here is his reply:

  I had a look at this, but the programme wasn't complete and it
  required a more modern C++ than
  I have here.

  I modified the code to try to get it to work.  I didn't succeed, but
  include the code as I left it.
  If I have a working example, I can probably get back to it.

I've attached the modified code (frx.cc), in case that's of any use to 
you.  I assume you are already familiar with the netcdf-cxx4 documentation
we make available:

  http://www.unidata.ucar.edu/software/netcdf/docs/cxx4/

I'm afraid that's all the help we can provide, unless you can send me a
self-contained example that demonstrates the problem in a way that it
will be easy for Lynton to work on.

--Russ



> I've included the code below, with minor modifications in /* */:
> 
> std::pair<uint64_t, uint64_t> test_random_ints_ncpp(
> std::fstream& test_fstream,
> uint_fast16_t num_rows,
> uint_fast16_t num_cols )
> {
> //This pair stores the read and write times for this test, in
> //that order.
> std::pair<uint64_t, uint64_t> times;
> 
> //Begin NetCDF C++ library testing.
> test_fstream.seekg( 0 );
> 
> //Create a new file.
> /*
> A temporary file is created for testing purposes using an
> in-house library.
> The path is temp_path
> */
> netCDF::NcFile ncpp_file( temp_path, netCDF::NcFile::replace );
> 
> //Arrange the dimensions.
> ncpp_file.addDim( "height", num_rows );
> ncpp_file.addDim( "width", num_cols );
> netCDF::NcDim dim_h = ncpp_file.getDim( "height" );
> netCDF::NcDim dim_w = ncpp_file.getDim( "width" );
> 
> //Set up the variable.
> //( row, col ), col varies fastest.
> std::vector<netCDF::NcDim> dim_vec { dim_h, dim_w };
> netCDF::NcVar value = ncpp_file.addVar( "value", netCDF::ncInt, dim_vec );
> 
> //The data will be compressed.
> value.setCompression( true, true, 1 );
> 
> //Set up chunked data access.
> std::vector<size_t> chunks { 1, num_cols };
> value.setChunking( netCDF::NcVar::nc_CHUNKED, chunks );
> 
> //Assign fill value.
> value.setFill( true, -1 );
> 
> //This conversion should not be included in the timing.
> std::vector<std::vector<int32_t> > ncpp_rows;
> ncpp_rows.reserve( num_rows );
> for ( uint_fast16_t row = 0; row < num_rows; ++row )
> {
> std::vector<int32_t> row_vec;
> row_vec.reserve( num_cols );
> //Tokenize the ints from the fstream and put them in
> //the vector.
> for ( uint_fast16_t column = 0; column < num_cols; ++column )
> {
> std::string text_int;
> std::getline( test_fstream, text_int, ' ');
> int32_t i = atoi( text_int.c_str() );
> row_vec.emplace_back( i );
> }
> //Store the row and move to the next line.
> ncpp_rows.emplace_back( row_vec );
> test_fstream.ignore( 1, '\n' );
> }
> 
> //Now write the rows.
> 
> //Remember, value's dimensions are ( row, column ).
> for ( uint_fast16_t cur_row = 0; cur_row < num_rows; ++cur_row )
> {
> std::vector<uint64_t> startp { cur_row, 0 };
> std::vector<uint64_t> countp { 1, num_cols };
> std::vector<int32_t> row_vec = ncpp_rows[cur_row];
> value.putVar( startp, countp, row_vec.data() );
> }
> 
> //NetCDF C++ Write test complete.
> /* times.second set */
> 
> /*
> The file is saved here for later examination.
> */
> 
> //Now the read test begins.
> netCDF::NcFile read_file ( temp_path, netCDF::NcFile::read );
> netCDF::NcVar read_value = read_file.getVar( "value" );
> 
> //We will not decompress before reading.
> for ( uint_fast16_t cur_row = 0; cur_row < num_rows; ++cur_row )
> {
> std::vector<uint64_t> start { cur_row, 0 };
> std::vector<uint64_t> count { 1, num_cols };
> uint64_t get_row[num_cols];
> read_value.getVar( start, count, get_row );
> }
> /* times.first set */
> 
> //Now the tests are complete.
> return times;
> }
> 
> 
Russ Rew                                         UCAR Unidata Program
address@hidden                      http://www.unidata.ucar.edu



Ticket Details
===================
Ticket ID: FRX-290710
Department: Support netCDF
Priority: High
Status: Closed
#include <blitz/array.h>
#include <string>
#include <iostream>
#include <netcdf>
extern "C" {
#include <sys/param.h>
#include <sys/times.h>
#include <sys/types.h>
}
#include <stdint.h>
#include <syslog.h>
#include <linux/stddef.h>
#include <pthread.h>
#include <unistd.h>
#include <iostream>
#include <fstream>

using namespace std;
using namespace netCDF::exceptions;

std::pair<uint64_t, uint64_t> test_random_ints_ncpp(
                              std::fstream& test_fstream,
                              uint_fast16_t num_rows,
                              uint_fast16_t num_cols )
{
    //This pair stores the read and write times for this test, in
    //that order.
    std::pair<uint64_t, uint64_t> times;

    //Begin NetCDF C++ library testing.
    test_fstream.seekg( 0 );

    //Create a new file.
    /*
        A temporary file is created for testing purposes using an
in-house library.
        The path is temp_path
    */
    string temp_path="netCDFFile.nc";
    netCDF::NcFile ncpp_file( temp_path, netCDF::NcFile::replace );

    //Arrange the dimensions.
    ncpp_file.addDim( "height", num_rows );
    ncpp_file.addDim( "width", num_cols );
    netCDF::NcDim dim_h = ncpp_file.getDim( "height" );
    netCDF::NcDim dim_w = ncpp_file.getDim( "width" );

    //Set up the variable.
    //( row, col ), col varies fastest.
    std::vector<netCDF::NcDim> dim_vec;
    dim_vec.push_back(dim_h);
    dim_vec.push_back(dim_w);
    netCDF::NcVar value = ncpp_file.addVar( "value", netCDF::ncInt,
dim_vec );

    //The data will be compressed.
    value.setCompression( true, true, 1 );

    //Set up chunked data access.
    std::vector<size_t> chunks;
    chunks.push_back(1);
    chunks.push_back(num_cols);
    value.setChunking( netCDF::NcVar::nc_CHUNKED, chunks );

    //Assign fill value.
    value.setFill( true, -1 );

    //This conversion should not be included in the timing.
    std::vector<std::vector<int32_t> > ncpp_rows;
    ncpp_rows.reserve( num_rows );
    for ( uint_fast16_t row = 0; row < num_rows; ++row )
    {
        std::vector<int32_t> row_vec;
        row_vec.reserve( num_cols );
        //Tokenize the ints from the fstream and put them in
        //the vector.
        for ( uint_fast16_t column = 0; column < num_cols; ++column )
        {
            std::string text_int;
            std::getline( test_fstream, text_int, ' ');
            int32_t i = atoi( text_int.c_str() );
            row_vec.push_back( i );
        }
        //Store the row and move to the next line.
        ncpp_rows.push_back( row_vec );
        test_fstream.ignore( 1, '\n' );
    }

    //Now write the rows.

    //Remember, value's dimensions are ( row, column ).
    for ( uint_fast16_t cur_row = 0; cur_row < num_rows; ++cur_row )
    {
      /*
      std::vector<uint64_t> startp;
      startp.push_back(cur_row);
      startp.push_back(0);
      std::vector<uint64_t> countp;
      countp.push_back(1);
      countp.push_back(num_cols);
      std::vector<int32_t> row_vec = ncpp_rows[cur_row];
      value.putVar( startp, countp, row_vec.data() );
      */
      std::vector<uint> startp;
      startp.push_back(cur_row);
      startp.push_back(0);
      std::vector<uint> countp;
      countp.push_back(1);
      countp.push_back(num_cols);
      std::vector<int32_t> row_vec = ncpp_rows[cur_row];
      value.putVar( startp, countp, row_vec.data() );
    }

    //NetCDF C++ Write test complete.
    /* times.second set */

    /*
        The file is saved here for later examination.
    */

    //Now the read test begins.
    netCDF::NcFile read_file ( temp_path, netCDF::NcFile::read );
    netCDF::NcVar read_value = read_file.getVar( "value" );

    //We will not decompress before reading.
    for ( uint_fast16_t cur_row = 0; cur_row < num_rows; ++cur_row )
    {
      /*
      std::vector<uint64_t> start;
      start.push_back(cur_row);
      start.push_back(0);
      std::vector<uint64_t> count;
      count.push_back(1);
      count.push_back(num_cols);
      */
      std::vector<uint> start;
      start.push_back(cur_row);
      start.push_back(0);
      std::vector<uint> count;
      count.push_back(1);
      count.push_back(num_cols);
      uint64_t get_row[num_cols];
      read_value.getVar( start, count, get_row );
    }
    /* times.first set */

    //Now the tests are complete.
    return times;
}



int main(int argc, char* argv[])
{
  fstream is;
  test_random_ints_ncpp(is,10,10);
  return 0;
}