Hi Christian,
I'm still trying to sort out the issue of parsing .grib2 files. I need to
be able to open .grib2 files without using the netcdfAll bundle. I'm
taking the approach of gathering all the CDM Dependancies
<http://www.unidata.ucar.edu/software/thredds/v4.5/netcdf-java/reference/JarDependencies.html#logging>
from
Maven Central, however, I'm still getting errors. Have you successfully
opened a .grib2 files by providing all necessary .jar files at your shell
command? If so, could you pass along an example?
Many thanks,
Annie
On Wed, Jul 30, 2014 at 10:00 AM, <netcdf-java-request@xxxxxxxxxxxxxxxx>
wrote:
> Send netcdf-java mailing list submissions to
> netcdf-java@xxxxxxxxxxxxxxxx
>
> To subscribe or unsubscribe via the World Wide Web, visit
> http://mailman.unidata.ucar.edu/mailman/listinfo/netcdf-java
> or, via email, send a message with subject or body 'help' to
> netcdf-java-request@xxxxxxxxxxxxxxxx
>
> You can reach the person managing the list at
> netcdf-java-owner@xxxxxxxxxxxxxxxx
>
> When replying, please edit your Subject line so it is more specific
> than "Re: Contents of netcdf-java digest..."
>
>
> Today's Topics:
>
> 1. Re: grib2 (Dennis Heimbigner)
>
>
> ----------------------------------------------------------------------
>
> Message: 1
> Date: Tue, 29 Jul 2014 15:04:15 -0600
> From: Dennis Heimbigner <dmh@xxxxxxxx>
> To: Christian Ward-Garrison <cwardgar@xxxxxxxx>
> Cc: Annie Bryant <anniebryant.burgess@xxxxxxxxx>,
> netcdf-java@xxxxxxxxxxxxxxxx
> Subject: Re: [netcdf-java] grib2
> Message-ID: <53D80C4F.3010007@xxxxxxxx>
> Content-Type: text/plain; charset=UTF-8; format=flowed
>
> There may be a way to fix this.
> You might 'unjar' tika-app-1.6-SNAPSHOT.jar
> and replace the old netcdf in that jar with the
> newer version, then rejar tika-app-1.6-SNAPSHOT.jar.
> Assuming no interface problems, this has a fair chance
> to work.
> =Dennis Heimbigner
> Unidata
>
> Christian Ward-Garrison wrote:
> > Hi Annie,
> >
> > It turns out that this is an issue with Tika. tika-app-1.6-SNAPSHOT.jar
> > actually includes netcdf-4.2.20, so you have 2 different versions of the
> > same library on the classpath, which is always bad. And unfortunately,
> > using Tika's bundled netcdf-4.2.20 alone won't work, because support for
> > reading GRIB files was only added to NetCDF-Java in 4.3+.
> >
> > This may be a problem that only the Tika developers know how to fix. I
> > suggest opening a ticket with them.
> >
> > Good luck!
> > Christian Ward-Garrison
> >
> >
> > On Mon, Jul 28, 2014 at 3:19 PM, Annie Burgess <anniebryant@xxxxxxxxx>
> > wrote:
> >
> >> Hi Christian,
> >>
> >> Your code words great - no problems when I compile and run. I've
> modified
> >> my code to use NetcdfDataset rather than NetcdfFile to open the .grib2
> >> file. However, I'm still getting an error in my code:
> >>
> >> % javac -classpath
> >>
> ../../../../tika-core/target/tika-core-1.6-SNAPSHOT.jar:../../../../netcdfAll-4.3.jar
> >> org/apache/tika/parser/grib/GribParser.java
> >>
> >> % java -classpath
> >>
> tika-app/target/tika-app-1.6-SNAPSHOT.jar:annie-parsers.jar:netcdfAll-4.3.jar
> >> org.apache.tika.cli.TikaCLI --text gdas1.forecmwf.2014062612.grib2
> >>
> >>
> >> -----------------------GripParser.java------------------------
> >> import java.io.ByteArrayOutputStream;
> >> import java.io.IOException;
> >> import java.io.InputStream;
> >> import java.util.Collections;
> >> import java.util.Set;
> >> import java.io.File;
> >>
> >> import org.apache.tika.exception.TikaException;
> >>
> >> import org.apache.tika.mime.MediaType;
> >> import org.apache.tika.parser.AbstractParser;
> >> import org.apache.tika.parser.ParseContext;
> >> import org.apache.tika.parser.Parser;
> >> import org.apache.tika.sax.XHTMLContentHandler;
> >> import org.xml.sax.ContentHandler;
> >> import org.xml.sax.SAXException;
> >>
> >> import ucar.nc2.NetcdfFile;
> >> import ucar.nc2.dataset.NetcdfDataset;/**
> >>
> >>
> >> public class GribParser extends AbstractParser {
> >>
> >> private final Set<MediaType> SUPPORTED_TYPES =
> >> Collections.singleton(MediaType.application("x-grib2"));
> >>
> >> public Set<MediaType> getSupportedTypes(ParseContext context) {
> >> return SUPPORTED_TYPES;
> >> }
> >>
> >> public void parse(InputStream stream, ContentHandler handler,
> >> Metadata metadata, ParseContext context) throws IOException,
> >> SAXException, TikaException {
> >>
> >> System.err.println(" Check 1 ");
> >>
> >> File gribFile = new File("gdas1.forecmwf.2014062612.grib2");
> >>
> >> NetcdfFile ncFile =
> NetcdfDataset.openFile(gribFile.getAbsolutePath(),
> >> null);
> >>
> >> System.err.println(" Check 2 ");
> >>
> >> }
> >> }
> >>
> >> ---------------------OUTPUT---------------------
> >>
> >> Check 1
> >> Exception in thread "main" org.apache.tika.exception.TikaException:
> >> TIKA-198: Illegal IOException from
> >> org.apache.tika.parser.grib.GribParser@483ad415
> >> at
> org.apache.tika.parser.CompositeParser.parse(CompositeParser.java:249)
> >>
> >> at
> org.apache.tika.parser.CompositeParser.parse(CompositeParser.java:243)
> >> at
> org.apache.tika.parser.AutoDetectParser.parse(AutoDetectParser.java:121)
> >> at org.apache.tika.cli.TikaCLI$OutputType.process(TikaCLI.java:141)
> >> at org.apache.tika.cli.TikaCLI.process(TikaCLI.java:420)
> >> at org.apache.tika.cli.TikaCLI.main(TikaCLI.java:111)
> >> Caused by: java.io.IOException: Cant read
> >> /Users/IGSWAHWSWBURGESS/Development/tikadev/tika/gdas1.forecmwf.
> 2014062612.grib2:
> >> not a valid CDM file.
> >> at ucar.nc2.NetcdfFile.open(NetcdfFile.java:734)
> >> at ucar.nc2.NetcdfFile.open(NetcdfFile.java:384)
> >> at
> ucar.nc2.dataset.NetcdfDataset.openOrAcquireFile(NetcdfDataset.java:687)
> >> at ucar.nc2.dataset.NetcdfDataset.openFile(NetcdfDataset.java:564)
> >> at org.apache.tika.parser.grib.GribParser.parse(GribParser.java:82)
> >>
> >> at
> org.apache.tika.parser.CompositeParser.parse(CompositeParser.java:243)
> >> ... 5 more
> >>
> >> ---------------------------------------------
> >>
> >> netcdfALL and .grib2 files are in the same director where I am running
> >> GribParser.java. This could totally be a classpath issue... I'm just
> >> stumped.
> >>
> >> Any other ideas?
> >>
> >> Thanks!
> >> Annie
> >>
> >>
> >> On Mon, Jul 28, 2014 at 12:40 PM, Christian Ward-Garrison <
> >> cwardgar@xxxxxxxx> wrote:
> >>
> >>> Hi Annie,
> >>>
> >>> I see you're using netcdfAll-4.3.jar. That actually already contains
> the
> >>> grib module, so it should be all you need. I created a minimal example:
> >>>
> >>> ------------------------------------- Foo.java
> >>> -------------------------------------
> >>>
> >>> import java.io.IOException;
> >>> import java.io.File;
> >>> import ucar.nc2.NetcdfFile;
> >>> import ucar.nc2.dataset.NetcdfDataset;
> >>>
> >>> public class Foo {
> >>> public static void main(String[] args) throws IOException {
> >>> File gribFile = new File("foo.grib2");
> >>> NetcdfFile ncFile =
> >>> NetcdfDataset.openFile(gribFile.getAbsolutePath(), null);
> >>> try {
> >>> System.out.println(ncFile.toString());
> >>> } finally {
> >>> ncFile.close();
> >>> }
> >>> }
> >>> }
> >>>
> >>> ------------------------------------- Shell commands
> >>> -------------------------------------
> >>>
> >>> javac -cp netcdfAll-4.3.jar Foo.java
> >>>
> >>> java -cp .;netcdfAll-4.3.jar Foo
> >>>
> >>>
> >>> That should work as long as netcdfAll-4.3.jar and a file named
> >>> "foo.grib2" are in the same directory as Foo.java. If you move things,
> >>> you'll obviously need to modify the commands. Does this example work
> for
> >>> you?
> >>>
> >>> Cheers,
> >>> Christian
> >>>
> >>>
> >>>
> >>> On Mon, Jul 28, 2014 at 1:52 PM, Annie Burgess <anniebryant@xxxxxxxxx>
> >>> wrote:
> >>>
> >>>> Hi Christian,
> >>>>
> >>>> Thanks for your response. I've cut down the code (pasted below) to a
> >>>> sort of bare-bones version that is ONLY trying to open the .grib2
> file as
> >>>> if it were a .nc file.
> >>>>
> >>>> I build apache tika from:
> >>>>
> >>>> svn co http://svn.apache.org/repos/asf/tika/trunk tika
> >>>> mvn install
> >>>>
> >>>> I pulled netcdfAll and toolsUI .jar files from:
> >>>> http://www.unidata.ucar.edu/downloads/netcdf/netcdf-java-4/index.jsp
> >>>>
> >>>> I pulled the grib .jar from:
> >>>> http://mvnrepository.com/artifact/edu.ucar/grib/8.0.29
> >>>>
> >>>> I compile the code as:
> >>>> [asc-227-196:src/main/java] AB% javac -classpath
> >>>>
> ../../../../tika-core/target/tika-core-1.6-SNAPSHOT.jar:../../../../toolsUI-4.3.jar:../../../../netcdfAll-4.3.jar:../../../../grib-8.0.29.jar
> >>>> org/apache/tika/parser/grib/GribParser.java
> >>>>
> >>>> I run the code as:
> >>>> [asc-227-196:~/Development/tikadev/tika] AB% java -classpath
> >>>>
> tika-app/target/tika-app-1.6-SNAPSHOT.jar:annie-parsers.jar:netcdfAll-4.3.jar:grib-8.0.29.jar:toolsUI-4.3.jar
> >>>> org.apache.tika.cli.TikaCLI --text gdas1.forecmwf.2014062612.grib2
> >>>>
> >>>> CODE:
> >>>>
> >>>> --------------------------------------------------
> >>>> package org.apache.tika.parser.grib;
> >>>>
> >>>> import java.io.ByteArrayOutputStream;
> >>>> import java.io.IOException;
> >>>> import java.io.InputStream;
> >>>> import java.util.Collections;
> >>>> import java.util.Set;
> >>>> import java.util.List;
> >>>> import java.util.Iterator;
> >>>>
> >>>> //JDK imports
> >>>> import org.apache.tika.exception.TikaException;
> >>>> import org.apache.tika.io.IOUtils;
> >>>> import org.apache.tika.metadata.Metadata;
> >>>> import org.apache.tika.metadata.Property;
> >>>> import org.apache.tika.metadata.TikaCoreProperties;
> >>>> import org.apache.tika.mime.MediaType;
> >>>> import org.apache.tika.parser.AbstractParser;
> >>>> import org.apache.tika.parser.ParseContext;
> >>>> import org.apache.tika.parser.Parser;
> >>>> import org.apache.tika.sax.XHTMLContentHandler;
> >>>> import org.xml.sax.ContentHandler;
> >>>> import org.xml.sax.SAXException;
> >>>>
> >>>> import ucar.grib.grib2.*;
> >>>> import ucar.nc2.*;
> >>>>
> >>>> /**
> >>>> * A {@link Parser} for <a
> >>>> * href="http://www.unidata.ucar.edu/software/netcdf/index.html
> >>>> ">NetCDF</a>
> >>>> * files using the UCAR, MIT-licensed <a
> >>>> * href="http://www.unidata.ucar.edu/software/netcdf-java/";>NetCDF
> for
> >>>> Java</a>
> >>>> * API.
> >>>> */
> >>>> public class GribParser extends AbstractParser {
> >>>>
> >>>> private final Set<MediaType> SUPPORTED_TYPES =
> >>>> Collections.singleton(MediaType.application("x-grib2"));
> >>>> /*
> >>>> * (non-Javadoc)
> >>>> *
> >>>> * @see
> >>>> *
> >>>> org.apache.tika.parser.Parser#getSupportedTypes(org.apache.tika.parser
> >>>> * .ParseContext)
> >>>> */
> >>>> public Set<MediaType> getSupportedTypes(ParseContext context) {
> >>>> return SUPPORTED_TYPES;
> >>>> }
> >>>> /*
> >>>> * (non-Javadoc)
> >>>> *
> >>>> * @see org.apache.tika.parser.Parser#parse(java.io.InputStream,
> >>>> * org.xml.sax.ContentHandler, org.apache.tika.metadata.Metadata,
> >>>> * org.apache.tika.parser.ParseContext)
> >>>> */
> >>>> public void parse(InputStream stream, ContentHandler handler,
> >>>> Metadata metadata, ParseContext context) throws
> IOException,
> >>>> SAXException, TikaException {
> >>>>
> >>>> System.err.println(" Check 1 ");
> >>>> String name = "/Users/IGSWAHWSWBURGESS/POLARCYBER/gdas1.forecmwf.
> >>>> 2014062612.grib2";
> >>>>
> >>>> if (name == null) {
> >>>> name = "";
> >>>> }
> >>>>
> >>>> NetcdfFile ncFile = NetcdfFile.open(name, null);
> >>>> System.err.println(" Check 2 ");
> >>>> }
> >>>> }
> >>>>
> >>>>
> >>>> OUTPUT:
> >>>>
> >>>> Check 1
> >>>>
> >>>> Exception in thread "main" org.apache.tika.exception.TikaException:
> >>>> Unexpected RuntimeException from
> >>>> org.apache.tika.parser.grib.GribParser@261a53b9
> >>>> at
> org.apache.tika.parser.CompositeParser.parse(CompositeParser.java:245)
> >>>> at
> org.apache.tika.parser.CompositeParser.parse(CompositeParser.java:243)
> >>>> at
> >>>>
> org.apache.tika.parser.AutoDetectParser.parse(AutoDetectParser.java:121)
> >>>> at org.apache.tika.cli.TikaCLI$OutputType.process(TikaCLI.java:141)
> >>>> at org.apache.tika.cli.TikaCLI.process(TikaCLI.java:420)
> >>>> at org.apache.tika.cli.TikaCLI.main(TikaCLI.java:111)
> >>>> Caused by: java.lang.RuntimeException: java.lang.NoSuchMethodError:
> >>>>
> ucar.grib.grib2.Grib2WriteIndex.writeGribIndex(Ljava/io/File;Ljava/lang/String;Lucar/unidata/io/RandomAccessFile;Z)Lucar/grid/GridIndex;
> >>>> at ucar.nc2.NetcdfFile.<init>(NetcdfFile.java:1326)
> >>>> at ucar.nc2.NetcdfFile.open(NetcdfFile.java:744)
> >>>> at ucar.nc2.NetcdfFile.openInMemory(NetcdfFile.java:670)
> >>>> at org.apache.tika.parser.grib.GribParser.parse(GribParser.java:93)
> >>>> at
> org.apache.tika.parser.CompositeParser.parse(CompositeParser.java:243)
> >>>> ... 5 more
> >>>> Caused by: java.lang.NoSuchMethodError:
> >>>>
> ucar.grib.grib2.Grib2WriteIndex.writeGribIndex(Ljava/io/File;Ljava/lang/String;Lucar/unidata/io/RandomAccessFile;Z)Lucar/grid/GridIndex;
> >>>> at
> >>>>
> ucar.nc2.iosp.grib.GribGridServiceProvider.writeIndex(GribGridServiceProvider.java:348)
> >>>> at
> >>>>
> ucar.nc2.iosp.grib.GribGridServiceProvider.getIndex(GribGridServiceProvider.java:292)
> >>>> at
> >>>>
> ucar.nc2.iosp.grib.GribGridServiceProvider.open(GribGridServiceProvider.java:118)
> >>>> at ucar.nc2.NetcdfFile.<init>(NetcdfFile.java:1308)
> >>>> ... 9 more
> >>>>
> >>>>
> >>>>
> >>>> Note, if I use a .nc file the code runs successfully.
> >>>>
> >>>> OUTPUT:
> >>>>
> >>>> Check 1
> >>>> Check 2
> >>>>
> >>>>
> >>>> I am sort of a java newbie, so please let me know if I've left out any
> >>>> critical information!
> >>>>
> >>>> Thank you for any help/insight you can give.
> >>>>
> >>>> Annie
> >>>>
> >>>>
> >>>> On Sun, Jul 27, 2014 at 10:44 PM, Christian Ward-Garrison <
> >>>> cwardgar@xxxxxxxx> wrote:
> >>>>
> >>>>> Hi Annie,
> >>>>>
> >>>>> This is the result of the GRIB module not being on the classpath when
> >>>>> you execute your Java program. Can you give me more details about
> your
> >>>>> setup? Can you provide you build file (Maven, Ant, Gradle, etc)?
> >>>>>
> >>>>> Cheers,
> >>>>> Christian
> >>>>>
> >>>>>
> >>>>> On Wed, Jul 23, 2014 at 5:06 PM, Annie Burgess <
> anniebryant@xxxxxxxxx>
> >>>>> wrote:
> >>>>>
> >>>>>> Greetings all,
> >>>>>>
> >>>>>> I am trying to create a script that will mimic the output of NCDump.
> >>>>>> I have successfully done this for NetCDF files, and now I am
> trying to
> >>>>>> apply it to grib2 files. I am using the NetCDF-java library in
> conjunction
> >>>>>> with Apache Tika to do this. Other posts have indicated I should
> be able
> >>>>>> to open my grib2 files, just as if they were .nc files. However, I
> >>>>>> continue to get the following error:
> >>>>>>
> >>>>>> "Caused by: java.io.IOException: Cant read gdas1.forecmwf.
> 2014062612.grib2:
> >>>>>> not a valid CDM file."
> >>>>>>
> >>>>>> To open the .nc files, this is the bit of code I use (with the
> >>>>>> exception of changing the .nc file to a .grib2 file):
> >>>>>>
> >>>>>> String name = "gdas1.forecmwf.2014062612.grib2";
> >>>>>>
> >>>>>> if (name == null) {
> >>>>>> name = "";
> >>>>>> }
> >>>>>>
> >>>>>> try NetcdfFile ncFile = NetcdfFile.openInMemory(name,
> >>>>>> os.toByteArray());
> >>>>>> // first parse out the set of global attributes
> >>>>>> for (Attribute attr : ncFile.getGlobalAttributes()) {
> >>>>>> Property property =
> resolveMetadataKey(attr.getName());
> >>>>>> if (attr.getDataType().isString()) {
> >>>>>> metadata.add(property, attr.getStringValue());
> >>>>>> } else if (attr.getDataType().isNumeric()) {
> >>>>>> int value = attr.getNumericValue().intValue();
> >>>>>> metadata.add(property, String.valueOf(value));
> >>>>>> }
> >>>>>> }
> >>>>>>
> >>>>>> Also, I am using the netcdfAll-4.3.jar at the command line. Does
> >>>>>> anyone have any insight as to *why *I'd be getting the 'not a valid
> >>>>>> CDM' error. I have checked the file using the NetCDF (4.3) GUI and
> the
> >>>>>> file looks good.
> >>>>>>
> >>>>>> Thank you for any insight you can give.
> >>>>>>
> >>>>>> Annie
> >>>>>>
> >>>>>>
> >>>>>> _______________________________________________
> >>>>>> netcdf-java mailing list
> >>>>>> netcdf-java@xxxxxxxxxxxxxxxx
> >>>>>> For list information or to unsubscribe, visit:
> >>>>>> http://www.unidata.ucar.edu/mailing_lists/
> >>>>>>
> >>>>>
> >>
> >
> >
> > ------------------------------------------------------------------------
> >
> > _______________________________________________
> > netcdf-java mailing list
> > netcdf-java@xxxxxxxxxxxxxxxx
> > For list information or to unsubscribe, visit:
> http://www.unidata.ucar.edu/mailing_lists/
>
>
>
> ------------------------------
>
> _______________________________________________
> netcdf-java mailing list
> netcdf-java@xxxxxxxxxxxxxxxx
> For list information or to unsubscribe, visit:
> http://www.unidata.ucar.edu/mailing_lists/
>
> End of netcdf-java Digest, Vol 66, Issue 12
> *******************************************
>
--
------------------------------------------------------------------------------------------
Ann Bryant Burgess, PhD
Postdoctoral Fellow
Computer Science Department
University of Southern California
Viterbi School of Engineering
Los Angeles, CA
Alaska Science Center/USGS
Anchorage, AK
Cell: (585) 738-7549
Office: (907) 786-7059
Fax: (907) 786-7150
E-mail: anniebryant.burgess@xxxxxxxxx
Office Address: 4210 University Dr., Anchorage, AK 99508-4626
-------------------------------------------------------------------------------------------
