Hey Christian,
Yes - we absolutely need a 'proper' fix! My first goal was just to make
sure the code to open and parse a .grib2 file was correct. Before anything
is committed to Apache Tika, we will make sure all is kosher.
Annie
On Tue, Jul 29, 2014 at 12:08 PM, Christian Ward-Garrison <cwardgar@xxxxxxxx
> wrote:
> Ah, interesting. You've changed the order of tika and netcdfAll-4.3 in the
> classpath listing, so now Java is finding the v4.3 classes BEFORE the v4.2
> classes. Both are still there, but java's not seeing the 4.2.
>
> As I'm sure you know, relying on such things for proper execution of your
> program is NOT recommended. I'm glad that you're trying to get a "proper"
> fix in place on the Tika side of things.
>
> Cheers!
> Christian
>
>
>
> On Tue, Jul 29, 2014 at 2:00 PM, Annie Burgess <anniebryant@xxxxxxxxx>
> wrote:
>
>> Hi Christian,
>>
>> Yes, I've discovered the same thing over the past 24 hours!
>>
>> When I call Tika at the command line, using the following, I now can
>> successfully open and parse a .grib2 file.
>>
>> *java -classpath
>> .:netcdfAll-4.3.jar:tika-app/target/tika-app-1.6-SNAPSHOT.jar:annie-parsers.jar
>> org.apache.tika.cli.TikaCLI --metadata gdas1.forecmwf.2014062612
>> <2014062612>.grib2*
>>
>> However, I also get the following errors (assumed because of the
>> duplicate versions) that print above my parsed text:
>>
>> SLF4J: Class path contains multiple SLF4J bindings.
>> SLF4J: Found binding in
>> [jar:file:/Users/IGSWAHWSWBURGESS/Development/tikadev/tika/netcdfAll-4.3.jar!/org/slf4j/impl/StaticLoggerBinder.class]
>> SLF4J: Found binding in
>> [jar:file:/Users/IGSWAHWSWBURGESS/Development/tikadev/tika/tika-app/target/tika-app-1.6-SNAPSHOT.jar!/org/slf4j/impl/StaticLoggerBinder.class]
>> SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an
>> explanation.
>> SLF4J: Actual binding is of type [org.slf4j.impl.JDK14LoggerFactory]
>>
>> So good news, I've got .grib2 parsing with Tika! Now, I'll take care of
>> the Java 4.3+ issue on the Tika side of things.
>>
>> Many thanks for your help.
>>
>> Annie
>>
>>
>> On Tue, Jul 29, 2014 at 11:53 AM, Christian Ward-Garrison <
>> cwardgar@xxxxxxxx> wrote:
>>
>>> Hi Annie,
>>>
>>> It turns out that this is an issue with Tika. tika-app-1.6-SNAPSHOT.jar
>>> actually includes netcdf-4.2.20, so you have 2 different versions of the
>>> same library on the classpath, which is always bad. And unfortunately,
>>> using Tika's bundled netcdf-4.2.20 alone won't work, because support for
>>> reading GRIB files was only added to NetCDF-Java in 4.3+.
>>>
>>> This may be a problem that only the Tika developers know how to fix. I
>>> suggest opening a ticket with them.
>>>
>>> Good luck!
>>> Christian Ward-Garrison
>>>
>>>
>>> On Mon, Jul 28, 2014 at 3:19 PM, Annie Burgess <anniebryant@xxxxxxxxx>
>>> wrote:
>>>
>>>> Hi Christian,
>>>>
>>>> Your code words great - no problems when I compile and run. I've
>>>> modified my code to use NetcdfDataset rather than NetcdfFile to open the
>>>> .grib2 file. However, I'm still getting an error in my code:
>>>>
>>>> % javac -classpath
>>>> ../../../../tika-core/target/tika-core-1.6-SNAPSHOT.jar:../../../../netcdfAll-4.3.jar
>>>> org/apache/tika/parser/grib/GribParser.java
>>>>
>>>> % java -classpath
>>>> tika-app/target/tika-app-1.6-SNAPSHOT.jar:annie-parsers.jar:netcdfAll-4.3.jar
>>>> org.apache.tika.cli.TikaCLI --text gdas1.forecmwf.2014062612.grib2
>>>>
>>>>
>>>> -----------------------GripParser.java------------------------
>>>> import java.io.ByteArrayOutputStream;
>>>> import java.io.IOException;
>>>> import java.io.InputStream;
>>>> import java.util.Collections;
>>>> import java.util.Set;
>>>> import java.io.File;
>>>>
>>>> import org.apache.tika.exception.TikaException;
>>>>
>>>> import org.apache.tika.mime.MediaType;
>>>> import org.apache.tika.parser.AbstractParser;
>>>> import org.apache.tika.parser.ParseContext;
>>>> import org.apache.tika.parser.Parser;
>>>> import org.apache.tika.sax.XHTMLContentHandler;
>>>> import org.xml.sax.ContentHandler;
>>>> import org.xml.sax.SAXException;
>>>>
>>>> import ucar.nc2.NetcdfFile;
>>>> import ucar.nc2.dataset.NetcdfDataset;/**
>>>>
>>>>
>>>> public class GribParser extends AbstractParser {
>>>>
>>>> private final Set<MediaType> SUPPORTED_TYPES =
>>>> Collections.singleton(MediaType.application("x-grib2"));
>>>>
>>>> public Set<MediaType> getSupportedTypes(ParseContext context) {
>>>> return SUPPORTED_TYPES;
>>>> }
>>>>
>>>> public void parse(InputStream stream, ContentHandler handler,
>>>> Metadata metadata, ParseContext context) throws IOException,
>>>> SAXException, TikaException {
>>>>
>>>> System.err.println(" Check 1 ");
>>>>
>>>> File gribFile = new File("gdas1.forecmwf.2014062612.grib2");
>>>>
>>>> NetcdfFile ncFile =
>>>> NetcdfDataset.openFile(gribFile.getAbsolutePath(), null);
>>>>
>>>> System.err.println(" Check 2 ");
>>>>
>>>> }
>>>> }
>>>>
>>>> ---------------------OUTPUT---------------------
>>>>
>>>> Check 1
>>>> Exception in thread "main" org.apache.tika.exception.TikaException:
>>>> TIKA-198: Illegal IOException from
>>>> org.apache.tika.parser.grib.GribParser@483ad415
>>>> at
>>>> org.apache.tika.parser.CompositeParser.parse(CompositeParser.java:249)
>>>>
>>>> at
>>>> org.apache.tika.parser.CompositeParser.parse(CompositeParser.java:243)
>>>> at
>>>> org.apache.tika.parser.AutoDetectParser.parse(AutoDetectParser.java:121)
>>>> at org.apache.tika.cli.TikaCLI$OutputType.process(TikaCLI.java:141)
>>>> at org.apache.tika.cli.TikaCLI.process(TikaCLI.java:420)
>>>> at org.apache.tika.cli.TikaCLI.main(TikaCLI.java:111)
>>>> Caused by: java.io.IOException: Cant read
>>>> /Users/IGSWAHWSWBURGESS/Development/tikadev/tika/gdas1.forecmwf.
>>>> 2014062612.grib2: not a valid CDM file.
>>>> at ucar.nc2.NetcdfFile.open(NetcdfFile.java:734)
>>>> at ucar.nc2.NetcdfFile.open(NetcdfFile.java:384)
>>>> at
>>>> ucar.nc2.dataset.NetcdfDataset.openOrAcquireFile(NetcdfDataset.java:687)
>>>> at ucar.nc2.dataset.NetcdfDataset.openFile(NetcdfDataset.java:564)
>>>> at org.apache.tika.parser.grib.GribParser.parse(GribParser.java:82)
>>>>
>>>> at
>>>> org.apache.tika.parser.CompositeParser.parse(CompositeParser.java:243)
>>>> ... 5 more
>>>>
>>>> ---------------------------------------------
>>>>
>>>> netcdfALL and .grib2 files are in the same director where I am running
>>>> GribParser.java. This could totally be a classpath issue... I'm just
>>>> stumped.
>>>>
>>>> Any other ideas?
>>>>
>>>> Thanks!
>>>> Annie
>>>>
>>>>
>>>> On Mon, Jul 28, 2014 at 12:40 PM, Christian Ward-Garrison <
>>>> cwardgar@xxxxxxxx> wrote:
>>>>
>>>>> Hi Annie,
>>>>>
>>>>> I see you're using netcdfAll-4.3.jar. That actually already contains
>>>>> the grib module, so it should be all you need. I created a minimal
>>>>> example:
>>>>>
>>>>> ------------------------------------- Foo.java
>>>>> -------------------------------------
>>>>>
>>>>> import java.io.IOException;
>>>>> import java.io.File;
>>>>> import ucar.nc2.NetcdfFile;
>>>>> import ucar.nc2.dataset.NetcdfDataset;
>>>>>
>>>>> public class Foo {
>>>>> public static void main(String[] args) throws IOException {
>>>>> File gribFile = new File("foo.grib2");
>>>>> NetcdfFile ncFile =
>>>>> NetcdfDataset.openFile(gribFile.getAbsolutePath(), null);
>>>>> try {
>>>>> System.out.println(ncFile.toString());
>>>>> } finally {
>>>>> ncFile.close();
>>>>> }
>>>>> }
>>>>> }
>>>>>
>>>>> ------------------------------------- Shell commands
>>>>> -------------------------------------
>>>>>
>>>>> javac -cp netcdfAll-4.3.jar Foo.java
>>>>>
>>>>> java -cp .;netcdfAll-4.3.jar Foo
>>>>>
>>>>>
>>>>> That should work as long as netcdfAll-4.3.jar and a file named
>>>>> "foo.grib2" are in the same directory as Foo.java. If you move things,
>>>>> you'll obviously need to modify the commands. Does this example work for
>>>>> you?
>>>>>
>>>>> Cheers,
>>>>> Christian
>>>>>
>>>>>
>>>>>
>>>>> On Mon, Jul 28, 2014 at 1:52 PM, Annie Burgess <anniebryant@xxxxxxxxx>
>>>>> wrote:
>>>>>
>>>>>> Hi Christian,
>>>>>>
>>>>>> Thanks for your response. I've cut down the code (pasted below) to a
>>>>>> sort of bare-bones version that is ONLY trying to open the .grib2 file as
>>>>>> if it were a .nc file.
>>>>>>
>>>>>> I build apache tika from:
>>>>>>
>>>>>> svn co http://svn.apache.org/repos/asf/tika/trunk tika
>>>>>> mvn install
>>>>>>
>>>>>> I pulled netcdfAll and toolsUI .jar files from:
>>>>>> http://www.unidata.ucar.edu/downloads/netcdf/netcdf-java-4/index.jsp
>>>>>>
>>>>>> I pulled the grib .jar from:
>>>>>> http://mvnrepository.com/artifact/edu.ucar/grib/8.0.29
>>>>>>
>>>>>> I compile the code as:
>>>>>> [asc-227-196:src/main/java] AB% javac -classpath
>>>>>> ../../../../tika-core/target/tika-core-1.6-SNAPSHOT.jar:../../../../toolsUI-4.3.jar:../../../../netcdfAll-4.3.jar:../../../../grib-8.0.29.jar
>>>>>> org/apache/tika/parser/grib/GribParser.java
>>>>>>
>>>>>> I run the code as:
>>>>>> [asc-227-196:~/Development/tikadev/tika] AB% java -classpath
>>>>>> tika-app/target/tika-app-1.6-SNAPSHOT.jar:annie-parsers.jar:netcdfAll-4.3.jar:grib-8.0.29.jar:toolsUI-4.3.jar
>>>>>> org.apache.tika.cli.TikaCLI --text gdas1.forecmwf.2014062612.grib2
>>>>>>
>>>>>> CODE:
>>>>>>
>>>>>> --------------------------------------------------
>>>>>> package org.apache.tika.parser.grib;
>>>>>>
>>>>>> import java.io.ByteArrayOutputStream;
>>>>>> import java.io.IOException;
>>>>>> import java.io.InputStream;
>>>>>> import java.util.Collections;
>>>>>> import java.util.Set;
>>>>>> import java.util.List;
>>>>>> import java.util.Iterator;
>>>>>>
>>>>>> //JDK imports
>>>>>> import org.apache.tika.exception.TikaException;
>>>>>> import org.apache.tika.io.IOUtils;
>>>>>> import org.apache.tika.metadata.Metadata;
>>>>>> import org.apache.tika.metadata.Property;
>>>>>> import org.apache.tika.metadata.TikaCoreProperties;
>>>>>> import org.apache.tika.mime.MediaType;
>>>>>> import org.apache.tika.parser.AbstractParser;
>>>>>> import org.apache.tika.parser.ParseContext;
>>>>>> import org.apache.tika.parser.Parser;
>>>>>> import org.apache.tika.sax.XHTMLContentHandler;
>>>>>> import org.xml.sax.ContentHandler;
>>>>>> import org.xml.sax.SAXException;
>>>>>>
>>>>>> import ucar.grib.grib2.*;
>>>>>> import ucar.nc2.*;
>>>>>>
>>>>>> /**
>>>>>> * A {@link Parser} for <a
>>>>>> * href="http://www.unidata.ucar.edu/software/netcdf/index.html
>>>>>> ">NetCDF</a>
>>>>>> * files using the UCAR, MIT-licensed <a
>>>>>> * href="http://www.unidata.ucar.edu/software/netcdf-java/";>NetCDF
>>>>>> for Java</a>
>>>>>> * API.
>>>>>> */
>>>>>> public class GribParser extends AbstractParser {
>>>>>>
>>>>>> private final Set<MediaType> SUPPORTED_TYPES =
>>>>>> Collections.singleton(MediaType.application("x-grib2"));
>>>>>> /*
>>>>>> * (non-Javadoc)
>>>>>> *
>>>>>> * @see
>>>>>> *
>>>>>> org.apache.tika.parser.Parser#getSupportedTypes(org.apache.tika.parser
>>>>>> * .ParseContext)
>>>>>> */
>>>>>> public Set<MediaType> getSupportedTypes(ParseContext context) {
>>>>>> return SUPPORTED_TYPES;
>>>>>> }
>>>>>> /*
>>>>>> * (non-Javadoc)
>>>>>> *
>>>>>> * @see org.apache.tika.parser.Parser#parse(java.io.InputStream,
>>>>>> * org.xml.sax.ContentHandler, org.apache.tika.metadata.Metadata,
>>>>>> * org.apache.tika.parser.ParseContext)
>>>>>> */
>>>>>> public void parse(InputStream stream, ContentHandler handler,
>>>>>> Metadata metadata, ParseContext context) throws
>>>>>> IOException,
>>>>>> SAXException, TikaException {
>>>>>>
>>>>>> System.err.println(" Check 1 ");
>>>>>> String name = "/Users/IGSWAHWSWBURGESS/POLARCYBER/gdas1.forecmwf.
>>>>>> 2014062612.grib2";
>>>>>>
>>>>>> if (name == null) {
>>>>>> name = "";
>>>>>> }
>>>>>>
>>>>>> NetcdfFile ncFile = NetcdfFile.open(name, null);
>>>>>> System.err.println(" Check 2 ");
>>>>>> }
>>>>>> }
>>>>>>
>>>>>>
>>>>>> OUTPUT:
>>>>>>
>>>>>> Check 1
>>>>>>
>>>>>> Exception in thread "main" org.apache.tika.exception.TikaException:
>>>>>> Unexpected RuntimeException from
>>>>>> org.apache.tika.parser.grib.GribParser@261a53b9
>>>>>> at
>>>>>> org.apache.tika.parser.CompositeParser.parse(CompositeParser.java:245)
>>>>>> at
>>>>>> org.apache.tika.parser.CompositeParser.parse(CompositeParser.java:243)
>>>>>> at
>>>>>> org.apache.tika.parser.AutoDetectParser.parse(AutoDetectParser.java:121)
>>>>>> at org.apache.tika.cli.TikaCLI$OutputType.process(TikaCLI.java:141)
>>>>>> at org.apache.tika.cli.TikaCLI.process(TikaCLI.java:420)
>>>>>> at org.apache.tika.cli.TikaCLI.main(TikaCLI.java:111)
>>>>>> Caused by: java.lang.RuntimeException: java.lang.NoSuchMethodError:
>>>>>> ucar.grib.grib2.Grib2WriteIndex.writeGribIndex(Ljava/io/File;Ljava/lang/String;Lucar/unidata/io/RandomAccessFile;Z)Lucar/grid/GridIndex;
>>>>>> at ucar.nc2.NetcdfFile.<init>(NetcdfFile.java:1326)
>>>>>> at ucar.nc2.NetcdfFile.open(NetcdfFile.java:744)
>>>>>> at ucar.nc2.NetcdfFile.openInMemory(NetcdfFile.java:670)
>>>>>> at org.apache.tika.parser.grib.GribParser.parse(GribParser.java:93)
>>>>>> at
>>>>>> org.apache.tika.parser.CompositeParser.parse(CompositeParser.java:243)
>>>>>> ... 5 more
>>>>>> Caused by: java.lang.NoSuchMethodError:
>>>>>> ucar.grib.grib2.Grib2WriteIndex.writeGribIndex(Ljava/io/File;Ljava/lang/String;Lucar/unidata/io/RandomAccessFile;Z)Lucar/grid/GridIndex;
>>>>>> at
>>>>>> ucar.nc2.iosp.grib.GribGridServiceProvider.writeIndex(GribGridServiceProvider.java:348)
>>>>>> at
>>>>>> ucar.nc2.iosp.grib.GribGridServiceProvider.getIndex(GribGridServiceProvider.java:292)
>>>>>> at
>>>>>> ucar.nc2.iosp.grib.GribGridServiceProvider.open(GribGridServiceProvider.java:118)
>>>>>> at ucar.nc2.NetcdfFile.<init>(NetcdfFile.java:1308)
>>>>>> ... 9 more
>>>>>>
>>>>>>
>>>>>>
>>>>>> Note, if I use a .nc file the code runs successfully.
>>>>>>
>>>>>> OUTPUT:
>>>>>>
>>>>>> Check 1
>>>>>> Check 2
>>>>>>
>>>>>>
>>>>>> I am sort of a java newbie, so please let me know if I've left out
>>>>>> any critical information!
>>>>>>
>>>>>> Thank you for any help/insight you can give.
>>>>>>
>>>>>> Annie
>>>>>>
>>>>>>
>>>>>> On Sun, Jul 27, 2014 at 10:44 PM, Christian Ward-Garrison <
>>>>>> cwardgar@xxxxxxxx> wrote:
>>>>>>
>>>>>>> Hi Annie,
>>>>>>>
>>>>>>> This is the result of the GRIB module not being on the classpath
>>>>>>> when you execute your Java program. Can you give me more details about
>>>>>>> your
>>>>>>> setup? Can you provide you build file (Maven, Ant, Gradle, etc)?
>>>>>>>
>>>>>>> Cheers,
>>>>>>> Christian
>>>>>>>
>>>>>>>
>>>>>>> On Wed, Jul 23, 2014 at 5:06 PM, Annie Burgess <
>>>>>>> anniebryant@xxxxxxxxx> wrote:
>>>>>>>
>>>>>>>> Greetings all,
>>>>>>>>
>>>>>>>> I am trying to create a script that will mimic the output of
>>>>>>>> NCDump. I have successfully done this for NetCDF files, and now I am
>>>>>>>> trying to apply it to grib2 files. I am using the NetCDF-java library
>>>>>>>> in
>>>>>>>> conjunction with Apache Tika to do this. Other posts have indicated I
>>>>>>>> should be able to open my grib2 files, just as if they were .nc files.
>>>>>>>> However, I continue to get the following error:
>>>>>>>>
>>>>>>>> "Caused by: java.io.IOException: Cant read gdas1.forecmwf.
>>>>>>>> 2014062612.grib2: not a valid CDM file."
>>>>>>>>
>>>>>>>> To open the .nc files, this is the bit of code I use (with the
>>>>>>>> exception of changing the .nc file to a .grib2 file):
>>>>>>>>
>>>>>>>> String name = "gdas1.forecmwf.2014062612.grib2";
>>>>>>>>
>>>>>>>> if (name == null) {
>>>>>>>> name = "";
>>>>>>>> }
>>>>>>>>
>>>>>>>> try NetcdfFile ncFile = NetcdfFile.openInMemory(name,
>>>>>>>> os.toByteArray());
>>>>>>>> // first parse out the set of global attributes
>>>>>>>> for (Attribute attr : ncFile.getGlobalAttributes()) {
>>>>>>>> Property property =
>>>>>>>> resolveMetadataKey(attr.getName());
>>>>>>>> if (attr.getDataType().isString()) {
>>>>>>>> metadata.add(property, attr.getStringValue());
>>>>>>>> } else if (attr.getDataType().isNumeric()) {
>>>>>>>> int value = attr.getNumericValue().intValue();
>>>>>>>> metadata.add(property, String.valueOf(value));
>>>>>>>> }
>>>>>>>> }
>>>>>>>>
>>>>>>>> Also, I am using the netcdfAll-4.3.jar at the command line. Does
>>>>>>>> anyone have any insight as to *why *I'd be getting the 'not a
>>>>>>>> valid CDM' error. I have checked the file using the NetCDF (4.3) GUI
>>>>>>>> and
>>>>>>>> the file looks good.
>>>>>>>>
>>>>>>>> Thank you for any insight you can give.
>>>>>>>>
>>>>>>>> Annie
>>>>>>>>
>>>>>>>>
>>>>>>>> _______________________________________________
>>>>>>>> netcdf-java mailing list
>>>>>>>> netcdf-java@xxxxxxxxxxxxxxxx
>>>>>>>> For list information or to unsubscribe, visit:
>>>>>>>> http://www.unidata.ucar.edu/mailing_lists/
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>
>>>>
>>>
>>
>>
>> --
>>
>> ------------------------------------------------------------------------------------------
>> Ann Bryant Burgess, PhD
>>
>> Postdoctoral Fellow
>> Computer Science Department
>> University of Southern California
>> Viterbi School of Engineering
>> Los Angeles, CA
>>
>> Alaska Science Center/USGS
>> Anchorage, AK
>>
>> Cell: (585) 738-7549
>> Office: (907) 786-7059
>> Fax: (907) 786-7150
>> E-mail: anniebryant.burgess@xxxxxxxxx
>> Office Address: 4210 University Dr., Anchorage, AK 99508-4626
>>
>> -------------------------------------------------------------------------------------------
>>
>
>
--
------------------------------------------------------------------------------------------
Ann Bryant Burgess, PhD
Postdoctoral Fellow
Computer Science Department
University of Southern California
Viterbi School of Engineering
Los Angeles, CA
Alaska Science Center/USGS
Anchorage, AK
Cell: (585) 738-7549
Office: (907) 786-7059
Fax: (907) 786-7150
E-mail: anniebryant.burgess@xxxxxxxxx
Office Address: 4210 University Dr., Anchorage, AK 99508-4626
-------------------------------------------------------------------------------------------
