There may be a way to fix this.
You might 'unjar' tika-app-1.6-SNAPSHOT.jar
and replace the old netcdf in that jar with the
newer version, then rejar tika-app-1.6-SNAPSHOT.jar.
Assuming no interface problems, this has a fair chance
to work.
=Dennis Heimbigner
Unidata
Christian Ward-Garrison wrote:
Hi Annie,
It turns out that this is an issue with Tika. tika-app-1.6-SNAPSHOT.jar
actually includes netcdf-4.2.20, so you have 2 different versions of the
same library on the classpath, which is always bad. And unfortunately,
using Tika's bundled netcdf-4.2.20 alone won't work, because support for
reading GRIB files was only added to NetCDF-Java in 4.3+.
This may be a problem that only the Tika developers know how to fix. I
suggest opening a ticket with them.
Good luck!
Christian Ward-Garrison
On Mon, Jul 28, 2014 at 3:19 PM, Annie Burgess <anniebryant@xxxxxxxxx>
wrote:
Hi Christian,
Your code words great - no problems when I compile and run. I've modified
my code to use NetcdfDataset rather than NetcdfFile to open the .grib2
file. However, I'm still getting an error in my code:
% javac -classpath
../../../../tika-core/target/tika-core-1.6-SNAPSHOT.jar:../../../../netcdfAll-4.3.jar
org/apache/tika/parser/grib/GribParser.java
% java -classpath
tika-app/target/tika-app-1.6-SNAPSHOT.jar:annie-parsers.jar:netcdfAll-4.3.jar
org.apache.tika.cli.TikaCLI --text gdas1.forecmwf.2014062612.grib2
-----------------------GripParser.java------------------------
import java.io.ByteArrayOutputStream;
import java.io.IOException;
import java.io.InputStream;
import java.util.Collections;
import java.util.Set;
import java.io.File;
import org.apache.tika.exception.TikaException;
import org.apache.tika.mime.MediaType;
import org.apache.tika.parser.AbstractParser;
import org.apache.tika.parser.ParseContext;
import org.apache.tika.parser.Parser;
import org.apache.tika.sax.XHTMLContentHandler;
import org.xml.sax.ContentHandler;
import org.xml.sax.SAXException;
import ucar.nc2.NetcdfFile;
import ucar.nc2.dataset.NetcdfDataset;/**
public class GribParser extends AbstractParser {
private final Set<MediaType> SUPPORTED_TYPES =
Collections.singleton(MediaType.application("x-grib2"));
public Set<MediaType> getSupportedTypes(ParseContext context) {
return SUPPORTED_TYPES;
}
public void parse(InputStream stream, ContentHandler handler,
Metadata metadata, ParseContext context) throws IOException,
SAXException, TikaException {
System.err.println(" Check 1 ");
File gribFile = new File("gdas1.forecmwf.2014062612.grib2");
NetcdfFile ncFile = NetcdfDataset.openFile(gribFile.getAbsolutePath(),
null);
System.err.println(" Check 2 ");
}
}
---------------------OUTPUT---------------------
Check 1
Exception in thread "main" org.apache.tika.exception.TikaException:
TIKA-198: Illegal IOException from
org.apache.tika.parser.grib.GribParser@483ad415
at org.apache.tika.parser.CompositeParser.parse(CompositeParser.java:249)
at org.apache.tika.parser.CompositeParser.parse(CompositeParser.java:243)
at org.apache.tika.parser.AutoDetectParser.parse(AutoDetectParser.java:121)
at org.apache.tika.cli.TikaCLI$OutputType.process(TikaCLI.java:141)
at org.apache.tika.cli.TikaCLI.process(TikaCLI.java:420)
at org.apache.tika.cli.TikaCLI.main(TikaCLI.java:111)
Caused by: java.io.IOException: Cant read
/Users/IGSWAHWSWBURGESS/Development/tikadev/tika/gdas1.forecmwf.2014062612.grib2:
not a valid CDM file.
at ucar.nc2.NetcdfFile.open(NetcdfFile.java:734)
at ucar.nc2.NetcdfFile.open(NetcdfFile.java:384)
at ucar.nc2.dataset.NetcdfDataset.openOrAcquireFile(NetcdfDataset.java:687)
at ucar.nc2.dataset.NetcdfDataset.openFile(NetcdfDataset.java:564)
at org.apache.tika.parser.grib.GribParser.parse(GribParser.java:82)
at org.apache.tika.parser.CompositeParser.parse(CompositeParser.java:243)
... 5 more
---------------------------------------------
netcdfALL and .grib2 files are in the same director where I am running
GribParser.java. This could totally be a classpath issue... I'm just
stumped.
Any other ideas?
Thanks!
Annie
On Mon, Jul 28, 2014 at 12:40 PM, Christian Ward-Garrison <
cwardgar@xxxxxxxx> wrote:
Hi Annie,
I see you're using netcdfAll-4.3.jar. That actually already contains the
grib module, so it should be all you need. I created a minimal example:
------------------------------------- Foo.java
-------------------------------------
import java.io.IOException;
import java.io.File;
import ucar.nc2.NetcdfFile;
import ucar.nc2.dataset.NetcdfDataset;
public class Foo {
public static void main(String[] args) throws IOException {
File gribFile = new File("foo.grib2");
NetcdfFile ncFile =
NetcdfDataset.openFile(gribFile.getAbsolutePath(), null);
try {
System.out.println(ncFile.toString());
} finally {
ncFile.close();
}
}
}
------------------------------------- Shell commands
-------------------------------------
javac -cp netcdfAll-4.3.jar Foo.java
java -cp .;netcdfAll-4.3.jar Foo
That should work as long as netcdfAll-4.3.jar and a file named
"foo.grib2" are in the same directory as Foo.java. If you move things,
you'll obviously need to modify the commands. Does this example work for
you?
Cheers,
Christian
On Mon, Jul 28, 2014 at 1:52 PM, Annie Burgess <anniebryant@xxxxxxxxx>
wrote:
Hi Christian,
Thanks for your response. I've cut down the code (pasted below) to a
sort of bare-bones version that is ONLY trying to open the .grib2 file as
if it were a .nc file.
I build apache tika from:
svn co http://svn.apache.org/repos/asf/tika/trunk tika
mvn install
I pulled netcdfAll and toolsUI .jar files from:
http://www.unidata.ucar.edu/downloads/netcdf/netcdf-java-4/index.jsp
I pulled the grib .jar from:
http://mvnrepository.com/artifact/edu.ucar/grib/8.0.29
I compile the code as:
[asc-227-196:src/main/java] AB% javac -classpath
../../../../tika-core/target/tika-core-1.6-SNAPSHOT.jar:../../../../toolsUI-4.3.jar:../../../../netcdfAll-4.3.jar:../../../../grib-8.0.29.jar
org/apache/tika/parser/grib/GribParser.java
I run the code as:
[asc-227-196:~/Development/tikadev/tika] AB% java -classpath
tika-app/target/tika-app-1.6-SNAPSHOT.jar:annie-parsers.jar:netcdfAll-4.3.jar:grib-8.0.29.jar:toolsUI-4.3.jar
org.apache.tika.cli.TikaCLI --text gdas1.forecmwf.2014062612.grib2
CODE:
--------------------------------------------------
package org.apache.tika.parser.grib;
import java.io.ByteArrayOutputStream;
import java.io.IOException;
import java.io.InputStream;
import java.util.Collections;
import java.util.Set;
import java.util.List;
import java.util.Iterator;
//JDK imports
import org.apache.tika.exception.TikaException;
import org.apache.tika.io.IOUtils;
import org.apache.tika.metadata.Metadata;
import org.apache.tika.metadata.Property;
import org.apache.tika.metadata.TikaCoreProperties;
import org.apache.tika.mime.MediaType;
import org.apache.tika.parser.AbstractParser;
import org.apache.tika.parser.ParseContext;
import org.apache.tika.parser.Parser;
import org.apache.tika.sax.XHTMLContentHandler;
import org.xml.sax.ContentHandler;
import org.xml.sax.SAXException;
import ucar.grib.grib2.*;
import ucar.nc2.*;
/**
* A {@link Parser} for <a
* href="http://www.unidata.ucar.edu/software/netcdf/index.html
">NetCDF</a>
* files using the UCAR, MIT-licensed <a
* href="http://www.unidata.ucar.edu/software/netcdf-java/";>NetCDF for
Java</a>
* API.
*/
public class GribParser extends AbstractParser {
private final Set<MediaType> SUPPORTED_TYPES =
Collections.singleton(MediaType.application("x-grib2"));
/*
* (non-Javadoc)
*
* @see
*
org.apache.tika.parser.Parser#getSupportedTypes(org.apache.tika.parser
* .ParseContext)
*/
public Set<MediaType> getSupportedTypes(ParseContext context) {
return SUPPORTED_TYPES;
}
/*
* (non-Javadoc)
*
* @see org.apache.tika.parser.Parser#parse(java.io.InputStream,
* org.xml.sax.ContentHandler, org.apache.tika.metadata.Metadata,
* org.apache.tika.parser.ParseContext)
*/
public void parse(InputStream stream, ContentHandler handler,
Metadata metadata, ParseContext context) throws IOException,
SAXException, TikaException {
System.err.println(" Check 1 ");
String name = "/Users/IGSWAHWSWBURGESS/POLARCYBER/gdas1.forecmwf.
2014062612.grib2";
if (name == null) {
name = "";
}
NetcdfFile ncFile = NetcdfFile.open(name, null);
System.err.println(" Check 2 ");
}
}
OUTPUT:
Check 1
Exception in thread "main" org.apache.tika.exception.TikaException:
Unexpected RuntimeException from
org.apache.tika.parser.grib.GribParser@261a53b9
at org.apache.tika.parser.CompositeParser.parse(CompositeParser.java:245)
at org.apache.tika.parser.CompositeParser.parse(CompositeParser.java:243)
at
org.apache.tika.parser.AutoDetectParser.parse(AutoDetectParser.java:121)
at org.apache.tika.cli.TikaCLI$OutputType.process(TikaCLI.java:141)
at org.apache.tika.cli.TikaCLI.process(TikaCLI.java:420)
at org.apache.tika.cli.TikaCLI.main(TikaCLI.java:111)
Caused by: java.lang.RuntimeException: java.lang.NoSuchMethodError:
ucar.grib.grib2.Grib2WriteIndex.writeGribIndex(Ljava/io/File;Ljava/lang/String;Lucar/unidata/io/RandomAccessFile;Z)Lucar/grid/GridIndex;
at ucar.nc2.NetcdfFile.<init>(NetcdfFile.java:1326)
at ucar.nc2.NetcdfFile.open(NetcdfFile.java:744)
at ucar.nc2.NetcdfFile.openInMemory(NetcdfFile.java:670)
at org.apache.tika.parser.grib.GribParser.parse(GribParser.java:93)
at org.apache.tika.parser.CompositeParser.parse(CompositeParser.java:243)
... 5 more
Caused by: java.lang.NoSuchMethodError:
ucar.grib.grib2.Grib2WriteIndex.writeGribIndex(Ljava/io/File;Ljava/lang/String;Lucar/unidata/io/RandomAccessFile;Z)Lucar/grid/GridIndex;
at
ucar.nc2.iosp.grib.GribGridServiceProvider.writeIndex(GribGridServiceProvider.java:348)
at
ucar.nc2.iosp.grib.GribGridServiceProvider.getIndex(GribGridServiceProvider.java:292)
at
ucar.nc2.iosp.grib.GribGridServiceProvider.open(GribGridServiceProvider.java:118)
at ucar.nc2.NetcdfFile.<init>(NetcdfFile.java:1308)
... 9 more
Note, if I use a .nc file the code runs successfully.
OUTPUT:
Check 1
Check 2
I am sort of a java newbie, so please let me know if I've left out any
critical information!
Thank you for any help/insight you can give.
Annie
On Sun, Jul 27, 2014 at 10:44 PM, Christian Ward-Garrison <
cwardgar@xxxxxxxx> wrote:
Hi Annie,
This is the result of the GRIB module not being on the classpath when
you execute your Java program. Can you give me more details about your
setup? Can you provide you build file (Maven, Ant, Gradle, etc)?
Cheers,
Christian
On Wed, Jul 23, 2014 at 5:06 PM, Annie Burgess <anniebryant@xxxxxxxxx>
wrote:
Greetings all,
I am trying to create a script that will mimic the output of NCDump.
I have successfully done this for NetCDF files, and now I am trying to
apply it to grib2 files. I am using the NetCDF-java library in conjunction
with Apache Tika to do this. Other posts have indicated I should be able
to open my grib2 files, just as if they were .nc files. However, I
continue to get the following error:
"Caused by: java.io.IOException: Cant read gdas1.forecmwf.2014062612.grib2:
not a valid CDM file."
To open the .nc files, this is the bit of code I use (with the
exception of changing the .nc file to a .grib2 file):
String name = "gdas1.forecmwf.2014062612.grib2";
if (name == null) {
name = "";
}
try NetcdfFile ncFile = NetcdfFile.openInMemory(name,
os.toByteArray());
// first parse out the set of global attributes
for (Attribute attr : ncFile.getGlobalAttributes()) {
Property property = resolveMetadataKey(attr.getName());
if (attr.getDataType().isString()) {
metadata.add(property, attr.getStringValue());
} else if (attr.getDataType().isNumeric()) {
int value = attr.getNumericValue().intValue();
metadata.add(property, String.valueOf(value));
}
}
Also, I am using the netcdfAll-4.3.jar at the command line. Does
anyone have any insight as to *why *I'd be getting the 'not a valid
CDM' error. I have checked the file using the NetCDF (4.3) GUI and the
file looks good.
Thank you for any insight you can give.
Annie
_______________________________________________
netcdf-java mailing list
netcdf-java@xxxxxxxxxxxxxxxx
For list information or to unsubscribe, visit:
http://www.unidata.ucar.edu/mailing_lists/
------------------------------------------------------------------------
_______________________________________________
netcdf-java mailing list
netcdf-java@xxxxxxxxxxxxxxxx
For list information or to unsubscribe, visit: http://www.unidata.ucar.edu/mailing_lists/
