Showing entries tagged [java]

Extensible NcML for AI/ML Ready Data on the THREDDS Data Server

Leo Matak
Leo Matak

I would like to begin by saying that this internship has definitely been one of the top highlights of my Ph.D. journey. I spent most of my working hours implementing the idea of server-side virtual data processing. This means that data on the THREDDS Data Servers (TDS) could be virtually processed without actually modifying the data. As such, the data integrity would remain intact, but it could be optimized for ML/AI.

[Read More]

THREDDS and Java 8 plans

THREDDS development is now switching to version 5.0 which will require Java 8. Version 5 is a major upgrade and some of the APIs will change. Deprecated classes will be moved to a legacy jar and will not be supported. If you are a developer, you will need to test the new version against your code. We expect to have an alpha release out by July for that purpose.
[Read More]

Introducing IDV 5.0 - Lynx!

Introducing IDV 5.0 - code name Lynx!

[Read More]

Integrating a SAX Parser with the Bison Parser Generator

A SAX (Simple API for XML) parser is a particular mechanism for parsing XML documents. Using a SAX parser has the advantage over the DOM-based parser in that it is not necessary to build the explicit DOM tree. On the other hand, it can be difficult to build a SAX parser because it requires management of complex state.

Combining SAX parsing with a GNU Bison generated parser is appealing because it allows the Bison parser to manage all of the state. Additionally, the .y file encapsulates the equivalent of a DTD but in a much more readable form. The combination makes using SAX parsing a lot simpler.

[Read More]

Reading NetCDF-4 With Java JNI

I finally got a JNI connection to the netCDF-4 C library.

The idea is to test the Java and C libraries against each other. Since they are independent implementations, agreement likely indicates correct results.

Because I wanted to do the work on my Windows development machine, I needed to wait until Ed could build a Windows DLL for netCDF-4. He had to go through the learning hump for Visual Studio etc. I did that 8-9 years ago, and I was glad not to do it again.

I bought a commercial package JNIWrapper, a companion product for my IDE, IntelliJ. It wasn't much money for an academic license ($150 ?), but still I expected that the documentation would be more than minimum. Its OK, but not that good. It is supposed to make it easy to write JNI to native code libraries.

Anyway, I couldn't get the damn thing to work on the netcdf.dll (it worked on other dlls) so I mistakenly thought that there was something wrong with the dll. In googling around to try to solve the problem, I ran across another JNI helper library, JNA, an open source package on java.net that is actively being maintained and supported. I switched over to it - still didnt work with netcdf.dll.

I finally downloaded a trial copy of PE Explorer, which analyzed netcdf.dll and showed all the netCDF-4 DLL dependencies. So all I had to do is to make sure to explicitly load those libraries, then it all worked. I liked PE Explorer, and would buy it if I had to work much with DLLs.

So I immediately got things working with netCDF-3 files (I think I've written this code 5 times before). The 2 libraries agreed on reading all the test files I threw at it, so that's a nice feeling. There was a moment of panic on one test file - then it turned out to be truncated, and the C library silently fills in the missing parts with zeros. In the Java library, Im returning garbage. I'm going to change the Java library to detect truncation and reduce the record count to reflect the actual number of complete records.

I like JNA, simple stuff is simple. There are some dark corners in trying to write portable JNI. The netCDF library uses size_t a lot, which is an integer whose size is system dependent, and possibly compile-flag dependent. I'm mapping size_t to  JNA's NativeLongByReference which should do the right thing most of the time. Mapping to structures and pointers to arrays of Structures etc gets pretty complicated.

Im currently working on the netCDF-4 specific bindings, and learning the C API (again). I should have done this when Ed was designing it, to give him feedback, but oh well. There are a number of JNI mappings that I'm not yet sure how to do.

If things get too hard in JNA, I will see if JNIWrapper does a better job. At the moment I don't really have a comparison between them.

Oh yeah, am I glad I get to work in Java and not C. Thanks to Ed and Russ and Dennis who use that lovely language so I don't have to!
Unidata Developer's Blog
A weblog about software development by Unidata developers*
Unidata Developer's Blog
A weblog about software development by Unidata developers*

Welcome

FAQs

News@Unidata blog

Take a poll!

What if we had an ongoing user poll in here?

Browse By Topic
Browse by Topic
« November 2024
SunMonTueWedThuFriSat
     
1
2
3
4
5
6
7
8
9
10
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
       
Today