Technical complexity is wonderful, if you are a tweak, but in reality,
we must distribute data so that the masses -- PIs and Techs with limited
time and/or computer skills -- can use and understand it. This is
unfortunate, in a way, as the average computer tweak would love to
implement grand, elegant, and complex systems to manage raw and
processed data. Taking some minutes to think though, tells one that the
best data storage system/format will be simple, succinct, and
sufficient. One should be able to look at a README file that comes with
the data and say, "Oh, I get it" in one minute. I would bet that as
soon as the complexity of a format exceeds the "one minute" test, HALF
of your potential users will toss it. There will also be those who are
computer USERS rather than MANIPULATORS who will toss it even sooner.
It's just like netCDF itself -- how many of you have tried to get
co-workers to use it, only to have them say no, or they will think about
it. Perhaps they don't want to change because their current system
works, following the "if it isn't broke, don't fix it" rule, or they are
just afraid because they don't understand the technology. If getting
someone to use netCDF is a challenge, imagine getting them to use yet
another convention on top of netCDF. It makes me think of the metric
system (here in the US). Imagine asking US carpenters to go metric,
then tell them that plywood will now come in 3x6 sheets (which are
really 91.44 by 182.88 cm) and that a 2x4 (which is really 1 1/2 x 3 1/2) is
now a 2 1/2 x 3 1/2 (which is 6.35 by 8.89 cm, but will be really 6 x 8.5).
"No thank you", says the average carpenter -- "Confusion on confusion,
complexity on complexity -- why bother? All I want to do is build a
house."
We must NEVER loose site of the end users of our data. To that end, we
must concentrate on what users DO with data, not just what we want it to
look like. If we don't have simple standards and tools that the average
user can use, no one is going to use them. Technology and the like is
supposed to make things easier, not more complex.
-Tim Holt
