[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

[LDM #YGD-150476]: pqinsert multiple files w/ same name

Brendon,

> First off, special thanks to Steve Emmerson for answering an emergency
> phone question on Tuesday, my query was answered nicely.  I know you
> folks prefer to work via email, but I was in a restricted machine room
> situation, Unidata answered the call nicely! :)
> 
> I'm setting up an LDM box to ingest RWIS data and relay to MADIS via the
> EXP feed.  Data files are generated every 5 minutes with date/time
> stamps as part of the filename.  My challenge is that the folks at MADIS
> want files with the same filename.
> 
> I found the following reference article which seems to cover my task:
> http://www.unidata.ucar.edu/support/help/MailArchives/ldm/msg04035.html
> 
> I can write a script which can modify the original filename w/timestamp
> (ExpAtmosData20061018.200201.txt) to satisfy what MADIS is looking for
> (NH_ExpAtmosData.txt).  I'm assuming if I add a line to the script that
> will insert the file with the following command:
> 
> "pqinsert -f EXP -p id NH_ExpAtmosData.txt"
> 
> LDM will keep the different files in the queue even though they would
> have the same filename.

The data in the files must be unique to each file.  If two files have the same 
data, then their MD5 checksums will be identical and the second file won't be 
inserted into the queue.

> My question is, does the "id" in the command
> above have to be a specific number, or can I just use the syntax as
> listed above and LDM will autogenerate a specific id #?  I would prefer
> not to have to worry about generating/keeping track of id numbers every
> time I ingest a file.  Or, should I be using the -s option?  Another
> idea was to expire the queue before inserting the file, but this seemed
> like a more cumbersome way to go, plus it prevents files from going out
> in the event of a network outage.

"id" in the command can be the same for each file.  There no prohibition 
against different data products having the same product-identifier.  If 
multiple files are inserted by a single pqinsert(1) command, however, then the 
product-identifiers will match the pattern "<id>.<sequence no>".

> Please let me know if I'm on the right track.

You appear to be on the right track.

Regards,
Steve Emmerson

Ticket Details
===================
Ticket ID: YGD-150476
Department: Support LDM
Priority: Normal
Status: Closed