Hi again,
I just remembered I ported this project over to Github shortly before my
departure; here is the pertinent script:
https://github.com/Unidata/idd-cataloging/blob/main/scripts/iddcat_ingest.py
Again, it's just a "relatively" simple example of taking streaming input
and putting it into a DB. In this case I'm running notifyme and look for
specific output to know that it's a product ID, your logic would likely be
a little different. My point is this is just one way to do it.
Another way might be to, as you suggested in your email, to have a PQACT
entry handle this somehow. Unlike my project where I start off by running
a script which Then invokes the input-generator, this is the other way
around. That's important to note because anytime an EXEC entry is invoked
it'll start a new process. Depending on what you're ingesting (and how
much/fast) this could lead to one of those snowball situations. YMMV, just
something cognizant of.
I'm not sure which of these or any other methods are best, it'll boil down
to your usage details and environment. I hope this helps, though!
Best,
-Mike
On Sat, Dec 14, 2024 at 2:37 PM Mike Zuranski <zuranski.wx@xxxxxxxxx> wrote:
> Hi Chris,
>
> Yes this is possible. What I've done in the past was to implement
> Postgres databases and wrote Python scripts to ingest the data. In my case
> all I was really doing was logging the product IDs coming over the IDD, but
> the core concept is the same. I genuinely can't speak for the status of
> LDM's ability to send info into a DB directly, it has never come up since
> I've been using the LDM. IMO, I wouldn't worry about the decoders and
> instead write something yourself; the operations you're describing seem
> "relatively" straight forward.
>
> I have some code that will likely help you out, I'll be back once I dig it
> out from my backups.
>
> Best,
> -Mike
>
>
> On Sat, Dec 14, 2024 at 12:09 PM Herbster, Christopher G. <
> herbstec@xxxxxxxx> wrote:
>
>> Hi all,
>>
>> I have a number of desires for having processed METAR data in a database
>> like format. We would like to do some forecast verifications for an
>> internal effort, as well as possibly create charts of recent/past
>> conditions like: https://www.weather.gov/wrh/timeseries?site=KDAB
>>
>> I suspect that some of you have significant experience doing this, and I
>> would really appreciate any sagely advice you can provide.
>>
>> I was expecting to implement something that would happen as a pqact entry
>> when data arrive, though I have not ruled out making hourly files that get
>> processed next hour. Data would be either from NOAAPort, IDD or (perhaps?)
>> local archives. Does it matter that the Unidata (supported) decoders look
>> like they are unchanged for many years? An Internet search showed many
>> GitHub options are out there. (Yikes!)
>>
>> I was also going to create a database entry of the images that flow
>> through our system as a way to query/load/delete files that live under our
>> web pages. Again, I would certainly benefit from any input folks would like
>> to provide.
>>
>> I am not yet married to a database on the backend, so input there is
>> welcome as well. We do have an Apache Hive system that is a candidate for
>> this, but I have tinkered with MariaDB in the past. (I am a rookie at
>> whatever I will use.)
>>
>> Is this already being done in EDEX? We had, and will have again, and EDEX
>> server running internally, so perhaps I can exploit that.
>>
>> Thanks in advance for any thoughts, suggestions, and/or tools.
>>
>> Chris
>>
>> *Christopher Herbster, Ph.D.*
>> *Associate Professor of Meteorology*
>> *Applied Aviation Science*
>> Daytona Beach Campus
>> College of Aviation Rm. 338
>>
>> 1 Aerospace Boulevard
>> Daytona Beach, FL 32114
>> 386.226.6444
>> *christopher.herbster@xxxxxxxx <christopher.herbster@xxxxxxxx>*
>>
>> *Embry-Riddle Aeronautical University*
>> Florida | Arizona | Worldwide
>>
>>
>> 386.226.6446 Weather Center
>> http://wx.erau.edu/
>>
>> Schedule at: http://wx.erau.edu/faculty/herbster/Schedules/
>>
>> _______________________________________________
>> NOTE: All exchanges posted to Unidata maintained email lists are
>> recorded in the Unidata inquiry tracking system and made publicly
>> available through the web. Users who post to any of the lists we
>> maintain are reminded to remove any personal information that they
>> do not want to be made public.
>>
>>
>> ldm-users mailing list
>> ldm-users@xxxxxxxxxxxxxxxx
>> For list information or to unsubscribe, visit:
>> https://www.unidata.ucar.edu/mailing_lists/
>>
>
>
> --
> - Mike Zuranski
>
--
- Mike Zuranski
