Re: [ldm-users] 20200423: Re: Efficiency of splitting pqacts

  • To: "Herzmann, Daryl E [AGRON]" <akrherz@xxxxxxxxxxx>
  • Subject: Re: [ldm-users] 20200423: Re: Efficiency of splitting pqacts
  • From: Steven Emmerson <emmerson@xxxxxxxx>
  • Date: Thu, 23 Apr 2020 17:51:43 -0600
Hi Daryl,

On Thu, Apr 23, 2020 at 1:37 PM Herzmann, Daryl E [AGRON] <
akrherz@xxxxxxxxxxx> wrote:

> I am sure Unidata will correct my ignorance / incorrect details, but my
> understanding is that an individual pqact process can only do 32 "things"
> at one time, or there's 32 slots available for work.
>

The "things" are file descriptors and the Unix standard only *guarantees* 32
of them. Modern O/S provide much more.


> Now, the above depends on the action.  If you run `PIPE -close`, the slot


The "slot" is a file descriptor.


> can be used for another product even with the PIPEd process still
> running...


The PIPE-d process might still be running, but the file descriptor will
have been closed -- so, yes, that descriptor can be reused.


>   This type of action can lead LDM to DOSing the server it is on as it
> will fire off as many PIPE'd processes that it can.


If the decoders take a long time to process the data-product, then, yes,
you can wind up with many of them. This is not a problem with pqact(1) *per
se*, but with inefficient decoders (pure Python, for example, is about
three orders of magnitude slower than C).

Regards,
Steve Emmerson
  • 2020 messages navigation, sorted by:
    1. Thread
    2. Subject
    3. Author
    4. Date
    5. ↑ Table Of Contents
  • Search the ldm-users archives: