Skip to Main Content
Spotfire Ideas Portal
Status Future Consideration
Categories performance
Created by Guest
Created on Mar 14, 2023

Parallel start execution of two or more scripts/jobs

dear sir/madam,


I have multiple scrips that are independent of each other, for example scripts to do some calculations and then insert/update/delete data into tables in target database.

For example script01, script02 and script03.

Each script can take 10-15 minutes. Execution in sequence take a lot of time probably 30-45 minutes. Instead of this I want to do this parallel, like a cache refresh for independent tables inside of a cache policy.

I would like to start scripts (script01, script02, script03) at once by a main script. Then main script should be able to start all three scripts, after each others and without waiting of execution of previous one.

call script01;

call script02;

call script03;

In this case then total time is will be +-runtime of longest script, about 15 minutes.

Option with three triggers is not really desired because the execution of those 3 scripts is a small part of a large processing some where between other operations.

Preferably we want also to finish the main script only when all 3 sub scripts are done (nice to have).

I tried implement this with using of standard TDV procedure /services/webservices/system/admin/"execute"/operations/executeProcedure but unfortunately parameter "isBlocking" (true or false) doesn't works properly, it waits for the execution to be over in case of true and in case of false. Bug?

Description of TDV8.3 for this procedure parameter:

If "isBlocking" is "true", then this operation will not return until the

processing associated with the execution has completed and the "outputs" element will be set.

Otherwise, this operation will return immediately and the "outputs" element will be unset.

Please implement this functionality or fix executeProcedure bug.

Any data processing, ETL or integration tool has this kind of functionality TDV have to provide this also


regards,


Roman


  • Attach files
  • Guest
    May 15, 2023

    Hi Roman,

    One possible workaround for this is to make sure all these procedures have a a similar return.
    You can then call then in a cursor using UNION ALL {OPTION PARALLEL} like so:

    Select Output from script01
    UNION ALL {OPTION PARALLEL}
    Select Output from script02
    UNION ALL {OPTION PARALLEL}
    Select Output from script03;

    The script will continue once it has responses from all three queries that should run in parallel

    With Kind Regards,
    Robbert