Very Long Running PHP Scripts - How to Make it Work With Wappler

Hi.
I found this post relevant to the point I wish to get some insights on: Long server action freezes up the entire website

Similar to what is described in the post, I too have created an extremely long running SA which synchronises my DB copy of Shopify product data. It may take anywhere from 3 to 10 minutes to complete.
The concept is pretty much similar:

  1. Fetch a set of product IDs from DB.
  2. For each ID, hit Shopify API to fetch product variant information.
  3. Upsert the response in another DB table.
  4. Keep repeating for each ID until done.
  5. Run this script periodically using CRON jobs.

While running this, what I discovered is that after 120 seconds, I was being returned a “connection timeout” error from the server.
I ran this SA directly in the browser/postman as an API call to test and also via cURL in terminal.
After a long discussion with my hosting provider, I discovered two things:

  1. There is something called webserver timeout, apart from PHP’s max execution timeout setting.
  2. An API call should not last more than 120 seconds - 2 minutes - as a recommended approach. Nothing wrong with it itself, but a long running call means blocked resource for other API calls and requests on the webserver.
  3. I played around with script timeout settings in Wappler SA, as well as PHP INI settings - but at the end, it was web server timeout that made the difference.

So here’s what I would like from the team and community to help me with:

  1. Has anyone had any experience with running long scripts in PHP via cron job or API call with Wappler or otherwise? What do you recommend?
  2. Is there any way to run a SA PHP file directly from terminal, without having to use cURL? SA provides an extremely easy and secure way of building out complex stuff. So I wish to use SA to build my set of tasks and not use core PHP.
  3. Is there any other way to handle the task of synchronizing data from a 3rd party API (like Shopify) into my own DB other than long running PHP script? I have though about breaking it into batches, but could not figure out a Wappler-based way of doing it.

Using PHP & Wappler are constants - I would not go around this unless the other method is extremely rewarding.

Community Page
Last updated: