Let's say there's a PHP script that does a lot of work, such as parsing, for several hours. However, the hosting has restrictions, like max_execution_time set to 600 and cannot be increased. I'm interested in methods to bypass this time limit.

One possible approach is to use AJAX requests with JavaScript, where the script processes only part of the data, like parsing only 10 pages. The use of JS allows for adding progress bars. However, this solution may not work for cron scripts. I'm curious about other options available, such as libraries or examples of implementation.

It's unfortunate that using PHP to solve these types of problems is difficult. Other possible solutions include message queues, executing (pseudo)multi-threaded code in PHP, or rewriting the script in another language like Perl, Python, C, C++, or Java.
Though it's not ideal, you could also use socket or file_get_contents to trigger recursive execution of the script.

To work within the given conditions, the only viable option is to break down the big task into smaller ones. A working process should be run with cron every five minutes, which will check if its previous counterpart is still active, and if so, continue working on the next small task. It's important to optimize the execution time to minimize downtime, without exceeding the five-minute limit. Additionally, it's worth checking if the web hosting capabilities allow for running several "workers" in parallel to increase efficiency.

However, hosting services often restrict the number of processes and memory, making it challenging to optimize performance.
Progress can be tracked by updating a file on disk or a database entry, and a web client can poll an AJAX script once a second for updates. If the restrictions cannot be bypassed, one option is to use a personal virtual server, like the minimum box on DigitalOcean, which costs $5 a month. By using a provided referral link, users can receive ten dollars credit to their account to try out the service.

Creating your own autoscaling system is not as difficult as it seems, and there's no need to rely on expensive options like Amazon, Google, or Microsoft Azure, which can be at least three times more costly than using your own resources. It's worth considering that you may already have everything you need and don't need to migrate to a different system or technology. Your current virtualization setup may suffice.

By implementing your own autoscaling system, you can save company resources, including electricity, disk space, and hardware machine slots. Additionally, you'll save time by not having to fix issues in the middle of the night. A properly functioning autoscaling system can keep the system running smoothly without the need for manual intervention.

It's important to have developers who can write the right applications for this setup. The previous presentation focused on writing correct code, and now the focus is on creating the right applications. All the necessary utilities can be found on GitHub.

The AJAX approach you mentioned is a solid start, but let's explore a few more strategies that could help you bypass this execution time limit.
You might consider breaking down your script into smaller chunks. This is similar to your AJAX idea but can also be implemented server-side. You can create a loop that processes a portion of your data and then stores the state of where it left off. By doing this, you can run the script multiple times with the help of, for example, cron jobs or a scheduled task. Each execution would pick up where the last one ended. This would allow the script to remain within the time constraints while effectively getting the job done over several iterations.

Another approach could involve using the PHP `ignore_user_abort` function, which allows the script to continue running even if the user that initiated the script disconnects. You can set it to true at the beginning of your script. However, remember this wouldn't help with the max_execution_time limit itself, it only ensures a more persistent execution.

You could also explore using message queues like RabbitMQ or Redis. In this setup, your main script can push tasks to the queue, with worker scripts retrieving and processing these tasks at their own pace. This can effectively distribute the workload and avoid long execution times on a single script.

Consider using PHP's built-in command line interface (CLI) for executing your scripts. The CLI version of PHP does not have the same execution time limits as web-based scripts, so you can run your long processes without running into the max_execution_time issue. You can schedule your CLI scripts using cron jobs, or even run them manually when you need them.

If you have access to your own server and are comfortable with PHP frameworks, you could look into something like Laravel's job queues. Laravel offers a rich set of tools for background processing, including retries and task scheduling. This can abstract some of the complexity away and help manage long-running tasks more efficiently.

If your script is already structured to use databases, you could implement a system of "checkpoints" within your data processing. For example, if parsing document pages, update a database record after processing a specific page or group of pages. The next execution of the script can then query this record to continue from the last completed page.
There are multiple strategies you can use to tackle the max_execution_time issue in PHP. Breaking your workload into manageable chunks, using background processes via CLI or message queues, and utilizing frameworks or libraries for task management can all lead to effective solutions. Evaluating what fits best with your current setup and requirements will be key for a successful implementation.

Reduce the number of queries: Minimize the database queries by combining them, avoiding unnecessary joins, and using indexed columns for fast lookup.

Use caching: Implement caching using tools like Redis or Memcached to store frequent query results and reduce database load.

Optimize SQL queries: Use EXPLAIN to analyze your SQL queries and ensure they are efficient.

A highly loaded PHP script may experience performance issues due to excessive resource consumption, leading to slow response times or timeouts. To optimize, consider refactoring code, reducing database queries, caching results, and implementing efficient algorithms. Additionally, monitor server performance to identify bottlenecks.