I'm thinking of creating a minimalistic panel that would allow adding and deleting users, setting up websites and databases. My main concern is developing an installer to automate the process of installing services and features on the server.
I want all the scripts to work in PHP and have a modern user interface. I'm considering using the scripts from VESTA CP, which are already implemented in PHP and can be executed via exec().

Do you have any advice on the best way to go about this?

At the current stage of designing, I am engaged in a similar activity. However, I have realized that developing a panel for multiple operating systems is not a viable option due to the extensive amount of time it would require.

It is crucial to comprehend the workings of the OS level, web server, database server, and other components to produce a secure and extendable panel without bugs. Simply copy-pasting scripts from VestaCP will not suffice as it requires a deeper understanding of the system.

It's important to recognize that each product has its own target audience. ISPmanager is a highly effective option within its specific niche. That said, if you're an advanced user who doesn't require a visual interface, there may be other products that suit your needs better than ISPmanager - but that doesn't mean ISPmanager is inferior.

An important consideration for hosting providers is that some customers will be accustomed to working with cPanel or ISPmanager. Even if your offering is objectively superior, these customers may find it difficult to adapt to something new. Ultimately, marketing decisions must be balanced with practical considerations.

Easily accessible ISPConfig exists along with numerous auto-installer scripts that can be found on Github. One can swiftly configure it by removing unwanted modules. However, if one chooses to recreate something that's already available, they'll have to invest a considerable amount of time and effort into the task.
This effort could well be spent on programming websites for clients or simply taking more time off. It's unlikely that an employer would appreciate paying for the recreation of something that already exists.

Let's acknowledge that VESTA CP scripts are a decent starting point. They're already written in PHP, which is a plus. However, you'll need to carefully evaluate their compatibility with your envisioned minimalistic panel's architecture. You might need to refactor or rewrite parts of the scripts to ensure they integrate smoothly with your project.

Now, regarding the installer itself. You're wise to aim for a modern user interface. I recommend exploring PHP frameworks like Laravel or Symfony, which offer robust tools for building web applications. These frameworks can help you create a sleek, user-friendly interface that will make your installer a joy to use.

Next, you'll need to tackle the installation process itself. This is where things can get messy. You'll need to write scripts that can:

Detect the server's operating system and architecture
Install required services and dependencies
Configure settings for each service
Integrate with your minimalistic panel's database
To make this process more manageable, consider breaking it down into smaller, modular scripts. This will allow you to focus on one aspect at a time, rather than trying to tackle the entire installation process at once.

Another crucial aspect is error handling. Your installer should be able to detect and report errors in a clear, concise manner. This will help you identify and troubleshoot issues more efficiently.

Finally, don't underestimate the importance of testing. You'll need to thoroughly test your installer to ensure it works seamlessly across various environments and scenarios. This might involve setting up test servers, simulating different installation scenarios, and gathering feedback from beta testers.
Creating a robust installer is a challenging task that requires careful planning, modular design, and thorough testing. By leveraging PHP frameworks, VESTA CP scripts, and your own expertise, you can build an installer that will make your minimalistic panel a joy to use. Just remember to stay focused, and don't be afraid to ask for help when needed.