When we talk about "the largest online school in Europe", there are many things that come to mind - 2,000 lessons per hour, 15,000 teachers, 120,000 students, and for infrastructure engineers like me, it means dealing with over 200 servers and hundreds of micro and not-so-micro services, SSLs, and certificates.
(https://assets-global.website-files.com/626ff4d25aca2edf4325ff97/627534f68e918f3a7f6c8976_152-header.jpeg)
Even though Let's Encrypt certificates are free and automated, they have a short validity period of only three months, which requires frequent updates. At first, we struggled with manual work and constant breaks, but we developed a simple and reliable method for updating our pile of certificates.
However, as we expanded and added more servers, we faced challenges such as inaccessible servers behind incoming traffic balancers requiring manual certificate copying and servers without HTTP that also require manual certificate copying.
We even resorted to using self-signed certificates for internal testing, which posed difficulties later on. To find a solution, we started monitoring certificate validity and considered using wildcard certificates from Let's Encrypt, but this required confirmation of domain ownership through AWS Route53 and expanding access details across all servers, including new ones.
While it's great to hear about the method for obtaining LE certificates, it would be even better if we could see a complete role/playbook and how it works instead of just parts of the code. I'm not the only one who would be interested in seeing it.
Additionally, I believe that using LE certificates in a web server may not be the most architecturally elegant solution.
Yes, the scheme is not the simplest, but the weakest link: programmers and testers. Although the creation of specialized test servers seems like a natural solution to the problem. In general, it seems that the whole development, i.e. the main activity of the company "dances" around certification. Three months, of course, is a very short time to regularly carry out this painstaking work manually. Well, since a solution has been found, maybe someone else will adopt this method. :D
To simplify the process, you mentioned using Let's Encrypt certificates, which are free and automated but have a short validity period. It's great that you have developed a reliable method for updating the certificates. However, as you expanded and added more servers, you faced additional challenges such as inaccessible servers and those without HTTP, which required manual certificate copying.
You also mentioned using self-signed certificates for internal testing, which caused difficulties later on. To find a solution, you started monitoring certificate validity and considered using wildcard certificates from Let's Encrypt. This would require confirming domain ownership through AWS Route53 and expanding access details across all servers, including new ones.
Based on the challenges you've described, here are a few suggestions that might help you manage SSL certificates more effectively:
1. Certificate Automation: Since Let's Encrypt certificates have a short validity period, it's crucial to automate the certificate renewal process. You mentioned already developing a reliable method for updating certificates, but if there is room for improvement or optimization, you might consider exploring more robust automation tools or frameworks.
2. Load Balancer Configuration: For servers behind incoming traffic balancers that require manual certificate copying, you can consider configuring the load balancer to handle SSL termination. This way, the load balancer takes care of SSL certificates and forwards the traffic to the backend servers with HTTP.
3. Server Connectivity: For servers without HTTP that also require certificate copying, you can explore alternative methods such as Secure File Transfer Protocol (SFTP) or Secure Shell (SSH) tunneling to securely copy certificates to those servers without relying on HTTP. This can help reduce the need for manual intervention.
4. Internal Testing: Instead of using self-signed certificates for internal testing, you might consider using a separate certificate authority within your infrastructure for generating trusted test certificates. This can help avoid difficulties when transitioning to production environments.
5. Wildcard Certificates: If it's feasible for your infrastructure, using wildcard certificates from Let's Encrypt can simplify certificate management. However, keep in mind that this approach requires domain ownership confirmation through AWS Route53 and ensuring access details are updated across all servers, including new ones.
6. Certificate Monitoring: Continuously monitoring the validity of certificates is essential. You can set up automated monitoring tools or external services that check certificate expiration dates and send alerts when renewals are needed. This allows you to proactively manage certificates and avoid unnecessary downtime due to expired certificates.
considerations for managing SSL certificates in your online school infrastructure:
1. Centralized Certificate Management: Consider implementing a centralized certificate management solution that can handle the distribution and renewal of SSL certificates across all your servers. Tools like HashiCorp Vault, Certbot, or even custom scripts can help automate certificate deployment and renewal.
2. Certificate Transparency Monitoring: Keep an eye on the Certificate Transparency (CT) logs to ensure that your SSL certificates are being correctly logged and maintained. This helps maintain the integrity of your certificate ecosystem and provides visibility into any unauthorized certificates issued for your domains.
3. Configuration Management Systems: Utilize configuration management systems like Ansible, Puppet, or Chef to enforce SSL certificate configurations across your server infrastructure. This ensures consistency and avoids manual errors when deploying or updating certificates.
4. Infrastructure as Code: Consider treating your infrastructure as code using tools like Terraform or CloudFormation. This allows you to define your SSL certificate configurations along with your infrastructure, making it easier to manage certificates alongside server provisioning.
5. Certificate Revocation: In case of compromised certificates or server decommissioning, make sure to promptly revoke certificates using the appropriate revocation mechanisms. This ensures that any compromised certificates are no longer valid and helps maintain the security of your infrastructure.
6. dоcumentation and Standardization: dоcument the process for managing SSL certificates in your infrastructure and establish standard procedures for requesting, updating, and renewing certificates. This helps ensure consistency and facilitates knowledge sharing among your infrastructure team.