Server that can poll and store data from 15,000 sensors

Started by kyouxngofi, Apr 18, 2023, 06:06 AM

Previous topic - Next topic

kyouxngofiTopic starter

My current project involves developing the solar energy control system, and I am personally responsible for creating the brains of this system from scratch. While the prototype is able to work autonomously, I need to develop a server that will display all the data from these brains on a web interface.
As I am also learning Debian in parallel with development, I prefer not to use third-party hosting.

The ultimate goal is to have 1500 houses equipped with this technology, wherein each house's brain would transmit sensor data such as temperature, voltage, current, illumination, etc. to a central computer. I need a database to store and retrieve all the pertinent data and to monitor the status of each brain through the web interface.

What hardware requirements do I need for this task? And how are such systems generally built?
Is there any resource or guide available that can help me understand how everything works?
  •  

doro

Collecting statistics can be done with any hardware, and you don't necessarily need a dedicated IP on the server or use MySQL (dynamic DNS is also an option).

As for the monitoring and display, any language that allows for programming can be used. You can put together a Bootstrap + PHP interface in a matter of hours. All in all, the task is straightforward.
  •  

john121

Based on your comment, it seems like you prefer starting from the beginning. In that case, you may require a library such as Boost Asio or any other comparable ones.

You can find numerous articles on Habr regarding this topic, like Asio or libevent. Additionally, connecting to a database is not difficult, as there are multiple C++ bindings available, including MySQL.

However, if dealing with "low-level" sockets, multithreading, and debugging seems daunting, then you could opt for Erlang, which will solve your problem perfectly.
  •  

addisoncave

Check out graphite.wikidot.com. It offers a user-friendly database and an easy-to-use API for adding data. It also includes a web interface, numerous aggregate functions, scalability, and compatibility with various systems and applications.

The platform enables plotting of any data value, and includes bulk data entry functionality for any time interval. Installation and mastery of the system are straightforward; it can accommodate small installations monitoring just a few values, as well as large-scale installations like google's, which collects parameters from tens of thousands of objects.
  •