Is MongoDB the right choice for me?
Greetings! I have an application that generates a significant amount of real-time logs.

Currently, these logs are saved in a file format. However, to ensure ease of use and accessibility on the web, I am interested in migrating them to a database. The log types are diverse and do not follow a particular structure; they could entail anything from user login events to server disconnects.

Would MongoDB be a suitable option to accomplish this task? Alternatively, is the conventional MySQL approach preferable?

I had been using MySQL for storage, and while it recorded everything quickly, there was one critical issue. With multiple message options, creating a table for each one was not feasible, so the structure was only defined at the top level, with some fields in the BLOB. As a result, search functionality suffered, and processing was slow and prone to timeout errors.

To address this, I made the decision to switch to MongoDB. Despite accumulating several million records since the transition, it only requires 3.4GB of RAM, with only half of that being used for indexes. An incremental MapReduce is scheduled for analysis, with searches taking less than a second and MapReduce aggregations averaging a couple of seconds.

Overall, the transition to MongoDB has been seamless and trouble-free, making it a highly recommended solution for complex and variable data structures. As an added suggestion, parallel recording in both databases for a brief period can aid in analysis and comparison.

To generate statistics using logs, they need to be translated into SQL, which may require rotation systems for large amounts of data. Writing to clean tables without indexes and then transferring to indexed ones is preferable. The data is structured, so creating a table for each type of logging is simpler than SQL-NoSQL rotation. Alternatively, leading to a normalized form during rotation is an option. However, if there is a possibility of needing to track certain information, such as how many users disable the server, using mongo may be worth considering.

On a related note, it's important to carefully consider the intended use and potential future needs of data when choosing a storage and retrieval system. This can save time and resources in the long run.

My experience with MongoDB was rather disappointing. The data being stored in JSON format led to a rapid depletion of space and made aggregation and map-reduce operations unreasonably slow, causing timeouts and practically halting database entry. So, my personal observation contrasts with the popular belief that MongoDB is faster than traditional SQL databases. Perhaps, I am missing something on how to optimize its performance.

On the other hand, MySQL provides a simpler and more dependable solution. For even better performance, one can opt for MariaDB without any compromise in terms of features.

In the case of real-time logs with diverse and unstructured formats, MongoDB offers a flexible schema design compared to the rigid structure of MySQL. This means that MongoDB can efficiently handle the variable log types without requiring predefined schemas, making it suitable for accommodating the dynamic nature of your log data. Additionally, MongoDB's ability to handle large volumes of unstructured data makes it an excellent choice for managing significant amounts of real-time logs.

On the other hand, if your application requires complex transactions and extensive relational data querying, the traditional approach of using MySQL might be more suitable. MySQL provides strong support for complex joins, transactions, and relational data integrity, which could be beneficial if your application needs these features for log analysis or reporting.

Considering the web accessibility aspect, both MongoDB and MySQL can be effectively integrated into web applications. However, MongoDB's JSON-based dоcument model aligns well with the typical data formats used in web development, allowing for seamless integration with web-based technologies.

If your priority is to efficiently manage diverse and unstructured real-time logs with a focus on scalability and flexibility, MongoDB would likely be the more suitable option for your application. Conversely, if your application heavily relies on complex transactions and relational data interactions, MySQL may be the preferred choice.
It's important to conduct a thorough analysis of your application's specific requirements and consider factors such as data volume, access patterns, and future scalability needs before making a final decision. Both databases have their strengths, and the optimal choice will depend on how well their characteristics align with your application's demands.