The algorithms responsible for dividing the workload within the various workstations balances incoming and outgoing requests for achieving the best response time. This type of database is useful when huge data needs to be stored in the database. It is physically not possible to save on a single machine. The bits like log files, data collected by tracking click-throughs in the application, and data generated by IoT can be stored somewhere else. They are often referred to as distributed databases.
The reason behind splitting up a database:
Size:
Certain databases are larger than they can be stored on a single physical drive. The data sets must be divided across various machines.
Demand:
The database performance is affected significantly when many users try to access the data. When the workload is divided among various machines. They can answer more requests and the uses don’t see the performance delay.
Redundancy:
With drive failures into consideration, it is a good step to create various copies and store them within various versions can provide a lot of protection to the valuable data.
Geographically redundant:
When there are natural disasters, power failure, or catastrophic fire, spreading the database across various machines can protect the data in such conditions.
Speed:
Network latency is often an issue when the database and the user queries are placed geographically apart. When the database is placed close to the user’s geographical location the response is faster. This is because the data does not have to travel very far. Speed is an important factor for projects where people work from different continents.
Computational load:
Machine learning applications can distribute large data sets across various systems to spread out the analytical work which can be substantial.
Privacy:
The database is divided to maximize privacy and eliminate the risk of data breaches. If different parts of the database are stored on different machines, then in case of a breach of one machine the rest of the data remains safe.
Politics:
In case when multiple groups use the same data set the governance can prove to be challenging. The data is stored across various versions. It can be useful if some data is with one group and some other data is governed by another group. This splitting of the database within various workstations is called sharding which can inspire strategies that range from simple to complex.
It is challenging to distributed database:
Information consistency remains the biggest challenge as far as the splitting of the data is concerned. If we consider a hypothetical airline booking system, the machine may respond to a database query for or a sold-out seat then another machine may respond to the query by saying that this is available. Some Distributed databases include the enforcement of consistency rules so that all the user queries are given the same answer regardless of which computer or cluster they contact.
In other distributed databases the database relaxes the consistency requirement by following the principle of “eventual consistency”. With this concept, the machines can be out of sync with each other and may return different answers. But eventually, these patients will catch up with each other and provide sem results. One machine may not hear about the new version of the data stored on another machine for some time.
Machines that belong to the same data center tend to reach consistency faster than those separated by long-distance or slow networks. The database developers need to choose between fast response and consistent answers. Tight synchronization within the distributed systems can increase the amount of computation and slow out the responses. But the answers will turn out to be more accurate. However, if the data is out of sync, it will speed up the performance but the accuracy will remain compromised. It is the art of choosing between the prioritization of speed or accuracy. For instance, banks know that the customers want correct account details more than faster responses.
The legacy approach to the distributed system:
Major database companies provide various options for the storage of distributed databases. Some of them support large machines with multiple processors, multiple disks, and large blocks of RAM. Technically the machine is a single computer but with individual processors. It co-ordinates the responses in ways as if the processors were separated by continents. Many organizations run their Oracle and SAP deployment using Amazon Web Services to take advantage of the computing power. AWS’s u-24tb1.metal looks like one machine but it has 448 processors inside along with 24 terabytes of RAM. It is optimized for large databases like SAP’s HANA which stores huge information in RAM for faster response.
All the major databases have options to replicate the database for the creation of distributed versions that are divided within more distinct machines. For example, the Oracle database has supported a wide range of replication strategies across collections of machines that can include non-Oracle databases. Lately, Oracle has started marketing a version with the name autonomous to signify that it can scale and replicate itself automatically as a response to loads.
MariaDB which is the fork of MySQL supports a variety of application strategies. It enables the data from primary nodes to pass copies of all the transactions to replicas that are set up to be read-only. That is, the replica can answer queries for information but does not store any new data. In a recent presentation, Max Mether, one of the co-founders of MariaDB, said that the company is working hard to add autonomous abilities to its database.
Upstarts handle the distributed database differently:
The increased use of cloud services hides the complexity of distributing databases for or configuration of the server and arranging for the connection. DigitalOcean offers managed versions of MySQL, PostgreSQL, and Redis. Clusters can be created with a certain size and with a single control panel to provide storage and failover. Some providers have added the ability to spread out clusters within different data centers around the world. Amazon’s RDS can configure clusters that span multiple areas known as “availability zones”.
Online file storage has also started to offer much of a similar kind of replication. The services that offer to store blocks of data don’t provide the indexing for complex searching of databases. Yet, they do provide replication as part of the deal. Some ideas work too much more complex calculations with Distributed Data sets. Tools like Hadoop and Spark are two popular open-source constellations of tools that match distributed computation with distributed data. Dene various companies that specialize in supporting versions that have been installed in-house or include configuration.
Groups that value privacy are also exploring complicated distributed operations like Interplanetary File System which is a project to spread web data among multiple locations for increased speed and redundancy.
Limitations of a distributed database:
Not all work requires the complexity of coordinating multiple machines. Some projects may be labeled big data by project managers even though the volume and computational load can be easily handled within a single machine. If a fast response time is not essential and if the size is not too large a simpler database with regular backups could be sufficient.
