Increasingly, organizations today are facing more and more Big Data challenges. Big data analytics reveals insights hidden previously by data too costly to process, such as peer influence among customers, revealed by analyzing shoppers’ transactions and social and geographical data in real time.
The meaning of real time can vary depending on the context in which it is used. Real-time denotes the ability to process data as it arrives, rather than storing the data and retrieving it at some point in the future. But “the present” also has different meanings to different users.
From the perspective of an online merchant, the present means the attention span of a potential customer. If the processing time of a transaction exceeds the customer’s attention span, the merchant doesn’t consider it real time. From the perspective of an options trader, however, real time means milliseconds. From the perspective of a guided missile, real time means microseconds. Real-time big data analytics is an iterative process involving multiple tools and systems.
Hardware failure is the norm rather than the exception. An HDFS instance may consist of hundreds or thousands of server machines, each storing part of the file system’s data. The fact that there are a huge number of components and that each component has a nontrivial probability of failure means that some component of HDFS is always non-functional.