My understanding of Big data and Cyber security

Its data everywhere and now it’s my turn to write about it.

All of us have phones,laptops and other internet connected devices. The reduction in the cost of storage devices (especially with the advent of cloud computing) has enabled organisations to store data from every source available, leading to the accumulation of data in volumes unimaginable. (BIG DATA!!!!!)

The operating system(OS) manages resources within the system. It determines the allocation of these resources to the multiple tasks and allow them access in an orderly manner. An OS is large and has complex programs, therefore security issues will be present.

Every organization has a network protected by firewalls, proxy servers, domain controllers, intrusion detection systems and Virtual Private Network (VPN) servers’ that contain information about situational responses of the OS or the system as such. This can be used for detecting operations that are prone to stealthy attacks. Big data not only helps to make data available only to authorized entities but also offers the transmission of data much more faster than conventional methods due to the advanced type of distributed file system(like Hadoop Distributed File System)

As far as computing world is concerned, systems are being programmed to become self-managing i.e. move towards autonomic computing. The system makes decisions on its own, using high-level policies. Constant checks are done by the system to optimize the functioning and updates are installed according to changing conditions. 

The big data repositories can collect data relating to previous cases of faults in operation. In analogy or case based reasoning, the system retrieves best matching old cases and suggests relevant problem descriptors. Some portion of the past is then used to form the solution to the new problem. The technological advances in storage and processing of big data like the development of new frameworks such as Hadoop help data scientists to predict the possible behavior from a large set of possible behaviors to counter any change in the system.


Advanced Persistent Threats(APT) occur in multiple stages. Therefore each action by the attacker provides an opportunity to detect behavioural deviations from the ideal. For example, in the delivery stage the attacker either attempts direct contact with the target or tries to compromise a 3rd party application the target uses.

It cannot be denied that managing big data systems involves various security issues. Improvised and complex resource management mechanisms are also needed to utilize the tools. Also platform layers for MapReduce, Hadoop Distributed File System(HDFS), YARN etc, need to be created to run them separately. Many open source and proprietary solutions have been proposed and developed to address these requirements. Also its efficiency in processing such data accumulation rate outweigh the liabilities.Reporting and visualization of predicted outcomes can be done effectively. Big data analysis systems, such as MapReduce and Spark provide the programming-interface and address the parallel computing needs in security analysis as they are said to be  highly resistant to failure.

Big data can be used to carry out predictive analysis on how personal systems work under threats and incorporate self resurrecting programs to solve minor issues without connecting to the manufacturer. According to an analysis report by Symantec in 2012, most successful APT attacks on small companies is due to the fact that they did not have a proper IT team monitoring the cyber security of their company network. Any organisation, large or small, must have a team in the IT department dedicated to analyzing the data transmission happening within the organisation. Companies that carry out critical business operation must protect their data from intentional attacks.

So from whatever I have read and learnt ,big data seems to be the next big thing.


Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )


Connecting to %s