"Big Data" is a buzz term that is thrown around in both the IT and Developer space. It is often associated with other terms like Hadoop, Business Intelligence (BI), NoSQL, unstructured data, and more. The question is, as a developer, should you care about big data and these associated terms?
One of the things we did in order to better understand the business drivers for big data as well as to understand the benefits and challenges, is a survey. When I say we, it wasn’t actually Developer.com, but rather our parent company, QuinStreet. The results were the report, QuinStreet Enterprise Research: 2014 Big Data Outlook (click to download).
This survey tells what people really think about big data. The result was that over 77 percent of the companies thought that big data analytics was important as a priority. Even so, there were a number of issue and concerns raised. Simply dealing with large amounts of data is a concern in itself. Understanding the technology that can process large amounts is also a concern. Over 46 percent of the companies surveyed stated that they were already working with over ten terabytes of data or more, and 21 percent are working with over 100 terabytes in a given month. Dealing with large amounts of data can mean that the underlying infrastructure has to scale to accommodate the volumes.
Of course, dealing with this over abundance of data requires tools, which is another challenge that was identified. There is a variety of tools that can be used, and determining which is the best, is a concern. In the survey, it is noted that 43 percent say they don’t have a lead tool vendor.
Big data might have challenges, but it also brings benefits. Many people see data transparency, timely data access, and thus more accurate business decisions as all being some of the benefits of big data analytics.
Big Data and Developers
If used right, big data can help businesses. The report gives many examples of business cases for using big data, it talks about types of appliances being used, and it talks about the technologies being considered. As with any technology being rolled into an organization, you can expect customization and integration tweaks to be needed. These tasks often fall on developers. As such, the more you know about big data now, the better you'll be prepared when big data comes to your organization — assuming it isn't happening already!
As a developer, if you are in a mid to large size organization, you should care about big data. Even if you are in a small organization, you might find that you need to pull external big data sources into what you are doing, so you also might need to care!
About the Report
The research study was conducted by QuinStreet via an email invitation to the subscriber base. The target of the survey was subscribers involved in big data purchasing decisions. There were 540 completed surveys, which gives the survey a +/- accuracy of 4.3 percent. To get a copy of the report, you can go here.
# # #