Obviously, DevOps has been a focus of my career for the last couple years, and within DevOps, I tend to be more Ops focused. This would make sense given my stint at Splunk and a couple of other operational intelligence based startups. However, over the last few months, I’ve had the opportunity to work with a startup that focuses specifically on big data platforms as a managed service. Having the opportunity to get an intimate look at these big data platforms, I’ve learned one thing; although big data has been around for while, and DevOps is the hot buzzword right now, big data isn’t going anywhere.
In my opinion, we are just scratching the surface of what big data platforms can do for us. With the combination of multiple big data platforms such as Hadoop, MongoDB, Elastic Search, Couchbase and Basho’s RiakCS, as well as a host of others, there is really no limit to what we are going to be able to do in the future. While working at NOAA, I saw a lot of statistical data modeling done in a very archaic way. I definitely see a play for big data and NoSQL technology to enhance this capability in the future. The combination of NoSQL technology and data science will offer the key to many questions. My peers and I are currently working on big data projects in multiple verticals, including finance, insurance, logistics, manufacturing and retail. Each of these industries will reap massive benefits from leveraging the combination of big data platforms with data science. One example that comes to mind is the use of big data in hard disk drive manufacturing. HGST, the largest manufacturer of Western Digital hard drives, came to my company looking for a way to leverage their data to enhance their production. During the initial engagement, HGST stated that they were only producing a 50% yield of hard drives from their assembly line. By leveraging their consolidated data in a big data platform, and being able to analyze it, they were able to increase their yield by 3%, which doesn’t seem like a big deal, but it translated into tens of millions per year in savings. You can download the white paper here.
Don’t confuse my message, as I believe vehemently that DevOps is a necessary evolution for all modern business, but big data is here, it’s here to stay, and any organization that doesn’t embrace that is bound for failure.