Big Data and Gardening

You can be a botanist, a horticulturist or a gardener—it depends on how knowledgeable you want to be about what you are growing. But what’s that got to do with Big Data?

Well, I was recently having a conversation with Matt Ritter from Athena Health about Big Data, analytics, the role of the mainframe, and much more, and he posed what I thought was a really interesting question: What is the line between Business Intelligence and Data Science? His passion on this subject impressed me, and it reminded me of the passion of another person—my mother.

No, my mother is not a data scientist, but she is incredibly passionate about gardening. She loves it with every fiber of her being—but she is “just” a gardener. She has no formal training in horticulture, so she dabbles in something that interests her and, of course, she gets a lot from that. I suppose if she wanted to go to the next level she could have gone back to school to become a horticulturist. If she had done that, I’m sure her passion for gardening would have expanded: she would certainly have a much deeper understanding of how to make her garden successful.

She could perhaps be able to identify problems with her garden, and even be able to select appropriate flora whose harmonious and interdependent nature would actively contribute to the success of her garden; like planting marigolds next to your strawberries—that’s true, check it out. From there, the ultimate would be to go all the way to becoming a botanist. This, of course, would require many more years of study, but she would have the ability to dramatically increase the effectiveness of her garden to the level that an “amateur” gardener would have no hope of achieving, perhaps even engineering new plant species to do precisely what she wants in her garden.

Big Data is a bit like this. Big Data holds the promise of great insight, but it’s by no means a magic wand. And you can approach it as a simple gardener does a garden, with simple amateur knowledge, or you can become the equivalent of a Big Data botanist—a data scientist—and really get into it. Now you certainly don’t need to become a botanist to be an accomplished gardener, and in the same way, you don’t need to be a professional Data Scientist before you can get some benefit from Big Data; so where do you start?

Like most in things in life, it depends. You can get some benefit from a garden right away by learning on the job. All it costs is time and effort, and it doesn’t adversely affect the rest of your life in any way. And you’ll get better at it over time. But if you want to know everything there is to know on the subject, or if your livelihood depends on it, you’d better obtain some expertise, either invest in education or hire expertise. The real trick is knowing when you’ve reached your limits as an amateur gardener, and when you need a horticulturist or a botanist.

And it’s the same with Big Data. It’s easy to start looking at business intelligence with some basic tools, but it takes data science to really understand things at a deeper level. The need varies and can depend on which industry you’re looking at: a retailer might simply want to know during which week of the month the highest concentration of sales occur. That is a pretty straightforward business intelligence problem, and can be answered without much data science expertise—the answers are in the data just waiting to be uncovered.

But understanding why it occurs on that week versus another week is another question, and knowing what is driving that behavior is where we start to get into real data science. So in retail, and many other industries, you can start to obtain benefits without much data science expertise, and dig deeper from there. However, there are some industries, like health care in which Mr. Ritter works, where there are compelling reasons, and a clear need to understand trends and issues at a deep level, so there can be a need for data science expertise right from the beginning.

The term Big Data is—well—big. It covers everything from IT hardware, to specialized IT software, to analytics, to business intelligence, to data science and to how we can create some real actionable intelligence from what we learn. As I mentioned in a previous article, you can start small with Big Data projects and that can apply to your data science expertise as well, depending on your particular industry, situation, or specific business intelligence needs. So where is the line between Business Intelligence and Data Science? Well, you decide where it is—but make sure you know that there is a line.

As for me, I sometimes feel lucky when I can actually get a tomato out of my garden.