A few notes from my Big Data and Analytics classes – MBA Learnings

Technology analyst Benedict Evans shared an interesting image from a classic 1960 film “The Apartment.” The scene is set in the office of a large insurance company in New York – drones laid out at desks almost as far at the eye can see. Each desk has a telephone, rolodex, typewriter and a large electro-mechanical calculating machine.


As Ben points out – “In effect, every person on that floor is a cell in a spreadsheet. The floor is a worksheet and the building is an Excel file, with thousands of cells each containing a single person. The links between cells are made up of a typewriter, carbon copies (‘CC’) and an internal mail system, and it takes days to refresh whenever someone on the top floor presses ‘F9’.”

(Incidentally, as the protagonists are a desk worker and an elevator attendant, this is actually a romance between a button and a spreadsheet cell.)

It is clear that the capabilities of analysis tools in 1960 was far below our ability to analyze them. So, Microsoft Excel and other spreadsheet programs offered huge benefit simply because they helped bridge the gap between the average manager’s ability to analyze data and the tools available to do so. This, in turn, spurred businesses to collect more data in the hope of extracting insights. So, over the late 90s and the 2000s, every junior consultant and investment banker became an Excel ninja. Being able to use the tool to the best extent possible added real value.

All was well. Until “big data” entered the picture.

Excel spreadsheets had a major capacity upgrade recently that finally allowed a million rows. However, that makes it massively inadequate for a real world “big data” dataset. So, what is “big data?”

The consensus is that big data refers to data sets that have 3 V’s – volume (i.e. size), velocity (speed of data in and out) and variety (range of data types and sources). These data sets are in the size of hundreds of millions of rows with inputs coming in every second. To imagine a big data set, imagine a massive spreadsheet that receives point-of-sale data for McDonalds or Wal-Mart in real-time.

The next question, then, is – how do we make sense of all of this? It is hard to have a simple answer to this “big” question. So, I’ll share a couple of observations from my big data and analytics classes –

  • As any person who has analyzed data in Excel will tell you, you can make your data tell any number of stories. The presence of large amounts of data doesn’t change that fact. If anything, it becomes easier to manipulate the data to tell the story you want.
  • Additionally, the biggest problem plaguing poor analysis – mixing correlation and causality – definitely doesn’t go away. While correlation can be instructive in itself (sophisticated retailers have used correlated buys successfully to push the right coupons), it is dangerous to imply causal relationships because of a number of reasons – e.g. there could be a third effect that causes both.
  • Big data has increased our ability to experiment with different campaigns and messages. However, unless the experiments are well designed and executed with groups that are perfectly random (or close), these results can be erroneous.
  • In some ways, this gets to the root of the fundamental issue with analysis – analysis, by itself, is generally useless. Analysis, supported by business judgment, can be incredibly powerful.
  • The effect of this issue gets magnified when we have huge amounts of data. There are often more variables than we know what to do with. And, while machine learning tools like neural networks can help us find relationships between them, they won’t mean much if they aren’t combined with good business judgment.

All this leads me to conclude that we’re now in a situation that is very different from the 1960s picture we started with. At that time, an average manager’s analysis capabilities were far ahead of the tools and data available. Now, it is safe to say that the tools and data available far outstrip the average manager’s analysis capabilities. Forget the average manager, it is safe to say that even the most sophisticated managers will struggle with driving the right analysis and then interpreting the results right. While we can expect the tools to become easier in 5-10 years, the for analysis and insights are not going to go away any time soon. If, by chance, you are wondering as to why I am not referencing sophisticated data science teams that exist to solve this problem in leading companies, I’d like to go back to the key driver of great analysis –  “good business judgment.” The ideal analyst is someone who combines amazing tool capability with business judgment. Very few of these people exist. Great analysis is driven by managers AND data scientists. And, for managers to work well with data scientists, they need to become good consumers of analysis.

So, if there’s one thing I’ve taken away from these classes, it is the importance of doing whatever it takes to get on board the big data train.

For those who don’t plan to attend graduate school, consider online courses in statistics that cover basic statistics tools. Open source tools like R make it easy for anyone to be analytically savvy.

And, if you are fortunate enough to attend a graduate school that is emphasizing big data and analytics, take full advantage of the opportunity.

I am certainly trying to do so.

A learning from Chocolat

My wife and I watched a nice Anglo-French movie called “Chocolat” this weekend. The setting was a conservative village in France ruled by a Mayor who used religion to maintain strict norms and alienate outsiders. The village community are forced to question their beliefs when an outsider who doesn’t adhere to their strict norms opens up a Chocolate shop near the square. Over time, the village, and, in particular, the Mayor, learn to appreciate and, eventually, accept her.

There was a lovely line in the movie that summed up the challenge the villagers faced. The village priest summed it up thus –

I think that we can’t go around measuring our goodness by what we don’t do – by what we deny ourselves, what we resist, and who we exclude. I think we’ve got to measure goodness by what we embrace, what we create, and who we include.

The village, to me, was a nice analogy of the human condition. Every time we find a system that works, we are tempted to hold on to it by resisting any new force that threatens it. It is tempting, as a result, to exclude anyone who thinks differently. But, a life lived well embraces what’s new, strives to create and be inclusive.

This isn’t a “good-to-have.” It is a necessity.

After all, what got us here won’t get us there.

Downtime focus

I asked myself a question – what do I focus on when there’s downtime these days, i.e., what sort of problems does my mind wander to?

The list was fascinating as it consisted of a list of problems that weren’t really problems in the big scheme of things.

One of the biggest benefits of being human is this ability to see ourselves and our thoughts from a different perspective. The reason this is hard to do is because we don’t often like what we see.

I certainly didn’t.

But, a problem we’re aware of is a problem that stands the chance to be solved. So, here’s to awareness… and an eventual solution.

Meaningful anxiety – The 200 words project

Last week, we found that stress was found to be a predictor of a meaningful life – contrary to our generally negative perception. Researchers at Yale found that this negative perception greatly decreased the quality of our lives.

So, how do we go about switching our perception of stress?
In a study, researchers had people come into a lab about to go into a job interview for the job of their dreams. In the stressful waiting area, researchers told some participants to focus how they were going to impress the interviewers. Another group was asked to think about how the job was connected to their values and to counter anxiety by thinking about why they cared about the job so much.

The “meaning” group were rated higher and were found to be more inspiring and uplifting.

Similar studies have found that the way to switch our perception of stress is to embrace it by digging deep to understand why we care and, then, to recognize that a meaningful life is also a stressful life.

Stress, like adversity, is an opportunity to learn, grow and express our values. And, most importantly, to trust that we can handle the challenge.

When you’re feeling stressed out, just make contact with the paradox of stress. Recognize that stress is not a signal that there’s something wrong with you but a sign that something you care about is at stake, an opportunity to think about what you care about. – Kelly McGonigal (paraphrased)

Source and thanks to: Kelly McGonigal @ 99U

What do you value

Every decision you make on how you manage your energy and time is an implicit answer to the question – what do you value?

There’s no globally right way to spend your energy or time. Anyone who tells you otherwise has no idea what they’re talking about. Right and wrong depend entirely on your priorities. And, your schedule is the only reliable indicator of your priorities.

There are 2 implications of this –

1. It helps greatly if you understand what you value. The clearer your priorities, the easier it is to make decisions that are right for you.

2. It adds a level of necessary seriousness to seemingly minor time decisions. If that scares you, that’s great. Welcome to the club.

As we live our days, so we live our lives.


A few friends and I were discussing what the true signs of confidence are. One of them commented that there are people who mask large amounts of insecurity with an illusion of supreme confidence.

I agree. I’ve come to find that the truest sign of confidence is vulnerability.

Vulnerability requires you to put yourself out there and say – “This is who I am and this is what I care about” with the knowledge that it might not work with many people. The most important words in that sense are “it might not work” – it is the lack of certainty that underscores vulnerability. As Brene Brown describes it in her wonderful TED talk, vulnerability involves the willingness to say, “I love you” first, the willingness to do something where there are no guarantees, the willingness to breathe through waiting for the doctor to call after your mammogram, and the willingness to invest in a relationship that may or may not work out.

The reason this is hard is because vulnerability is also the core of shame, fear and our struggle for worthiness.

That’s also why it says so much. You can only be authentic and vulnerable if you are comfortable saying – “I know I am not perfect but I know I am enough.”

There are a million ways to do something wrong

There are just a few ways to get things right.

The best practice sales systems were built by understanding top performing sales people.
The best practice training regimes for athletes were built by studying top performing athletes like Jerry Rice.

So, when you get into a new system where you want to be successful, first, identify folks who’re doing well. Next, figure out how they approach their art. Finally, get introduced to more folks like them (like attracts like – stars attract stars) and build relationships with them. You are, after all, the average of the five folks you spend most of your time with. So, if you want to be an elite athlete, start hanging out in circles with elite athletes.

And, once you understand what they do well, copy shamelessly. Let there be no ego in adopting best practices. You will, in time, adapt their approach to your own style and even make them better.

That’s how innovation happens.

What should Volkswagen do? – Part II – MBA Learnings

I’d shared a framework we discussed in a crisis management class 3 weeks ago – a few days after the Volkswagen emissions debacle came out in the press. I thought it was time for part II.

I woke up to the following paragraph as part of my “Economist Espresso.”

“Volkswagen’s boss in America offered a congressional hearing a “sincere apology” for the company’s use of “defeat devices” which helped diesel engines cheat in emissions tests. Stressing that he was not an engineer, Michael Horn blamed “a couple of software engineers” for the modification, of which he said he had no prior knowledge. German prosecutors searched the carmaker’s headquarters.”

While I could have looked for a longer article with more details, this paragraph was very instructive. And, I couldn’t resist throwing in the picture as well.

There are 2 things I would like to call out –

1. The crisis framework I shared offered 4 dimensions for an effective response to a crisis – transparency, expertise, commitment and empathy. It is safe to say that this response failed on all 4.

2. However, that is not what’s most shocking about this response. It is the complete lack of spine that indicates a total failure of leadership. Let us, for a moment, try and forget the fact that he claimed no prior knowledge of something as massive as this. The fact that he felt it was acceptable to blame “a couple of software engineers” just blows my mind.

There are few tenets of leadership that are as fundamental as – “Take responsibility when things go wrong and give credit when things go well.”

A crisis can be a great opportunity to dig deep, go back to your values and show the world what you are made of.

Volkswagen seems to have missed the memo.

Lollipop moments and changing the world

When Drew Dudley was a student at college, he changed one girl’s life without even realizing it.

The girl was intimidated to start college and told her parents she couldn’t do it. They ended up convincing her to go to her first day, but said that they would fully support her if it didn’t feel right and she decided to quit. So she went the first day and, sure enough, she felt intimidated as she was standing in line to pick up her room key. Just as she was about to turn to parents to tell them she couldn’t do it, she saw Drew come out of the student union building. He was dressed like an idiot with a stupid hat on and was handing out lollipops to raise awareness for a charity. He came up and stopped near the girl.

He then confronted another nervous freshman guy standing next to this girl, telling him, “You, you need to give this lollipop to the beautiful girl standing next to you.” The freshman guy turned beet red, took the lollipop and awkwardly handed it to the girl next to him. Drew then turned to her parents and said, “Look at that, look at that! First day away from home, and, already, she’s taking candy from a stranger!” Everyone in line burst out laughing, and the girl finally felt a bit at ease and decided to stay .

Four years later, on his last day, the same girl came up to Drew and told him the story. She told him that it was the most important moment of her life, that she was felt intimidated and was ready to quit school before it even started. She told him she couldn’t have been happier that she stayed at school and it was all because of his little joke that made her feel welcome. And, she also told him that she was still dating the boy who gave her the lollipop and that they were getting married in a year!

Drew racked his brain and couldn’t even remember telling the joke and giving out the lollipops.

We’ve all probably had lollipop moments in our lives. In Drew’s words, perhaps we need to redefine leadership as being about lollipop moments – how many of them we create, how many of them we acknowledge, how many of them we pay forward, and how many of them we say thank you for.

Changing the world simply means affecting the people we come across in our everyday lives, because that literally is all “our world” is.

HT Jim and Drew Dudley’s talk on Everyday leadership

I make mistakes

I woke up this morning and found myself going through a highlight reel of some of the dumbest things I said in public in the past few months.

This is an occupational hazard when you put yourself out there and occasionally speak off-the-cuff.

For a moment, I felt a lot of empathy toward public figures who say something stupid and find it picked up and misinterpreted in every possible by media outlets everywhere. They probably don’t need a mental highlight reel as they’re likely constantly reminded of that moment of stupidity.

Of course, the other side of the story is that good stuff they say is also broadcast all over the world. If you put yourself out there enough, mistakes will pop up.

The challenge is not to avoid mistakes but to avoid the illusion of perfection when things are, momentarily, going your way.

The best way to learn from mistakes is to simply say – “I make mistakes.. and that’s okay.”

Change and betterment come with acceptance.

But, the biggest benefit of this approach is that we also learn to be truly empathetic and understanding.

It is only when we forgive ourselves can we truly learn to forgive others. That is the true beauty of making and embracing mistakes – they teach us to be human.