Data Science in the Wild

At this point, you might get tired of me emphasizing the decision-making aspect of data science as the main point of why it is important. It’s time we move from the general and abstact towards particular examples and applications in various industries. This will lead us to acknowledge how prevalent is AI (that we haven’t fully defined yet) in firms, services and technologies we use every day.

During a lecture, I usually ask students

Can you give some examples of businesses, sevices, technologies, problems, and domains which you suspect do have AI/ML algorithms and models behind the scenes?

See below some very good answers and argumentations provided by the students last year, then we’ll examine in more details one of them. Of course, there is always that one person, very passionate about sports or blockchain.

Do some reverse engineering!

When reading this section, I want to get you into a mindset of the reverse engineer: step back and think deeply about products and services you use every day:

  • Put yourself in the shoes of the business making those decisions and building those systems
  • What were the goals, user/client needs?
  • What was the firm’s objective?
  • What constraints did they hit? Why it was a difficult problem?
  • What made it an appropriate use-case for ML, Statistics, and AI?
  • What are some potential approaches they settled on?
  • What would be a baseline, simple, naive solution?
  • Dynamic pricing in Bolt and Uber, which takes into account the weather, especially if it rains, peak hours: balancing demand and supply. It is at the intersection of ML and economic theory, as they are a platform or marketplace. Prices also change with respect to competitors, so we see aspects of an oligopoly behavior. 1
  • Stock markets and trading bots: at the intersection of economics, finance and AI. I would add the “good old” and boring portfolio management and venture capital enterprises.
  • Management consulting: what market to enter, whether and how to build a new product (product development). Lots of use-cases in marketing and market research firms.
  • Medicine Applications: developing new vaccines and drugs, aided by AI and designing clinical trials for novel treatments.
  • Banks and insurance: risk management, predicting credit defaults on mortgages and business loans. Chatbots for customer support, for most frequent and simple questions.
  • Automotive: routine tasks like automated parking, the race towards self-driving, autonomous or semi-autonomous cars, safety warnings. Predictive maintainance is tackling a problem where they leverage predictions to replace risky parts before they go out of function.
  • Liverpool F.C. won a title, and a key part of their success was leveraging AI and ML to discover new tactis on the field with the highest payoffs. 2
  • NBA teams invested a lot in the data infrastructure and decision-making capabilities: LA Lakers found the best player at the moment for a particular position they were lacking and would play well along with the team. Rockets won the regular championship divison by going all in on the 3-point shot.3 Golden State Warriors simply revolutionised basketball with data, before everyone else was doing it – giving them a competitive edge. 4

1 When it doesn’t work out – I’m pretty upset at their data scientists and domain experts. Here is where ethical issues creep up: jacking up prices, monopolies, drivers struggling to make a living wage.

2 TODO: Reference article and maybe dataset for more details

3 Moreyball: The Houston Rockets and Analytics – an article in Harvard’s MBA Digital Innovation

4 Check out the following video by The Economist on how data transformed the NBA. For more details on the statistical methodology, I enjoyed an youtube channel called Thinking Basketball and their playlist about the statistical methodology.

In all of the examples above, those businesses and systems do have to make decisions, under uncertainty from multiple sources, trying to solve complex problems at a large scale, which would be impossible to do manually even with an army of employees.

I would like to add a few more examples, from an insider and practitioner’s perspective, which might not be as impressive and a bit routine, but no less important. Keep in mind, that if at a closer look, the service seems to do something relatively intelligent very fast, specialized AI might be involved behind the scenes.

  • Demand Planning: How many SKUs (items) should I order for manufacturing, to satisfy the demand (that last item on the shelf, minimizing lost sales) and to minimize excess inventory.
  • Logistics and Supply Chain: routing, distribution center operations and automations for order fulfillment, return management
  • Recommender systems for music, videos, books, products, news in social media, services, platforms, and e-commerces like facebook, instagram, tiktok, youtube, spotify, amazon, emag. You can find recommendations in surprising places, like google maps.
  • Programmatic Advertisement: finding best placement for ads on various platforms, right now dominated by meta and google
Classroom Case Studies for a Deeper Understanding

We will explore some challenges and applications from this list in a series of case studies and labs. The idea is to improve our ability to identify opportunities and formulate problems from the point of view of an organisation, such that we can match those with the methods, models and algorithms discussed in the course.

I’ll have to introduce a lot of new concepts when the language we developed so far will turn out to be insufficient to talk about and understand what’s happening inside these firms. Thus, each case study is an opportunity to play around with a novel idea. 5

  • In the first deep-dive, we will look at Uber and Bolt, with publicly available information, trying to figure out what do their data scientists do.
  • Then, we will look at the lingerie e-commerce where I work at, AdoreMe and a different set of problems we’re facing.

5 If you have a pretty good idea about what is AI, Analytics, ML, Deep Learning, Big Data, Causal Inference and when to use one approach or another, feel free to skip the history and terminology and go straight to the case studies.