Why did you study all of that

Author

Mihai Bizovi, VP of Decision Science

Why did we have to go through all those excruciating months and years doing mathematical analysis, linear algebra, probability, statistics, econometrics, operations research, and lots of economics? What about computer science and low-level programming in C? What about more abstract subjects like cybernetics and game theory?

It might be the case for you, as it is for me, that you didn’t engage in detail with some of those topics for a decade! I haven’t written a line of C in production, nor used the Heine-Borel theorem in practice – and that’s alright.

It was very frustrating for me, because it wasn’t clear how they fit together, what is the common thread, and more importantly – what part of the theory is relevant. It would’ve been great to know what works well in practice and why. I will try to be as clear as possible which ideas are helpful in solving the kind of problems in businesses we’re interested in, and which ones are designed to enhance our conceptual understanding.

Let’s draw a map, stop at each field and in a sentence explain why we learned it and how it contributes to decision science, ML, and AI. I mentioned from the very beginning that decision science is an applied, quantitative, and interdisciplinary field, but it is not just an union of those subjects – the inspirations and tools are quite carefully picked.

flowchart TD
  LA[Linear Algebra] --> OR[Operations Research]
  MA[Mathematical Analysis] --> OR
  MA --> SD[Systems Dynamics]
  %% CS[CS Algorithms] --> OR
  
  PT[Probability] --> MS[Statistics] --> EC[Econometrics]
  EC --> Caus[/Causal Inference\]
  EC --> TS[Time Series]

 %% subgraph 1
  Caus --- DM[/Data Mining\] --- ML[/Machine Learning\] 
  ML --- Caus
 %% end

  OR --> ML
  MS --> ML
  MA --> PT
  SD --> Caus

  Caus --- Econ[[Economics]]  
  Econ --- GT[Game Theory]
  Econ --- DT[Decision Theory]

  style Caus fill:#f7f5bc
  style ML fill:#f7f5bc
  style DM fill:#f7f5bc

  DM --- FSDA[/Full-Stack Apps\]
  FSDA --- DB[Databases/SQL]
  FSDA --- OOP[OOP]
  Econ --- TS

  style FSDA fill:#f7f5bc
Figure 1: Think of this as a stuctural organization of the fields and courses you studied before. Some are more useful in analytics, some in ML and some in making causal inferences, that is, based on data + theory. This diagram is organized around those three central pillars and perspectives.

Note that this map is a “model”, a simplification. It is overwhelming and impossible to make it accurate or detailed or exhaustive – as it becomes unreadable.

Fundamentals

Note that all subjects below can be studied in various levels of depth and mathematical rigor. Where do you stop heavily depends on what you want to do in your carreer.

Linear Algebra is a language of data. Most data types (structured, nested, images, video, text, voice, etc) can be represented as multidimensional arrays and tensors. The vast majority of machine learning and mathematical models can be reduced to operations on matrices, which is important if we want to implement them in code. Moreover matrix decomposition algorithms like SVD, Cholesky are useful tools in many practical applications. 1

1 On a personal note, linear algebra was my gateway drug to applied mathematics and machine learning. Without that course it is possible I would’ve never discovered a passion for math

My perspective over linear algebra is computational and geometric, in the sense of the “space” the data points live in and transformations of that space. It’s also a very concise and elegant way to define models, loss functions, and optimization problems.

I highly recommend the following resources: Essence of Linear Algebra (conceptual), Coding the Matrix (computational), Linear Algebra Done Right (abstract, mathematical), Mathematics for ML (applied), and S. Boyd’s course (applied).

Mathematical Analysis is all about change, formalizing how a function behaves with respect to its arguments and parameters. It is an essential building block in probability theory,2 optimization, and deep learning (automated differentiation).

2 Many mathematically rigorous degrees also require real analysis, measure theory, and even functional analysis

I argue that in order to understand any complex system, be it a firm, an economy, the climate, or environnment, we have to model how it evolves in time. This suggests the importance of differential equations and systems dynamics, modeling the feedback loops. All of this would be very awkward to reason about without mathematical analysis.

However, be warned, mathematical analysis is a rabbit-hole! The only courses which I can recommend without you going into years of proof-based study, is “Calculus, Applied!” from Harvard and John Hopkins’ specialization, which has some cool modeling applications with Python code!

Probability theory is the language and logic of uncertainty. In statistics, we change our mind and actions in the face of evidence. Just to highlight how important these two complementary fields are, it is hard for me to find an applied domain or aspect of life where probability and statistics isn’t useful (at least, in principle). I dare you to find one!

Moreover, our everyday intuitions make us terrible at probability, which is another reason why we need a more formal tool to reason and make decisions under uncertainty. Why then so many people hate it as a subject? Is it dry teaching, fear of math and abstraction? In the module two, I recommend a simulation and story-based approach to probability, with many resources which will make studying it fun and practically relevant. 3

3 In my case, Taleb Nassim’s “Fooled by Randomness” was what motivated me to study probability seriously. I do highly recommend it

Applied Mathematics

Here is an explanation for why other subjects are needed – which students from backgrounds other than quantitative economics or computer science might’ve not encountered. This is a point in which you see reseachers specializing.

Econometrics, in my personal opinion, tries to separate the signal from noise and make inferences about the causal processes in economic phenomena. It specializes statistics to the domain of economics, which presents new modeling challenges. Remember that in order to draw causal conclusions, we need a scientific theory, not just data alone. It’s not sufficient to check if statistical assumptions hold, but we have to make clear our causal and methodological assumptions.

You should be careful in introductory econometrics, as you can claim only association with the tools you will learn. Even in literature, there is this weird tension between an emphasis that most conclusions aren’t causal, but them treating them as such for policy-making. Leaving the causal inference aside for a moment, I can safely recommend “Mostly Harmless Econometrics” and “Econometrics with R”.

Time Series Analysis (senso largo), bridges the gap between theoretical dynamical models (systems of differential equations) with statistics and probability (stochastic processes). It adapts those tools to make inferences and predictions about phenomena which evolve in time, that are dynamic in their nature. I like the metaphor of data assimilation, which is actually an entire field trying to introduce the empirical dimension to all kinds of dynamical systems.

I don’t particularly like any books I read or courses I took on time series. However, I can recommend N. Vandeput’s “Demand Forecasting Best Practices” and his other writing. For practitioners, R. Hyndman’s “Forecasting: Principles and Practice (3ed)” and I. Svetunkov’s “Forecasting and Analytics with ADAM” are a good starting point.

Operations Research is about optimization with constraints, allocation of limited resources, and efficiency. The easy part is to apply an integer or linear programming algorithm to an already formulated problem. The hard part is to reduce a messy real world problem at a large scale to that formulation, especially under uncertainty and nonlinearity.

I’d argue that without implementing a problem in code with established OR libraries and inferring the parameters of the problem from data – we don’t really know how to do optimization. The only free resources on Operations Research with code which I know of are C. Kwon’s “Julia Programming for Operations Research (2ed)” and Huber’s “Algorithms for optimization”. Others are strictly focused on optimization for ML and deep learning.

A word of encouragement

None of those courses were useless. Think of how can we take parts from each of those prerequisites which are relevant for decision science, so that we have more tools to solve the complicated problems we encounter.

A note on Economics

In mathematically-oriented courses, you can think of economics as optimization with constraints, which are given by limited resources and our positive or normative theories of decision-making. In the most general sense, economics studies behavior and interactions of economic agents.

When we make decisions, we like to think of ourselfs as objective, but we have lots of biases and blind spots which prevent us to see the reality clearly. For example, we find patterns and regularities which are not real, but treat the conclusions as causal; we tend to seek data which supports our hypothesis and ignore contradictory evidence, etc.

  • This kind of knowledge about human behavior helps us to be wiser and practice active open-mindedness 4
  • In other words, we want to prevent self-deception and self-sabotage towards achieving the stated goals and objectives – and even evaluate if the goals chosen well

4 Actively open-minded thinking (AOT) is measured by items that tap the willingness to consider alternative opinions, sensitivity to evidence contradictory to current beliefs, the willingness to postpone closure, and reflective thought.

Moving on to more practical things, the question is not how much economics a decision scientist should know, but what kind of economics. This list will be wildly different for people going into quantitative finance, banking, insurance, or public policy.

If you studied in a program which provides a strong background in quantitative economics, econometrics, and operations research – I have some good news and bad news. The good news is that you know all the mathematics you will need in practice and you’re well versed in formulating good economic research questions. The bad news is that you probably need a lot of effort to improve your programming skills and that the kinds of economics you studied don’t translate well inside a firm.

Please, do not mistake corporate finance, marketing, management, operations, supply chain, logistics, CRM, and production as fluff. There is a good reason people do business degrees and get MBAs. If you look at a top marketing journal, you will be surprised at the complexity of models and innovations in econometrics and machine learning needed to tackle those challenges. Moreover, when you combine marketing with behavioral economics and psychology, it adds another layer of nuance and understanding of consumer behavior. We can’t learn this all from data alone.

Depending on your role in the firm, you might need to read about demand planning, advertisement, pricing, manufacturing, etc, in order to speak the language of decision-makers and stakeholders. I’m leaving you with a list of youtube playlists and channels which I found helpful to keep up-to-date with the way modern businesses operate: