Data Science Audit & Roadmap Development
You have been collecting data from day 1, using business intuition until you have amassed the data needed to train the AI models to help your company really scale. However, machine learning also present more risks than traditional software, not only due to issues of scale and increasing privacy concerns, but also due to the lack of good established processed in developing data products.
From understanding their data generative processes, to assessing the technologies enabling data-driven decision making and modeling. Over a 72 hr period, we will liaise with your business and data science leads, conduct data/code reviews, assess processes around machine learning and analytics projects, and interview technical talent at each level of the hierarchy. The goal is not to determine whether the data is “good or bad,” but rather to provide insights about how you can set up your data initiatives to succeed.
1. Org Analysis
A useful data audit starts with a thorough understanding of where data are used in the organization. Are data professionals centralized or distributed? Do you have an established data science team or citizen data scientists in each functional group? After going through an org chart with the CEO, we meet with the data science and analytics leads to understand their philosophy on hiring, managing, and driving results. We collaboratively build a schedule, picking the key individual collaborators to interview and setting common-sense boundaries on code review. This process builds trust and reveals deeper learnings than the typical adversarial approach.
2. Technology Review
Technology review – Tech DDs often frustrate target teams because the wrong talent is used. We don’t throw full stack generalists or junior talent at audits. All Brandt data audits are executed by a Chief Data Scientist quality data scientist, hand-picked for the problem domain and modeling techniques they’ll be encountering.
- Data Platform – in this process, we analyze how approachable the data are in driving business value through via insights or models. This review encompasses database audits, application analytics, and documentation as well as interviews with key individual contributors to assess how much friction there is between a question and a data-backed answer.
- Machine Learning Pipelines – the dirty secret in data science is that a lot of work is done in ad-hoc notebooks, and most results are not reproducible. With this review, we look to assess the resilience of the model-building process. We interview key data leads to understand the process of going from research to production. We also look at how data science interfaces with the rest of the business, and collaborate on creating processes that minimize the risk of data scientists going down rabbit-holes.
- Data Quality – Garbage In. Garbage Out. This is a core tenant of machine learning. Because data storage has become inexpensive, companies now store all of the data they can get their hands on. However, simply storing data does not mean it will be forward-compatible with future use cases. In this review, we analyze the collection and processing of data to assess if the data you collect are structured to be usable to produce insights and train models as your product evolves.
Much of the value of any tech audit comes from placing it in the context of the goals of the organization rather than simply labeling the code “good” or “bad.” For example: is your mental model of the technical talent and assets in your organization correct? Is your roadmap/timeline feasible given the condition of the data and current product team? If you attract investment, how can you properly attract and evaluate talent? If my data scientists develop amazing models, can my engineering team implement them in the product?
"Brandt & Co. have provided significant assistance, guidance, and tangible help for us at our early stage startup. We will be lucky to have them advising us as we scale. Jourdan is extremely competent, intelligent, and knows how to leverage his experience and extensive network to help his clients."
Jason Ovryn, COO, Carry | $1M Seed Round
"Over the years Jourdan has provided consistent and clear guidance on a variety of tough founder decisions from product scope, go-to-market strategy to fundraising. My team frequently leaves our discussions with Jourdan energized and with a much better defined action plan."