Data Efficiency in 2023: Learnings from the 2008 Financial Crisis

Robert Harmon is a seasoned data architect who currently works at Firebolt. With over a decade of experience in ad tech and tech support, Robert has carved a niche for himself in the data world. He worked with large publishers like Microsoft and CA, managing tech support teams. However, the experience of pulling data for managing TaxPORT drew him into the data world.

Summary

• 2023 marks the importance of data efficiency in the business world
• Need for return on investment and scarce resources drive organizations to increase efficiency
• Similarities with 2008 financial crisis, which taught the importance of sticking to fundamentals (efficiency and team management)
• Hiring right contributors and effective management of data systems crucial for efficiency and freedom
• Importance of interface between operational and data teams, rather than democratizing the workload
• Building a robust operational data store is essential

Reviving the Forgotten Fundamentals of Data Management: Addressing the Challenges of Today's Data Systems

Since the early days of transactional systems and operational data stores, data management has come a long way. However, with the advent of big data and the increasing reliance on data lakes, we still need to remember some fundamental principles that once made data management more predictable and efficient.

Staying True to the Fundamentals

• Fundamentals of data management: accuracy and consistency
• Shift away from fundamentals due to fast-paced data generation and processing
• Need for more constraint and schema in data lakes
• Issues arising from shift: problems with changes in operational systems causing chaos and confusion
• Resolving issues takes time, time that could be better spent on other tasks.

The Operational Side of Data Management

Despite the challenges faced in big data and data lakes, the operational side of data management remains relatively unchanged. 

• SQL Server continues to be used for transactions and running smoothly
• Operational side of data management faces issues in feeding data into downstream systems
• Historically, transactional systems fed into operational data stores for predictability and efficient error resolution
• Current model: dumping data from operational systems into data lakes with no constraints or schema
• Lack of predictability due to changes in operational systems causing chaos and disruption in the data lake.

The Importance of Proper Database Design for Data Quality

Data growth has been exponential in recent years, and it is crucial that the stored data is accurate, relevant, and easily accessible. This requires proper database design and data management techniques. In this article, we will explore the challenges that come with data growth and why proper database design is so important.

• Data growth leads to more complexity in data management
• Challenge of ensuring accuracy and accessibility of large volume of data
• Proper database design is crucial
• Cloud data warehouses provide data storage solution but also introduce new challenges
• Absence of constructs like operational data store and constraints leads to data quality issues
• Systems and processes needed to ensure high quality data stored in cloud data warehouses.

Watch the full interview