Robert Harmon is a seasoned data architect who currently works at Firebolt. With over a decade of experience in ad tech and tech support, Robert has carved a niche for himself in the data world. He worked with large publishers like Microsoft and CA, managing tech support teams. However, the experience of pulling data for managing TaxPORT drew him into the data world.
Summary
2023 marks the importance of data efficiency in the business world
Need for return on investment and scarce resources drive organizations to increase efficiency
Similarities with 2008 financial crisis, which taught the importance of sticking to fundamentals (efficiency and team management)
Hiring right contributors and effective management of data systems crucial for efficiency and freedom
Importance of interface between operational and data teams, rather than democratizing the workload
Building a robust operational data store is essential
Reviving the Forgotten Fundamentals of Data Management: Addressing the Challenges of Today's Data Systems
Since the early days of transactional systems and operational data stores, data management has come a long way. However, with the advent of big data and the increasing reliance on data lakes, we still need to remember some fundamental principles that once made data management more predictable and efficient.
Staying True to the Fundamentals
Fundamentals of data management: accuracy and consistency
Shift away from fundamentals due to fast-paced data generation and processing
Need for more constraint and schema in data lakes
Issues arising from shift: problems with changes in operational systems causing chaos and confusion
Resolving issues takes time, time that could be better spent on other tasks.
The Operational Side of Data Management
Despite the challenges faced in big data and data lakes, the operational side of data management remains relatively unchanged.
SQL Server continues to be used for transactions and running smoothly
Operational side of data management faces issues in feeding data into downstream systems
Historically, transactional systems fed into operational data stores for predictability and efficient error resolution
Current model: dumping data from operational systems into data lakes with no constraints or schema
Lack of predictability due to changes in operational systems causing chaos and disruption in the data lake.
The Importance of Proper Database Design for Data Quality
Data growth has been exponential in recent years, and it is crucial that the stored data is accurate, relevant, and easily accessible. This requires proper database design and data management techniques. In this article, we will explore the challenges that come with data growth and why proper database design is so important.
Data growth leads to more complexity in data management
Challenge of ensuring accuracy and accessibility of large volume of data
Proper database design is crucial
Cloud data warehouses provide data storage solution but also introduce new challenges
Absence of constructs like operational data store and constraints leads to data quality issues
Systems and processes needed to ensure high quality data stored in cloud data warehouses.
Watch the full interview
Feel like chatting instead?
We're in early-stages of development and always looking to hear from analytics professionals. Catch us via our calendar below.
.png)
.jpg)
