I recently had a conversation with someone about building a data warehouse and how to go about doing it. Over the last year, I’ve been through the same journey starting from scratch so this conversation was an opportunity for me to have an outside-in view whilst discussing the topic with a fellow technical expert. What stood out for me was the importance of understanding the end user, and understanding the business requirements.
It’s easy for us “tech geeks” to go and create maps of existing complex database structures and architecture landscapes whilst debating on-premises vs. cloud solutions, AWS vs. GCP vs. MS Azure, inflexible data warehouses or more flexible data lakes, data orchestration, data flows, and pipelines. Sometimes we need to take our tech geek hat off and listen to the business. Much in the same way that a good salesperson listens to the needs of the customer with their current pain points before deciding a course of action or selling their solution, IT must do the same. What business issues are we trying to solve? What questions is the business trying to seek answers for? How can the provided data be used to progress the company forwards? Before we throw a new piece of technology or data infrastructure at the company and its employees we must put the company and its employees first.
Selling (or even just presenting) technical ideas & potential solutions to non-technical people is a tough task. “Why do we need this now?” “But my MS Access databases have worked fine for the last 20 years.” It’s easy for IT to be seen as needlessly rocking the boat, so gaining buy-in from all stakeholders is critical. Having the support from engaged end-users will be a significant factor in the success of implementation and adoption of new technologies.
The Amazon approach
When asked about the success of Amazon, Jeff Bezos has always been clear. In everything Amazon do, and every decision they make as a company, the customer is always at the centre. A strong customer-focused approach has been the cornerstone of their business model. Bezos has often emphasised that the key to Amazon's success is an obsession with customer satisfaction, ensuring that they are always listening to and understanding the needs and desires of their customers.
Incorporating this approach into building a data warehouse means prioritising the end user's needs above all else. It involves continuously seeking feedback and understanding how data is used within the organisation, what information is most valuable to different stakeholders, and how the data warehouse can make accessing this information as seamless and effective as possible.
For IT professionals and organisations looking to emulate Amazon's success, the lesson is clear: success comes from putting the customer—or in this case, the end user and the business needs—at the heart of every decision. This requires a shift from a purely technical focus to a more holistic view that encompasses business objectives and user requirements
Once the end-users and business needs have been fully understood and mapped out, this should give IT the framework to start to build out the correct data architecture that will support the business. With a full mapping of the current landscape of data and supporting databases, one can assess each asset and decide what steps to take for it. Decisions here would include to either replace & start again, leave as it would form part of another major project, or upgrade & integrate it into the new data warehouse architecture. This phase of evaluation is critical, as it ensures that every piece of existing data and technology is considered for its contribution to the new system. During this process, IT must also consider compliance, data governance, and security requirements to ensure the new architecture not only supports business needs but also aligns with regulatory standards.
The next step involves choosing the right technology stack for the data warehouse. This decision will be influenced by several factors including scalability, performance, cost, and compatibility with existing systems. Whether it's opting for a fully cloud-based solution, a hybrid model, or leveraging on-premises resources, the goal is to build a robust, flexible architecture that can handle current and future data loads. It's also the stage where the design of data models and schemas takes place, ensuring that data is organised in a way that supports efficient storage, retrieval, and analysis.
As the construction of the data warehouse progresses, rigorous testing phases are essential. These tests are not just about ensuring the integrity of data and the performance of the system but also about verifying that the data warehouse meets the user requirements that were identified at the outset. User acceptance testing, in particular, should involve end-users to confirm that the system delivers the information and insights needed in an accessible format.
The final and ongoing phase involves monitoring, maintaining, and continually improving the data warehouse. This includes keeping an eye on performance metrics, user feedback, and changing business needs to ensure the data warehouse remains a critical, responsive asset to the business. By establishing a feedback loop with end-users and stakeholders, IT can ensure that the data warehouse evolves in line with the company's goals, maintaining its relevance and value over time.
❤️ Enjoyed this article?
Forward to a friend and let them know where they can subscribe (hint: it's here).
