Challenges and Solutions for Dealing with SpreadMarts in Organizations

What is SpeardMart?

SpeardMart refers to a collection of files, generally in a spreadsheet format like Excel, that are created and maintained by an individual or a group of people in an organization to analyze data and reporting purposes. These collections of files usually fulfill the expectations that we have from a Data Mart or Data Warehouse differently, and in some articles, they are referred to as shadow systems of data warehouses.

In most companies and organizations, there are powerful individuals in non-IT departments who strive to advance their goals and develop reporting systems by implementing and using tools that, for various reasons, do not require decision-making and consultation with IT units. These individuals typically collect their data in various file formats and perform data cleaning processes and formulas according to their experiences and knowledge, as well as the capabilities of the tools they use, regardless of whether the outcomes of all these activities could conflict with the patterns and standards of the organization in terms of data management and governance. In many organizations, after the creation of such information marts, we witness the growing and diverse use of these data sources, which can act like destructive bombs, suffocating any opportunity for data stability and reliability. According to research conducted by reputable sites, more than 90% of organizations have SpreadMarts.

One of my experiences in a large financial organization was that two people from two different teams presented indicators with the same definition but different figures in a meeting with board members. After review, it was clear that both individuals had correctly followed the process, but they had collected data from different networks.

In summary, the existence of these SpreadMarts currently poses significant risks to organizations. Individuals and business owners make decisions based on flawed data and plan incorrect analyses based on those assumptions, which affects the future of the business.

Cheap, Fast, and Easy

The data that individuals need for their tasks may not be available in the data warehouse, so they have to import and process the data themselves to obtain the information. Advanced BI tools may not be available in the organization, specialized data teams may not be accessible, or the data may not be displayed in the format that managers want. SpreadMarts are a short-term solution for a business need that is created very cheaply and quickly, within a day or two by an experienced analyst using a program they are fully proficient in, without the need to coordinate with other departments and spending days or weeks. There's no alternative!

However, there is a high cost for maintaining them in the long term, which is why many managers have made significant investments to solve this problem, and various organizations use customized architectures to improve data processes and become more data-driven, although many of these efforts have also failed.

Spreadsheets and other desktop tools are an important part of any organization's technology suite. The problem arises when individuals use these tools as data management systems for collecting, transforming, and maintaining corporate data for decision-making, planning, and process integration and monitoring. When this happens, SpreadMarts proliferate, weakening data stability and increasing risk.

The technical solution for SpreadMarts is to centrally manage and store data and logic in an integrated manner and then allow people to access these data using their chosen tools. In other words, the presentation layer should be separated from the logic and data. When this is done, business users can still access data for reporting and analysis purposes and manipulate them, but they do not create new data or logic for organizational consumption. We usually know this process as managed business intelligence.

Changing organizational culture usually requires senior managers to articulate the importance of having integrated, cohesive, and unified organizational data and to apply incentives and penalties to guide correct behaviours. Ultimately, change takes time, sometimes more than 3 years, but the right organizational levers can accelerate the process. Therefore, the topic of organizational maturity is raised. Information technology and business needs are rapidly advancing, while this growth in the layer of maturity and organizational culture is slower, so organizational managers and business owners must accept that reaching a goal and a mature organization has stages that must be followed for the organization to reach the desired maturity, and there is no shortcut or tool for this matter.