4 Key Practices for Aligning Your AI and BI Teams

Read time: 7 minutes

Hi there,

Today, let’s dive into some best practices I've seen in my experience to better align data science (AI) and BI teams in organizations.

The truth is, despite the shared goal of extracting value from data, too many AI and BI teams face a significant disconnect. This can lead to delayed insights, missed opportunities, and even wrong conclusions.

Fortunately, there are ways to bring these teams closer together, even if they're in different business units.

Let's explore them!

Note: The following article is adapted and reposted with permission from my original contribution to the AtScale blog. You can read the full version here: https://www.atscale.com/blog/data-science-bi-best-practices/ 

You can’t afford disconnected AI and BI teams

In many organizations, data science and BI teams are historically located in different departments, have different levels of leadership, and lack common goals. This hinders effective collaboration, resulting in confusion, errors, and diminished trust-especially among business users in organizations with low analytical maturity.

That’s a huge problem:

If even the core data teams can't speak the same language and collaborate effectively, how can business users effectively collaborate with data teams?

In addition to organizational challenges, data science and BI teams often struggle to collaborate effectively due to:

• Different languages

• Different tools

• Different data models

• Different data cultures

To overcome these hurdles, there are four key best practices for data science and BI teams to work closely together, as shown in the figure below:

Let’s unpack them piece by piece!

Best Practice 1: Align Language

One of the biggest challenges for data science and BI teams is the lack of a common language. Data Science and BI teams often use different terms and concepts, which can lead to confusion and misunderstanding.

For example, a data scientist may use the term “model” to refer to a predictive machine learning model, while an BI analyst may use the term to refer to a relational or dimensional data model in an OLAP cube.

Similarly, terms like “measures”, “dimensions” or “report filters” are everyday language for BI professionals, while data scientists may argue about the right set of “training examples,” the best “predictive features,” or “sampling methods.”

The lack of a common language can lead to a lack of trust between teams and poor communication overall, which in turn leads to poor collaboration, as it can be difficult for teams to share and understand each other’s work.

Using a shared vocabulary

To establish a common language between data science and BI teams, organizations should create a glossary of terms used by both teams. This glossary can include definitions of common terms as well as explanations of how they are used in the context of data science and BI.

This glossary should become the first point of reference for members of both teams so that they no longer have to secretly Google and guess what the other party might mean. In addition, cross-functional training sessions can be held to ensure that both teams have a clear understanding of the terms and concepts used by the other team.

Maintaining a centralized documentation

Furthermore, having clear documentation and procedures in place, where the team can refer to, can also help to align the language.

This documentation may include data dictionaries, data lineage, data governance policies and procedures, and other important documents that teams may need to refer to.

Best Practice 2: Align Tools

Another challenge faced by data science and BI teams is the use of different tools and technologies.

Data Scientists often use programming languages such as Python or R, while BI analysts typically use tools such as Tableau and Power BI and the ETL stack built around them. These different tools can make collaboration difficult, as it can be challenging for teams to understand and share each other’s work.

For example, a data scientist may use Python to build a predictive model, but the BI team may have difficulty understanding what the code is doing and how to interpret the results of the model — let alone integrate it into their ETL pipeline.

On the other hand, a BI team may use Power BI to build a dashboard. Meanwhile, the data scientist may not be able to access the transformations and KPI calculations that are done there, so they must painstakingly reconstruct the results in Python and verify that they’re getting the same results shown in the BI dashboards.

Define a core data stack

To overcome this challenge, organizations can encourage the use of common tools for both teams — each for their specific tasks. For example, data scientists could be trained in BI tools to use their data visualization capabilities, and BI analysts could be trained to read (and perhaps even write) Python or R code. An even more elegant solution would be for data science and BI teams to commit to using a common technology as much as possible. Oftentimes, that’s SQL.

While it may be more tedious for a Data Scientist to write a data preparation pipeline in SQL than in pandas, the advantage is that BI teams can easily use this pipeline. This not only increases efficiency and improves data quality, but also allows both teams to understand and interpret each other’s work.

Needless to say, having a common data platform for data storage and management is an important catalyst. By using a common platform, teams can access the same data and collaborate more easily. This also helps with data governance and security compliance.

Best Practice 3: Align Data Models

Nothing separates data science and BI teams as much as their data models.

Data scientists typically require data in a denormalized, granular form, while BI analysts typically require normalized, dimensional, and relational models.

These different models can lead to confusion and errors when teams share data.

For example, a data scientist might use a complex model to predict customer behavior but might use a different logic to create critical business entities, such as defining customers, calculating their revenue, or assigning customer-specific attributes. These different definitions can create a lot of confusion and distrust among teams — and ultimately lead to poor results.

Using a shared semantic layer

To overcome this challenge, organizations should use a centralized, well-managed semantic layer that includes feature stores for models used in production and serves as a source of truth for both data science and BI teams. Data models used by both teams should be documented, including the purpose of the model and the appropriate owners who can answer further questions. This ensures that both teams have a clear understanding of the data models and how they are used.

In addition, joint data modeling sessions can be held to ensure that both teams have a clear understanding of the data. In these sessions, data scientists and BI analysts can work together to discuss the data, identify any gaps or inconsistencies, and develop a common understanding of the data.

This ensures that both teams are working with the same data and that the insights gained are accurate and actionable.

Best Practice 4: Align Culture

The cultural differences between data science and BI team members often hinder effective collaboration.

Data scientists and BI analysts often have different communication styles, problem-solving approaches, and ways of working. These cultural differences can lead to misunderstandings and delays in problem solving.

For example, data scientists have a more experimental and research-oriented approach, while BI Analysts have a more regimented and business-oriented approachThis is due to their different backgrounds.

BI teams and processes have typically evolved over decades of working with data warehouses and well-managed reports. Data science teams, on the other hand, have often been hired with a more agile mindset, commonly incubated in “data labs” trying to explore and push the boundaries of what’s possible with data. This can lead to different priorities and a lack of understanding of each other’s perspectives.

Let people collaborate in cross-functional projects

To align cultures, companies should encourage cross-functional project teams and organize team-building events. The best way to better understand each other is to work on projects together. Then, data science and BI teams can develop a better understanding of each other’s roles and responsibilities and learn to appreciate the value each team brings to the business.

Regular communication and knowledge sharing sessions are also very helpful. This will give a space for the data science and BI teams to share their insights, experiences, and best practices. This can help break down barriers and build trust between teams.

By aligning cultures, organizations can ensure that the data science and BI teams work together more effectively and contribute to an overall more data-centric and data-driven culture across the organization.

Conclusion

To get data science and BI teams in sync, we discussed four best practices for bringing teams closer together and producing more value: aligning language, aligning tools, aligning data models, and aligning culture:

• Align language: Ensure a common understanding of the terms and concepts used in the field.

• Align tools: Ensure that data science and BI teams work together with high efficiency.

• Align data models: Ensure that data science and BI teams both produce insights that are accurate and actionable.

• Align culture: Increase the effectiveness of data science and BI teams and make sure their combined output is larger than the sum of its parts.

Following these best practices for data science and BI teams, organizations can overcome the often historically induced challenges of working together and create a more seamless and effective business ecosystem.

After all, when the data departments are not working together effectively among themselves, who else will?

I hope you enjoyed today's deep dive into the organizational aspects of making data projects a success.

Let me know your feedback - I'm always looking forward to it!

See you next Friday,

Tobias

Want to learn more? Here are 3 ways I could help:

1. Read my book: Improve your AI/ML skills and apply them to real-world use cases with AI-Powered Business Intelligence (O'Reilly).

2. Book a meeting: Let's get to know each other in a coffee chat.

3. Follow me: I regularly share free content on LinkedIn and Twitter.