How to Integrate Data from Multiple Sources for Unified Analytics

Author: Steve Lock

If integrating multiple data sources has been feeling massively confusing, this article contains the clarity you need.

We’re going to simplify the overwhelm and highlight some of the best approaches so you can make the right decisions for your project.

When integrating multiple data sources into a single analytics platform, there are three main approaches to choose from:

1. Data Visualization Tool: You can select a data visualization tool such as Tableau, Qlik, Looker or Power BI that supports the data sources you want to ingest.
2. Custom Data Platforms: If you have significant engineering resources available, or if you’re a larger organization with more complex needs, you can consider building your own modern data platform, with custom data pipelines and data warehousing.
3. Third-Party Connectors: Alternatively, there’s a large ecosystem of third-party connectors available. For example, if you want to collect data from one of your systems and it’s not directly supported by your data visualization tool, you may find there is a third-party connector that does the job. They can also be a good choice if you want to send data to your data warehouse, reducing the maintenance and engineering resources required.

Data Visualization Tool

This is a great option for small to medium size organizations, especially if you’re using data sources that are widely supported and already have a data visualization tool in place.

For example, Salesforce/SFDC, HubSpot, MailChimp, Google Analytics and Google Ads tend to have great support without the need for data engineers. If you are planning to ingest data from these sources and there isn’t a massive volume of data, you can achieve quick wins from this approach.

Custom Data Platforms

Custom solutions are best suited to the following scenarios:

1. The data sources you need aren’t supported by data visualization tools or third-party connectors.
2. You’re a large enterprise and reducing the number of third-parties that can access your data is attractive.
3. You’re a startup or smaller organization with significant engineering resources.
4. The volume of data is large and requires data warehousing.
5. There is significant customization involved that isn’t supported out of the box without engineering e.g. complex data transformations, looking up data across different systems etc.

This is a much heavier lift compared to the other options, and definitely needs strategic technical expertise and implementation guidance. If you have a significant budget or technical resource available, the huge upside with this option is that it allows you to tailor the systems to perfectly meet your needs.

Third-Party Connectors

In recent years, there has been a massive explosion in third-party connectors. The main use cases are:

1. Your data visualization tool doesn’t support a data source you need, and you’re willing to pay a monthly fee to a third-party who’ll collect the data on your behalf, without the need for engineering support.
2. You want the benefits of a custom data platform, but your engineering resource and budget is limited. For example, you could use a service such as Fivetran, Stitch Data, or Talend to send data to a data warehousing solution that your data visualization tool can connect to.
3. You have engineering resources, but you’re concerned about maintenance and monitoring, and would prefer to outsource the data pipelines to a vendor. For example, you don’t want your engineers spending time supporting changes to APIs and source data.

Conclusion

If you’re on a budget and your data sources are supported by your visualization tool, this is a good option and can be helpful to prototype what you will need in future, larger projects.

If you have the budget and data is critical to your organization, custom data solutions can enable you to build almost anything you could ever need.

Researching the third-party connector ecosystems is a high value exercise for every organization as you can weigh up engineering investment and approaches, or even factor in to selecting key systems such as CRMs. This reduces the risk in building a custom data pipeline to then find out you could have purchased a solution more cost effectively after the fact.

PS. If the Custom Data Platform option appeals to your enterprise, make sure you are implementing these 8 CDP Best Practices.