Data goals: maximise the value from data

‍

Flibco is a Transportation Management Software, custom developed, which runs on a semi-structured (NoSQL JSON) database.‍ With flibco being operational for certain time, they were looking to extract maximum value and insights from the underlying datasets the platform generates. These data-driven insights would enable Flibco to make impactful business decisions with fact-based answers from their data such as:

‍

Productivity:

How much time do drivers spend actually driving around passengers?

‍

Compliance:

Does the scheduled vehicle of any given trip comply with specific passenger needs? (e.g. disabilities, child seats, etc.).

‍

Human Resources:

How much overtime did a driver perform in any given month? What does that mean in terms of salary calculation?

‍

Sales:

How is Flibco performing against their KPI’s for bookings, tickets, trips, etc.?

‍

‍

Data Challenges: Transforming raw data into activation-ready insights

‍

Removing data silos to increase data clarity

When making impactful data-driven decisions, you first need to have the full picture in front of you. Bringing data together from various data sources, allowed Flibco.com to better connect the dots between different data sets and build an holistic view of the software.

‍

‍

Data transformation & modelling to increase data quality

Getting easier, faster answers from data, required Flibco.com to significantly improve the quality of their data first. Flattening the JSON datasets, handling different UTC timezones, and achieving some level of modularity in the data were among the bigger challenges of Flibco’s data journey.

‍

‍

Driving a company-wide understanding of data flows

When people work with data, they need to trust the data they’re seeing. And that requires people to understand what data they’re seeing, and where it's coming from. Troubleshooting at Flibco.com was difficult due to lack of both proper documentation and a company-wide understanding of data flows.

Accelerating the Flibco Data Journey with the Modern Data Stack

‍

To accelerate Flibco's data journey & increase the level of data maturity, a modern data stack infrastructure was implemented to allow for better, faster & more effective data-driven decision-making at scale.

‍

‍

Extract & loading data in under 5 minutes with Fivetran

Fivetran manages data delivery from source to destination with fully managed connectors, so Flibco didn’t have to worry about engineering.

‍

Fivetran offers 150+ zero-configuration connectors that launch in minutes, allowing Flibco to load data into their data warehouse without hassle. Centralising all their data and empowering their team with faster insights.

‍

‍

Removing Data Silos with Snowflake

Snowflake’s support for data warehousing and analytics provides a low-maintenance, cost-effective way for Flibco to consolidate all their data silos into a single source of truth they can query to get results fast.

‍

This streamlines data engineering, delivering performance and simplicity so Flibco and their data teams can now focus on getting value from their data instead of managing the pipelines and infrastructure. Win-win.

‍

Data transformation within the warehouse with dbt

Dbt is a SQL execution tool that helps Flibco.com apply DevOps best practices to data transformation workflows.

‍

It does not extract, hold or write any data; it does not even provide the CPU for the transformations. Now, it simply holds Flibco’s data logic and transformations (written as SQL scripts) and sends them to the (usually Cloud-) database for execution.

‍

‍

Intuitive analytics & data exploration with Tableau

Tableau offers robust data connectivity options, allowing users to connect to various data sources, from databases to cloud platforms, ensuring flexibility in data integration.

‍

So, once placed on top of the Snowflake data warehouse, Tableau helped Flibco’s data users visualise and explore all of their data without having to write code. They can now dive deeper, pivot analysis, and change chart types without starting over or breaking their flow.

‍

‍

‍

Key results

‍

‍

Saving big costs with Fivetran

Having a standard connector available out-of-the-box for both the semi-structured source and the Snowflake destination, Fivetran was set-up in a matter of hours.

‍

Almost no expertise was required from Biztory’s side. Fivetran’s tool was simple enough for Flibco’s internal colleagues to handle without too much trouble, which means the operating costs are based almost entirely on the tool’s usage, rather than Data Engineer FTEs.

‍

Big cost saver.

‍

‍

Snowflake eats JSON for breakfast

With Snowflake’s all-powerful lateral flatten() function we had the JSON arrays and objects normalised in no time.

‍

Being the CPU powerhouse that it is, Snowflake was also able to reduce runtimes of flows that used to take over an hour, to under 5 minutes. This without even requiring the Medium or Large “warehouses” (=CPU pools), so essentially in first gear.

‍

Under the hood, snowflake stores all timestamps as an amount of milliseconds since 01-01-1970 in UTC time (as do many databases). By setting sessions or account timezones, Snowflake is then able to both present the actual time in the specified timezone (by adding or subtracting the required amount of hours from the UTC value), as well as
perform datediff() calculations on the actual UTC values.

‍

Eg. Snowflake knows the difference in time between 01:00 AM on October 29th 2023 (=UTC + 02:00) and 01:00 AM on October 30th 2023 (=UTC + 01:00) is 25 hours and not 24!

‍

‍

Fast and rigorous progress enabled by dbt

As mentioned above, dbt does not extract, hold or write any data; it does not even provide the CPU for the transformations. It simply holds Flibco’s data logic and transformations and sends them to the Cloud-database for execution.

‍

In this pretty straightforward capacity, it is however capable of transforming Flibco’s Data Modeling flows to true DevOps-proof Data Engineering!

‍

Some of the areas where dbt delivers value are:

• DML/DDL & Materializations
• Data Lineage & Documentation
• Troubleshooting & Tracking Upstream Data
• Scheduling
• Advanced Coding
• Automated Testing & CI/CD
• Version control & DEV/PRD Environments

‍

‍

Conclusion

In summary, Flibco's adoption of a Modern Data Stack infrastructure, featuring tools like Fivetran, Snowflake, dbt, and Tableau, has significantly enhanced their data handling capabilities. This transformation allows them to make informed, impactful business decisions and continues to propel them as a driving force in the mobility sector of tomorrow.