Report: Data Engineers Spend 2 Days a Week Fighting Poor Data Quality Fires

Couldn't attend Transform 2022? Check out all the summit sessions in our on-demand library now! Look here.

Past industry surveys have shown that most data professionals struggle with poor data quality, despite the time and resources invested in addressing it. To dig deeper into this insight and help data leaders understand where and how to invest resources, data reliability company, Monte Carlo, partnered with Wakefield Research to interview more than 300 data professionals to reveal the state data quality in the wider market.

Results of the study show that teams are still struggling with data quality and wasting valuable time, money and resources resolving issues that arise. Respondents indicated that, on average, 40% of their time is spent evaluating or verifying the quality of data. Additionally, more than half of respondents indicated that building or repairing pipelines took the longest time during the day.

When data incidents have occurred, the majority of data workers (62%) said their detection time was five to eight hours. The average data incident resolution time was nine hours.

Poor data quality has also been shown to affect the bottom line; 47% of respondents believed that bad data affected 25% or more of their company's revenue.

Of the 300 data professionals surveyed:

75% take four hours or more to detect a data quality incident. About half said it took an average of nine hours to fix the problem once identified. 58% said the total number of incidents had increased somewhat or greatly over the past year, often due to more complex pipelines, larger data teams, larger data volumes, and other factors.

However, there is positive news. About 90% of respondents said they were already investing or planning to invest in data quality solutions, such as data observability, within six months.

The survey of over 300 data professionals was conducted between April 28 and May 11, 2022, using an email invitation and an online survey.

>

Read the full Monte-Carlo report.

VentureBeat's mission is to be a digital marketplace for technical decision makers to learn about transformative enterprise technologies and transact business. Learn more about membership.