“Le mieux est l’ennemi du bien” or loosely translated to “Perfect is the enemy of good” is a saying that most attribute to Voltaire. It’s a cautionary phrase about the perceived value of chasing perfection versus the reality of it. The phrase is something that can apply to numerous domains, including Data Engineering.
Suggesting that perfection is not the goal often invites skepticism from data engineers. After all, developing data products requires precision and a detail-oriented approach. Further, most Data Engineers have been a part of rushed implementations only to have it come back to haunt them. Sprinkle in a natural aversion to risk and there are multiple reasons why people would challenge the notion that stopping short of perfection is a good idea.
However, this is not a plea for shortcuts. Instead, I’m advocating for delivering faster and only implementing the valuable features.
The concept of delivering faster is similar to Agile’s principle of shipping a Minimum Viable Product (MVP). The value of quickly putting a product in the hands of a user cannot be overstated. For example, if your leaders require three metrics to run the business and you have only one completed, ship it. Driving the organization with some insight is always better than driving with none.
The only thing that might be worse than waiting to ship a viable product is developing the remaining features in the pursuit of perfection. These remaining features can often take longer to implement than others or provide a reduced value or both. For example, if adding a dashboard visualization for a single user requires breaking model and pipeline changes, the payoff probably does not warrant the effort. All of a solution’s proposed features should be scrutinized for their projected return on investment, but be particularly vigilant about ones at the bottom of the priority list.
Data Engineering is a challenging endeavor that is often fraught with complexity. By delivering faster and only implementing the valuable features, you can reduce the risk and increase your chance of success.