As organizations grow, so too does the specialization of teams addressing specific aspects of the organization’s needs. The same holds true for data organizations. A new data science team might be established to generate novel insights, or the data engineering team might split off a group for managing business critical data streams. Within these teams you may have many different roles too such as data engineer, data modeler, business analyst, or data scientist. This specialization can differ significantly from company to company, but the expansion of these groups always comes with the same growing pains.
While all organizational growth may lead to disagreements between groups, with data teams there’s also a more fundamental shift in problem solving which is challenging to identify. A team that once worked well together can start to splinter over how to solve a problem, and it’s easy for a gulf to form between the various teams. Too often the assumption is that the distance between teams is caused by bad hires — an assumption that can derail resolutions — when in actuality it’s the result of each team’s specialization. The approach these developers take to perform their jobs can vary dramatically and what feels like an obvious path to solve a data problem for one individual may be counterintuitive for another team.
Let’s look at a hypothetical scenario: you have three data teams collaborating to bring some intelligence to your website. One team, a newly established data science team, has developed a model for the purchase path that will suggest closely related products to upsell in the cart. A data engineer is assigned to the project who is responsible for collecting raw website data and preparing it for use in model training. And finally, an application engineer is assigned to the project to integrate the model into the website and deploy changes to the production site.
Continue reading this post on our Medium blog.
