The global pandemic has driven home the fact that data is vital to the success of every organisation. Companies across the UK are realising the importance of scaling and growing their analytics capabilities, something that has become even more critical in the ‘covid’ era.
According to the 2020 UK Business Data Survey, 81% of all businesses surveyed say they handle digitised personal data, digitised non-personal data, or both, and use of data increases considerably as businesses become larger.
This means organisations need to bring in the right talent to make sure their data is being used appropriately and effectively. But what does it take as a data engineer to keep up with the innovation needs of today’s fast-paced businesses? And are we expecting too much from today’s analytics talent?
‘The solvers of all business problems’
Despite widespread recognition of the value of data in business, budgetary constraints, skills challenges, education around data, and the best use of employees’ time, remain key challenges. There is also the risk of data teams being seen as ‘the solvers of all business problems’ – and they become overloaded with irrelevant questions.
In fact, what we’re seeing more than ever are the data teams are suddenly becoming the ‘perceived source of all business decisions’, while the CEO, the CTO, the CFO are just ‘hammering them’ with questions and requests.
What this is doing is creating a bottleneck issue whereby engineers are becoming bombarded with requests that could often be solved by analysts or self-serve BI tools. This means their resources are taken up on things that aren’t the most efficient use of their time or driving real value for the business.
Great data engineers are highly coveted and hard to come by, so companies have to start thinking smarter about their data strategy and how to prioritise requests for their engineers in order to ensure companies are making the most use of their teams and talent.
Understanding the role data plays in business success is essential for making the most of the talent and skills a data engineer has within a business. Tools and technology are a crucial part of that success, but the value of it will only be realised when enabled by the right people and process.
The need for data engineers to drive better business decisions
Decisions need to be driven by the actual real-life data, rather than intuition or passive observation, says Carly Metcalfe, Director of Data Engineering at Forth Point.
“This will lead to more accurate, more optimal decisions being made on the ground,” Carly explains. “Data-driven decision makers should have a deep understanding of what purpose data serves, why the problems we seek to solve exist in the first place and how data can ultimately be used to improve the lives of people around the world.”
By focusing on the real-world problems that data can solve and providing tangible benefits from decisions arising from this data, the UK’s image as a leader in data-driven engineering and leadership will be boosted, Carly adds.
Organisations can foster a data-driven culture through:
Having a transparent data strategy in place;
Buy in from the top of the organisation;
Data Governance measures; and
Data assets data lineage being documented and made available.
Critical skills of a modern data engineer
According to Carly, to be a good data engineer, you need three key things: communication, problem-solving skills and agility.
For communication, she explains, “Understanding the challenges facing a client and communicating this information to the rest of the team are vital skills required to ensure that projects run smoothly.
“A great data engineer will be able to interpret complex requirements, analyse possible solutions and suggest the benefits and potential downsides to any approach.”
For problem solving, Carly says the majority of a data engineer’s time can be spent optimising or problems solving issues with data rather than building the pipeline.
When it comes to agility Carly highlights that building data pipelines can be a challenging and complex task.
“Compatibility issues may arise if the structure of the data provided ends up being different to that which was agreed upon,” she adds. “A client may also prefer a specific tool or methodology. An agile data engineer can pivot and adjust the approach as required to suit the requirements.”
Understanding the tools
According to Carly, the amount of tools and applications in the workplace right now can make it overwhelming for data engineers to do their job.
“The sheer scale of tools and applications that are out there can make it challenging at times to get to grips with all the latest technology and concepts,” she says. “The volume of data that needs to be processed can also cause challenges of its own.”
Currently in Scotland, particularly Edinburgh, Forth Point has recognised there is a shortage of data engineers, notes Carly.
“I would say we need to have more data engineering degrees and graduate schemes. We are fortunate to have universities such as Edinburgh and Napier which offer data engineering degrees, she says. “But we need more courses that can teach the foundational skills required to be a data engineer.”
The evolution of the modern data team
The future looks exciting. Companies that are moving beyond traditional BI are those that are disrupting markets. They’ve invested in modern data teams to optimise insights across the organisation, driving growth and ROI in the process.
Organisations that have a data driven mindset are encouraging frequent discussion and offering plenty of opportunity for learning, embracing failure and successes and keeping a strong focus on data security.
Where data teams of the past may have had three traditional participants: database administrators responsible for data warehousing capabilities and capacity, data analysts focused on data modelling, and BI architects in charge of building dashboards and related self-service reporting capabilities, today the team is so much more.
Typically a team that partners with the business but has its own specialised skills, this group does everything from outlining business rules to modelling data to building single sources of truth.
The role of a great engineer in this kind of team is to be the strategic thinker – what are the patterns that are emerging from the adhoc requests? How do we build the sustainable assets that will allow us to scale our delivery to the business without growing costs? How can we make life easier for us and for our downstream users?
The answers to these questions may include advanced techniques like applying machine learning or some statistical and predictive technologies to determine patterns or gaps where your organisation simply does not have ideal data. It’s about blending the strategic and theoretical, with the tactical and practical.
A great data engineer will also increasingly be blending model data with raw data from multiple sources. Ideally, your data team can take raw data coming in from a new system and join it with existing model data, such as account, opportunity, or service data, to answer questions that may not have been anticipated in a current data model.
If data is blended from multiple sources into a single warehouse, does that process occur before or after it arrives at the warehouse? For example, a more modern approach would blend the data post warehousing. Data engineering includes elements of problem solving and solution design that have traditionally been associated with architectural roles.
Unlike a traditional BI team that provides only a data model, today’s data team operates as a core function of the business and recommends a strategy built on data. In a modern data team engineers, analysts and BI developers are working cohesively in an organisation to provide up to date and precise information. The group has a seat at the table, offering informed opinions — based on data — about what the company should be doing.
(Source: AI News What data skills do you need to drive business success? (artificialintelligence-news.com))
Chinese
Japanese
Korean
