fbpx

The skills challenges facing the data industry & how to tackle them

Female data engineer in a data centre

ARTICLE SUMMARY

Data is becoming ever-increasingly important in our day to day lives – everything from healthcare to entertainment is driven by data.

Data is becoming ever-increasingly important in our day to day lives – everything from healthcare to entertainment is driven by data.

It will come as no surprise, given how all encompassing data is, that the data industry is huge! A recent report has estimated that the data and business analytics industry will be worth around $274.3 billion by 2022.

But with great power, comes great responsibility – and the data industry is no stranger to its share of challenges. A lack of skilled professionals, data growth issues and securing data are just a few of the problems the industry is facing. 

In this episode, Taylor McGrath, VP of Data Labs at Rivery, gives us an insight into the data industry, the opportunities and the challenges it’s facing, and expected trends for 2023.

Rivery is a data pipeline and integration platform company, and Taylor manages the company’s data operations and data advisory, as well as its data solutions marketplace. She is passionate about the data landscape and empowering her teams to take products to market. 

Prior to this role, Taylor led Rivery’s solutions engineering, customer success, and support functions, helping the company to grow rapidly since she joined in 2020. 

hello everyone and thank you for tuning in again I am Kaylee boatsman the content director at chican code and
today we’re going to be discussing skills challenges facing the data industry and how to tackle them a recent
report estimated that the data and business analytics industry will be worth around
274.3 billion dollars by 2022 but with great power comes great responsibility
and the data industry is no stranger to its share of challenges a lack of
skilled professionals data group issues and securing data are just a few of the problems the industry is facing
thankfully I have the fabulous Taylor McGrath VP of data Labs at Rivery to
delve into this topic with me today welcome Taylor hi thank you thanks for having me thank
you so much for joining us can we um just set the scene a little bit with um a little bit of background about
yourself if that’s okay sure yeah um so I I’m currently the VPD
Labs at at Rivery um I I grew up in a small state in the
US called Connecticut and um kind of started started my data
career out of school as like uh you know a business intelligence analyst
um doing a lot of like Services work for companies where we were kind of doing
end-to-end business intelligence projects um kind of stayed there for a number of
years and started managing kind of end-to-end data projects all the way from like building a data warehouse to
you know the ETL pipelines that kind of create data models to then the the
business intelligence whether that’s dashboards reporting on on top of the data sets
um and then about three years ago I joined Rivery where I’m currently at now and we are a an elt pipeline platform so
we’re a SAS technology so basically went from like delivering the services to now
working for like the product that kind of can make a lot like a lot of that much easier
um so I started there um really actually very customer facing during our like customer success sales engineering uh so really kind of feeling
you know the pain that a lot of our our prospects and customers were feeling uh and then just recently you know
relocated to to London and um taking up our data Labs team which is all of our
like internal analytics as well as um working with like technical
Partnerships other other platforms that we can like integrate with with River ER our product can you tell us a little bit
about Rivery and what it does sure uh so Rivery is a SAS elt platform
which basically means it is a tool for building data pipelines in the cloud
um so what that covers uh is your data ingestion so getting data from Source
systems such as business applications like a CRM um marketing platforms operational data
and a database or data you know in file formats getting that into a data lake or
data warehouse in the cloud kind of build start to build that centralized
analytical source of Truth for the rest of your Downstream applications such as
like AI tools reporting tools to leverage so that kind of everyone in the
company right is kind of looking at data with the same definitions from the same place so a river recovers is the getting
of data into the warehouse all also the transforming of data in a data warehouse and then
also what is now called reverse ETL or data Activation so pushing data back out
to Source systems the world is getting much more um you know uh or combining more and
more of the analytical with operational so data activation is a very big uh it
topic currently and then kind of the orchestration of all of that so kind of building dependencies and saying you
know first I want this to happen and then if this condition is met run this process so kind of being that kind of
control panel for for all of your data pipelines is this River so when you joined um when you wanted to
move into the data industry did you was that something that that you knew you wanted to do did you just sort of um you
know think there’s a need there there’d be a really good opportunity for me or is that you know it did something
that direction good question I absolutely like absolutely not I had no idea that I wanted to like you know
would would love the data industry and end up there I think and uh I definitely can you know touch on this further you
know like growing up and and you know going through the classic kind of stem classes and education I think and it’s
it’s probably also a part because the the data industry is relatively new a lot of the um uh roles that exist there
are very new this was just not something I was really aware of you know in stem here you know there’s math there’s you
know engineering and so that’s what I ended up going to school for and just kind of assumed I would get a role in
like a more physical engineering role um and uh kind of you know when I was 22
I I had to make decisions like do I do I go be a consultant uh in in a city or do
I you know go move out to like Nebraska and work in like a power like a plant and uh you know optimize the
manufacturing flows there so I chose a city and um you know really introduced me that
first role in like business intelligence where really I was kind of building reports building dashboards into like
what the data industry was and that was about 10 years ago and I think even since then it’s evolved a crazy amount
um so yeah I I really had no idea coming out of school and I think part of that is just lack of awareness of kind of you
know where like Academia the education system is like you know it’s hard to keep up with how quick the data industry
moves um to kind of align education there and what’s possible to the actual roles that
exist um yeah yeah because it’s always so interesting to hear how people not only
end up in technology but how they end up in their particular roles because you’re absolutely right when you’re at school a
lot of the roles in technology they don’t even exist yet so um it’s always interesting to hear people’s entry point
into technology and and how they you know make their way into into um
um discipline such as as the data industry and what role does data play in
our lives today do you think um I I mean I think yeah a huge role I
think um there’s you know there’s the area of like business right where where data can play the role in terms of like driving
decisions backed by data like mapping certain data processes to business value
um but I also think like when you break that down you’re analyzing data points you know based on predictions or based
on what happened in the past making prediction based on what happened in the past like that that is a part of our our day-to-day life I think you know even if
it’s like not something in an Excel spreadsheet or you know a database necessarily we’re kind of curating all
these like data points of what’s happened to us in the past and making a decision based on um you know data that yeah may be very
biased to us but still is um you know different than just uh some gut feeling or you know some some
emotional decisions I think um it’s present all around and where where it applies to businesses we
actually have these these data points um like logged and and can kind of curate what what they actually mean yeah
because I think um I think you need to say we we hear a lot
about data and big data and careers in that area but I don’t think we always
quite understand exactly what that entails and not just what a career looks
like but I suppose what big data is itself it’s it’s something that we hear a lot especially in the tech industry
and but but what exactly is Big Data uh yes big data I think is very much you
know it’s a big it’s a big buzzword for sure um and definitely when I when I started
out early my career it was like the buzzword about you know uh 10 ish years ago
um and really like the tech definition is that businesses were seeing data with
um like greater it’s like I think the three V’s they say greater volume greater variety and greater velocity
right so we’re seeing things happening faster getting into more near real time
um bigger sizes of data um you know crazy sizes of data and then
variety I think is key too where data is not just coming in and very clean formats uh you know in with certain
level of quality it has to kind of be prepped and it’s coming in and in all different file formats or video formats
or whatever the case may be um so it’s kind of like how with all these things happening how do we
actually extract value out of these like big fast like varied data sets
um but I do think like when the word when the buzzword started like you know it was very early on in kind of the the
data uh world and so you had things uh you know when I first kind of joined the
industry we were kind of at the tail end of like the uh the Hadoop realm and kind
of these hdfs like very on-prem systems it was the start of like migrating things to the cloud
um was kind of the beginning of the Big Data era so I think really it’s like we had all this kind of you know
um data not really knowing what to do with it and the kind of early developments of tech that went along with it but it
really it’s just to me big data is like big aspirations right we just want to do big things with kind of all these crazy
amounts and crazy types of like data points now at our fingertips yeah
yeah and I think that that’s that’s the daunting thing I think if you’re not in the industry you’re probably thinking as
you mentioned your job in data is to extract the value like you say you know
the way that it comes in it’s not clean you you know you have to have the skills
to be able to extract that value from the data and I think that’s probably the thing that those of us on the outside
are thinking you know what kind of skills do I even need to have to to go
into to a sector like that yeah I mean I think like at the root of
it is some curiosity some desire for problem solving you know I think you
know that that’s that’s the biggest thing for me is is kind of continually being curious and kind of asking the why
and then where I think you know that kind of moves from just being purely technical problem solving to value is
when you can start to build skills to communicate and tell a story with some analysis that you did or your team did
uh because that’s where I think the real advancement in the data industry happens
is when you can kind of tie the technical to the value um so that’s you know I think a core a
core piece of advice um that probably I would tell myself as an early like analyst was you know
there’s one you know there’s something to be said sure about technically doing your job well and solving for technical
problems but really the the key is taking it to that next level where you can speak to how what you did just
impacted you know the bottom line of your your company yeah and do you think the industry is
facing skills challenges as well oh definitely I think there’s
um you know the amount uh demand is way outpacing Supply I’d say and uh and
that’s just because it’s it’s moving so quickly and I think um there’s you know endless kind of
innovation happening and um I think you know deficits in in
skills across the board I say especially data leadership just because it doesn’t it hasn’t existed for that long so there
aren’t you know these tenured professionals that have uh kind of withered you know 20 25 years of of data
roles because they really haven’t existed at least not in the way they do today um being very
um more I guess top top down where companies are actually saying yes like
um data needs to be a part of how we make decisions and how we build value
um just it hasn’t existed for that long it’s kind of just been a very uh you know in the world of let’s make sure we
can get you know some static report uh to you know to our shareholders but yeah
yes I agree because the tech industry moves so fast there’s always been that
challenge of you know trying to link education with the workplace um which I know in technology is
something that I’ve written about for for years I know they tried to um uh change things in schools and they
tried to you know make sure that children were more prepared for for going into jobs and Technology but it is
something there’s always going to be that Gap because as you mentioned technology moves incredibly fast data
roles are very new by the time we actually figure that out there’ll be something new that comes along anyway
and a lot of our community our first and second job and so I highly doubt that a
lot of those were studying for data roles at school or a university anyways
there’s always going to be that skills challenge um I think once you’re in the industry
if the industry is doing a good job to try and highlight the available opportunities that are there in the data
industry and I think that’s where people start to to connect the dots there about how how else do you think we can address
these types of challenges in the data industry yeah I mean I think you hit on on some
of it I think what’s um what’s probably a less realistic goal is kind of going all the way down into
you know early education programs because like you said it moves super fast so by the time these things are
initiated it could be very different in the real world um but I do think we can go like kind of
deeper down into entry-level grad level programs in terms of you know mapping at
least at least awareness of what’s possible and how to kind of like jump maybe from from one role to another uh
and then transparency around what it means for advancement to like what are the possibilities after your a data
analyst right like what what is next what what are all the other things kind of you can do to kind of progress and
Advance your career so I think it’s kind of that like mapping um
mapping positions to you know different skill sets early on uh and I do think
um a little bit what I said before is kind of earlier on in in like the
careers of data Engineers data analysts like entry entry level data roles let’s say
um to um kind of show the significance importance of uh mapping what you’re doing to the
actual business value so I think for me at least that’s something that took I’d say like years to really understand the
importance of just because you know it wasn’t really it’s something I had to learn through like trial and error and
and experience of um you know kind of you know how to how to how to communicate something you did
how to present something you did in a way that you know someone um someone
on the business side we will actually care about um and I think that’s that’s a huge
thing another thing for addressing these challenges is I’d say um at least with my experience of like
moving from role to roll or company to company um you know uh I think your your network
is is huge that’s something that uh I think they’ll be kind of uh an
investment of um not only like Talent education of entry level but also uh the importance
of uh building a network of Professionals in kind of different areas
that you you may want to jump to yeah yeah I absolutely agree you say
about mapping your job to the business value and I think that that is something
um you know just communicating with how do you get more people into an industry if they don’t quite understand
um the value that you can bring not just to your business but we talk to a lot of people that you know say I didn’t
realize until I fell into Tech the difference that I can make in people’s lives every day and the the good that
can be done with technology and especially with data I think that’s that’s a big thing that people don’t
quite understand how you can you know change people’s lives and solve real
world problems and I think it just it probably isn’t communicated very well within something like the data industry
you just think I’m going to be sitting looking through you know lots and lots of raw data none of it’s going to make
much sense and it’s not gonna um have much value or change um for anybody
yeah exactly and I think a lot of it um a lot of even I think content that
you see of the data industry is very uh technology forward very like Tech
driven so you’ll see like you know uh a lot about you know the new best of breed
products to like solve all these data challenges and I think um you know maybe not enough Focus about
what that actually means for for businesses for for Value uh and I think
that’s that’s something that exists you know across the board in the data industry is kind of focusing on like infrastructure and architecture and the
how we’re going to solve this instead of the the why yes yeah and we spoke um a little bit
about the challenges there but what what about the opportunities in the data industry um what what kind of
opportunities is the industry um facing I mean I I think they’re endless I think
um there’s so much Innovation happening uh so much demand for for kind of uh
data Talent which I think means uh only good things for everything industry kind
of um kind of being becoming very uh diverse and open to kind of any
um sort of like data leadership from atypical backgrounds upbrings education paths I think because it’s such a net
new industry of roles there’s really no um stat rules for like how you have to
get here uh so I think that’s that’s um a real big positive in terms of
opportunity yeah yeah and I think um we discussed this a little bit before about
um the challenges of of getting people into the data industry and probably not
quite you know realizing the opportunities available to them um or what or career in the industry
would look like we know across all the companies are struggling to get females into
technology anyway um but how do companies encourage more women into Data you know into the data
industry and into data and roles um you know is that something that you’ve seen yourself in in your career
and how do we overcome that yeah I think productivity I think with
the in terms of like you know um to actually kind of promote diversity
inclusion like it you have to be focused and proactive in in terms of your you
know it starts with you know kind of the HR recruiting right in terms of we want
to be proactive about the way that that we hire people and that we grow our culture as a company
um so I think uh productivity as well as like you know communication and transparency
um for for existing employees and kind of what what our values are in terms of
uh core values of of a company and then also like supporting managers and already existing leadership
on on um those changes on on those values so I think really it has to come from the top
and it has to be very like proactive and targeted that this is a goal
yeah yeah and once those ladies zoom in I think you’re absolutely right um being
more targeted and proactive and then once they’re in making sure that those ladies are visible and that they talk
about their job and and what it entails and um you know even on a level that you I
think some ladies I speak to they think you know my role actually isn’t very inspiring they probably can’t see
um actually that their story and you know what it what it is that they do day to day is actually
um you know very extremely interesting to people looking to come into the industry I think it’s important to be
visible and to share your story with with other females looking to come into
that um area of Technology um did you face any challenges coming in
at all or you know have you faced any challenges now you’re in the sector or
now in a very male dominated um sector yeah it’s true I mean it’s definitely
um you know it’s just the truth there’s it’s very male dominated um and I think as as I’ve grown in my
career I’ve um there’s definitely points where you know you feel you kind of have to uh
vouch for what you’re doing defend kind of defend what you’re doing especially when you get to the point where you have to say this you know this is the why of
what I’m doing uh in terms of you know some data analysis or you know some some initiative right so I think ownership is
is very heavy um and that continues as as you grow uh
for sure and then um yeah I think I I’ve seen a change in a positive way in terms of you know it’s
still very male dominated but I think there there is proactivity there is kind
of you know the um the efforts to kind of grow
um female uh data people in in uh in all levels of of a company oh yeah
yeah and I hear it a lot from ladies that I speak to that
um ladies in technology not just in technology right across the board always feel like they have to prove themselves
a little bit more and maybe to enter an industry or even when when they’re um in
a role um is that is that something that you found yourself in in uh in in the area
that you work definitely I think and and I don’t know you know um if you know it’s Society imposed or
some of it’s maybe self-imposed but I think I definitely feel you know a heaviness to prove myself I think all
the time in terms of um especially when like you know advancing in roles I think that only
increases uh in terms of denied I need to prove myself you know even more because I I haven’t been here yet and um
kind of that that imposter syndrome of it all I think is uh you know it’s something that I think you know you can
have um things you do to minimize but I you know I don’t think ever really goes away
completely yes oh God yes imposter syndrome loves to see pain doesn’t it
every every company that we move to it sits in um for our listeners if you wanted to
move into a career in data um are there any skills that you you know you would pick um that stand out
that you think they should probably um already have or go off and try try
and learn themselves yeah I mean I think there’s you know in terms of
um I guess non-technical skills I would definitely say um just kind of having the building that
that Curiosity for for Learning and kind of always wondering why because I think
that is the easiest way to start tying what you’re doing to business value is kind of keep asking okay why are we
doing this why is it actually important and then you’ll eventually get to the point of what the ROI is for whatever
task you’re on or maybe there is none and that’s that’s a good insight as well
um so I think for um non-technical skills I would definitely say curiosity
um communication is huge uh as well um for technical skills at least in the
data world I think there are um you know the trend right now is a lot
of Technologies and products to make life easier right to do maybe things no
code low code so I still think the fundamentals are very important but I don’t think you necessarily have
to go as deep into the weeds in terms of you know certain certain languages or uh
unless it’s a very specialized role you know that you want to do at least with my background I started in business
intelligence which was very much just um like dashboarding reporting
um so just understanding kind of uh the basics of um of data and data modeling and and you
know what granularity means and things like that but then getting kind of deeper into the data engineering side
and kind of learning you know all the ins and outs of of SQL and data
warehousing so I think you know you you can take it as deep as you want but I
think the fundamentals are key across the board because we do have lots of
products making making our lives much easier but even with those you know it’s not a perfect story The the like you
still need to kind of understand um the fundamentals of you know data modeling governance and and Analysis Etc
but yeah yeah because I think from those of us that don’t work in data it sounds quite
intimidating to think you know to to go to go into a discipline like that but it’s it’s great that you know to hear
you say actually you don’t have to be that technical to get started and it is up to you about you know how far you go
into it from the sounds of it you really just need to um be somebody that likes to solve
problems and that’s really your starting point um and then you know take it take it as
far as you want I think it just it sounds quite daunting as a career um but I think a lot of Technology jobs
actually sound quite daunting until you’re in the sector um and then realize actually what it’s
about um uh we are we nearly running out of time so I’m going to ask you one last
question about um uh the future if that’s okay um what what trends will we see in the
data industry in 2023 we’ve already mentioned that it moves incredibly fast
so definitely I think uh so I’ll I’ll guess
I’ll answer in two parts first I guess the the technical uh Trends
um I think you know we have a lot of innovation happening um you’ll see you know endless you know
companies whether you know startups all the way to Enterprises of um kind of different Technologies
um I do think we’ll see a large amount of uh consolidation this year meaning
um kind of looking for for Tech that can do uh more than just one thing in in the
data World um kind of the bundling of Technologies I’ll say uh and I’m I’m hoping and I
think this is also kind of the reason why is it’ll lead to less focus on the infrastructure of it all and more on
aligning data practices to the business value from from top down um in terms of the the non-tech uh
non-technologies I would say um to that same realm of you know mapping and kind of I think the theme of
a lot of what we talked about today mapping what you’re doing in the data world to what it actually means for for
a business is kind of that spread of data literacy across all parts of the business and
understanding that yes these are things like you know we have coders and Engineers kind of doing you know in a
back room and they’re on their computer but um this is what it actually means for us Revenue wise or you know something in
terms of business value um and with that I think there has to be you know investments in in Talent
education whether that’s a Content like this you know YouTube uh LinkedIn Etc
it’s very easy to like spread knowledge now so I do think they’ll just be increased
um increased content in that realm and increased opportunities uh you know in a positive way to like learn
um about the data industry and get into it if that’s you know if that’s the desire yeah yeah and in terms of um
obviously we discussed that things move incredibly fast and as you’ve mentioned there the industry is due to change a
lot in um 2023 how do you see your role already at Rivery
um you know changing and from what you do now to you know where you see things going in the future and you know what
what um River we plan to do with data uh next year and Beyond
I’ve been lucky enough with referee to kind of you know move move all around the the company so like I said it
started with like the customer facing type roles and now doing our uh or leading our data team so we’re a small
team of data people currently so I think our you know the goals for next year the predictions for next year are a bit of
growth but that also means um kind of grappling with you know how you scale a data team you know if you
have more than just a couple analysts uh you know what what that means in terms of how do we prioritize tasks how do we
continue to map what we’re doing uh improve the the results of what we’re doing and
um uh kind of have have that ownership as as as um you know data team which is you know
usually considered a cost center of sorts to prove that the ROI what we’re doing so I think it’s it’s going to be
questions of uh scale and also of Roi is is what’s on my radar for this year
fabulous okay well we we are out of time it flies by when we’re talking about
something um incredibly interesting and thank you so much our community is really gonna
um find that very useful whether they’re already in the data world or they’re looking to move into a career into it so
thank you so much Taylor for for joining us today yeah thank you thanks for having me so much thank you and for
everybody as always thank you so much for listening and we hope to see you again next time

RELATED ARTICLES

Preparing for a job interview can feel daunting, but with the right approach, you can turn your anxiety into confidence. Whether you’re a seasoned professional...
Discover Lisa Iudiciani, Lead Software Engineer at Vista's journey of career reinvention, balancing technical growth with leadership development, and her insights on mentorship for aspiring...
In this episode of Spilling the T, Seena Samani, Head of Data and Analytics at Mindshare UK, dives into the dynamic world of data and...
Amanda Whicher, Technology Director at Hays UK&I, highlights the importance of job interviews as a two-way street, where both employers and candidates assess each other....