Hear from Rosie Lang, Senior Principal Data Engineer, at Quantexa, and find out about her journey.
Can you start by sharing a bit about your background and what led you to Quantexa’s Academy Programme?
I studied Chemistry at university. Towards the end of university, I realised I did not enjoy the lab-based experimental work, but I loved the problem-solving aspects of the degree.
For my first job, I worked at a technical consultancy. I had done some basic computational modeling at university, but I had no programming experience at all. Over the 5 years, I worked on numerous projects and picked up the tools the client was working on; a mixture of SQL, Python, scripting, and SAS.
I was excited to start working at Quantexa as it allowed me to pick up a new language and understand more formal software engineering practices.
I have now been working at Quantexa for over 6 years and I am still enjoying it. I have been lucky enough to work on a few international projects but have now settled back in my hometown of Bournemouth on the sunny south coast. I have recently had a period of maternity leave and am now adjusting back to work with a little one to entertain when I am not working.
What were your initial thoughts about having to learn Scala after being familiar with Python?
When I joined scala was a relatively new language and it is still fairly uncommon, so a lot of colleagues were in a similar situation. Having worked in a consultancy environment I was used to adapting to different technologies.
In a way, I found adapting to a more formal software engineering practice harder than the language change. I was more familiar with notebook analysis and had no experience of what a Production Release was.
What was your experience like during the academy programme?
The academy is intense but enjoyable. I joined Quantexa as an experienced hire along with several others. We went through the academy together and it was great to have people to bounce ideas off and form a community with. Everyone had different levels of knowledge and skills, and we often worked together.
How can they best prepare themselves for the transition from Python to Scala?
There are a huge number of online tutorials and exercises. I would recommend spending some time on them, partly to give yourself more confidence that you can make the transition. However, before starting at Quantexa you do not need to do any additional preparation. The training is very comprehensive.
Technology moves so fast, that you are constantly having to adapt and pick up new skills, even within a language, version updates and new 3rd party libraries are constantly released.
I really enjoy that Quantexa stays ahead of technology changes and proactively uses new tools and technologies, and the part of the job I most enjoy is having to pick up new skills.
Can you describe a specific project or assignment during the Academy Programme where you applied Scala? What were the technical challenges you encountered, and how did you overcome them?
During the Academy programme, after the initial training you are asked to create a mock project using open source data. This involves using Scala to join, process and cleanse the data, as well as Quantexa tools and libraries.
I found the build process of the academy confusing. Having primarily used python in the form of notebooks I was not experienced with packaging up code into shareable artifacts. We used a tool called Gradle to collect our dependencies and build our jars. I needed support debugging when things did not work. Quantexa has a great portal called “rnd-support” where you can ask for support and see questions (and answers) from other colleagues. I found this extremely helpful for debugging and reassuring that others have encountered similar issues.
Did you use any particular libraries or frameworks in Scala during your training?
A lot of the libraries I use are Quantexa-developed libraries, specifically written for our purposes. There are loads of interesting data quality, data generation, network building, graph analysis, and scoring libraries that we use day to day. The library that I found hardest to understand is Shapeless, a library that allows generic programming in Scala. I have spent a long time trying and failing to understand vague error messages and have often had to ask for help from colleagues.
Could you share a screenshot of some code you worked on?
Most of the code I worked on is on sensitive client data, but here is an example of an ETL from our “Project-example” (used as a guide for best practice and training).
In this code, we are joining up two datasets into a single nested dataset and collecting some population metrics.
One of the nice features of Scala is the ease you can work with nested datasets. We often receive data from clients from a relational database and have to join them together to get a full picture.
Metric collection is also particularly important. At this stage of the data processing, we collect population statistics of key fields. In the later stages of the data processing, we apply cleansing and parsing. We then collect further population statistics so we can monitor how much data we have lost and ensure we have not lost anything valuable.
How did your understanding of functional programming principles evolve as you transitioned from Python to Scala?
Slowly! As I do not have a formal engineering background, I did not start off knowing any programming principles let alone functional ones. The training Quantexa offers helps and through conversations with colleagues, I have learnt a huge amount. After a few years of developing in Scala, I am now working on a project in Python, and I can now see the contrast between the two languages and why the functional nature of Scala is such an advantage.
How did Quantexa support you through the learning process?
As well as the Academy programme, Quantexa offers continuous learning. Everyone is encouraged to take an hour out on Friday afternoon for personal development. In addition, there are frequent “Drink and Discover” sessions – informal knowledge sharing run by developers. We also have Slack and other channels for asking technical questions and for assistance.
How well do you feel the academy prepared you for your role at Quantexa?
The academy was a great introduction to some of the techniques and technologies we use at Quantexa. It also gives an idea of what life is like on a project. It takes a bit of time to understand how everything fits together but by the end everything is clear.
What advice would you give to candidates who are considering joining Quantexa’s Academy Programme?
If you have a STEM (or similar) background but no prior Scala coding assignment, don’t worry, we’ll give you the support and training you need. Have a read about Quantexa and the problems we solve for our clients. If you think that sounds interesting, then come and join us.
