There’s a hot job emerging in the tech industry, and it revolves around mastering a new big-data tool run by a startup out of Philadelphia.
Called analytics engineers, people in this role use an open-source tool called dbt, built by Dbt Labs, to clean and maintain large data sets. The process is designed to make the jobs of analysts and data scientists — who are used to crunching unstructured and messy data sets — much easier and faster.
Companies like Amazon, Apple, Netflix, Discord, and Instacart are hiring analytics engineers. The median salary for an analytics engineer listed on Glassdoor, which relies on aggregated data, is about $127,000. One job listing for a senior analytics engineer at Discord suggests the role pays more than $200,000 a year, while one from CircleCI says it pays $128,000 to $160,000. (Companies must list salary bands in New York and Colorado.)
Becoming an analytics engineer doesn’t necessarily require a lot of new skills for existing analysts. But it does require a mindset shift, said Emily Hawkins, Data Engineering Manager for Data Platforms at Drizly. Hawkins runs a course on co:rise explaining how to use dbt.
“If you just have a data dump in an Excel sheet, you’re probably not thinking about how it got into that format,” she said. “You have to think about how to get data from this very raw source that hasn’t been processed or transformed into something that someone could dump in an Excel sheet and do some analysis.”
Still, people interested in the role will need some technical (and soft) skills to get started.
Dbt is mainly responsible for popularizing the role thanks to its ability to streamline the data-crunching process. Analytics engineers use dbt to transform data into a more useful format for people internally looking to derive insights.
Dbt relies on a language called SQL to pull data from a data warehouse like Snowflake. It then transforms that data into a more useful form, such as by aggregating parts of it or cleaning dates. Analysts can use the data from that dbt transformation instead of constantly rerunning those aggregations, which can be expensive and time-consuming.
Co:rise offers an analytics-engineering course on how to best use dbt run by Hawkins and Jake Hannan, a manager of analytics engineering at Gopuff. Dbt Labs also offers courses in building expertise with dbt.
Because the tool is very new, Hawkins, who works in analytics engineering at the alcohol-delivery platform Drizly, said that immersing yourself in the community is an excellent way to build experience and expertise.
“When I joined Drizly, I had never used dbt and had to learn it from scratch and learn it from nothing and implement it,” she said. “For me, I had to put myself out there and ask questions in the dbt community. But I think now there are more resources available with courses, and I know dbt has some online tutorials.”
Much of the appeal of the analytics engineer is that it expands what an analyst can do with SQL. It’s one of the oldest and most popular database languages, and analysts are typically required to know it.
The role also allows SQL experts to take on some of the tasks data engineers typically handle using SQL. Those tasks have traditionally required expertise in Python, which, while nice to have, is no longer a necessity, Hannan said.
“I think what Snowflake and dbt have done well is SQL is the source of truth for everything,” he said. “But what is interesting is that both Snowflake and dbt are working on Python support. You can write Python scripts in dbt and it’ll compile to SQL and run it. But it won’t be a hard requirement.”
Both Udemy and Coursera offer courses on SQL for analytics and data science. IBM, Microsoft, and Oracle also run certification programs for SQL.
Analytics engineers are popular because they essentially simplify working with data warehouses like Snowflake or Amazon Redshift. Their work results in clean, easy-to-use tables that analysts can use to complete ad hoc requests or build dashboards with Tableau or Looker.
So it makes sense that they’d have a strong understanding of some of the inner workings of data warehouses like Snowflake or Redshift.
Snowflake has a Snowpro certification, while Amazon offers a big-data specialization.
Analytics engineers often start as analysts because they’re the target audience for what analytics engineers build. Hawkins, for example, was a business-intelligence analyst before moving into analytics engineering.
She said one of the most critical factors is to remember that analytics engineers are trying to simplify massive data sets for analysts and not trying to be overly technical.
“A typical analyst would write this bespoke query 400 lines long; it gets that one answer — that’s what the stakeholder does today,” she said. “With analytics engineers, it’s a different paradigm: You get models that serve 90% to 95% of the use cases. I’m not creating this one query that solves this use case. I want to create value for everyone. Being an analyst is good as a background because you know what analysts tend to look for.”
An essential part of data processing is ensuring that everything is running in order and that things aren’t failing. That process is called orchestration, and several tools are designed to manage it, like Airflow, Prefect, or Dagster.
Hawkins said that while it helps to have experience with orchestration tools, it isn’t vital. But analytics engineers at smaller companies may be expected to take on more data-engineering tasks, including implementing those orchestration tools.
Laurent Paris, a senior vice president of research and development at Astronomer, the startup behind Airflow, previously told Insider that the creators of Airflow were trying to make the product easier to use for analytics engineers.
“You could be an analytics engineer at a company and you just live and breathe SQL,” Hannan said. “Other companies, you might own the full end-to-end of ingesting raw source, going to the other side of the spectrum, and creating dashboards. Each company has pretty different jobs.”
Astronomer offers certification with Airflow, the most popular open-source tool. Companies like Udemy also offer courses on Airflow.
Keep reading
For you
