Job Details

Team is completely Python based (Java a plus, not needed)

Need Python and Spark EMR

AWS Glue is good but comes later in migration

Needing someone Senior in Data Engineering / Architecture

Come in and drive solution design at team level and beyond

Nobody interviewed was high level enough to feel they could do this

• This year, data tech team is going through migration to converge current platforms into Spark and Glue ETL
• Expertise in Batch ETL
• Apache Spark is current work, Glue comes later based on priority
• Converging a lot of existing jobs, Spark and non-Spark jobs, Apache Spark running on EMR, eventually onto Glue. Coming in with Glue is helpful ,but main expertise in Spark EMR is needed first. They want to get to the point where everything is running on Glue, but that's a little ways away. First is getting Spark EMR jobs in place.

• Best practices for how do you performance test an ETL job?
• Senior contractor should holistically know these things
• How do you test these pipelines for governance and best practices?