Position: (Senior) Data Engineer
Location: Canary Wharf, London (2 days per week onsite, 3 days per week remote)
Reports to: Head of Data Analytics
NO VISA SPONSORSHIP
As the Data Engineer, you will own the client´s data ecosystems and will work with developers, solution architects, technical BAs, data scientists, and other SMEs to define the optimum data architecture for the business.
You will build the required ETL/ELT pipelines for the collection, preparation and storage of data in a form that is optimised, secure and reliable.
You will have a specific focus on developing the data inputs that are utilised by the organisation’s internal forecasting model of the GB power markets; designing a framework that scales effectively along with the source data.
- Design data infrastructure to support modelling and data analytics
- Build and maintain ETL/ELT pipelines to make data accurate and easy to use
- Work to ingest and transform data sets from a variety of data sources
- Explore ways to enhance data quality and reliability
- Assist with the establishment of a data culture across the organisation
- Drive better data governance through the creation and embedding of principles and processes e.g. Logical Data Model & Flow Diagram, Data Dictionary, Data Semantic Layers
- Set service level indicators and monitor execution of data workflows and configure alerts
- Apply dimensional data modelling concepts and practices to develop conceptual, logical and physical data models to support insight delivery
- Identify data quality issues through data profiling, analysis and stakeholder engagement
Required Skills & Experience:
- At least 2 years of designing data infrastructure to support modelling and data analytics
- Experience building, modelling, and maintaining data pipelines
- Strong experience in SQL optimisation, performance tuning, data modelling and SQL/database design skills.
- Hands-on experience within the Azure Data ecosystem, with Azure Databricks, Data Factory, Data Lake, and Synapse. Certifications are a plus.
- Strong competence in Python or Scala, ideally PySpark experience
- Experience in Microsoft Azure Datawarehouse architecture as well as Data Warehousing tools and database systems
- Experience of ETL and ELT processes, working with database architecture and business intelligence tools
- Experience with database administration
- Able to develop and optimise queries in SQL
- Experience in creating data pipelines using Azure Data Factory
- Experience with Azure: ADLS, Databricks, Stream Analytics, SQL DW, COSMOS DB, Analysis Services, Azure Functions, Serverless Architecture, ARM Templates
- Experience in writing PowerShell scripts
- Able to work with large datasets and extrapolate conclusions
Nice to have:
- Experience in building and maintaining reliable and scalable ETL on big data platforms as well as experience working with varied forms of data infrastructure inclusive of relational databases such as SQL, Hadoop, Spark and column-oriented databases
- Data Engineering experience in Microsoft stack
- Circa £55k - 75k GBP per annum depending on experience
- 25 days' annual leave and bank holidays
- Recognition schemes allowing colleagues to say thanks
- Company contribution to your pension scheme
- Family-friendly policies, including enhanced company maternity/paternity and shared parental benefits
- Employee assistance programme for free, confidential support for your professional or personal life, including financial management and family care
- Special leave such as study leave, sabbatical or public duties
- Three days of paid leave a year for volunteering to support your local community
- Season ticket loan scheme to support your commute
- Access to “Work Perks” offering deals, discounts and cashback on your purchases
- Family savings on days out and English Heritage or gym discounts through our partners.
Contact Person: Elvis Eckardt