Senior Data Engineer

About BeInspired and Team Inspire

BeInspired is a Technology, Transformation & Growth delivery partner. Through Team Inspire, we mobilise specialist consultants and multidisciplinary teams to deliver critical workstreams and complex programmes.|We are working with a leading public sector organisation to find an experienced Senior Data Engineer to develop, integrate and optimise the data infrastructure, services and pipelines underpinning critical digital capabilities within a central government environment.

The role combines hands-on engineering with technical leadership, ensuring data connectors, integration services and underlying data foundations are reliable, scalable, secure and suitable for advanced analytics and AI-enabled applications.

The consultant will design and implement complex data flows, connect operational and analytical systems, improve data reliability and performance, and establish reusable engineering solutions.

Key responsibilities

• Design, develop, test and maintain complex data pipelines, connectors and integration services.
• Ensure data connectors and integrations required for AI-enabled services are correctly implemented, monitored and maintained.
• Build robust data flows linking operational systems, analytics platforms, business intelligence solutions and downstream applications.
• Design and lead the implementation of scalable batch and streaming data integration solutions.
• Optimise data-processing code, pipeline efficiency, database performance and data reliability.
• Develop reusable components and identify opportunities to consolidate existing data flows.
• Lead appropriate database management, troubleshooting and operational improvement activity.
• Establish suitable metadata management arrangements and maintain technical documentation.
• Monitor data services, diagnose integration failures, resolve problems and implement preventative measures.
• Apply engineering standards, appropriate testing, source control and maintainable development practices.
• Select suitable technologies and architectural approaches for secure, resilient and future-proofed data products.
• Collaborate with architects, engineers, analysts, AI specialists, delivery leads and other stakeholders.
• Communicate technical considerations and delivery dependencies to both technical and non-technical audiences.
• Provide senior technical guidance and contribute to multidisciplinary delivery.

Essential experience

• Substantial hands-on experience in senior data engineering roles delivering complex enterprise or public-sector data solutions.
• Proven design, development and maintenance of production-grade data pipelines and integration services.
• Strong understanding of APIs, data connectors, data ingestion, transformation and system-to-system integration.
• Experience building reliable data foundations supporting analytics, BI or AI-related workloads.
• Strong SQL and practical programming/scripting capability, ideally Python.
• Experience with relational databases, data modelling, performance optimisation and troubleshooting.
• Knowledge of batch processing, streaming architectures, and modern data engineering patterns.
• Experience with cloud-based data services and scalable data platforms.
• Strong understanding of metadata, data quality, security, monitoring and operational resilience.
• Ability to work with multidisciplinary teams and communicate effectively across technical and business functions.
• Current, active UK Security Check (SC) clearance, confirmed at application.

Desirable experience

• Delivering data integrations or connectors supporting AI, machine learning or generative AI services.
• Azure, AWS or other enterprise cloud data platforms.
• Databricks, Spark, orchestration tools or equivalent technologies.
• Event-driven architectures, streaming services and reusable API integration patterns.
• CI/CD, infrastructure as code and automated data engineering testing.
• Data governance, lineage, catalogue and metadata platforms.
• UK central government digital delivery and relevant security requirements.
• Existing Cabinet Office-sponsored SC clearance.

Successful delivery

The consultant will help ensure that the data connectors, integration services and underlying data infrastructure needed by digital and AI capabilities are correctly designed, implemented and operated.

Success means more dependable connections between systems, improved pipeline performance, well-managed integration dependencies, reusable data assets and reduced operational risk.