Robert Half International DevOps Engineer in Newton, Massachusetts
Principal Data Engineer
The Principal Data Engineer / Lead will play a key role in in the Design and Development of Data models, datasets, and ETL/ELT pipelines from a variety of data sources to the Data Warehouse / Data Lake on Google Cloud Platform (GCP) for TechTarget’s flagship Priority Engine product. Priority Engine relies heavily on this foundational data layer and is rapidly moving in the direction of more real-time streaming data flows.
The Data Engineering Lead will provide technical leadership and strategic direction for Priority Engine’s data platform. The position will be responsible for the delivery and technical quality of our programs, taking on complex projects, and lead the work of individual contributors. It will also partner with cross-functional stakeholders to drive initiatives forward, and mentor team members in technical design, project leadership, and running team processes. The Data Engineering Lead will be responsible to build and maintain scalable data infrastructure.
The candidate must have a keen sense for the business drivers, have a vision for the data strategy and help execute it. The candidate must be familiar with industry trends and best practices around data engineering on the cloud/on-premise and apply them appropriately.
• Design, develop, and implement end-to-end data solutions (storage, integration, processing, access) in Google Cloud Platform (GCP)
• Create ETL/ELT pipelines that transform and process terabytes of structured and unstructured data in real-time
• Design data models for optimal storage and retrieval to support sub-second latency
• Deploy and monitor large database clusters that are performant and highly available
• Deeply immerse in our Priority Engine Data Pipelines, to acquire full understanding and ability to converse with business representatives and individual contributors alike
• Work cross-functionally with data scientists, backend engineers, and product managers to design and implement new data models to support the product
• Incorporate solid understanding of cloud-based offerings in the space
• Some upcoming technical challenges include scaling our data ingestion pipelines across a growing number of GCP, AWS, and data center-based data sources, reducing the latency of our product data ingestion pipelines through moving batch jobs into a streaming architecture, and extending our data lake architecture for the growing ecosystem of data ingestion and creation tools
• Be a hands-on contributor on the team
• Collaborate with product/application architects to develop holistic solutions
• Implement operational procedures (logging, monitoring, alerting, etc.) for dependable running of pipelines/jobs
• Autonomous. You own what you work on. You move fast, take ownership, and get things done. The Data lake/warehouse is a central and essential service for multiple initiatives, and you need to be able to multitask and juggle priorities on a regular basis with a deep understanding of business impacts of prioritization decisions
• BS Degree in Computer Science or a related field
- Linux, Google Cloud Platform
• 10+ years industry experience, 5+ years of experience building real-time and distributed system architecture, from whiteboard to production
• Strong programming skills in Python or Java and SQL
• Versatility: 8+ years’ experience across the entire spectrum of data engineering, including:
o Cloud data stores (e.g., GCP BigQuery, GCP BigTable, GCP FireStore, GCP CloudSQL, ELK)
o Data pipeline and workflow orchestration tools (e.g., Dataflow, Pentaho, Airflow, Azkaban)
o Data processing technologies (e.g., GCP BigQuery, Spark)
o Data messaging technologies (e.g., GCP PubSub, Kafka)
o Deployment and monitoring large database clusters in public cloud platforms (e.g., Docker, Terraform)
• Unix/Linux Shell scripting
• Demonstrated knowledge of industry trends and standards
Technology Doesn't Change the World, People Do.®
Robert Half is the world’s first and largest specialized talent solutions firm that connects highly qualified job seekers to opportunities at great companies. We offer contract, temporary and permanent placement solutions for finance and accounting, technology, marketing and creative, legal, and administrative and customer support roles.
Robert Half puts you in the best position to succeed by advocating on your behalf and promoting you to employers. We provide access to top jobs, competitive compensation and benefits, and free online training. Stay on top of every opportunity – even on the go. Download the Robert Half app and get 1-tap apply, instant notifications for AI-matched jobs, and more.
Questions? Call your local office at 1.888.490.4429. All applicants applying for U.S. job openings must be legally authorized to work in the United States. Benefits are available to contract/temporary professionals. Visit https://roberthalf.gobenefits.net/ for more information.