Senior Databricks Engineer for Ciklum Digital

Gdańsk, Poland

Apply

Ciklum is a Software Engineering and Solutions Company. Our 3,000+ IT professionals are located in the offices and delivery centres in Ukraine, Belarus, Poland and Spain.

As Ciklum employee, you'll have the unique possibility to communicate directly with the client when working in Extended Teams. Besides, Ciklum is the place to make your tech ideas tangible. The Vital Signs Monitor for the Children’s Cardiac Center as well as Smart Defibrillator, the winner of the US IoT World Hackathon, are among the cool things Ciklumers have developed.

Ciklum is a technology partner for Google, Intel, Micron, and hundreds of world-known companies. We are looking forward to seeing you as a part of our team!

Read more about the client

Description

On behalf of Ciklum Digital, we are looking for a Senior Databricks Engineer to join the Gdansk team on a full-time basis.

Project description:

  • The project is about Data Processing & Analytics, the team will be using tools like: Azure Data Services (ADFv2, ADLS, Cosmos DB etc..), Azure Databricks, Spark, Python, SQL, Power BI. Team will be working closely with the Client’s Data & AI practice members.
  • It is a great opportunity to learn Azure best practices. Working with UK’s No. 1 Azure development partner, developers will get further experience in using containers inside Azure, combined with Azure native PaaS components. Azure environments will be built according to best practices. Azure DevOps fully automated CI/CD will be built and utilized during the project. Fast-paced, agile environment.

Responsibilities

  • Responsible for the building, deployment, and maintenance of mission-critical analytics solutions that process data quickly at big data scales
  • Contributes to design, code, configurations, and documentation for components that manage data ingestion, real-time streaming, batch processing, data extraction, transformation, and loading across multiple data storages
  • Working in an agile environment
  • Participate in daily stand-ups, articulating the stories worked upon, blocking issues and their status
  • Participate in sprint planning, estimating stories, providing proposed solutions and identifying stories that are not ready for planning
  • Developing solutions and unit tests from requirement and acceptance criteria
  • Working with cloud services
  • Ability to debug issues with a development or PR environment, and understand if the environment is representative of the production infrastructure.
  • Contribute to CoE activities and community building, participate in conferences, provide excellence in exercise and best practices

Requirements

  • 2+ years of contribution to production deployments of large backend data processing and analysis systems as a team lead
  • 3+years (Proficient) with both Python (in a Spark environment) and SQL
  • Experience with scripting languages such as PowerShell or bash
  • Databricks hands-on experience
  • Knowledge of using debugging tools, profilers and linters
  • Understanding of Test Driven Development
  • Good working knowledge of Azure data services e.g.:  Azure SQL Database options, Azure Cosmos DB, Data Lake, Azure Data Lake Storage,  Data Factory, Databricks
  • Good working knowledge of Azure Key Vault, Azure Functions, Logic Apps, Power BI
  • Able to work with Business Analysts to ensure stories are implementable and have workable acceptance criteria
  • Able to discuss problem solutions in a reasoned and dispassionate way
  • Able to implement business requirements as software components with accompanying unit tests and integration tests as required.
  • Good knowledge of Azure DevOps Pipelines is ideal, but knowledge of any other Enterprise product/tool for CI/CD is also considered, e.g.: Terraform, Jenkins, etc
  • Able to manage branching and pull requests and engage in peer code reviews
  • Ability to modify CI/CD pipelines as required, without adversely impacting the work of colleagues
  • Knowledge of  delivering solutions in the Databricks/Spark environment – architecture, execution and debugging

Desirable

Certifications:

  • DP-900 Azure Data Fundamentals
  • Databricks Certified Associate Developer for Spark 2.4/3.0

Personal skills

  • Curious mind and willingness to work with client in consultivitive manner to find areas to improve
  • Intermediate +++ English
  • Good analytical skills
  • Good team player, motivated to develop and solve complex tasks
  • Self-motivated, self-disciplined and result-oriented
  • Strong attention to details and accuracy

What's in it for you

Centre of Excellence is ultimately a community that allows you to improve yourself and have fun. Our centers of excellence (CoE) bring together all Ciklumers from across the organization to share best practices, support, advice, industry knowledge, and to create a strong community.

  • Close cooperation with client
  • A constant flow of new projects
  • Dynamic and challenging tasks
  •  Ability to influence project technologies
  • Projects from scratch
  • Team of professionals: learn from colleagues and gain recognition of your skills
  • European management style
  • Continuous self-improvement

Client video presentation