Website Virtusa
Job Title : Data Engineer
Qualification : B.E/B.Tech
Experience : Freshers
Location : Gurgaon
Company Profile :
In an ever-evolving digital landscape, businesses need a partner to help them solve complex challenges and navigate the environment of today and tomorrow. Virtusa is that partner.We are builders, makers, and doers with the technical skills and domain expertise to transform your business at scale and speed without disruption.Our unique Engineering First approach blends deep industry expertise and empowered, agile teams, to create holistic solutions that seamlessly move the business forward. We help clients engage with new technology paradigms to creatively build solutions that drive them to the forefront of their industries.
Responsibilities/Skills :
- Understand the data architecture design that supports scaling, high availability, end-to-end security for data in motion and data at rest, and performance scalability
- Understand the data modeling process of discovering, analyzing, transforming, performance tuning and enabling
- Experience in working with AWS enterprise-scale cloud platforms
- Experience with data governance that includes Data quality, security and compliance, Data transparency
- Experience working with Event based architecture & real-time services.
Activities: - Develop and implement scalable data pipelines from multiple sources
- Implement data pipelines to transform data using real-time events, and batch methods
- Develop, construct, test and maintain data pipelines which includes connecting object storage, API, JDBC, SFTP
- Implement strategy on data reliability, efficiency, performance, and quality
- Perform peer design reviews, code reviews, pair programming and functional testing to ensure quality releases
- Deploy data pipelines following best practice DevOps principles and incorporating automated testing.
- Automating monitoring of data pipelines and data integrity
- Implement and maintain of data security policies
- Showcasing new pipelines and data models to solutions team and end users
- Strong programming experience in Python / Scala / Java
- Strong understanding of Data Warehouse methodologies and concepts including modelling and partitioning
- Experience working on AWS environment including S3, EC2, Arora / RDS, SQS, IAM etc.
- Experience working processing large datasets using Spark
- Strong SQL background with experience with at least one Relational database including query optimisation, indexing concepts
- Experience working Hive metastore
- Experience with Streaming data concepts using Kinesis or Kafka
- Experience with CI/CD Methodologies and Building Infrastructure as a Code.
- Working experience with multiple file formats such as CSV, Parquet, Avro, JSON, XML
- Experience Data Pipelining with tools like Apache Airflow / Argo DAGs for orchestration workflow
- Knowledge around RClone and Solace
Join us on Telegram For More Updates: https://t.me/nareshit
To apply for this job please visit www.virtusa.com.
