Hadoop Job Vacancies in Mumbai

Hadoop Lead RoleExperience 9+ years- Strong architectural experience with Hortonworks, Cloudera Hadoop distributions on Appliance based and on-premise clusters.- Expertise in providing technical solutions for data lakes design and data ingestion in Hadoop.- Expertise in data modelling, data governance and architecture of large size databases.- Expertise in understanding complex data models, large scale data migrations and application development.- Expertise in designing data pipelines solutions for structured, semi-structured and unstructured data in Hadoop.- Expertise in Developing solutions for batch and real time processing data processing in Hadoop.- Hands on experience in query writing using HiveQL, Impala QL and HBase commands.- Able to provide technical guidelines assistance to development team if they face any problems related to environment- Able to coordinate with cloudera team for trouble shooting, OS / network related problems with respective teams.- Proficiency in Cloudera Manager architecture, cloudera cluster environment and cloudera manager.- Proficiency in data ingestion tools like Sqoop, Flume, Kafka, Nifi(HDF), UNIX shells scripting and python- Proficiency in building data warehouse on top of Hadoop in Hive. Defining data models and data mapping from the source to enterprise model.- Proficiency in creating solutions in Spark (Scala/Pyspark) for batch and real time processing.- Proficiency in developing solutions for NoSQL databases like HBase, Cassandra and MongoDB.- Knowledge of Data Security / Governance and Data Lineage handling on Hadoop clusters.- Strong experience on SQL Server and Oracle, MySQL.- Expertise in understanding Java , REST API concepts and troubleshooting java based services.

Technologies / Skills: Advanced SQL, Python and associated libraries like Pandas, Numpy etc., Pyspark , Shell scripting, Data- Modelling, Big data, Hadoop, Hive, ETL pipelines and IaC tools like Terraform etc.Responsibilities: Efficient communication skills to coordinate with users, technical teams and DataSolution architects. Document technical design documents for given requirements or JIRA stories. Communicate results and business impacts of insight initiatives to key stakeholders to collaboratively solve business problems. Working closely with the overall Enterprise Data & Analytics Architect and Engineering practice leads to ensure adherence with the best practices and design principles. Assures quality, security and compliance requirements are met for supported area. Develop fault-tolerance data pipelines running on cluster Ability to come up with scalable and modular solutionsRequired Qualification: 1-8 yrs of hands-on experience developing data pipelines for Data Ingestion or transformation using Python (PySpark) /Spark SQL in AWS cloud Experience in development of data pipelines and processing of data at scale using technologies like EMR, Lambda, Glue, Athena, Redshift, Step Functions. Advanced experience in writing and optimizing efficient SQL queries with Python and Hive handling Large Data Sets in Big-Data Environments Experience in debugging, tunning and optimizing PySpark data pipelines Should have implemented concepts and have good knowledge of Pyspark data frames, joins, partitioning, parallelism etc. Understanding of Spark UI, Event Timelines, DAG, Spark config parameters, in order to tune the long running data pipelines. Experience working in Agile implementations Experience with Git and CI/CD pipelines to deploy cloud applications Good knowledge of designing Hive tables with partitioning for performanceThanks and RegardsHR TEAM

Profile - Data Engineer ExecutiveQualification - Graduate With Good Communication SkillExperience - Minimum 1 Year RequiredCandidate Should Have Knowledge of AWS,Spark, Py- Spark, Python, HarkSalary - 24 LPA TO 42 LPA Gender - Male & Female Can ApplyLocation - Pen IndiaDuties & Responsibilities-----Analyze and organize raw data.Build data systems and pipelines.Evaluate business needs and objectives.Interpret trends and patterns.Conduct complex data analysis and report on results.Prepare data for prescriptive and predictive modeling.Build algorithms and prototypes.Only Serious Candidate Apply

Brief about the Company:AdZapier Corporation is a global technology and enablement services company with a vision to transform data into value for everyone. Through a simple open approach, in connecting systems and data, we provide the data foundation for the worlds best marketers. By making it safe and easy to activate, validate, enhance, and unify data. We provide marketers with the ability to deliver relevant messages at scale and tie those messages back to actual results. Our products and services enable individual-based marketing, allowing our clients to generate a higher ROI and drive better omni-channel customer experiences.Position Description:Join our Information Technology team where you will work on new technologies and find ways to meet our customers needs and make it easy for them to do business with us.You will use functional expertise to act as an advisor to management and make recommendations on more complex projects. You will use professional concepts and company policies & procedures to solve a wide range of difficult problems creatively and practically.ResponsibilitiesYou will be responsible for operations and administration of Cloudera Hadoop platform.You will work independently on day to day monitoring and operations of Data Analytics platform. You will be required to develop automation using scripting languages. After initial training, you will be able to handle critical operation tasks as well as on demand requests.Minimum Requirements: 5+ years of experience in Software Development including Big Data Analytics area Experience in Hadoop Big Data Platform Operations and Administration High Proficiency working with Hadoop platform including Hadoop, Hive, Spark/Scala, Java, Kafka, Flume etc. Experience with any scripting language such as BASH, Scala or Python Good understanding of file formats including JSON, Parquet, Avro, and otherWork Hours - 2.30 pm noon to 11.30 pm (Mon-Fri) (US Shift)

At least 3 years of experience developing ETL processes.- Strong in Informatica design concepts using its products.- Hands-on knowledge of Mapplets, Mappings, Workflows, and Applications.- Proficient in Creating Mappings, workflows and implementing ETL concepts.- Solid data warehousing concepts - dimensional modelling, facts, dimensions, helper tables, SCD concepts etc.- Strong ETL and data modelling experience.- Experience in development of database processes using Oracle SQL, Hive, MapReduce.- Sound Unix shell scripting and command level experience.- Knowledge of Hadoop, Map Reduce, Hive, Spark is an advantage.- Excellent knowledge of debugging, tuning and optimising the performance of database queries.- Thorough knowledge of software methodologies, distributed networking, databases, communications, and multiprocessing applications.- Experience in Netezza- Actively participate in business requirements sessions, design review and test case review meetings.Basic understanding of any programming language. As developer, should worked on change requests or enhancements by making some code changes. .:Good to have Understanding of SAS programming language.

ResponsibilitiesCreate and maintain optimal data pipeline architectureAssemble large, complex data sets that meet functional / non-functional business requirements.Build the infrastructure required for optimal extraction, transformation, and loading of data from a wide variety of data sources using Open Source and AWS big data technologiesBuild analytics tools that utilize the data pipeline to provide actionable insights into customer acquisition, operational efficiency and other key business performance metrics.Work with stakeholders including the Executive, Product, Data and Design teams to assist with data-related technical issues and support their data infrastructure needs.Work with data and analytics experts to strive for greater functionality in our data systems.QualificationsExperience building and optimizing big data pipelines, architectures and datasets.Experience performing root cause analysis on internal and external data and processes to answer specific business questions and identify opportunities for improvement.Experience interacting with customers and various stakeholders.Strong analytical skills related to working with unstructured datasets.Build processes supporting data transformation, data structures, metadata, dependency and workload management.Working knowledge of message queuing, stream processing, and highly scalable big data lakes.Strong project management and organizational skills.Experience supporting and working with cross-functional teams in a dynamic environment.They should also have experience using the following software/tools:Big data technologies: Hadoop, Spark, Kafka, etc.Relational SQL and NoSQL databases, including Postgres and Cassandra.Data pipeline and workflow management tools: Airflow, NiFi etc.Cloud services: AWS - EMR, RDS, Redshift, Glue. Azure - Databricks, Data Factory. GCP - Dataproc, Pub/SubStream-processing systems: Storm, Spark Streaming, Flink etc.