116

Python Data Engineer Jobs

filter
  • Location
  • Role
  • Functional Area
  • Qualification
  • Experience
  • Employer Type

AI/ML Engineer

United Technology

  • 5 - 7 yrs
  • 12.0 Lac/Yr
  • Coimbatore
Python Numpy Pandas TensorFlow Docker Git CICD AzureAWS FAST API Aiml Engineer Data Engineer
We are looking fo AI/ML Engineer with 5+ year experience in Coimbatore.Skills Set:Strong Python (Pandas, NumPy, Scikit-learn)-Deep Learning using TensorFlow or PyTorch-Expertise in time-series modeling and feature engineering-Experience with ETL/ELT pipelines, Docker, Git, CI/CD-Cloud exposure (Azure/AWS) and hybrid deployments-API development (FastAPI preferred)
View all details
  • 0 - 1 yrs
  • 8.0 Lac/Yr
  • Female
  • Mall Road Amritsar
Data Integration Data Warehousing SQL Informatica ETL Hadoop Big Data Python
We are looking for a motivated Data Engineer to join our team. This part-time position allows you to work from home and is suitable for individuals with little to no experience. The ideal candidate will help us manage and process data to ensure it meets the needs of the business.**Key Responsibilities:**- **Data Collection:** Gather data from various sources to prepare for analysis. Its important to ensure the data is accurate and up-to-date.- **Data Cleaning:** Clean and organize raw data to make it usable. This involves removing errors and inconsistencies, which is crucial for reliable analysis.- **Data Storage:** Help in storing data in databases or cloud storage systems. Proper organization helps in easy access and retrieval of data when needed.- **Collaboration:** Work with other team members to understand their data needs. Communication is key to delivering the right data for their projects.- **Support:** Assist in monitoring data systems and providing technical support. Being proactive in identifying issues helps keep the data flow smooth.**Required Skills and Expectations:**Candidates should have a basic understanding of data management principles. Familiarity with data cleaning tools and database management systems is a plus. The ability to learn new software quickly and a strong attention to detail are essential. Good communication skills are important for working with teammates and understanding project requirements. We encourage fresh graduates and those with relevant qualifications to apply.
View all details

AI/ML Engineer

Kasa Talent Pvt Ltd

  • Fresher
  • 4.0 Lac/Yr
  • Pune
Data Analysis C++ Python LLM AWS Google Cloud Azure AI SQL Data Cleaning
We are seeking a talented AI/ML Engineer to design, develop, and deploy machine learning models that solve real-world business problems.Key ResponsibilitiesDevelop, train, and optimize machine learning and deep learning models.Design and implement AI solutions for automation, prediction, and data analysis.Work with large datasets to clean, preprocess, and engineer features.Deploy models into production environments and monitor performance.Build scalable ML pipelines and integrate models with applications.Conduct experiments, model evaluations, and performance tuning.Collaborate with cross-functional teams including data engineers and product managers.Stay updated with the latest research and advancements in AI/ML.Note: Only Pune-based candidates can eligible to apply.
View all details
  • 3 - 8 yrs
  • Bangalore
Web Scraping Python
Data Extraction Engineer designs extraction systems (and not just scripts). They build and maintain a next-generation data acquisition platform that treats web scraping as a declarative, specification-driven discipline. Instead of hard-coding XPaths for every site, Web Scraping Developer defines what data is neededusing schemas, natural language descriptions, or visual blueprintsand lets intelligent pipelines figure out how to get it.Key Responsibilities:Specification-Driven Extraction Engineering-Design and maintain declarative extraction specificationsusing Pydantic models, JSON schemas, or domain-specific languagesthat describe exactly which fields to capture, their types, and validation rules.Implement pipelines that translate these specifications into executable extraction plans, leveraging both classical (Scrapy, Playwright) and AI-augmented (LLM-based semantic parsing) backends.Build reusable specification libraries for recurring data types (product prices, tariff codes, regulatory texts) to accelerate onboarding of new sources.Autonomous & Self-Healing Systems-Deploy self-healing spiders that automatically detect website layout changes and repair themselves using Model Context Protocol (MCP) servers (e.g., Scrapy MCP Server, Playwright MCP).Integrate semantic extraction (Scrapy-LLM, custom LLM pipelines) to eliminate selector brittlenessspiders rely on field descriptions, not fragile XPaths.Orchestrate complex, multi-step browsing workflows with agentic frameworks (BMAD/TEA, AutoGPT-like agents) that reason about page state, adapt to anti-bot measures, and correct their own behaviour in real time.Platform Thinking & Reusability-Move beyond one-off scrapers: build a component-based extraction platform where selectors, login handlers, and pagination logic are shared, versioned, and tested.Implement monitoring, alerting, and automatic rollback for failed extraction runs.Champion ethical crawling by designrate limiting, robots.txt respect, and compliance with GDPR/CCPA are built into the specification layer, not retrofitted.Collaboration & Continuous Innovation-Partner with data scientists and domain experts to refine extraction specifications for complex, unstructured domains (e.g., legal texts, tariff classifications).Evaluate and pilot emerging tools to push automation coverage beyond 90%.Document and evangelise specification-driven best practices across the engineering organisation.Candidate Profile:Education and Experience -Bachelors degree in Computer Science3+ years of experience in web scraping or data extractionSkills and competences-Specification-Driven Extraction Experience defining extraction requirements via schemas (Pydantic, JSON Schema) and executing them through both traditional crawlers and LLM-based semantic parsers.SelfHealing & Semantic Extraction Handson use of ScrapyLLM, Scrapy MCP Server, or similar systems that decouple field definitions from page structure.Agentic Workflows Familiarity with frameworks that give LLMs browser control (Playwright + MCP, BMAD/TEA) to handle complex, nondeterministic crawling tasks.Classical Scraping Fundamentals You still know how to write a Scrapy spider or a Playwright script when needed, but you actively seek to replace that work with reusable, specification-driven components.Data Validation & Storage Ability to define validation rules within specifications and land clean data into SQL/NoSQL databases or data lakes.Python proficiency: the focus is on an extraction engineer who happens to use Python.HTTP, DOM, XPath, CSS.Basic API integration and authentication flows.Preferred / Nice-to-Have Skills:Contributions to open-source scraping or AI-automation projects.Experience training or fine-tuning small LLMs for domain-specific extraction.Familiarity with data privacy engineering (GDPR, CCPA) baked into specification design.DevOps light Docker, CI/CD for testing extraction specifications.Mindset & Approach (Non-Negotiable):Strong belief that the future of scraping is declarative, not imperative. Youd rather write a schema that says extract the price than debug an XPath when a website redesigns.Looking to shift from code that scrapes to systems that understand extraction.
View all details

Get Personalized Job Matches

Based on your experience, skills, interests, and career goals to help you find the most relevant opportunities faster. Register Now!
  • 4 yrs
  • 15.0 Lac/Yr
  • Tharamani Chennai
Machine Learning Large Language Models LLM Evaluation LLM Optimization Clustering Customer Segmentation SQL Python Data Pipeline Development ML Algorithms Airflow
Role OverviewWe are seeking a highly skilled Machine Learning Engineer with 4+ years of experience to bridge the gap between Generative AI and actionable business intelligence. The ideal candidate will be responsible for optimizing Large Language Model (LLM) performance and developing advanced clustering models for customer and product segmentation.Location : Shollinganallur , ChennaiKey ResponsibilitiesLLM Optimization & EvaluationOptimize accuracy and reduce latency of LLM-powered applicationsDesign and implement robust LLM evaluation frameworksWork with LLM APIs and integrate them into production workflowsAdvanced Analytics & ModelingDevelop clustering models for customer and product segmentationIncorporate complex business logic into ML modelsApply statistical and machine learning techniques to derive actionable insightsData Engineering & OrchestrationBuild and maintain scalable data pipelines using SQLDevelop and manage Python-based DAGs (e.g., Airflow)Ensure efficient data processing and workflow automationTechnical RequirementsMandatory SkillsStrong experience in LLM EvaluationHands-on expertise in Clustering TechniquesAdvanced proficiency in SQLExperience building Python-based DAGs (Airflow or similar tools)Experience working with LLM APIsPreferred (Bonus) SkillsExperience in self-hosting and fine-tuning LLMsKnowledge of Time Series ForecastingExperience in Dimension Reduction techniquesFamiliarity with Market Basket AnalysisHands-on experience with Spark and Alteryx
View all details
  • 4 - 10 yrs
  • 5.0 Lac/Yr
  • Bangalore
ETL ELT SQL Python Dbt Spark Hadoop Cloud Data CICD Data Security Data Warehousing
Design, build, and maintain ETL/ELT data pipelines and data lake solutions to support analytics and AI/ML use cases. Ensure data quality, performance, and reliability across enterprise data platforms.Key ResponsibilitiesPipeline DevelopmentData Lake EngineeringPerformance & OptimizationCollaboration & SupportRequired Skills & Experience 4+ years of experience in data engineering or ETL development. Proficiency in SQL and Python (or Scala/Java) for data transformations. Hands-on with ETL tools (Informatica, Talend, dbt, SSIS, Glue, or similar). Exposure to big data technologies (Hadoop, Spark, Hive, Delta Lake). Familiarity with cloud data platforms (AWS Glue/Redshift, Azure Data Factory/Synapse, GCP Dataflow/BigQuery). Understanding of workflow orchestration (Airflow, Oozie, Prefect, or Temporal).Preferred Knowledge Experience with real-time data pipelines using Kafka, Kinesis, or Pub/Sub. Basic understanding of data warehousing and dimensional modeling. Exposure to containerization and CI/CD pipelines for data engineering. Knowledge of data security practices (masking, encryption, RBAC).Education & Certifications Bachelors degree in Computer Science, IT, or related field.Preferred certifications:o AWS Data Analytics Specialty / Azure Data Engineer Associate / GCP Data Engineer.o dbt or Informatica/Talend certifications.
View all details
  • 6 - 12 yrs
  • 16.0 Lac/Yr
  • Bangalore
Python GCP Developer
Job Title:Data Engineer Location: Bangalore Experience: 6+ years Notice Period : Immediate to 21daysMUST-HAVE TECHNICAL SKILLSSkillSkill DepthPython for Data PipelinesIndependently written ingestion/transformation scripts, including pagination, exception handling, logging, and dataframe-level operations using Pandas, JSON, or GCP SDKsDBT (Data Build Tool)Authored and executed DBT models and tests using YAML files and Jinja macros; contributed to CI test configs and schedule integrationGCP (BigQuery, GCS, CloudSQL)Hands-on experience in at least two of the above tools in pipeline execution e.g., used BigQuery for SQL transformation and GCS for raw/processed layer segregationAWS LambdaIntegrated serverless functions to automate trigger points like new file upload, API call chaining or job completion; used boto3 or GCP Pub/Sub hooksData Quality & ValidationDeveloped or plugged-in validation layers for ingestion such as record count matching, null/duplicate flagging, recon table populationCloud-Native ModelingAdapted pre-existing logical models to ingestion logic, ensuring correct joins, partitioning strategy, and target-layer conformity (Star/Snowflake)Version Control & AgileParticipated in Git branching workflows and sprint-based delivery (JIRA or similar); able to push/pull/test with basic conflict resolution
View all details

Data Scientist

The Best Services & Enterprise's

  • 6 - 10 yrs
  • 13.0 Lac/Yr
  • Mumbai
Data Scientist Python Media Mix Models MTA MMM Data Engineering Python Data Engineer Data Management
Job Title: Data ScientistExperience: 7+ YearsLocation: USA/Canada (Remote) or Offshore (Remote)Working Hours: USA ESTJob Description:We are looking for experienced Data Scientists with proven expertise in building Media MixModels (MMM) and Multi-Touch Attribution (MTA) models for a long-term engagementwith Univision. The ideal candidates should have a strong background in AdTech, datascience, and analytics, with the ability to derive actionable insights from large datasets inthe media and OTT domain.Key Responsibilities: Develop advanced MMM (Media Mixed Modeling) and MTA (Multi-touch Attribution)models to optimize marketing and advertising strategies Analyze large volumes of structured and unstructured data to uncover trends,correlations, and actionable insights Build and deploy machine learning models and predictive algorithms to solve complexbusiness problems Gather data from diverse sources, clean and transform it for analysis Apply statistical techniques to validate models and ensure accuracy and reliability Automate analytical workflows and repetitive tasks using AI tools and scripting languages Create compelling visualizations, dashboards, and reports to communicate insights acrossteams Collaborate with cross-functional teams including AdSales, Data Engineering, andAnalytics Stay updated on the latest developments in AI, machine learning, and media analyticsRequired Skills & Experience: Specific MMM Experience: Bayesian methods, causal inference, incrementality testing MTA Expertise: Attribution modeling, customer journey analysis, touchpoint optimization Tools: Python (scikit-learn, statsmodels), R (prophet, CausalImpact), SQL,Tableau/PowerBI 6+ years of experience in data science, analytics, or machine learning roles At least 3 years of experience in AdTech or AdSales systems Hands-on experience in developing MMM and MTA models Strong understanding of OTT, digital media, and advertising ecosystems Proficiency in programming languages like Python, R, or SQL for data manipulation andmodeling Experience working with large datasets, data pipelines, and BI/reporting tools Familiarity with statistical methods, experiment design, and model evaluation metrics Excellent problem-solving, communication, and stakeholder management skil
View all details
  • 3 - 5 yrs
  • 1.5 Lac/Yr
  • Indore
Cisco Networking Network Automation Python VXLAN Data Center Networking Apic Developer
We are looking for 3 Cisco Voice Engineer Posts in Indore with deep knowledge in Cisco Networking, Network Automation, Python, APIC (Cisco Application Policy Infrastructure Controller), VXLAN, Data Center Networking and Required Educational Qualification is : B.E, B.Tech, M.Tech
View all details
  • 3 - 6 yrs
  • 10.0 Lac/Yr
  • Baner Pune
FastAPI MongoDB AWS Services GitHub Actions and CI CD Pipelines Lambda S3 EC2 RESTful API Design Microservices Event-driven Architecture Performance Tuning Caching Security Best Practices Docker and Containerized Applications Problem-solving Skills Ability to Lead Team Good Communication Skills Data Science Knowledge
About Netra LabsAt Netra Labs, we redefine enterprise AI with our groundbreaking platform, Ground Truth. Our platform transforms expertise into powerful AI agents, enabling businesses to automate complex tasks efficiently. With a user-friendly interface and seamless integration with any language model, Ground Truth empowers system integrators, innovators, and developers to rapidly build and deploy AI solutions. Our commitment to security, scalability, and ROI ensures our clients can trust us with their AI-driven workflows.Role OverviewWe are looking for a highly skilled Python Engineer to lead our backend team and drive the development of scalable, secure, and high-performance AI-powered applications. The ideal candidate will have expertise in data science, a deep understanding of backend development, and hands-on experience with cloud services and DevOps practices. You will work closely with cross-functional teams, ensuring seamless integration between AI models, data pipelines, and enterprise applications.Key Responsibilities Work with the backend development team, ensuring best practices in coding, architecture, and performance optimization. Design, develop, and maintain scalable backend services using Python and FastAPI. Architect and optimize databases, ensuring efficient storage and retrieval of data using MongoDB. Integrate AI models and data science workflows into enterprise applications. Implement and manage AWS cloud services, including Lambda, S3, EC2, and other AWS components. Automate deployment pipelines using Jenkins and CI/CD best practices. Ensure security and reliability, implementing best practices for authentication, authorization, and data privacy. Monitor and troubleshoot system performance, optimizing infrastructure and codebase. Collaborate with data scientists, front-end engineers, and product team to build AI-driven solutions. Stay up to date with the latest technologies in AI, backend development, and cloud computing.Required Skills & Qualifications 3+ years of experience in backend development with Python. Strong experience in FastAPI or other modern Python web frameworks. Proficiency in MongoDB or other NoSQL databases. Hands-on experience with AWS services (Lambda, S3, EC2, etc.). Experience with GitHub Actions and CI/CD pipelines. Data Science knowledge with experience integrating AI models and data pipelines. Strong understanding of RESTful API design, microservices, and event-driven architecture. Experience in performance tuning, caching, and security best practices. Proficiency in working with Docker and containerized applications. Excellent problem-solving skills and ability to lead a team. Strong communication skills to interact with stakeholders and cross-functional teams.Preferred Qualifications Experience with Machine Learning frameworks such as TensorFlow or PyTorch. Knowledge of GraphQL, WebSockets, or gRPC. Familiarity with Terraform or Kubernetes for infrastructure as code. Experience with big data processing frameworks such as Apache Spark.
View all details
  • 5 - 10 yrs
  • Hyderabad
Python API Spark
Data Engineer HyderabadExperience: 48 years Location: Bangalore Employment Type: Full-timeAbout the RoleWere seeking a skilled Data Engineer to design, build, and maintain scalable data pipelines and integrations across multiple systems. Youll work with Python, GCP (BigQuery), Spark, and API integrations, ensuring data quality and seamless workflows. Experience with Ascend.io is a plus.Key ResponsibilitiesDesign and develop ETL/ELT pipelines and automated workflows.Integrate data from APIs, Oracle EBS, and cloud platforms.Leverage Google Cloud Platform (GCP) and BigQuery for data analytics.Utilize Apache Spark or similar frameworks for large-scale data processing.Ensure data accuracy, consistency, and security across systems.Collaborate with business and data teams to deliver reliable data solutions.Monitor and troubleshoot pipeline performance.RequirementsStrong Python and workflow orchestration experience (Airflow, Prefect, etc.).Hands-on with GCP / BigQuery and big data frameworks (Spark).Experience with API integrations (REST, SOAP, GraphQL).Understanding of ETL optimization, CI/CD, and Agile workflows.Excellent problem-solving and communication skills.Nice to HaveExperience with Ascend.io.Knowledge of SQL/NoSQL, Docker, Kubernetes.Exposure to machine learning pipelines.Why Join UsWork on cutting-edge data engineering and cloud-based integration projects.Collaborative, innovative team environment.Competitive compensation and strong career growth.
View all details
Machine Learning Research Assistant Artificial Intelligence Python Data Science R&D PhD Research Deep Learning Academic Research AI Engineer Research Programmer Neural Networks Model Development Research and Development
Role: Research Programmer(Python)Industry Type: IT Services & ConsultingDepartment: Research & DevelopmentEmployment Type: Full Time, PermanentSalary: NegotiableJob Description:We are seeking a Machine Learning Research Assistant to support our Research and Development (R&D) projects in the field of AI and data-driven PhD research. The role involves developing, testing, and implementing machine learning models, data analysis, and assisting in the preparation of academic research papers and technical documentation.Key Responsibilities: Assist in designing and implementing machine learning algorithms and deep learning models for research applications. Conduct data preprocessing, feature engineering, and model evaluation. Perform literature reviews to identify trends and methods relevant to ongoing research projects. Support PhD researchers with technical coding assistance and research data analysis. Prepare research reports, documentation, and result visualizations. Collaborate with the R&D team to explore new technologies and research directions in AI and ML.Required Skills: Strong programming skills in Python, NumPy, Pandas, TensorFlow, or PyTorch. Knowledge of machine learning algorithms, data analytics, and model optimization. Experience with research writing or academic publications is a plus. Strong analytical, problem-solving, and communication skills.Best Regards,Anand Kumar LHRPanacorp Software Solutions
View all details
  • 5 - 10 yrs
  • 40.0 Lac/Yr
  • Hyderabad
AWS Python AWS Data Engineer Terraform ETL Tool CI CD
About the RoleWe are looking for a highly skilled and experienced Senior Data Engineer to join our team in Hyderabad. The ideal candidate will bring strong technical expertise in building scalable data platforms and pipelines using modern technologies such as Python, Scala, AWS, Redshift, Terraform, Jenkins, and Docker. This role demands a hands-on professional who thrives in a fast-paced, collaborative environment and is eager to solve complex data problems.Key ResponsibilitiesDesign, build, and optimize robust, scalable, and secure data pipelines and platform components.Collaborate with data scientists, analysts, and engineering teams to ensure seamless data flow, integration, and availability across systems.Develop infrastructure as code using Terraform to automate provisioning and environment management.Manage containerized services and workflows using Docker.Set up, manage, and optimize CI/CD pipelines using Jenkins for continuous integration and deployment.Optimize performance, scalability, and reliability of large-scale data systems on AWS.Write clean, modular, and efficient code in Python and Scala to support ETL, data transformation, and processing tasks.Support data architecture planning and participate in technical reviews and design sessions.Must-Have SkillsStrong hands-on experience with Python, Scala, SQL, and Amazon Redshift.Proven expertise in AWS cloud services and ecosystem (EC2, S3, Redshift, Glue, Lambda, etc.).Experience implementing Infrastructure as Code (IaC) with Terraform.Proficient in managing and deploying Docker containers in development and production environments.Hands-on experience with CI/CD pipelines using Jenkins.Strong understanding of data architecture, ETL pipelines, and distributed data processing systems.Excellent problem-solving skills and ability to mentor junior engineers.Nice-to-HaveExperience working in regulated domains like healthcare or finance.Exposure to Apache Airflow, Spark, or Databricks.Familiarity with data quality frameworks and observability tools.
View all details

GEN AI - AIML

Welkin Soft Tech Pvt. Ltd.

  • 8 - 12 yrs
  • 18.0 Lac/Yr
  • Bangalore
GEN AI AI ML Python LLM Optimization AI Engineer Integration Data Engineer Analysis SQL Cloud Computing Data Base Natural Language Processing
Job Opening: Generative AI & LLMs / Distinguished Gen AI Engineer Location: Remote Experience: 7+ yearsShare your profilestosandhya@welkinsofttech.com To apply, send your profile to sandhya@welkinsofttech.com / hr@welkinsofttech.com or connect with us here. Key Responsibilities:LLM Development: Design, fine-tune, and implement large language models (e.g., GPT, BERT, T5) for applications like personalized learning, content generation, and semantic search.Generative AI Solutions: Drive innovation with Gen AIdeveloping tools like adaptive learning paths, resume builders, and AI-written job descriptions.Machine Learning: Create predictive models and recommendation engines that align user profiles to skills and job opportunities.Token Optimization: Work with OpenAI and other services to manage token efficiency and usage costs.AI Integration: Collaborate with product and engineering teams to integrate AI features seamlessly into the Elefy platform.Data Engineering: Build and maintain robust data pipelines using Python, Node.js, and MongoDB.Data Analysis: Analyze large datasets to surface actionable insights for user engagement and platform growth.Visualization & Reporting: Build dashboards using Tableau, Power BI, or Matplotlib to communicate insights to stakeholders.Documentation: Ensure clear and comprehensive documentation for models, pipelines, and workflows. Who Were Looking For:Experience:8+ years in data science or AI, with 3+ years hands-on with LLMs or Gen AI in production settings.Proven track record of delivering ML models in scalable, real-world applications.Skills:Languages: Python (must), R, SQLFrameworks: PyTorch, TensorFlow, Hugging Face, Scikit-learnPrompt Engineering: Few-Shot Learning, Dynamic Prompting, Role Play, Chain-of-Thought (nice to have)Cloud: Azure (preferred), AWS, or GCPDatabase: MongoDB or similar NoSQL/SQL systemsKnowledge & Tools:Deep NLP & LLM expertise (e.g., GPT, BERT, T5, etc.)Containerization, APIs, CI/CD, and Azure-native cloud toolsStrong visual storytelling via Tableau, Power BI, or Python-based plotsAgile & cross-functional collaboration mindset Bonus Points:Experience with ethical AI, bias mitigation, and explain abilityFamiliarity with skill-based learning platforms or EdTech ecosystemsJoin us at Elefy and be part of a team thats reshaping the future of learning with AI.
View all details

Data Engineer

Guiding Consulting

  • 10 - 12 yrs
  • Bangalore
SQL Python Spark Data Integration ETL AWS ETL Tool Data Warehousing Azure Server
Job Description:Yrs of Exp : 10 + yrsMode : 3 days a weekLocation: BangaloreWork Type : PermanentKey ResponsibilitiesDesign and Development:Architect, implement, and optimize scalable data solutions.Develop and maintain data pipelines, ETL/ELT processes, and workflows to ensure the seamless integration and transformation of data.Collaboration:Work closely with data scientists, analysts, and business stakeholders to understand requirements and deliver actionable insights.Partner with cloud architects and DevOps teams to ensure robust, secure, and cost-effective data platform deployments.Data Management:Manage and maintain data lakes, data warehouses, and real-time analytics systems.Ensure high data quality, integrity, and security across the organization.Performance Optimization:Monitor and enhance system performance, troubleshoot issues, and implement optimizations as needed.Leverage Microsoft Fabrics advanced analytics and AI capabilities for innovative data solutions.Best Practices & Leadership:Lead and mentor junior engineers to foster a culture of technical excellence.Stay updated with industry trends and best practices, especially in the Microsoft ecosystem.Required:Bachelors or Masters degree in Computer Science, Data Engineering, or a related field.10+ years of experience in data engineering, with a proven track record of working on large-scale data platforms.Expertise in Microsoft Fabric and its components (e.g., Synapse, Data Factory, Azure Data Lake, Power BI).Strong proficiency in SQL, Python, and Spark.Experience with cloud platforms, particularly Microsoft Azure.Solid understanding of data modeling, data warehousing, and ETL/ELT best practices.Excellent problem-solving, communication, team management and project management skills.Preferred:Familiarity with other cloud platforms (e.g., AWS, GCP).Experience with machine learning pipelines or integrating AI into data workflows.Certifications in Microsoft Azure or related technologies.
View all details
  • 4 - 6 yrs
  • 18.0 Lac/Yr
  • Pune
SQL ETL Azure Pyspark Databricks Python
Responsibilities: Design, develop, and deploy data solutions on Azure, leveraging SQL Azure,Azure Data Factory, and Databricks. Build and maintain scalable data pipelines to ingest, transform, and load datafrom various sources into Azure data repositories. Implement data security and compliance measures to safeguard sensitiveinformation. Collaborate with data scientists and analysts to support their data requirementsand enable advanced analytics and machine learning initiatives. Optimize and tune data workflows for performance and efficiency. Troubleshoot data-related issues and provide timely resolution. Stay updated with the latest Azure data services and technologies andrecommend best practices for data engineering.Qualifications: Bachelors degree in computer science, Information Technology, or related field. Proven experience as a data engineer, preferably in a cloud environment. Strong proficiency in SQL Azure for database design, querying, and optimization. Hands-on experience with Azure Data Factory for ETL/ELT workflows. Familiarity with Azure Databricks for big data processing and analytics. Experience with other Azure data services such as Azure Synapse Analytics,Azure Cosmos DB, and Azure Data Lake Storage is a plus. Solid understanding of data warehousing concepts, data modeling, anddimensional modeling. Excellent problem-solving and communication skills.
View all details

Looking For ML Engineer

The Supreme Consultancy

Machine Learning Data Analysis Python ML Engineer Data Science Data Analyst Problem Sloving Deep Learning Deep Learning Engineer
Mandatory Criteria (Can't be neglected during screening) : Looking for Only BTech and BE candidates. Candidate should have Hands-on development experience as Data Analyst and/or ML Engineer. Candidate must have Coding experience in Python. Need candidates with atleast 1-2years of ML experience. Candidate should have Good Experience with ML models and ML algorithms. Need Experience with statistical modelling of large data sets. Looking for Immediate joiners or max. 30 days of Notice Period candidates. The candidates based out of these locations - Bangalore, Pune, Hyderabad, Mumbai, will be preffered. Kindly note Salary bracket will vary according to the exp. of the candidate - - Experience from 4 yrs to 5 yrs - Salary range - 15 LPA - 21 LPA max.- Experience from 6 yrs to 7 yrs - Salary range - 21 LPA - 25 LPA- Experience of 8 yrs to 9 yrs - Salary range - 30 LPA - 32 LPA- Experience 10 yrs to 12 yrs - Salary upto 40 LPA max.What You will do: Play the role of Data Analyst / ML Engineer Collection, cleanup, exploration and visualization of data Perform statistical analysis on data and build ML models Implement ML models using some of the popular ML algorithms Use Excel to perform analytics on large amounts of data Understand, model and build to bring actionable business intelligence out of data that is available in different formats Work with data engineers to design, build, test and monitor data pipelines for ongoing business operationsBasic Qualifications: Only BTech and BE candidates. Experience: 4+ years. Hands-on development experience playing the role of Data Analyst and/or ML Engineer. Experience in working with excel for data analytics Experience with statistical modelling of large data sets Experience with ML models and ML algorithms Coding experience in PythonNice to have Qualifications: Experience with wide variety of tools used in ML Experience with Deep learningBenefits: Competitive salary. Hybrid work model. Learning and gaining experience rapidly. Reimbursement for basic working set up at home. Insurance (including a top up insurance for COVID).
View all details
  • 4 - 10 yrs
  • 36000/Yr
  • Missouri +1 USA
Data Warehousing Data Management Data Integration SQL Data Extraction ETL Tool Hadoop AWS Big Data Python
Role OverviewThis position requires a detail-oriented data engineer who can independently architect and implement data pipelines, while also serving as a trusted technical partner in client engagements and stakeholder meetings. Youll work hands-on with PySpark, Airflow, Python, and SQL, driving end-to-end data migration and platform modernization efforts across Azure and AWS.In addition to technical execution, youll contribute to sprint planning, backlog prioritization, and continuous integration/deployment of data infrastructure. This is a senior-level individual contributor role with direct visibility across engineering, product, and client delivery functions.Key ResponsibilitiesLead design and development of enterprise-grade data pipelines and cloud data migration architectures.Build scalable, maintainable ETL/ELT pipelines using Apache Airflow, PySpark, and modern data services.Write efficient, modular, and well-tested Python code, grounded in clean architecture and performance principles.Develop and optimize complex SQL queries across diverse relational and analytical databases.Contribute to and uphold standards for data modeling, data governance, and pipeline performance.Own the implementation of CI/CD pipelines to enable reliable deployment of data workflows and infrastructure (e.g., GitHub Actions, Azure DevOps, Jenkins).Embed unit testing, integration testing, and monitoring in all stages of the data pipeline lifecycle.Participate actively in Agile ceremonies: sprint planning, daily stand-ups, retrospectives, and backlog grooming.Collaborate directly with clients, stakeholders, and cross-functional teams to translate business needs into scalable technical solutions.Act as a technical authority within the teamleading architectural decisions and contributing to internal best practices and documentation.
View all details

Hiring For Data Engineer

International Recruiters

DataPipelines Etlprocesses Etldeveloper BusinessIntelligence Business Intelligence Tool Data Integration BI Analyst Business Intelligence Analyst SQL SQL Developer Data Engineer Python For ETL API Development Data Scripting Dataware Housing Azure Administrator Ssis Developer Azure Data Engineering
Location: Guindy, Chennai, Tamil Nadu, IndiaExperience: 36 yearsReports To: BI ManagerJob Type: Full-timeKey Responsibilities Build and maintain scalable data pipelines and ETL processes for BI needs. Optimize data models and ensure clean, reliable, and well-structured data for Power BI. Integrate data from various internal systems (web apps, databases, accounting platforms). Collaborate with BI analysts and managers to meet reporting requirements. Automate data refreshes and performance-tune dashboards. Maintain metadata, data dictionaries, and lineage documentation.Required Skills Strong SQL development and performance tuning skills. Hands-on experience with Power BI dataset structuring and integration. Familiarity with Python for ETL, automation, or data cleaning tasks. Knowledge of APIs and scripting for data ingestion. Understanding of data warehousing and modelling techniques (star/snowflake schemas). Experience working with Azure, SSIS, or any cloud-based data services is a plus. Familiarity with version control systems like Git
View all details
Python SQL ML Docker AWS Cloud Engineer
Level of skills and experience:5 years of hands-on experience in using Python, Spark,Sql.Experienced in AWS Cloud usage and management.Experience with Databricks (Lakehouse, ML, Unity Catalog, MLflow).Experience using various ML models and frameworks such as XGBoost, Lightgbm, Torch.Experience with orchestrators such as Airflow and Kubeflow.Familiarity with containerization and orchestration technologies (e.g., Docker, Kubernetes).Fundamental understanding of Parquet, Delta Lake and other data file formats.Proficiency on an IaC tool such as Terraform, CDK or Cloud Formation.Strong written and verbal English communication skill and proficient in communication with non-technical stakeholderst
View all details
  • 5 - 7 yrs
  • 12.0 Lac/Yr
  • Chennai
Snow Flake Developer Dbt Dagster SQL Python Git Cicd Pipelines Data Modeling Datawarehouse Architecture Claude Copilot Data Extraction
We are looking for Senior Data Engineer (Snowflake / dbt / Dagster / AI-Assisted Development) with 5+ Year Experience in Chennai.Design and optimize data pipelines from SQL Server to Snowflake Work with healthcare data formats including EDI 835 / 837 if applicable Use AI tools (LLMs, code assistants, automation agents) to improve engineeringproductivity and quality
View all details

Data Engineer

United Technology

  • 1 - 3 yrs
  • 4.0 Lac/Yr
  • Chennai
Data Integration Data Engineer Hadoop ETL SQL Informatica Apache AWS Big Data Python
We are looking Data Engineer with 1 to 3 years experience in Chennai.Immediate joiners preferred
View all details

Looking For Data Engineer

BSRI Solutions Pvt Ltd

  • 3 - 5 yrs
  • 16.0 Lac/Yr
  • Chennai
Python Pyspark Developer Scala SQL Hive Hadoop Google Cloud Platform Kafka Developer Infrastructure AS Code GitHub Agile Methodology ETL
Required Qualifications : 3+ years of demonstrated ability with Hive, Python, Spark/Scala, SQL, etc. Google Cloud Platform Experience, Big Query, Cloud Storage, Dataproc, Data Flow, Cloud Composer, Cloud SQL, Pub Sub, Terraform, etc. Experience with Hadoop Ecosystem, Kafka, PCF cloud services Familiar with big data and machine learning tools and platforms Experience with BI tools, such as Alteryx, Data Stage, QlikSense, etc. Design data pipelines and data robots, take a vision and bring it to life Master data engineer; mentors others; works closely with IT architects to set strategy and design projects Provide extensive technical, and strategic advice and guidance to key stakeholders around the data transformation efforts Redesign data flows to prevent recurring data issues Strong analytical and problem-solving skills Possess excellent oral and written communication skills, as well as facilitationand presentation skills, and engaging presentation style. Ability to work as a global team member, as well as independently, in achanging environment and prioritize. Ability to establish and maintain coordinated and effective working relationships with application implementation teams, IT project teams, business customers, and end users. Ability to deliver work within deadlines. Experience with agile/lean methodologies Experience working independently and with minimal supervision Experience with Test Driven Development and Software Craftsmanship Experience with GitHub, Accurev, or other version-control systems Experience with Putty Experience with Datastage Strong Communications skills Ability to illustrate and convey ideas and prototypes effectively with team and partners Presence demonstrating confidence, ability to learn quickly, influence, and shape ideas Key Skills Required - Data Engineer- Python / PySpark / Scala- SQL & Hive- Hadoop Ecosystem- Data Pipeline Design & ETL Development- Google Cloud Platform (BigQuery, Dataproc, Dataflow, Cloud Storage)- Kafka / Streaming Data Processing- Terraform (Infrastructure as Code)- DataStage or Similar ETL Tools- Version Control (GitHub or equivalent)- Agile Methodologies- Strong Analytical & Problem-Solving Skills- Stakeholder Collaboration & CommunicationNice to Have:- Cloud Composer, Cloud SQL, Pub/Sub- BI Tools (Alteryx, QlikSense)- Machine Learning Platform Exposure- Test Driven Development (TDD)- Mentoring & Technical Leadership
View all details

Looking For Senior Data Engineer

BSRI Solutions Pvt Ltd

  • 5 - 9 yrs
  • 30.0 Lac/Yr
  • Chennai
Databricks Pyspark Java Web Services SQL Python
Job Title: Senior Data Engineer Experience: 5+ YearsLocation: Chennai (Hybrid)Budget: Up to 30 LPAJob SummaryWe are looking for a highly skilled developer with strong hands-on experience in Databricks, PySpark, Python/Java, Web Services, and SQL. The ideal candidate will work closely with architects, tech leads, and business teams to design, build, optimize, and support scalable data-driven solutions. This is a long-term role with a strong focus on performance, cost optimization, and production support.Key Responsibilities / Essential Job FunctionsUnderstand end-to-end system architecture and support operations through monitoring and dashboardsCollaborate with Architects and Tech Leads on solution design and implementationContinuously monitor and optimize system cost and performanceWork closely with Business Analysts on integration requirementsCoordinate with TCOE teams for defining test scenarios and supporting testing activitiesTroubleshoot performance and functional issues across environmentsDocument key technical decisions and maintain detailed design documentationHandle fast-paced project deliveries and support production issues as requiredQuickly learn and adapt to new technologies and frameworksEnsure prompt and effective communication with stakeholdersOther ResponsibilitiesCreate, document, and maintain project artifactsFollow industry standards, methodologies, and best practicesSafeguard company assets and sensitive information (PI)Report any suspected security or compliance issues promptlyAdhere to company compliance and governance policiesMaintain focus on customer service, efficiency, quality, and business growthCollaborate effectively with cross-functional teamsPerform other duties as assignedMinimum Qualifications & Job RequirementsMinimum 5+ years of IT development experienceStrong analytical and problem-solving skillsProven experience in solution design, code reviews, and mentoring junior engineersStrong backend development experience on data-driven projectsExcellent SQL and database skillsStrong team player with good communication skillsMandatory Technical SkillsPython / PySparkSQL / PL-SQLDatabricksJava or C#Preferred / Good-to-Have SkillsAzure CloudKafkaNode.jsAzure Data Factory
View all details
  • 3 - 9 yrs
  • 18.0 Lac/Yr
  • Chennai
Automation Testing DBT Test Pytest SQL Validators Complex SQL Agile Ceremonies Relational Databases ETLELT Workflows Data Pipelines Testing Airflow DAG Executions Selenium Cucumber Playwright JIRA Data Validation ISTQB Python
Role: Quality Assurance Engineer (Data & API)SUMMARYThe Quality Assurance Engineer partners with Engineering, Data, and Developmentteams to ensure client requirements are implemented correctly and reliably acrosshighvolume data pipelines. This role reviews requirements, designs test strategies, andexecutes automated and manual validation-emphasizing shiftleft practices andrepeatable automation. Candidates are former developers or Software DevelopmentEngineer in Test (SDET) who can effectively test large datasets, craft complex SQL fordata comparisons, and collaborate within a single, shared sprint cadence.ESSENTIAL DUTIES AND RESPONSIBILITIES Configure, implement, and maintain automated testing frameworks for data and APIvalidation (DBT tests, Pytest, SQL validators). Translate JIRA user stories and acceptance criteria into comprehensive test plans,scenarios, and data validation scripts. Ensure requirements traceability by mapping test cases and validation scriptsdirectly to client requirements and acceptance criteria, maintaining cleardocumentation throughout the lifecycle. Design and execute unit, integration, smoke, regression, and endtoend tests alignedto the recommended QA & automation framework. Validate large datasets for completeness, accuracy, timeliness, lineage, and schemaconformance, author complex SQL for data comparison. Coordinate with Engineering to enable shiftleft testing - QA participates ingrooming, planning, and daily standups; quality is a shared responsibility. Assist with user acceptance testing (UAT) and production validation, includingpostrelease smoke testing and regression cycles. Analyze test outputs, identify defects, document issues, and drive rootcauseanalysis; champion environment parity (VAL mirrors PROD). Contribute to release governance: freeze windows, QA gates, rollback plans, andrelease logs; ensure test evidence is captured. Continuously improve tools, templates, and processes; propose JIRA automation toautocreate QA stories from engineering stories. Develop domain knowledge of client data products, pipelines, integrations, and KPIs. Maintain comprehensive test documentation, including test cases, scripts, andresults, to support audits and compliance. Conduct performance and security testing as required to ensure robust, scalable,and secure solutions. Advocate for user experience and usability in all testing activities.AGILE & COLLABORATION EXPECTATIONS Active participant in Agile ceremonies (grooming, planning, standups, demos, retros)within a single, shared sprint cadence. Story readiness includes defined test criteria and data availability; QA estimatestracked alongside development. Promote predictable velocity by integrating automated tests into CI; definition ofdone includes -tested and validated.- Stay current with emerging QA tools, technologies, and best practices; proactivelyrecommend improvements to processes and frameworks.REQUIRED QUALIFICATIONS Former software developer or SDET with handson coding experience (Pythonpreferred). Expertise in SQL and relational databases; able to design complex validationqueries for large datasets. Demonstrated experience testing data pipelines, ETL/ELT workflows, and APIs inhighvolume environments. Practical knowledge of Airflow (or similar orchestration), DBT, and CI systems;experience validating DAG executions. Proficiency with automated testing tools (Selenium/Cucumber/Playwright for UIwhere applicable; Pytest for services; DBT for data). Proficiency with version control systems (e.g., Git) and defect tracking tools (e.g.,Jira, Azure DevOps). Experience with JIRA and Agile development methodology; comfortable with sharedsprint delivery. Strong critical thinking; challenges ambiguous requirements and drives clarity inacceptance criteria. Excellent communication skills; able to partner with Product Owners and Engineersto define testable stories.PREFERRED / DOMAIN EXPERIENCE Experience with liveevent data, broadcast schedules, media assets, or fanengagement platforms. Experience with bigdata validation (Snowflake/BigQuery/Redshift) and performancetesting of queries at scale. Familiarity with API contract testing, data lineage tools, and dataset comparisonframeworks. ISTQB or equivalent certification is a plus. Experience with cloud-based data platforms (AWS, Azure, GCP) is a plus.
View all details
View More Jobs