Data Engineer Jobs

As a Senior Data Engineer, you will be at the forefront of our data transformation, specifically architecting and building the next generation of our Financial Transaction Processing engine within Microsoft Fabric. You will lead the migration of legacy financial pipelines into a unified Lakehouse architecture, ensuring that every pence of revenue and every byte of roaming data is accounted for with 100% accuracy.This role requires a blend of high-level architectural design, hands-on pipeline engineering, and a deep understanding of telecom financial domains (billing, reconciliation, and inter-carrier settlements).Key ResponsibilitiesArchitectural Design: Lead the design and implementation of a Medallion Architecture (Bronze, Silver, Gold) within MS Fabric to handle high-velocity telecom transactions.Data Pipeline Engineering: Develop end-to-end ETL/ELT workflows using Fabric Data Factory, Spark Notebooks (PySpark), and Dataflows Gen2.Financial Migration: Lead the migration of legacy SQL-based financial workloads into Fabric Lakehouses and Warehouses, ensuring zero data loss and maintaining historical integrity.Configuration & Governance: Configure Fabric workspaces, capacities, and OneLake security. Implement rigorous data governance and Row-Level Security (RLS) to protect sensitive financial PII.Performance Optimization: Tune large-scale Spark jobs and SQL queries to process millions of Call Detail Records (CDRs) and financial events with minimal latency.Analytics & Reporting: Build high-performance Semantic Models and Direct Lake datasets for real-time financial reporting and margin analysis.Mandatory Skills & QualificationsTechnical ExpertiseMicrosoft Fabric: Expert-level knowledge of the Fabric ecosystem (Lakehouse, Eventhouse, Data Factory, and OneLake).Pipeline Development: Strong experience with Azure Data Factory or Fabric Pipelines, including complex orchestration and incremental loading patterns.Coding & Transformation: Advanced proficiency in PySpark/Python and T-SQL for complex financial logic (e.g., multi-currency conversion, tax calculations).Architecture: Deep understanding of Delta Lake, Star Schema modeling, and SaaS-based data platform design.DevOps: Hands-on experience with Git integration, CI/CD for Fabric items, and Azure DevOps deployment pipelines.Domain KnowledgeTelecom Finance: Familiarity with telecom-specific data such as CDRs, roaming settlements, prepaid/postpaid billing cycles, and Churn analytics.Financial Accuracy: Experience building automated reconciliation frameworks to catch revenue leakage or transaction mismatches.Required Experience8+ years in Data Engineering3+ years hands-on with Microsoft FabricProven experience delivering large-scale data migration programmes in UK telecom or retailStrong SQL & PySpark expertiseExperience with financial data reconciliation and ledger alignmentStrong stakeholder engagement across Finance, Billing, and Technology teamsExperience operating within UK enterprise governance frameworksPreferred ExperienceCertifications: DP-600 (Microsoft Fabric Analytics Engineer) or DP-203 (Azure Data Engineer).Legacy Knowledge: Experience migrating from on-prem SQL Server or older Azure Synapse environments.Real-Time Intelligence: Knowledge of KQL (Kusto) for monitoring real-time transaction streams.Skills to be evaluated onMicrosoft FabricAzure-Data-FactoryFabric-AnalyticsDatabricksMandatory Skills : Microsoft Fabric

We are looking for a Gen AI Developer with 5 to 9 years of experience to join our team in Guindy. The successful candidate will work on developing innovative artificial intelligence solutions.**Key Responsibilities:**- **AI Model Development:** Create and implement machine learning models to solve complex problems, ensuring they meet performance expectations.- **Data Analysis:** Analyze large datasets to extract meaningful insights and improve AI functionalities, enhancing the effectiveness of AI solutions.- **Collaboration with Teams:** Work closely with cross-functional teams, including data scientists and software developers, to integrate AI capabilities into existing systems and products.- **Testing and Optimization:** Conduct thorough testing of AI models and applications, optimizing them for better performance and user experience.- **Continuous Learning:** Stay updated with the latest trends and advancements in AI technology, and apply new knowledge to improve current practices.**Required Skills and Expectations:**- Candidate should hold a Bachelors degree in Computer Applications (B.C.A), Bachelor of Science (B.Sc), or Bachelor of Engineering (B.E).- A strong understanding of machine learning algorithms, natural language processing, and deep learning frameworks is essential.- Proficiency in programming languages such as Python, R, or Java is required to develop and implement AI models.- Excellent problem-solving skills and the ability to work independently as well as part of a team are necessary.- Strong communication skills to effectively collaborate with team members and stakeholders are important for success in this role.Work Location: Chennai Guindy Tamil Nadu 626001Interested For Job Please Call Immediately & Share you updated profileCall/Whatsapp: Mr. Pradeep HR careerturn5techhrsolution@gmail.com

Key Responsibilities:Build and manage data pipelines and ETL processesWork with large datasets using tools like Spark, Hadoop, or SQLEnsure data quality and performance optimizationRequirements:Experience in Python/SQLHands-on with ETL tools and big data technologiesUnderstanding of data warehousing conceptsBrief Summary:Develops scalable data systems to support analytics and business insights.

Minimum 4+ years of hands-on experience in data engineering or data management, preferably in the energy or industrial domain. Design, develop, and maintain scalable ETL/ELT pipelines to process structured and unstructured data from diverse sources including SAP ECC / S/4HANA, SQL Server, Oracle, other RDBMS, Excel/CSV files, Parquet, JSON, XML, and APIs. Collaborate with business and functional teams to define data integration and transformation strategies aligned with enterprise objectives. Optimize data pipelines, storage, and query performance for high-volume datasets across cloud or on-prem environments. Implement robust data quality, validation, and monitoring frameworks to ensure accuracy and reliability. Develop and automate data workflows, versioning, and deployment pipelines to streamline data operations. Support the deployment, monitoring, and governance of data infrastructure and warehouse/lakehouse environments. Work closely with Data Architects to establish best practices for data modeling, warehousing, and lineage tracking. Enable incremental data processing (CDC) and efficient handling of batch and near real-time data pipelines. Collaborate with analytics teams to enable insightful visualization and reporting solutions. Prepare and maintain technical documentation for pipelines, transformations, and data flows. Collaborate with business stakeholders to understand requirements and ensure alignment with business goals.Qualification and Experience: BE/B.Tech / Science Graduate in Computer Science, Information Technology, or related field. Strong foundation in data modeling, performance tuning, and ETL/ELT orchestration. Experience working with enterprise data sources such as SAP ECC / S/4HANA, SQL/Oracle databases, and file-based data (Excel, CSV, Parquet, JSON). Experience with one or more cloud platforms such as Azure, AWS, or GCP. Familiarity with tools such as Microsoft Fabric, Databricks, Power BI, or equivalent tools is an advantage. Experience in data migration or transformation projects is a strong advantage. Understanding of data governance, data quality, and metadata/lineage concepts is a plus.Key Deliverables: ETL/ELT pipelines for all assigned data sources Data ingestion from SAP, databases, files, and APIs Bronze/Silver/Gold data layer implementation Data quality checks and monitoring setup Optimized and scalable data pipelines Develop the Power BI dashboard. Technical documentation and runbooks

We are seeking a skilled Mid-Level Data Engineer to design, build, and optimize scalable data pipelines and architectures. The ideal candidate will have strong experience in data processing, ETL development, and cloud-based data platforms, supporting data-driven decision-making across the organization.

We are looking for a highly analytical Lead Process Engineer to drive operational excellence and process transformation within lending operations. The role involves leveraging advanced statistical methods, process intelligence, and AI-assisted tools to improve efficiency, reduce variation, and enable scalable business operations. This position also includes mentoring junior team members while collaborating with cross-functional stakeholders.Key Responsibilities:Lead end-to-end process improvement initiatives using DMAIC methodologyApply statistical analysis, DoE, and SPC techniques to enhance process performanceDesign and optimize operational workflows across lending value streamsBuild and maintain process intelligence dashboards for real-time insightsConduct FMEA-based risk assessments and implement mitigation strategiesDrive AI-assisted process improvements using GenAI tools and automation frameworksPerform capacity forecasting and time series modeling for scaling operationsCollaborate with Product, Data, and Operations teams for transformation initiativesPresent ROI-driven insights and business cases to senior stakeholdersMentor and develop Process Analysts and Junior EngineersRequired Skills & Qualifications:7-9 years of experience in process engineering, quality engineering, or operations improvementStrong expertise in Lean Six Sigma methodologies (DMAIC execution experience mandatory)Hands-on experience in statistical tools: DoE, SPC, regression, hypothesis testingExperience with process mining / intelligence tools (Power BI, Celonis, etc.)Proven experience in FMEA and risk assessment frameworksStrong capability in capacity planning and time series forecastingExposure to AI/GenAI tools in operational environmentsProficiency in Excel (advanced), Python or Minitab, and data visualization toolsStrong stakeholder management and communication skills

Key ResponsibilitiesTechnical Documentation: Maintaining project blueprints, technical manuals, and compliance records.Progress Monitoring: Tracking the project timeline against the master schedule (using tools like MS Project or Excel) and highlighting potential delays.Resource Coordination: Ensuring that parts, tools, and materials arrive at the site or production line exactly when the team needs them.Quality Assurance: Assisting in site visits or shop floor inspections to ensure that work is being done according to the technical specifications.Communication Bridge: Facilitating information flow between departments-for example, explaining design changes to the production team or clarifying technical issues to the client.Risk Mitigation: Identifying small technical issues early on before they snowball into costly project delays.

As a Data Engineer, your responsibilities will include collecting and analyzing data to help inform business decisions. You will be responsible for data management, ensuring that data is accurate and up-to-date. This will involve using tools such as Informatica, PLSQL, SQL, and Oracle SQL to manipulate and query large datasets.Your skills should include a strong understanding of data analysis techniques, such as data mining and statistical analysis.

Data Extraction Engineer designs extraction systems (and not just scripts). They build and maintain a next-generation data acquisition platform that treats web scraping as a declarative, specification-driven discipline. Instead of hard-coding XPaths for every site, Web Scraping Developer defines what data is neededusing schemas, natural language descriptions, or visual blueprintsand lets intelligent pipelines figure out how to get it.Key Responsibilities:Specification-Driven Extraction Engineering-Design and maintain declarative extraction specificationsusing Pydantic models, JSON schemas, or domain-specific languagesthat describe exactly which fields to capture, their types, and validation rules.Implement pipelines that translate these specifications into executable extraction plans, leveraging both classical (Scrapy, Playwright) and AI-augmented (LLM-based semantic parsing) backends.Build reusable specification libraries for recurring data types (product prices, tariff codes, regulatory texts) to accelerate onboarding of new sources.Autonomous & Self-Healing Systems-Deploy self-healing spiders that automatically detect website layout changes and repair themselves using Model Context Protocol (MCP) servers (e.g., Scrapy MCP Server, Playwright MCP).Integrate semantic extraction (Scrapy-LLM, custom LLM pipelines) to eliminate selector brittlenessspiders rely on field descriptions, not fragile XPaths.Orchestrate complex, multi-step browsing workflows with agentic frameworks (BMAD/TEA, AutoGPT-like agents) that reason about page state, adapt to anti-bot measures, and correct their own behaviour in real time.Platform Thinking & Reusability-Move beyond one-off scrapers: build a component-based extraction platform where selectors, login handlers, and pagination logic are shared, versioned, and tested.Implement monitoring, alerting, and automatic rollback for failed extraction runs.Champion ethical crawling by designrate limiting, robots.txt respect, and compliance with GDPR/CCPA are built into the specification layer, not retrofitted.Collaboration & Continuous Innovation-Partner with data scientists and domain experts to refine extraction specifications for complex, unstructured domains (e.g., legal texts, tariff classifications).Evaluate and pilot emerging tools to push automation coverage beyond 90%.Document and evangelise specification-driven best practices across the engineering organisation.Candidate Profile:Education and Experience -Bachelors degree in Computer Science3+ years of experience in web scraping or data extractionSkills and competences-Specification-Driven Extraction Experience defining extraction requirements via schemas (Pydantic, JSON Schema) and executing them through both traditional crawlers and LLM-based semantic parsers.SelfHealing & Semantic Extraction Handson use of ScrapyLLM, Scrapy MCP Server, or similar systems that decouple field definitions from page structure.Agentic Workflows Familiarity with frameworks that give LLMs browser control (Playwright + MCP, BMAD/TEA) to handle complex, nondeterministic crawling tasks.Classical Scraping Fundamentals You still know how to write a Scrapy spider or a Playwright script when needed, but you actively seek to replace that work with reusable, specification-driven components.Data Validation & Storage Ability to define validation rules within specifications and land clean data into SQL/NoSQL databases or data lakes.Python proficiency: the focus is on an extraction engineer who happens to use Python.HTTP, DOM, XPath, CSS.Basic API integration and authentication flows.Preferred / Nice-to-Have Skills:Contributions to open-source scraping or AI-automation projects.Experience training or fine-tuning small LLMs for domain-specific extraction.Familiarity with data privacy engineering (GDPR, CCPA) baked into specification design.DevOps light Docker, CI/CD for testing extraction specifications.Mindset & Approach (Non-Negotiable):Strong belief that the future of scraping is declarative, not imperative. Youd rather write a schema that says extract the price than debug an XPath when a website redesigns.Looking to shift from code that scrapes to systems that understand extraction.

Role: Data Engineer 2.0Location - RemoteExperience: Min. 5 YearsNotice Period- Immediate to 15 Days or serving notice periodKey Responsibilities:Design and implement manual test strategies for real-time streaming use cases using Azure Service Bus, Event Hubs, Kafka, and Azure Functions.Validate Spark Streaming applications, including unbounded data flows, streaming DataFrames, checkpoints, and streaming joins.Develop test plans for containerized microservices deployed on Kubernetes, ensuring scalability and fault tolerance.Test data ingestion and transformation workflows across open table formats like Delta Lake, Apache Iceberg, and Hudi.Good to Have:Monitoring and troubleshooting system performance using observability stacks such as Prometheus, Grafana, and ELK.functional and performance testing on analytical databases and query engines such as Trino, StarRocks, and ClickHouse.Testing and validation of data products designed under data mesh architecture, ensuring domain-oriented data quality and governance.

We are seeking a talented AI/ML Engineer to design, develop, and deploy machine learning models that solve real-world business problems.Key ResponsibilitiesDevelop, train, and optimize machine learning and deep learning models.Design and implement AI solutions for automation, prediction, and data analysis.Work with large datasets to clean, preprocess, and engineer features.Deploy models into production environments and monitor performance.Build scalable ML pipelines and integrate models with applications.Conduct experiments, model evaluations, and performance tuning.Collaborate with cross-functional teams including data engineers and product managers.Stay updated with the latest research and advancements in AI/ML.Note: Only Pune-based candidates can eligible to apply.

Dear Candidate ,Give dynamic start to your career with VMS Group SECRET TECHNOLOGIES INDIA , we are looking for freshers and experienced candidates who want to make good career in IT or NON It field.Job Type: Full time.Eligibility criteria : Fresher /Experience 0-5 Year.Location : Pune. OR PAN INDIADesired Qualification - Any Bachelor degree Minimum BA ,B.Com - Commerce, B.E , B.Tech, M.Sc , M.C.A B.C.A ,Any Graduates, Post Graduates, can also apply. Should be comfortable to work any location of pune or Pan India.Your application for theTesting & DevelopmentBack Office / Data EntryTechnical Support, Non-Voice ProcessBDE (canadian, US, UK)-In HouseBusiness Development Executive, On FieldOperations/Customer Service , Customer Service (Web)BPO MNC and Domestic ( Hindi Marathi or Any local language)Accounts & Finance /Financial AnalystPosition stood out to us and we would like to invite you for a Face to face interview Telephone interview at our office to get to know you a bit better.You will meet with the MR. Prem Sir or Aishwarya Ma'am.The interview will last about 20 minutes and you'll have the chance to discuss the above position and learn more about our company. Please carry your updated resume at list Two CopiesDesired Candidate - Candidate should be Dynamic, Enterprising, Presentable with excellent communication and written skills.If candidate is interested please visit our office. Monday to Saturday 10 am to 5 pm bitwin anytime Meet- Prem Sir / Ms. Aishwarya Ma'am Interview Venue-building A -15th , city vista, foundation road, Kharadi, Pune, Maharashtra 411014We will take 2 rounds of an interview.-Contact No. 7066165593 Aishwarya Ma'am 7066147637 Prem Sir 1. HR Round and simple logic checking P.I round.2. Technical HR Final round by Senior HR Manager.Thanks & Regards SECRET TECHNOLOGIES INDIA VMS GROUP Placement / Contract Labour / Real EstateAddress -building A -15th , city vista, foundation road, Kharadi, Pune, Mahara

Key Responsibilities: Cloud EngineerDesign, implement, and manage CI/CD pipelinesAutomate deployments and infrastructure using tools like Terraform/AnsibleMonitor system performance and ensure high availabilityRequirements:Experience with AWS/AzureKnowledge of Docker, Kubernetes, and scriptingStrong problem-solving skillsBrief Summary:Responsible for automating and optimizing cloud infrastructure and deployment processes.

Key Responsibilities Requirement Analysis: Collaborate with stakeholders to understand businessrequirements and data sources, and define the architecture and design of dataengineering models to meet these requirements. Architecture Design: Design scalable, reliable, and efficient data engineering models,including algorithms, data pipelines, and data processing systems, to support businessrequirements and quantitative analysis. Technology Selection: Evaluate using POCs and recommend appropriate technologies,frameworks, and tools for building and managing data engineering models, consideringfactors like performance, scalability, and cost-effectiveness. Data Processing: Develop and implement data processing logic, including data cleansing,transformation, and aggregation, using technologies such as AWS Glue, Batch, Lambda. Quantitative Analysis: Collaborate with data scientists and analysts to develop algorithmsand models for quantitative analysis, using techniques such as regression analysis,clustering, and predictive modeling. Model Evaluation: Evaluate the performance of data engineering models using metricsand validation techniques, and iterate on models to improve their accuracy andeffectiveness. Data Visualization: Create visualizations of data and model outputs to communicateinsights and findings to stakeholders.Data Engineering: Understanding of Data engineering principles and practices, includingdata ingestion, processing, transformation, and storage, using tools and technologiessuch as AWS Glue, Batch, Lambda. Quantitative Analysis: Proficiency in quantitative analysis techniques, including statisticalmodeling, machine learning, and data mining, with experience in implementingalgorithms for regression analysis, clustering, classification, and predictive modeling. Programming Languages: Proficiency in programming languages commonly used in dataengineering and quantitative analysis, such as Python, R, Java, or Scala,

Key ResponsibilitiesTechnical Design & Drafting: Using CAD software to create layouts for Medical Gas Pipeline Systems (MGPS), nurse call systems, or ambulance interior configurations.Site Supervision: Managing on-site installation teams (technicians, welders, and fitters) to ensure work meets ISO and AIS-125 safety standards.Project Scheduling: Developing timelines (Gantt charts) to ensure hospital wings or ambulance fleets are delivered on time.Quality Control & Testing: Conducting pressure tests on gas lines and load-testing stretchers/cots before handover to the client.Procurement Coordination: Working with the supply chain to ensure specialized medical-grade materials (like degreased copper pipes or reinforced alloys) are available on-site.Regulatory Compliance: Ensuring every aspect of the project adheres to healthcare building codes and medical device regulations.

System Monitoring & Operation:Monitor computer systems, networks, and peripheral devices to ensure optimal performance and identify any anomalies or errors.Execute routine batch jobs, scheduled tasks, and system processes as per established procedures.Perform system startup and shutdown procedures as required.Maintain logs of system activities, incidents, and resolutions.Data Entry & Management:Accurately input data into various software applications, databases, or spreadsheets (e.g., MS Excel, specialized ERP/CRM software, Tally).Verify data for accuracy and completeness, correcting any errors or inconsistencies.Organize and maintain digital files and documents in a structured manner.Perform data backup and recovery procedures as per schedule to prevent data loss.Printing & Documentation:Manage and operate printers, scanners, and other office equipment.Handle large-scale printing jobs, including reports, invoices, labels, and other essential documents.Ensure proper paper handling, toner/ink replacement, and basic maintenance of printing devices.Assist in maintaining organized physical and digital records.Basic Troubleshooting & Support:Perform first-level troubleshooting for common hardware and software issues (e.g., printer jams, network connectivity problems, application errors).Escalate complex technical issues to IT support or senior personnel as needed.Assist users with basic computer-related queries and provide guidance on software usage.System Maintenance & Security:Assist in routine system maintenance tasks, such as disk cleanup, anti-virus scans, and software updates.Ensure adherence to data security protocols and confidentiality policies.Report any suspicious activities or security breaches.Compliance & Reporting:Adhere to company policies and procedures regarding computer operations and data handling.Generate routine operational reports as requested by management.

We are looking for a highly experienced Python Architect to design and lead large-scale, enterprise-grade Python solutions for AI, data, and analytics platforms. The role requires deep hands-on expertise in Python across architecture, development, and production systems, supporting end-to-end AI and data-driven solutions.Key Responsibilities:Architect scalable, high-performance systems using Python as the core technologyDefine technical architecture for AI, data engineering, and analytics platformsLead design and development of microservices, APIs, and backend frameworks in PythonEnsure production-ready implementations with strong focus on performance, security, and reliabilityCollaborate with data science, ML, DevOps, and product teams to operationalize AI solutionsEstablish coding standards, best practices, and architectural guidelinesReview code, mentor senior engineers, and provide technical leadershipDrive innovation and continuous improvement in Python-based system designRequired Skills & Qualifications:1218 years of experience working fully and extensively in PythonStrong expertise in Python system architecture and large-scale backend designExperience supporting AI/ML, data engineering, or analytics platformsSolid understanding of microservices, REST APIs, and distributed systemsHands-on experience with cloud platforms (AWS/Azure/GCP)Strong problem-solving skills and ability to lead complex technical initiativesExcellent communication and stakeholder collaboration skills

Job Description: We are looking for Data Engineers to join our team. You will use various methods to transform raw data into useful data systems. For example, youll create algorithms and conduct statistical analysis. Overall, youll strive for efficiency by aligning data systems with business goals. To succeed in this position, you should have strong analytical skills and the ability to combine data from different sources. Data engineer skills also include familiarity with several programming languages and knowledge of machine learning methods. Job Requirements: Participate in the customers system design meetings and collect the functional/technical requirements. Build up data pipelines for consumption by the data science team. Skillful in ETL process and tools. Clear understanding and experience with Python and PySpark or Spark and SCALA, with HIVE, Airflow, Impala, and Hadoop and RDBMS architecture. Experience in writing Python programs and SQL queries. Experience in SQL Query tuning. Experienced in Shell Scripting (Unix/Linux). Build and maintain data pipelines in Spark/Pyspark with SQL and Python or SCALA. Knowledge of Cloud (Azure/AWS/GCP, etc..) technologies is additional. Good to have knowledge of Kubernetes, CI/CD concepts, Apache Kafka Suggest and implement best practices in data integration. Guide the QA team in defining system integration tests as needed. Split the planned deliverables into tasks and assign them to the team. Needs to Maintain/Deploy the ETL code and follow the Agile methodology Needs to work on optimization wherever applicable. Good oral, written and presentation skills. Preferred Qualifications: Degree in Computer Science, IT, or a similar field; a Masters is a plus. Hands-on experience with Python and Pyspark Or Hands-on experience with Spark and SCALA. Great numerical and analytical skills. Working knowledge of cloud platforms such as MS Azure, AWS, etc..

Responsibilities: Designing and implementing data pipelines to collect, clean, and transform data from various sources Building and maintaining data storage and processing systems, such as databases, data warehouses, and data lakes Ensuring data is properly secured and protected Developing and implementing data governance policies and proceduresCollaborating with business analysts, data analysts, data scientists, and other stakeholders to understand their data needs and ensure they have access to the data they need Sharing knowledge with the wider business, working with other BAs and technology teams to make sure processes and ways of working is documented. Collaborate with Big Data Solution Architects to design, prototype, implement, and optimize data ingestion pipelines so that data is shared effectively across various business systems. Ensure the design, code and procedural aspects of solution are production ready, in terms of operational, security and compliance standards. Participate in day-to-day project and agile meetings and provide technical support for faster resolution of issues. Clearly and concisely communicating to the business, on status of items and blockers. Have an end-to-end knowledge of the data landscape within the company. Skills & Experience: 10+ years of design & development experience with big data technologies like Azure, AWS or GCP Preferred is Azure & Databricks, with experience in Azure DevOps Experience in data visualising technology in DL, like PowerBI Proficient in Python, PySpark and SQL Proficient in querying and manipulating data from various DB (relational and big data). Experience of writing effective and maintainable unit and integration tests for ingestion pipelines. Experience of using static analysis and code quality tools and building CI/CD pipelines. Excellent communication, problem-solving, and leadership skills, and be able to work well in a fast-paced, dynamic environment.