Python Data Engineer Job Vacancies in Bangalore

Key Responsibilities:Build and manage data pipelines and ETL processesWork with large datasets using tools like Spark, Hadoop, or SQLEnsure data quality and performance optimizationRequirements:Experience in Python/SQLHands-on with ETL tools and big data technologiesUnderstanding of data warehousing conceptsBrief Summary:Develops scalable data systems to support analytics and business insights.

Data Extraction Engineer designs extraction systems (and not just scripts). They build and maintain a next-generation data acquisition platform that treats web scraping as a declarative, specification-driven discipline. Instead of hard-coding XPaths for every site, Web Scraping Developer defines what data is neededusing schemas, natural language descriptions, or visual blueprintsand lets intelligent pipelines figure out how to get it.Key Responsibilities:Specification-Driven Extraction Engineering-Design and maintain declarative extraction specificationsusing Pydantic models, JSON schemas, or domain-specific languagesthat describe exactly which fields to capture, their types, and validation rules.Implement pipelines that translate these specifications into executable extraction plans, leveraging both classical (Scrapy, Playwright) and AI-augmented (LLM-based semantic parsing) backends.Build reusable specification libraries for recurring data types (product prices, tariff codes, regulatory texts) to accelerate onboarding of new sources.Autonomous & Self-Healing Systems-Deploy self-healing spiders that automatically detect website layout changes and repair themselves using Model Context Protocol (MCP) servers (e.g., Scrapy MCP Server, Playwright MCP).Integrate semantic extraction (Scrapy-LLM, custom LLM pipelines) to eliminate selector brittlenessspiders rely on field descriptions, not fragile XPaths.Orchestrate complex, multi-step browsing workflows with agentic frameworks (BMAD/TEA, AutoGPT-like agents) that reason about page state, adapt to anti-bot measures, and correct their own behaviour in real time.Platform Thinking & Reusability-Move beyond one-off scrapers: build a component-based extraction platform where selectors, login handlers, and pagination logic are shared, versioned, and tested.Implement monitoring, alerting, and automatic rollback for failed extraction runs.Champion ethical crawling by designrate limiting, robots.txt respect, and compliance with GDPR/CCPA are built into the specification layer, not retrofitted.Collaboration & Continuous Innovation-Partner with data scientists and domain experts to refine extraction specifications for complex, unstructured domains (e.g., legal texts, tariff classifications).Evaluate and pilot emerging tools to push automation coverage beyond 90%.Document and evangelise specification-driven best practices across the engineering organisation.Candidate Profile:Education and Experience -Bachelors degree in Computer Science3+ years of experience in web scraping or data extractionSkills and competences-Specification-Driven Extraction Experience defining extraction requirements via schemas (Pydantic, JSON Schema) and executing them through both traditional crawlers and LLM-based semantic parsers.SelfHealing & Semantic Extraction Handson use of ScrapyLLM, Scrapy MCP Server, or similar systems that decouple field definitions from page structure.Agentic Workflows Familiarity with frameworks that give LLMs browser control (Playwright + MCP, BMAD/TEA) to handle complex, nondeterministic crawling tasks.Classical Scraping Fundamentals You still know how to write a Scrapy spider or a Playwright script when needed, but you actively seek to replace that work with reusable, specification-driven components.Data Validation & Storage Ability to define validation rules within specifications and land clean data into SQL/NoSQL databases or data lakes.Python proficiency: the focus is on an extraction engineer who happens to use Python.HTTP, DOM, XPath, CSS.Basic API integration and authentication flows.Preferred / Nice-to-Have Skills:Contributions to open-source scraping or AI-automation projects.Experience training or fine-tuning small LLMs for domain-specific extraction.Familiarity with data privacy engineering (GDPR, CCPA) baked into specification design.DevOps light Docker, CI/CD for testing extraction specifications.Mindset & Approach (Non-Negotiable):Strong belief that the future of scraping is declarative, not imperative. Youd rather write a schema that says extract the price than debug an XPath when a website redesigns.Looking to shift from code that scrapes to systems that understand extraction.

Design, build, and maintain ETL/ELT data pipelines and data lake solutions to support analytics and AI/ML use cases. Ensure data quality, performance, and reliability across enterprise data platforms.Key ResponsibilitiesPipeline DevelopmentData Lake EngineeringPerformance & OptimizationCollaboration & SupportRequired Skills & Experience 4+ years of experience in data engineering or ETL development. Proficiency in SQL and Python (or Scala/Java) for data transformations. Hands-on with ETL tools (Informatica, Talend, dbt, SSIS, Glue, or similar). Exposure to big data technologies (Hadoop, Spark, Hive, Delta Lake). Familiarity with cloud data platforms (AWS Glue/Redshift, Azure Data Factory/Synapse, GCP Dataflow/BigQuery). Understanding of workflow orchestration (Airflow, Oozie, Prefect, or Temporal).Preferred Knowledge Experience with real-time data pipelines using Kafka, Kinesis, or Pub/Sub. Basic understanding of data warehousing and dimensional modeling. Exposure to containerization and CI/CD pipelines for data engineering. Knowledge of data security practices (masking, encryption, RBAC).Education & Certifications Bachelors degree in Computer Science, IT, or related field.Preferred certifications:o AWS Data Analytics Specialty / Azure Data Engineer Associate / GCP Data Engineer.o dbt or Informatica/Talend certifications.

Job Title:Data Engineer Location: Bangalore Experience: 6+ years Notice Period : Immediate to 21daysMUST-HAVE TECHNICAL SKILLSSkillSkill DepthPython for Data PipelinesIndependently written ingestion/transformation scripts, including pagination, exception handling, logging, and dataframe-level operations using Pandas, JSON, or GCP SDKsDBT (Data Build Tool)Authored and executed DBT models and tests using YAML files and Jinja macros; contributed to CI test configs and schedule integrationGCP (BigQuery, GCS, CloudSQL)Hands-on experience in at least two of the above tools in pipeline execution e.g., used BigQuery for SQL transformation and GCS for raw/processed layer segregationAWS LambdaIntegrated serverless functions to automate trigger points like new file upload, API call chaining or job completion; used boto3 or GCP Pub/Sub hooksData Quality & ValidationDeveloped or plugged-in validation layers for ingestion such as record count matching, null/duplicate flagging, recon table populationCloud-Native ModelingAdapted pre-existing logical models to ingestion logic, ensuring correct joins, partitioning strategy, and target-layer conformity (Star/Snowflake)Version Control & AgileParticipated in Git branching workflows and sprint-based delivery (JIRA or similar); able to push/pull/test with basic conflict resolution

Job Opening: Generative AI & LLMs / Distinguished Gen AI Engineer Location: Remote Experience: 7+ yearsShare your profilestosandhya@welkinsofttech.com To apply, send your profile to sandhya@welkinsofttech.com / hr@welkinsofttech.com or connect with us here. Key Responsibilities:LLM Development: Design, fine-tune, and implement large language models (e.g., GPT, BERT, T5) for applications like personalized learning, content generation, and semantic search.Generative AI Solutions: Drive innovation with Gen AIdeveloping tools like adaptive learning paths, resume builders, and AI-written job descriptions.Machine Learning: Create predictive models and recommendation engines that align user profiles to skills and job opportunities.Token Optimization: Work with OpenAI and other services to manage token efficiency and usage costs.AI Integration: Collaborate with product and engineering teams to integrate AI features seamlessly into the Elefy platform.Data Engineering: Build and maintain robust data pipelines using Python, Node.js, and MongoDB.Data Analysis: Analyze large datasets to surface actionable insights for user engagement and platform growth.Visualization & Reporting: Build dashboards using Tableau, Power BI, or Matplotlib to communicate insights to stakeholders.Documentation: Ensure clear and comprehensive documentation for models, pipelines, and workflows. Who Were Looking For:Experience:8+ years in data science or AI, with 3+ years hands-on with LLMs or Gen AI in production settings.Proven track record of delivering ML models in scalable, real-world applications.Skills:Languages: Python (must), R, SQLFrameworks: PyTorch, TensorFlow, Hugging Face, Scikit-learnPrompt Engineering: Few-Shot Learning, Dynamic Prompting, Role Play, Chain-of-Thought (nice to have)Cloud: Azure (preferred), AWS, or GCPDatabase: MongoDB or similar NoSQL/SQL systemsKnowledge & Tools:Deep NLP & LLM expertise (e.g., GPT, BERT, T5, etc.)Containerization, APIs, CI/CD, and Azure-native cloud toolsStrong visual storytelling via Tableau, Power BI, or Python-based plotsAgile & cross-functional collaboration mindset Bonus Points:Experience with ethical AI, bias mitigation, and explain abilityFamiliarity with skill-based learning platforms or EdTech ecosystemsJoin us at Elefy and be part of a team thats reshaping the future of learning with AI.

Job Description:Yrs of Exp : 10 + yrsMode : 3 days a weekLocation: BangaloreWork Type : PermanentKey ResponsibilitiesDesign and Development:Architect, implement, and optimize scalable data solutions.Develop and maintain data pipelines, ETL/ELT processes, and workflows to ensure the seamless integration and transformation of data.Collaboration:Work closely with data scientists, analysts, and business stakeholders to understand requirements and deliver actionable insights.Partner with cloud architects and DevOps teams to ensure robust, secure, and cost-effective data platform deployments.Data Management:Manage and maintain data lakes, data warehouses, and real-time analytics systems.Ensure high data quality, integrity, and security across the organization.Performance Optimization:Monitor and enhance system performance, troubleshoot issues, and implement optimizations as needed.Leverage Microsoft Fabrics advanced analytics and AI capabilities for innovative data solutions.Best Practices & Leadership:Lead and mentor junior engineers to foster a culture of technical excellence.Stay updated with industry trends and best practices, especially in the Microsoft ecosystem.Required:Bachelors or Masters degree in Computer Science, Data Engineering, or a related field.10+ years of experience in data engineering, with a proven track record of working on large-scale data platforms.Expertise in Microsoft Fabric and its components (e.g., Synapse, Data Factory, Azure Data Lake, Power BI).Strong proficiency in SQL, Python, and Spark.Experience with cloud platforms, particularly Microsoft Azure.Solid understanding of data modeling, data warehousing, and ETL/ELT best practices.Excellent problem-solving, communication, team management and project management skills.Preferred:Familiarity with other cloud platforms (e.g., AWS, GCP).Experience with machine learning pipelines or integrating AI into data workflows.Certifications in Microsoft Azure or related technologies.

Level of skills and experience:5 years of hands-on experience in using Python, Spark,Sql.Experienced in AWS Cloud usage and management.Experience with Databricks (Lakehouse, ML, Unity Catalog, MLflow).Experience using various ML models and frameworks such as XGBoost, Lightgbm, Torch.Experience with orchestrators such as Airflow and Kubeflow.Familiarity with containerization and orchestration technologies (e.g., Docker, Kubernetes).Fundamental understanding of Parquet, Delta Lake and other data file formats.Proficiency on an IaC tool such as Terraform, CDK or Cloud Formation.Strong written and verbal English communication skill and proficient in communication with non-technical stakeholderst

Work Mode: Hybrid6-9 years of overall IT experience, preferably in cloud environments.Minimum of 5 years of hands-on experience with AWS cloud development projects.Design and develop AWS data architectures and solutions.Build robust data pipelines and ETL processes using big data technologies.Utilize AWS data services such as Glue, Lambda, Redshift, and Athena effectively.Implement infrastructure as code (IaC) using Terraform.Proficiency in SQL, Python, and other relevant programming/scripting languages.Experience with orchestration tools like Apache Airflow or AWS Step Functions.Strong understanding of data warehousing concepts, data lakes, and data governance frameworks.Expertise in data modeling for both relational and non-relational databases.Excellent communication skills are essential for this role.

Invitation for B2B Partnerships: Seeking Software Development Support! We are looking to collaborate with companies that can provide 10 skilled Data Engineers to support our data engineering requirements pipeline for Euoropean clients on a contract basis.Requirements:Expertise in data engineering, including data processing and ETL.Proficiency in SQL and NoSQL databases.Experience with Hadoop or Spark.Experience in AWS or Azure.Snowflake is an added advantage.If your company is equipped to provide top-notch Data Engineers, we invite you to submit your proposal with terms and conditions.Please contact us by email or Whatsapprakanalytics@gmail.com (or) 9900173022

We have immediate requirement for FreshersRole : Software Engineer , Network Engineer. Desktop Support EngineerSalary : 3 Lac - 4LacLocation : Chennai, Coimbatore, Bangalore.Training : 2 Week in Online trainingsend resumes to hr.alphamanpower@gmail.comOnly Tamilnadu candidate can apply

Job Summary:We are seeking an experienced Scala Developer with a strong background in Kafka and Big Data technologies. The ideal candidate will have extensive experience in designing and implementing scalable data solutions, with a focus on performance and reliability. You will work closely with our data engineering and AI teams to build and maintain high-performance data pipelines and applications.Key Responsibilities:Develop and maintain scalable applications using Scala and related technologies.Design, implement, and manage data pipelines utilizing Kafka and other Big Data tools.Collaborate with cross-functional teams to define, design, and ship new features.Optimize application performance, ensuring high throughput and low latency.Monitor and troubleshoot data processing workflows, ensuring data integrity and reliability.Participate in code reviews, provide feedback, and improve code quality.Stay up-to-date with the latest trends and best practices in Big Data and functional programming.Required Skills:Strong proficiency in Scala with at least 4+ years of hands-on experience.Extensive experience with Kafka: stream processing, Kafka Connect, Kafka Streams, etc.Solid understanding of Big Data technologies including Hadoop, Spark, HDFS, and Hive.Proficiency in SQL and NoSQL databases.Experience with ETL pipelines and data integration workflows.Familiarity with data warehousing concepts and cloud platforms (AWS, GCP, Azure).Knowledge of containerization (Docker) and orchestration tools (Kubernetes) is a plus.Excellent problem-solving skills and a proactive attitude.Qualifications:Bachelors or Masters degree in Computer Science, Engineering, or a related field.4+ years of experience in Scala development, with a focus on Big Data and Kafka.Proven track record of building and managing scalable, reliable data systems.Strong communication skills and ability to work in a collaborative environment.

Specific accountabilities: 6+ years of experience as a C++ software developer Knowledge of C++ standards and specifications Familiarity with C++ compatible languages such as C, Java, and Python Extensive expertise in software deployment across several platforms and operating systems Excellent analytical and problem-solving abilities Design, build, and maintain efficient and reliable C++ code. Work closely with other engineers to develop new products. Work on extensions of existing products Suggest and implement improvements on current products. Plan, design, develop, manage, document, test, deploy, and support new and existing modules. Hands on of algorithm complexities Development of the next generation software Basic Database systems - indexes, filtersMandatory Skill Set Year of experience required-5+ in C++ Programming Language Qualifications-BE/B.Tech

Are you a Mid/Senior level T-Shaped AWS expert with specialization in DevOps and Data Engineering space? If yes, We have an exciting opportunity just for you.One of our reputed European Client is looking for AWS engineers to help them build secure, resilient and cost-effective solutions on AWS platform to reap the benefits from their investment in AWS platform and services.We are looking for self-motivated, highly experienced engineers, possessing great analytical and excellent communication skills for this client facing role.What do we expect from you?Role: Data Engineer (AWS)*Mandatory*Experience in Developing Data Pipelines that process large volumes of data using Python, PySpark, Pandas etc, preferably on AWS.Experience in ingesting batch and streaming data from various data sources.Experience in writing complex SQL using any RDBMS (Oracle, PostgreSQL, SQL Server etc.)Experience in developing ETL, OLAP based and Analytical Applications.Ability to quickly learn and develop expertise in existing highly complex applications and architectures.Comfortable working in Agile projects*Desirable*Exposure to AWS platform's data services (AWS Lambda, Glue, Athena, Redshift, Kinesis etc.)Knowledge of DevOps and CD/CD tools.Experience in handling unstructured dataKnowledge of Financial Markets domainKeywords: Data Engineer, Data Pipelines, Data Ingestion, AWS Lambda, AWS Athena

Profile - Data Engineer/AnalystExperience 5+ YearsBudget 25 LPAOnsite -Bangalore Timeline Immediate.The current tools and tech stack relevant to the role are:Elasticsearch: datastore where processed data is stored and would be queried fromKibana, Grafana: platforms for building and hosting visualizations and dashboardsStrong proficiency with Python, especially towards data handling and analytics:Data handling and manipulation libraries (i.e. pandas, numpy)Visualization libraries (e.g. seaborn, matplotlib)Dashboarding libraries to build and host dashboards (e.g. Bokeh)Basic proficiency with modeling (e.g. sklearn, linear regression)Python is also currently used for its libraries in connecting and querying from Elasticsearch, as well as reporting through Slack.

I require a person to assist us with a Java Udemy Course. If the person has the training videos ready, it's the best. If not, the person should be willing to create a course. The person has to create the full course.To Apply please go to GerardYadGG LinkedIn's Page, you will find the website link there, you can apply there. Please apply there to be considered.100% worldwide remote.ID PI 206

Primary skillset: Experience working with distributed technology tools fordeveloping Batch and Streamingpipelines using SQL,Spark, Python, PySpark [4+ years], Airflow [3+ years], Scala [2+ years]. Able to write code which is optimized for performance. Experience in Cloud platform, e.g., AWS, GCP, Azure, etc. Able to quickly pick up new programming languages, technologies, and frameworks. Strong skills building positive relationships across Product and Engineering. Able to influence and communicate effectively, both verbally and written, with team members and business stakeholders Experience with creating/ configuring Jenkins pipeline for smooth CI/CD process for Managed Spark jobs, build Docker images, etc. Working knowledge of Data warehousing, Data modelling, Governance and Data Architecture

Mandatory Skills:Apache Spark and either PySpark or Scala: Extensive hands-on experience with Spark for large-scale data processing and analysis. Proficiency in either PySpark or Scala for developing Spark applications.Databricks: Strong expertise in using Databricks for big data analytics, data engineering, and collaborative work on Apache Spark.Github: Proficient in version control using Git and GitHub for managing and tracking changes in the codebase.Data Warehousing (DWH): Experience with one or more of the following DWH technologies Snowflake, Presto, Hive, or Hadoop. Ability to design, implement, and optimize data warehouses.Python: Advanced programming skills in Python for data manipulation, analysis, and scripting tasks.SQL: Strong proficiency in SQL for querying, analyzing, and manipulating large datasets in relational databases.Data Streaming or Data Batch: In-depth knowledge and hands-on experience in both data streaming and batch processing methodologies.Good to Have:Kafka: Familiarity with Apache Kafka for building real-time data pipelines and streaming applications.Jenkins: Experience with Jenkins for continuous integration and continuous delivery (CI/CD) in the data engineering workflow.Responsibilities:Design, develop, and maintain scalable and efficient data engineering solutions using Apache Spark and related technologies.Collaborate with cross-functional teams to understand data requirements, design data models, and implement data processing pipelines.Utilize Databricks for collaborative development, debugging, and optimization of Spark applications.Work with various data warehousing technologies such as Snowflake, Presto, Hive, or Hadoop to build robust and high-performance data storage solutions.Develop and optimize SQL queries for efficient data retrieval and transformation.Implement both batch and streaming data processing solutions to meet business requirements.Collaborate with other teams to integrate data engineering

We have vacant of 5 Data Engineer Jobs in Bangalore, Noida, Experience Required : 9 Years Educational Qualification : Other Bachelor Degree Skill Data Warehousing, ETL, Python, AWS, SCALA, data engineer

Job DescriptionThe Lead Data Engineer is responsible for finding trends in Data sets and Developing Algorithms to help make raw data more useful to the Organization. This role requires a significant Set of Technical Skills, including a deep Knowledge of SQL Database Design and Multiple Programming languages along with team management skillsiSOCRATES is looking to add a Lead Data Engineer to its growing Analytics team. The Lead Data Engineer will lead the team that is responsible for finding trends in Data sets and Developing Algorithms to help make raw data more useful to the Organization. The Lead Data Engineer will report directly to the Senior Director Qualifications Overall, 8-12 years of experience in data & analytics Have strong experience in SQL/NoSQL databases Design and build production data pipelines from ingestion to consumption within a big data architecture, using Python/Scala/Java Design, build and operationalize large-scale enterprise data solutions and applications using one or more of AWS data and analytics services in combination with Mandatory - Redshift, S3, Athena, Good to have - Glue, Snowflake Design and implement data engineering, ingestion, and curation functions on AWS cloud using AWS native or custom programming Perform detailed assessments of current state data platforms and create an appropriate transition path to AWS cloud Should be able to understand the data very well, create a pattern on structured /non-structured data Strong knowledge of Data warehouse, Data Modelling (Preferably Dimensional Modelling Kimball) Proficiency in both ETL & ELT AWS Certified Data Analytics Specialist / AWS Certified ML Specialist will be an added advantage Exposure to Media & Marketing is a plus Having knowledge of GCP is a plus Having knowledge of 1 or more visualization tools is a plus

Data Engineer/AnalystExperience 5+ yearsBudget Upto 25 LPAOnsite -Bangalore Timeline Immediate.The current tools and tech stack relevant to the role are:Elasticsearch: datastore where processed data is stored and would be queried fromKibana, Grafana: platforms for building and hosting visualizations and dashboardsStrong proficiency with Python, especially towards data handling and analytics:Data handling and manipulation libraries (i.e. pandas, numpy)Visualization libraries (e.g. seaborn, matplotlib)Dashboarding libraries to build and host dashboards (e.g. Bokeh)Basic proficiency with modeling (e.g. sklearn, linear regression)Python is also currently used for its libraries in connecting and querying from Elasticsearch, as well as reporting through Slack.

I require a person to assist us with Java Developer Udemy Course. If the person has the training videos ready, it's the best. If not, the person should be willing to create a course. The person has to create the full course.To Apply please go to GerardYadGG LinkedIn's Page, you will find the website link there, you can apply there. Please apply there to be considered.100% worldwide remote.ID PI 203

I require a person to assist us with a Python Developer Udemy Course. If the person has the training videos ready, it's the best. If not, the person should be willing to create a course. The person has to create the full course.To Apply please go to GerardYadGG LinkedIn's Page, you will find the website link there, you can apply there. Please apply there to be considered.100% worldwide remote.ID: PL300Thank You.

I require a person to assist us with a Data Science Udemy Course. If the person has the training videos ready, it's the best. If not, the person should be willing to create a course. The person has to create the full course.To Apply please go to GerardYadGG LinkedIn's Page, you will find the website link there, you can apply there. Please apply there to be considered.100% worldwide remote.ID PI 201