Principal Data Engineer (since 2025-07; phase 1 EPAM team lead of 4; now IC outstaffed into client ingestion team)
EPAM Systems · Client: Fortune-50 consumer-electronics company (under NDA)
- Led the pipeline-migration workstream of a multi-PB Hadoop/HDFS → AWS S3 platform migration: dependency-phased migration of ~220 Spark pipelines (ingestion, compaction/latest, hybrids) to Spark-on-Kubernetes in ~6 months — evaluated dependency lineage and migrated leaf-first so each phase gated on the previous one's verified success. AWS-funded program.
- Personally migrated 20–30 of the pipelines, including converting legacy custom Hadoop MapReduce jobs (Wikipedia-corpus data prep for LLM training) to Spark, alongside newer JSON-flattening pipelines.
- Designed the data-parity sign-off process for the migration: HDFS and S3 ran overlapping; both copies landed on a common platform where Spark-notebook comparison jobs proved result equivalence before each dataset was signed off and cut over.
- Dual role — EPAM team lead (4 engineers), co-ran the migration process with the delivery manager while contributing as a hands-on engineer. Framework-conformant jobs migrated fast; final months were the complex custom tail (e.g. Wikipedia LLM-training data-prep jobs).
- Ingestion-team engineer for the client's core pipeline: Cassandra → Kafka (table archiver + direct-event producers) → Kafka → S3 incremental → per-dataset compaction/latest jobs. Owns schema updates and job changes across the fleet; platform includes an in-house dataset-registry system + schema/config/metrics services.
- Drove a self-service epic for data analysts: a realtime-data enrichment wizard that generates schema-update tickets as structured JSON — machine-readable so schema changes can be AI-automated end-to-end (agent flow → PR creation). Built the companion AI chat assistant (web) on the Anthropic API with streaming, tool calling, and guardrails.
- Migrating the Kafka-export layer from Spark to Flink: replacing ~1-hour long-running pseudo-streaming Spark jobs (Kafka → S3 JSON, secondary job JSON → deduped Parquet, external watchdog scheduler restarts) with Flink-native pipelines. In progress.
- Participated in the table-format/catalog evaluation (AWS Glue vs Nessie vs in-house service); Iceberg-on-S3 selected, migration underway.
- AI-assisted engineering suite around Claude Code, used daily for a near-fully-automated workflow (task gathering → prioritization → ticket flow): an IaC tool, then a Terraform provider, managing Claude Code instances from config with profile support (self-adjusting in realtime); an "AI artifactory" pipeline that watches team + OSS AI repos, curates changes through security/poisoning review and performance evals, and installs the vetted results; a router skill choosing best model×effort config per repeatable task; a local RAG system replacing memory files (vector search, no size limits); a ticket workflow with team creation, communication, and retrospectives.
- Stepped in as the single point of contact for the customer during a delivery-manager gap (Aug–Sep), keeping the ingestion workstream aligned and delivering while the team was without a DM; integrated into the client-side team quickly and picked up the ingestion domain.
- Onboarded and mentored engineers joining the ingestion team — the go-to person for ramping new joiners on the domain and codebase.
- As AI champion on the programme, drove and evidenced the account's engineering AI-maturity to Level 2.1 (RAG Green, above the target of 2): authored the L2 evidence artefact and the Level-3 readiness roadmap — an agentic multi-agent Spark-to-AWS migration factory scoped over the ~1,000-job backlog.
- Built the team's AI-usage metrics tooling (per-day spend, DAU/WAU, limit-hit detection) and instrumented the team — daily-active AI usage measured at 87.9%, past the Level-2 >70% bar — and designed its released-only build->review->scan->publish CI/CD pipeline, closing a laptop supply-chain security risk.
- Grew a reusable AI factory — a 108-skill shared library, 39 committed AI-context files, and 4 team plugins published to a released-only internal marketplace — and delivered an AI-SDLC enablement session demonstrating these capabilities to the team.