BOSTON, MA
Data Engineer
Building the cloud data platform behind financial, payments, and product analytics. Architected a Databricks, Delta Lake, and Snowflake lakehouse consolidating 25+ enterprise systems, engineered real-time Kafka and Spark Structured Streaming pipelines that cut end-to-end latency from 95 minutes to under 18 minutes across 15M+ daily financial events, and built the RAG and feature engineering layers supporting enterprise search, fraud detection, and forecasting.
- Architected a cloud-native Lakehouse and Snowflake analytics platform using Databricks, PySpark, Delta Lake, AWS S3, and AWS Glue, consolidating financial, payments, customer, accounting, and product usage data from 25+ enterprise systems for analytics, AI-driven insights, and executive reporting across 200+ finance, product, and business stakeholders.
- Engineered real-time ingestion and transformation pipelines using Apache Kafka, Spark Structured Streaming, AWS Lambda, and Apache Airflow to process customer transactions, payment events, tax filings, and product activity, reducing end-to-end latency from 95 minutes to under 18 minutes while handling 15M+ financial events daily.
- Implemented vector-based knowledge retrieval using embedding models, metadata filtering, and RAG to enable secure enterprise search across structured and unstructured datasets, with prompt evaluation, hallucination mitigation, and response monitoring for reliable AI-powered applications.
- Partnered with AI Platform, Data Science, and Product Engineering to develop an AI-powered financial knowledge assistant using OpenAI, LangChain, RAG, and AWS Bedrock, enabling semantic search across tax documentation, accounting policies, engineering runbooks, and data catalogs, reducing incident resolution time from 3 hours to under 70 minutes.
- Built feature engineering pipelines and ML-ready datasets using Python, MLflow, Databricks Feature Store, and Scikit-learn, preparing 45M+ historical customer and financial records to support fraud detection, customer segmentation, financial forecasting, and personalization initiatives.
- Optimized cloud infrastructure and deployment workflows using Terraform, Docker, Kubernetes, GitHub Actions, and CI/CD, automating environment provisioning and secure platform releases, reducing deployment time from 6 hours to under 40 minutes across development, staging, and production environments.