I'm Priyam Choksi

Data Engineer & AI Engineer with five-plus years modernizing legacy, high-stakes data systems into reliable, AI-ready platforms across finance, healthcare, and enterprise.

Open to remote or onsite anywhere in the US. Core stack: Python, SQL, Spark, dbt, Airflow, Snowflake, AWS, LangGraph
Priyam Choksi

$1M+ in recovered revenue and cost savings, five peer-reviewed papers on datasets up to 133M rows, and a Harvard research pipeline made 192× faster.

HARVARD MEDICAL SCHOOL · BRIGHAM AND WOMEN'S HOSPITAL · NORTHEASTERN UNIVERSITY · INTUIT · HEEVA INFRA · CODENEST SOLUTION
Tech stack
Python
SQL
Spark
Databricks
Airflow
dbt
Kafka
Snowflake
Redshift
BigQuery
AWS
Terraform
Docker
PyTorch
LangChain
Tableau
EXPERIENCE
DRAWN AS A LINEAGE GRAPH · DRAG ANY NODE, THE EDGES FOLLOW · MARKS ARE GENERATIVE, CLICK TO REDRAW FULL DETAIL →
2019 – 2021ETL / BI

Data Engineer

CODENEST SOLUTION
ETL pipelines across ERP, CRM, and REST APIs, processing 750GB+ of business data weekly.
750GB/wkprocessed
80+reports
2.8h→1.6hETL runtime
2021 – 2023DE → SENIOR

Senior Data Engineer

HEEVA INFRA
Modernized a Postgres and Excel stack into a dbt and Airflow platform; promoted within the first year.
$894Krecovered
17.5M/motx, <5m fresh
12→2failures
2023 – 2025GRAD SCHOOL

M.S. Information Systems

NORTHEASTERN UNIVERSITY
GPA 3.7, Boston.
SEP – DEC 2024CLINICAL RESEARCH

Research Data Analyst Co-op

HARVARD MEDICAL SCHOOL
Genomic pipeline and ensemble models behind five peer-reviewed publications.
48h→15mpipeline
5.8TB133M rows
96%AUC
2025TEACHING

Graduate Teaching Assistant

NORTHEASTERN UNIVERSITY
Graduate coursework in business process engineering; 150+ students.
FEB 2026 →● CURRENT

Data Engineer

INTUIT
Lakehouse and real-time pipelines for financial, payments, and product analytics.
95m→18mlatency
15M+/dayevents
25+systems merged
ABOUT

A little bit about me

Who I am and what I do.

Who I am

I'm a Data Engineer with 5+ years building production data platforms and the LLM and agent systems that run on top of them, across finance, healthcare, and enterprise. I'm also a published medical-AI researcher with five peer-reviewed papers on risk-prediction modeling and generative imaging, and an ESC Young Investigator Award.

What I did

Most of my work has been a version of the same problem in different industries: a legacy system nobody trusts, feeding decisions that actually matter. At Brigham and Women's Hospital and Harvard Medical School, that meant re-architecting a genomic pipeline from 48-hour batch cycles to 15-minute distributed PySpark runs over 5.8TB of biobank, EHR, and metabolomic data. The pipeline and ensemble models I delivered there sit behind five publications and surfaced six novel biomarkers across a 133M-row dataset.

Before that, at an MEP engineering consulting firm, I replaced a Postgres-and-Excel reporting stack with version-controlled dbt and Airflow into Snowflake, re-engineering 217 ad hoc stored procedures into tested models and cutting deployment cycles from three days to four hours. I also built a real-time Kafka and Lambda pipeline moving 17.5M monthly transactions into Redshift, and tuned it to cut compute 30%. The ML cost-forecasting models I built there flagged budget overruns three weeks early and recovered $894K across a 12-client portfolio.

What I do

Today I own the cloud data platform at Intuit, architecting a Databricks, Delta Lake, and Snowflake lakehouse that consolidates 25+ enterprise systems for 200+ finance and product stakeholders, and engineering Kafka and Spark Structured Streaming pipelines that cut latency from 95 minutes to under 18 minutes across 15M+ daily financial events. I've also partnered with AI Platform and Data Science to ship a RAG-based financial knowledge assistant on OpenAI and AWS Bedrock, cutting incident resolution time from 3 hours to under 70 minutes.

The AI side is not a hobby project. I shipped an agentic triage system that resolves 74% of 2,000+ monthly tickets without human escalation, published a first-author paper on text-conditional chest X-ray generation using latent diffusion models, and built Trade Arena, a multi-agent LLM trading benchmark across 11 agents and seven models.

What I'm looking for

Data Engineer and AI Engineer roles in healthcare, fintech, or SaaS where I own the data platform and build the AI systems on top of it. Open to remote or onsite anywhere in the US.

Tech stack, in full
Languages

Python (PySpark, FastAPI, pandas), SQL, Bash

Data Engineering

Spark, Databricks, Airflow, dbt, Kafka, Delta Lake, Parquet, ETL/ELT, Kimball / star schema, Great Expectations, Pydantic

Databases & Warehouses

Snowflake, Amazon Redshift, BigQuery, PostgreSQL, MySQL, MongoDB, Oracle, Apache Hive

Cloud & DevOps

AWS, Azure, Terraform, Docker, Kubernetes, CI/CD (Jenkins, GitHub Actions)

ML

XGBoost, Random Forest, scikit-learn, PyTorch, MLflow, latent diffusion models

GenAI

LLMs, multi-agent systems, RAG (hybrid dense + BM25 + reranking), LangChain / LangGraph, FAISS, MCP

BI & Visualization

Tableau, Power BI, Looker, Jupyter

Tools

Git, JIRA, Confluence

Experience

Five roles, 2019 to now. The short version lives on the home page as a lineage graph; this is the full record.

FEB 2026 → PRESENT
BOSTON, MA

Data Engineer

INTUIT

Building the cloud data platform behind financial, payments, and product analytics. Architected a Databricks, Delta Lake, and Snowflake lakehouse consolidating 25+ enterprise systems, engineered real-time Kafka and Spark Structured Streaming pipelines that cut end-to-end latency from 95 minutes to under 18 minutes across 15M+ daily financial events, and built the RAG and feature engineering layers supporting enterprise search, fraud detection, and forecasting.

  • Architected a cloud-native Lakehouse and Snowflake analytics platform using Databricks, PySpark, Delta Lake, AWS S3, and AWS Glue, consolidating financial, payments, customer, accounting, and product usage data from 25+ enterprise systems for analytics, AI-driven insights, and executive reporting across 200+ finance, product, and business stakeholders.
  • Engineered real-time ingestion and transformation pipelines using Apache Kafka, Spark Structured Streaming, AWS Lambda, and Apache Airflow to process customer transactions, payment events, tax filings, and product activity, reducing end-to-end latency from 95 minutes to under 18 minutes while handling 15M+ financial events daily.
  • Implemented vector-based knowledge retrieval using embedding models, metadata filtering, and RAG to enable secure enterprise search across structured and unstructured datasets, with prompt evaluation, hallucination mitigation, and response monitoring for reliable AI-powered applications.
  • Partnered with AI Platform, Data Science, and Product Engineering to develop an AI-powered financial knowledge assistant using OpenAI, LangChain, RAG, and AWS Bedrock, enabling semantic search across tax documentation, accounting policies, engineering runbooks, and data catalogs, reducing incident resolution time from 3 hours to under 70 minutes.
  • Built feature engineering pipelines and ML-ready datasets using Python, MLflow, Databricks Feature Store, and Scikit-learn, preparing 45M+ historical customer and financial records to support fraud detection, customer segmentation, financial forecasting, and personalization initiatives.
  • Optimized cloud infrastructure and deployment workflows using Terraform, Docker, Kubernetes, GitHub Actions, and CI/CD, automating environment provisioning and secure platform releases, reducing deployment time from 6 hours to under 40 minutes across development, staging, and production environments.
2025
BOSTON, MA · HYBRID

Graduate Teaching Assistant

NORTHEASTERN UNIVERSITY

Supported graduate coursework in business process engineering across in-person and online cohorts, covering process modeling, workflow optimization, and operational analysis for 150+ students.

  • Facilitated process-mapping exercises so students could evaluate operational bottlenecks and document requirements, and introduced AI-assisted process analysis using prompt engineering and LLM workflows to summarize documentation and flag repetitive activities.
  • Coordinated assignment design, grading, and student feedback across cohorts, including industry guest speakers, and developed reusable feedback frameworks for a cohort distributed across multiple time zones.
SEP – DEC 2024
BOSTON, MA · ON-SITE

Research Data Analyst

BRIGHAM AND WOMEN'S HOSPITAL · HARVARD MEDICAL SCHOOL · INTERNSHIP

Top-ranked cardiovascular research hospital and Harvard Medical School affiliate, where scientists use genetic data from 500,000+ patients across three continents. Curated a 5.8TB multi-omics corpus down to a 300GB project-scoped dataset, cut its processing time from 48 hours to 15 minutes, delivered the pipeline and ensemble models (96% AUC) behind 5 peer-reviewed publications across AHA, EULAR, and ESC, and built a RAG assistant that cut manual data curation 80%.

  • Rebuilt a genomic pipeline from 48-hour batch cycles to 15-minute distributed runs, replacing a sequential CSV-and-notebook workflow with a PySpark pipeline on a Slurm HPC cluster integrating 5.8TB of UK Biobank, MGB EHR, and metabolomic data, with Python schema validation catching integrity failures across 133M rows before publication.
  • Built a curated egress layer consolidating 220+ clinical features from 6 heterogeneous sources into analytics-ready tables, cutting weekly research prep 85%, and migrated 2TB+ from CSV and Excel to Parquet, reducing storage 60% and query latency 75%.
  • Designed and built an AI-powered RAG chat assistant from a legacy workflow, cutting manual data curation effort 80% and giving clinicians and cross-functional teams plain-language access to clinical and genomic findings.
  • Delivered the data pipeline and ensemble models (XGBoost and Random Forest, 96% AUC) behind 5 peer-reviewed publications and an ESC Young Investigator Award, surfacing 6 novel biomarkers for clinical-trial stratification across a 48,628-participant, 133M-row dataset.
  • Developed HIPAA-compliant clinical data pipelines in SQL, Python, Pandas, and REDCap integrating 500,000+ patient observations across patient, laboratory, and study datasets for principal investigators and clinical researchers.
  • Designed self-serve Tableau and Looker dashboards for 3 clinical teams, cutting ad-hoc reporting requests 70%.
MAY 2021 – JUL 2023 · 2 YRS 3 MOS
MUMBAI · HYBRID

Senior Data Engineer

HEEVA INFRA
DATA ENGINEER MAY – DEC 2021 → PROMOTED TO SENIOR JAN 2022

MEP engineering consulting firm delivering multi-quarter data engagements across hospital and institutional clients. Modernized a legacy Postgres and Excel reporting stack into a version-controlled dbt and Airflow platform, engineered a real-time Kafka and AWS Lambda pipeline handling 17.5M monthly transactions with sub-5-minute data freshness, and recovered $894K across a 12-client portfolio with ML cost-forecasting models that flagged budget overruns 3 weeks early.

  • Modernized a legacy Postgres and Excel-based reporting stack into a version-controlled dbt and Airflow platform, replacing 217 ad hoc stored procedures with staged, tested models and an ETL pipeline from SAP into Snowflake, cutting deployment cycles from 3 days to 4 hours.
  • Engineered a real-time Kafka and AWS Lambda pipeline ingesting 17.5M monthly transactions into Amazon Redshift with sub-5-minute data freshness, then tuned sort keys and distribution styles across 8 high-volume tables to cut compute 30% ($22K/year) and query time 45%.
  • Modeled a Kimball star-schema warehouse across 18 Postgres, MySQL, and REST/gRPC sources and automated 50+ dbt models in Airflow with Great Expectations checks, cutting pipeline failures from 12 to 2 and retiring Excel reporting for 3 departments.
  • Recovered $894K across a 12-client portfolio and rescued 2 accounts from escalation, by building Python machine-learning cost-forecasting models that flagged 7 engagements 3 weeks before budget overruns.
  • Collaborated with software engineers, QA analysts, and business analysts in an Agile/Scrum environment to develop reusable ETL components, maintain source-to-target mappings, and support CI/CD deployments, contributing to the successful delivery of more than 25 production releases across enterprise projects.
AUG 2019 – APR 2021
MUMBAI · ON-SITE

Data Engineer

CODENEST SOLUTION

Built scalable ETL pipelines across ERP, CRM, and REST API sources for multiple enterprise client engagements, processing 750GB+ of business data weekly, developing dimensional models supporting 80+ reports, and tuning pipeline performance to cut ETL runtime from 2.8 hours to 1.6 hours.

  • Built scalable ETL pipelines using Python, SQL, Talend, Apache Spark, and MySQL to ingest and transform data from ERP systems, CRM platforms, and REST APIs, processing over 750GB of business data each week across multiple enterprise client engagements.
  • Developed dimensional data models and optimized SQL transformation workflows for sales, customer, and finance datasets, reducing dashboard refresh time from 55 minutes to under 20 minutes while supporting more than 80 operational and executive reports.
  • Integrated structured and semi-structured data from relational databases, flat files, and third-party APIs using Apache NiFi and batch processing frameworks to automate ingestion across 15+ production data sources, improving pipeline reliability across client implementations.
  • Implemented automated data validation and reconciliation processes using Python, SQL, Excel, and testing frameworks to validate over 1.2 million records during each production cycle, improving data accuracy and reducing downstream reporting issues for analytics teams.
  • Enhanced pipeline performance using SQL optimization, indexing strategies, partitioning, and Spark execution tuning, reducing ETL runtime from 2.8 hours to 1.6 hours while consistently meeting client reporting SLAs and improving data availability for business intelligence teams.

Projects

Eight selected builds. Each figure below is drawn in ink from the project's own numbers.

Multi-agent trading platform · Live

Trade Arena

  • An 11-agent analysis pipeline in five phases: technical, news, fundamental, and macro analysts; bull and bear researchers; debate synthesis; three risk evaluators; and a risk manager with veto power, turning market data into risk-adjusted BUY / HOLD / REJECT calls in about 3 minutes per ticker across 20 tickers and 35,000+ workflow outputs.
  • Underneath the agents, a real data platform: 500K+ daily records from five sources (Yahoo Finance, Finnhub, NewsAPI, SEC EDGAR, Reddit) streamed through Kafka into Snowflake, 10K+ SEC filings indexed in FAISS for retrieval, and 60+ technical indicators modeled in dbt for backtests.
  • Strategy Arena: four game-theoretic strategies competing for a shared $1M capital pool over 1,800 tournament rounds with z-score reallocation after every round, real strategic interdependence rather than isolated backtests. The LLM-driven Signal Follower returned +14.93% at 73.2% signal accuracy, +2.48% alpha over buy-and-hold.
  • LLM Arena: seven models given identical capital, data, and prompts for 383 rounds. Llama-3.3-70B won at +32.63%, edging Qwen3-32B at +32.19%, with trade counts ranging from 25 to 1,408 across models.
  • Key finding: no strategy dominates all conditions, market regime matters more than strategy selection; mean reversion won bear markets while signal-following won bulls.
SPARK · SNOWFLAKE · KAFKA · DBT · AIRFLOW · LANGGRAPH · MCP · RAG · FAISS
Multi-agent RAG · Travel + banking domains

Agentic Customer Triage System

  • Routes customer queries into 27 categories at 91% accuracy, with reviewer fallback on low-confidence predictions, by chaining classifier and escalation agents behind a FastAPI service.
  • Resolved 74% of 2,000+ monthly tickets without human escalation, using a RAG agent that matches tickets to a knowledge base and returns sourced answers.
  • A triage agent routes each query to one of two domains: travel runs as a LangGraph state graph with flight, hotel, car-rental, and excursion sub-agents over Qdrant, where safe tools auto-execute and sensitive tools pause for explicit user confirmation; banking runs a three-agent chain (reformulation → ChromaDB retrieval → validation) scoring every answer's confidence 0–100 with source attribution.
  • Conversation memory via the LangGraph checkpointer, LangSmith tracing, Docker Compose packaging, and a documented AWS production design: services on EKS, an S3 bronze/silver/gold medallion layer, Airflow-scheduled embedding refresh, and ElastiCache session state.
PYTHON · LANGGRAPH · LANGCHAIN · FASTAPI · CLAUDE · OPENAI · QDRANT · CHROMADB · DOCKER
Agentic RAG · Enterprise IT support

Enterprise AI IT Support Assistant

  • Built a production-grade agentic RAG assistant on FastAPI, OpenAI, MongoDB, and React: a custom five-stage state graph (query rewrite → intent classification → answer generation → groundedness check → fallback routing) automating enterprise IT support end to end.
  • Chose hybrid TF-IDF + BM25 retrieval over a vector database, judging the ops overhead not worth it for a knowledge base of a few dozen articles, and made groundedness scoring a first-class signal to catch and reroute low-confidence answers before they reach the user.
  • Engineered a self-improving feedback framework that captured user ratings, retrieval confidence, latency, and failure patterns to generate actionable recommendations for continuous optimization of AI system performance.
PYTHON · FASTAPI · OPENAI · MONGODB · REACT · TF-IDF · BM25
LLM fine-tuning · Text-to-SQL

Text-to-SQL LLM Fine-Tuning

  • Fine-tuned Mistral-7B, LLaMA-3-8B, and Phi-3-mini with LoRA and Unsloth on the Gretel.ai synthetic text-to-SQL dataset, keeping trainable parameters under 1% of each base model.
  • Lifted exact-match SQL accuracy on held-out queries from 0.05–0.20 to 0.35–0.40 across all three models, with LLaMA-3-8B improving the most and Phi-3-mini reaching perfect syntax validity post-tuning, measured with RAGAS semantic-equivalence scoring and SQLGlot syntax validation.
  • Developed an interactive Streamlit application to compare model outputs in real time, enabling side-by-side benchmarking and analysis of text-to-SQL generation performance.
PYTHON · HUGGING FACE · UNSLOTH · LORA · RAGAS · SQLGLOT · STREAMLIT
Generative medical imaging · First-author publication

Medical Text-to-Image Generation via Latent Diffusion

  • A 151.8M-parameter latent diffusion system: a 26.2M VAE compressing X-rays into an 8-channel 32×32 latent space at 0.11 reconstruction MSE, a BioBERT text encoder (108.9M parameters, 593K trainable), and a 39.7M UNet with cross-attention at 8×8, 16×16, and 32×32 resolutions.
  • Two-stage training on ~7,000 Indiana University chest X-rays with paired radiological reports: 200 VAE epochs, then 480 diffusion epochs with classifier-free guidance (10% null conditioning) and DDIM sampling at inference.
  • Results: SSIM 0.82, PSNR 22.3 dB, FID 18.7, ahead of GAN, VQGAN+CLIP, and pixel-diffusion baselines in the comparison; 256×256 generation in 0.66 seconds at 20 steps on a mid-range GPU.
  • Ships as a Streamlit app with generator, dataset explorer, model dashboard, and a radiological enhancement pipeline (windowing, CLAHE, unsharp masking, film-style presets). Published as first author.
PYTHON · PYTORCH · STREAMLIT · BIOBERT · HUGGING FACE DIFFUSERS
Research infrastructure · BWH / Harvard Medical School

Multi-omics Data Processing Pipeline

  • The problem: researchers waited 48–72 hours per run. A sequential bash-and-text-file pipeline over 5TB+ of multi-omics data (RNA sequencing, protein expression, metabolomics, clinical metadata) processed one sample at a time and requested 500GB of memory for 50GB jobs, so most of the wait was cluster queue, not computation.
  • The rebuild: PySpark partitioned by sample ID for parallel processing, Parquet in place of text files (60% smaller, columnar), right-sized resource requests after profiling actual usage, and Jupyter notebooks so researchers could test locally before submitting to the ERIS HPC cluster.
  • The results: processing 48–72h → 15 minutes including all joins and statistics; queue waits 2–6h → 5–10 minutes; storage −60%; researcher throughput from 2–3 analyses per week to 10–15.
  • The real lesson: this wasn't a big-data problem, it was a badly organized data problem. The cluster had the power; the pipeline wasn't using it. Interface kept simple and documented so the team could maintain it after the internship. Implementation details remain confidential per BWH/HMS policy.
PYTHON · PYSPARK · APACHE SPARK · PARQUET · SLURM · JUPYTER · BASH
Data engineering · AdventureWorks, end to end

Data Warehouse Integration & Business Intelligence

  • Four regional source systems on four platforms (SQL Server US, MySQL EU, PostgreSQL AU, Oracle CA) consolidated into a central warehouse, AdventureWorksDW_neu, built in parallel on MySQL as the Talend target and SQL Server as the SSIS target.
  • Parallel ETL pipelines in both Talend and SSIS: extraction with error handling at each step, business-rule transformation and cross-region standardization, and bulk loads with indexing tuned for analytical query patterns.
  • Star schema for sales and purchasing analysis: FactInternetSales, FactStoreSales, and FactPurchases over dimensions for store, customer, currency, promotion, and sales territory, with Type 1 slowly changing dimensions and reject tables feeding reconciliation reports.
  • BI layer on top: materialized views for hot query patterns, vendor ranking by purchase volume, and sales performance by territory, product, and channel, structured for Power BI consumption.
SQL SERVER · MYSQL · POSTGRESQL · ORACLE · TALEND · SSIS · POWER BI
Machine learning · Capstone grown into a platform

Diabetes Prediction & Analysis Platform

  • Prediction engine: logistic regression at 95.97% accuracy and 0.9587 ROC-AUC on a 100,000-record clinical dataset (glucose, BMI, HbA1c, age, lifestyle factors), with StandardScaler and OneHotEncoder preprocessing and L2 regularization against overfitting.
  • Evaluated five algorithms and wrote the comparison up as a research paper in the repo: SVM and random forest matched or edged the accuracy but were harder to explain, XGBoost's overhead wasn't justified, and logistic regression won on interpretability, which is what matters in a medical context.
  • Streamlit platform with separate prediction, EDA, and educational modules: real-time predictions with visual feedback, cached model loads, and session-state management to stay responsive with multiple visualizations running.
  • Ships with input validation for medical ranges, error handling, and clear disclaimers. A 96% model users can understand beats a 98% black box.
PYTHON · STREAMLIT · SCIKIT-LEARN · PANDAS · NUMPY · PLOTLY
MORE ON GITHUB ↗

Research

Five publications in cardiovascular and medical AI research. Figures are stylized in the site's ink language; the numbers on them are real.

★ ESC YOUNG INVESTIGATOR AWARD · PREVENTIVE CARDIOLOGY 2025
2026 · AMERICAN JOURNAL OF PREVENTIVE CARDIOLOGY

Use of an AI metabolomics score to predict cardiovascular risk among patients with psoriasis in the MGB and UK Biobanks

KOTANIDIS CP, SCHUERMANS A, CHOKSI P, ET AL.

An AI-derived plasma metabolomics score stratifying cardiovascular risk in psoriasis patients, validated across the Mass General Brigham and UK Biobank cohorts. Recognized with the ESC Young Investigator Award in Preventive Cardiology, 2025.

2026 · MEDRXIV · PREPRINT

Plasma metabolite associations with incident heart failure with reduced and preserved ejection fraction

CHEBROLU B, CHOKSI P, ET AL.

Maps plasma metabolite associations with incident heart failure separately for reduced and preserved ejection fraction, showing which markers track which phenotype. Preprint on medRxiv, 2026.

2025 · CIRCULATION · AHA SCIENTIFIC SESSIONS

Plasma metabolomics and machine learning identify causal metabolic contributors to incident heart failure

CHEBROLU B, CHOKSI P, ET AL.

Machine learning over plasma metabolomics in 38,628 MGB Biobank participants to isolate causal metabolic contributors to incident heart failure. Presented at AHA Scientific Sessions 2025 and published in Circulation.

2025 · ANNALS OF THE RHEUMATIC DISEASES · EULAR

Plasma metabolomics and incident major adverse cardiovascular events in patients with rheumatoid arthritis

KOTANIDIS CP, CHOKSI P, ET AL.

Metabolomic signatures predicting major adverse cardiovascular events in patients with rheumatoid arthritis. Published in Annals of the Rheumatic Diseases and presented at EULAR 2025.

FIRST AUTHOR
2025 · PREPRINTS

A resource-efficient approach to text-conditional chest X-ray generation using latent diffusion models

CHOKSI P

A 151M-parameter latent diffusion model for text-conditional chest X-ray generation, trained on 7,000 images with consumer-grade resources: VAE with 8× compression, BioBERT conditioning through cross-attention, 480 epochs to 0.11 reconstruction MSE. Solo first-author work.

READING

Bookshelf

Books and pieces of wisdom I've enjoyed reading. Tap any cover for what I actually thought of it.

Currently reading
The shelf