I'm Priyam Choksi

Data Engineer & AI Engineer with four-plus years modernizing legacy, high-stakes data systems into reliable, AI-ready platforms across healthcare, finance, and consulting.

Open to remote or onsite anywhere in the US. Core stack: Python, SQL, Spark, dbt, Airflow, Snowflake, AWS, LangGraph
About me Contact me
Priyam Choksi

$1M+ in recovered revenue and cost savings, five peer-reviewed papers on datasets up to 133M rows, and a Harvard research pipeline made 192× faster.

HARVARD MEDICAL SCHOOL · BRIGHAM AND WOMEN'S HOSPITAL · NORTHEASTERN UNIVERSITY · REBECCA EVERLENE TRUST · HEEVA INFRA · FASTTRACK SOFTWARE
Tech stack
Python
SQL
Spark
Databricks
Airflow
dbt
Kafka
Snowflake
Redshift
BigQuery
AWS
Terraform
Docker
PyTorch
LangChain
Tableau
EXPERIENCE
DRAWN AS A LINEAGE GRAPH · DRAG ANY NODE, THE EDGES FOLLOW · MARKS ARE GENERATIVE, CLICK TO REDRAW FULL DETAIL →
2020 – 2021ECOM / ERP

Data Analyst

FASTTRACK SOFTWARE
Analytics and ML for eCommerce, ERP, and CRM platforms moving 2.5M+ monthly transactions.
18%→3%stockouts
+$12K/morecovered
$500K+MRR kickoffs
2021 – 2023DE → SENIOR

Senior Data Engineer

HEEVA INFRA
Modernized a Postgres and Excel stack into a dbt and Airflow platform; promoted within the first year.
$894Krecovered
2.5M/motx <200ms
12→2failures
2023 – 2025GRAD SCHOOL

M.S. Information Systems

NORTHEASTERN UNIVERSITY
GPA 3.7, Boston.
SEP – DEC 2024CLINICAL RESEARCH

Research Data Analyst Co-op

HARVARD MEDICAL SCHOOL
Genomic pipeline and ensemble models behind five peer-reviewed publications.
48h→15mpipeline
5.8TB133M rows
96%AUC
APR – DEC 2025TEACHING

Graduate Teaching Assistant

NORTHEASTERN UNIVERSITY
Two graduate Business Process Engineering courses (INFO 7374, 7260); 150+ students.
FEB 2026 →● CURRENT

Data Engineer

REBECCA EVERLENE TRUST
Financial-aid data platform serving 453 partner institutions.
92%RAG accuracy
5TB+→ Snowflake
−40%errors
ExperienceFive roles in full detail · 2020 → now→ ProjectsSix selected · Trade Arena, agentic triage, medical diffusion, more→ ResearchFive publications · ESC Young Investigator Award→
ABOUT

A little bit about me

Who I am and what I do.

Who I am

I'm a Data Engineer with 4+ years building production data platforms and the LLM and agent systems that run on top of them, across healthcare, finance, and SaaS. I'm also a published medical-AI researcher with five peer-reviewed papers on risk-prediction modeling and generative imaging, and an ESC Young Investigator Award.

What I did

Most of my work has been a version of the same problem in different industries: a legacy system nobody trusts, feeding decisions that actually matter. At Brigham and Women's Hospital and Harvard Medical School, that meant re-architecting a genomic pipeline from 48-hour batch cycles to 15-minute distributed PySpark runs over 5.8TB of biobank, EHR, and metabolomic data. The pipeline and ensemble models I delivered there sit behind five publications and surfaced six novel biomarkers across a 133M-row dataset.

Before that, at an MEP engineering consulting firm, I replaced a Postgres-and-Excel reporting stack with version-controlled dbt and Airflow into Snowflake, re-engineering 217 ad hoc stored procedures into tested models and cutting deployment cycles from three days to four hours. I also built a real-time Kafka and Lambda pipeline moving 2.5M monthly transactions into Redshift, and tuned it to cut compute 30%. The ML cost-forecasting models I built there flagged budget overruns three weeks early and recovered $894K across a 12-client portfolio.

What I do

Today I own the ingestion platform at Rebecca Everlene Trust Company, consolidating 5TB+ of financial-aid data from 453 institutional sources into Snowflake, with a Pydantic validation gate in CI that cut system-wide transformation errors 40% and a hybrid-retrieval RAG system answering staff policy questions across 1,000+ documents at 92% accuracy.

The AI side is not a hobby project. I shipped an agentic triage system that resolves 74% of 2,000+ monthly tickets without human escalation, published a first-author paper on text-conditional chest X-ray generation using latent diffusion models, and built Trade Arena, a multi-agent LLM trading benchmark across 11 agents and seven models.

What I'm looking for

Data Engineer and AI Engineer roles in healthcare, fintech, or SaaS where I own the data platform and build the AI systems on top of it. Open to remote or onsite anywhere in the US.

Priyam Choksi Get in touch
Tech stack, in full
Languages

Python (PySpark, FastAPI, pandas), SQL, Bash

Data Engineering

Spark, Databricks, Airflow, dbt, Kafka, Delta Lake, Parquet, ETL/ELT, Kimball / star schema, Great Expectations, Pydantic

Databases & Warehouses

Snowflake, Amazon Redshift, BigQuery, PostgreSQL, MySQL, MongoDB, Oracle, Apache Hive

Cloud & DevOps

AWS, Azure, Terraform, Docker, Kubernetes, CI/CD (Jenkins, GitHub Actions)

ML

XGBoost, Random Forest, scikit-learn, PyTorch, MLflow, latent diffusion models

GenAI

LLMs, multi-agent systems, RAG (hybrid dense + BM25 + reranking), LangChain / LangGraph, FAISS, MCP

BI & Visualization

Tableau, Power BI, Looker, Jupyter

Tools

Git, JIRA, Confluence

Experience

Five roles, 2020 to now. The short version lives on the home page as a lineage graph; this is the full record.

FEB 2026 → PRESENT
CHICAGO, IL · REMOTE

Data Engineer

REBECCA EVERLENE TRUST COMPANY

Nonprofit platform helping students and young adults access education and financial opportunities; building the data foundation for the ASPECTS higher-education platform across 453 partner institutions. Consolidated 5TB+ of financial-aid data into a single Snowflake platform, added a Pydantic validation layer in CI that cut transformation errors 40%, and shipped a hybrid-retrieval RAG system over 1,000+ policy documents at 92% accuracy.

  • Built a Databricks and Spark pipeline consolidating 5TB+ of financial-aid data from 453 institutional sources into Snowflake, replacing fragmented per-institution feeds with one trusted source and using Delta Lake staging to absorb schema drift before transformation into analytics marts.
  • Reduced system-wide transformation errors 40% by adding a Pydantic validation layer to a Jenkins and GitHub Actions CI pipeline that rejects type mismatches and null violations before data reaches Snowflake.
  • Designed a hybrid RAG pipeline (dense vectors plus BM25 with reranking) on multi-region AWS that answers questions across 1,000+ financial-aid policy documents at 92% accuracy, provisioned with Terraform.
APR – DEC 2025 · 9 MOS
BOSTON, MA · HYBRID

Graduate Teaching Assistant

NORTHEASTERN UNIVERSITY
SEP – DEC 2025 · INFO 7374 · ADVANCED BUSINESS PROCESS ENGINEERING · PROF. SHANNON PETTIFORD
  • Supported a graduate course in advanced business process engineering covering process modeling, workflow optimization, and operational analysis: assignment design, grading, student feedback, and coordinating guest speakers from industry.
APR – SEP 2025 · INFO 7260 · BUSINESS PROCESS ENGINEERING (ONLINE)
  • Delivered teaching support for an online graduate cohort distributed across multiple time zones: student communications, grading analytical assignments, and developing reusable feedback frameworks.
SEP – DEC 2024
BOSTON, MA · ON-SITE

Research Data Analyst

BRIGHAM AND WOMEN'S HOSPITAL · HARVARD MEDICAL SCHOOL · INTERNSHIP

Top-ranked cardiovascular research hospital and Harvard Medical School affiliate, where scientists use genetic data from 500,000+ patients across three continents. Curated a 5.8TB multi-omics corpus down to a 300GB project-scoped dataset, cut its processing time from 48 hours to 15 minutes, delivered the pipeline and ensemble models (96% AUC) behind 5 peer-reviewed publications across AHA, EULAR, and ESC, and built a RAG assistant that cut manual data curation 80%.

  • Rebuilt a genomic pipeline from 48-hour batch cycles to 15-minute distributed runs, replacing a sequential CSV-and-notebook workflow with a PySpark pipeline on a Slurm HPC cluster integrating 5.8TB of UK Biobank, MGB EHR, and metabolomic data, with Python schema validation catching integrity failures across 27M EHR rows before publication.
  • Built a curated egress layer consolidating 220+ clinical features from 6 heterogeneous sources into analytics-ready tables, cutting weekly research prep 85%, and migrated 2TB+ from CSV and Excel to Parquet, reducing storage 60% and query latency 75%.
  • Designed and built an AI-powered RAG chat assistant from a legacy workflow, cutting manual data curation effort 80% and giving clinicians and cross-functional teams plain-language access to clinical and genomic findings.
  • Delivered the data pipeline and ensemble models (XGBoost and Random Forest, 96% AUC) behind 5 peer-reviewed publications and an ESC Young Investigator Award, surfacing 6 novel biomarkers for clinical-trial stratification across a 48,628-participant, 133M-row dataset.
  • Designed self-serve Tableau and Looker dashboards for 3 clinical teams, cutting ad-hoc reporting requests 70%.
MAY 2021 – JUL 2023 · 2 YRS 3 MOS
MUMBAI · HYBRID

Senior Data Engineer

HEEVA INFRA
DATA ENGINEER MAY – DEC 2021 → PROMOTED TO SENIOR JAN 2022

MEP engineering consulting firm delivering multi-quarter data engagements across hospital and institutional clients. Modernized a legacy Postgres and Excel reporting stack into a version-controlled dbt and Airflow platform, engineered a real-time Kafka and AWS Lambda pipeline handling 2.5M monthly transactions at sub-200ms latency, and recovered $894K across a 12-client portfolio with ML cost-forecasting models that flagged budget overruns 3 weeks early.

  • Modernized a legacy Postgres and Excel-based reporting stack into a version-controlled dbt and Airflow platform, replacing 217 ad hoc stored procedures with staged, tested models and an ETL pipeline from SAP into Snowflake, cutting deployment cycles from 3 days to 4 hours.
  • Engineered a real-time Kafka and AWS Lambda pipeline ingesting 2.5M monthly transactions into Redshift at sub-200ms latency, then tuned sort keys and distribution styles across 8 high-volume tables to cut compute 30% ($22K/year) and query time 45%.
  • Modeled a Kimball star-schema warehouse across 18 Postgres, MySQL, and REST/gRPC sources and automated 50+ dbt models in Airflow with Great Expectations checks, cutting pipeline failures from 12 to 2 and retiring Excel reporting for 3 departments.
  • Recovered $894K across a 12-client portfolio and rescued 2 accounts from escalation, by building Python machine-learning cost-forecasting models that flagged 7 engagements 3 weeks before budget overruns.
MAY 2020 – APR 2021
MUMBAI · ON-SITE

Data Analyst

FASTTRACK SOFTWARE

Consumer e-commerce platform handling 2.5M+ monthly transactions across web and mobile, with category mix shifting through pandemic-era buying behavior.

  • Cut inventory stockouts from 18% to 3% across 35+ SKUs, giving planners a reliable demand signal in place of manual reorder guesses, with an LSTM demand-forecasting model served via a REST API.
  • Recovered an estimated $12K in monthly revenue by querying 2.5M+ monthly transactions in SQL, identifying a 15% drop-off at the payment confirmation step, tracing the issue to a tracking gap, and delivering a root-cause breakdown that engineering shipped within sprints.
  • Recommended a marketing retention budget reallocation toward the top 20% of customers driving 54% of monthly revenue, improving retention spend efficiency over 2 quarters, by segmenting 18 months of purchase history with RFM analysis in SQL.
  • Supported product analytics across 2 client kickoffs worth $500K+ in MRR, creating 4 presentations for cross-functional partners in product, engineering, and marketing across 3 parallel workstreams.

Projects

Six selected builds. Each figure below is drawn in ink from the project's own numbers.

Multi-agent trading platform · Live

Trade Arena

TRADEARENA.SITE ↗GITHUB ↗
  • An 11-agent analysis pipeline in five phases: technical, news, fundamental, and macro analysts; bull and bear researchers; debate synthesis; three risk evaluators; and a risk manager with veto power, turning market data into risk-adjusted BUY / HOLD / REJECT calls in about 3 minutes per ticker across 20 tickers and 35,000+ workflow outputs.
  • Underneath the agents, a real data platform: 500K+ daily records from five sources (Yahoo Finance, Finnhub, NewsAPI, SEC EDGAR, Reddit) streamed through Kafka into Snowflake, 10K+ SEC filings indexed in FAISS for retrieval, and 60+ technical indicators modeled in dbt for backtests.
  • Strategy Arena: four game-theoretic strategies competing for a shared $1M capital pool over 1,800 tournament rounds with z-score reallocation after every round, real strategic interdependence rather than isolated backtests. The LLM-driven Signal Follower returned +14.93% at 73.2% signal accuracy, +2.48% alpha over buy-and-hold.
  • LLM Arena: seven models given identical capital, data, and prompts for 383 rounds. Llama-3.3-70B won at +32.63%, edging Qwen3-32B at +32.19%, with trade counts ranging from 25 to 1,408 across models.
  • Key finding: no strategy dominates all conditions, market regime matters more than strategy selection; mean reversion won bear markets while signal-following won bulls.
SPARK · SNOWFLAKE · KAFKA · DBT · AIRFLOW · LANGGRAPH · MCP · RAG · FAISS
Multi-agent RAG · Travel + banking domains

Agentic Customer Triage System

GITHUB ↗
  • Routes customer queries into 27 categories at 91% accuracy, with reviewer fallback on low-confidence predictions, by chaining classifier and escalation agents behind a FastAPI service.
  • Resolved 74% of 2,000+ monthly tickets without human escalation, using a RAG agent that matches tickets to a knowledge base and returns sourced answers.
  • A triage agent routes each query to one of two domains: travel runs as a LangGraph state graph with flight, hotel, car-rental, and excursion sub-agents over Qdrant, where safe tools auto-execute and sensitive tools pause for explicit user confirmation; banking runs a three-agent chain (reformulation → ChromaDB retrieval → validation) scoring every answer's confidence 0–100 with source attribution.
  • Conversation memory via the LangGraph checkpointer, LangSmith tracing, Docker Compose packaging, and a documented AWS production design: services on EKS, an S3 bronze/silver/gold medallion layer, Airflow-scheduled embedding refresh, and ElastiCache session state.
PYTHON · LANGGRAPH · LANGCHAIN · FASTAPI · CLAUDE · OPENAI · QDRANT · CHROMADB · DOCKER
Generative medical imaging · First-author publication

Medical Text-to-Image Generation via Latent Diffusion

PAPER (DOI) ↗GITHUB ↗
  • A 151.8M-parameter latent diffusion system: a 26.2M VAE compressing X-rays into an 8-channel 32×32 latent space at 0.11 reconstruction MSE, a BioBERT text encoder (108.9M parameters, 593K trainable), and a 39.7M UNet with cross-attention at 8×8, 16×16, and 32×32 resolutions.
  • Two-stage training on ~7,000 Indiana University chest X-rays with paired radiological reports: 200 VAE epochs, then 480 diffusion epochs with classifier-free guidance (10% null conditioning) and DDIM sampling at inference.
  • Results: SSIM 0.82, PSNR 22.3 dB, FID 18.7, ahead of GAN, VQGAN+CLIP, and pixel-diffusion baselines in the comparison; 256×256 generation in 0.66 seconds at 20 steps on a mid-range GPU.
  • Ships as a Streamlit app with generator, dataset explorer, model dashboard, and a radiological enhancement pipeline (windowing, CLAHE, unsharp masking, film-style presets). Published as first author.
PYTHON · PYTORCH · STREAMLIT · BIOBERT · HUGGING FACE DIFFUSERS
Research infrastructure · BWH / Harvard Medical School

Multi-omics Data Processing Pipeline

  • The problem: researchers waited 48–72 hours per run. A sequential bash-and-text-file pipeline over 5TB+ of multi-omics data (RNA sequencing, protein expression, metabolomics, clinical metadata) processed one sample at a time and requested 500GB of memory for 50GB jobs, so most of the wait was cluster queue, not computation.
  • The rebuild: PySpark partitioned by sample ID for parallel processing, Parquet in place of text files (60% smaller, columnar), right-sized resource requests after profiling actual usage, and Jupyter notebooks so researchers could test locally before submitting to the ERIS HPC cluster.
  • The results: processing 48–72h → 15 minutes including all joins and statistics; queue waits 2–6h → 5–10 minutes; storage −60%; researcher throughput from 2–3 analyses per week to 10–15.
  • The real lesson: this wasn't a big-data problem, it was a badly organized data problem. The cluster had the power; the pipeline wasn't using it. Interface kept simple and documented so the team could maintain it after the internship. Implementation details remain confidential per BWH/HMS policy.
PYTHON · PYSPARK · APACHE SPARK · PARQUET · SLURM · JUPYTER · BASH
Data engineering · AdventureWorks, end to end

Data Warehouse Integration & Business Intelligence

GITHUB ↗
  • Four regional source systems on four platforms (SQL Server US, MySQL EU, PostgreSQL AU, Oracle CA) consolidated into a central warehouse, AdventureWorksDW_neu, built in parallel on MySQL as the Talend target and SQL Server as the SSIS target.
  • Parallel ETL pipelines in both Talend and SSIS: extraction with error handling at each step, business-rule transformation and cross-region standardization, and bulk loads with indexing tuned for analytical query patterns.
  • Star schema for sales and purchasing analysis: FactInternetSales, FactStoreSales, and FactPurchases over dimensions for store, customer, currency, promotion, and sales territory, with Type 1 slowly changing dimensions and reject tables feeding reconciliation reports.
  • BI layer on top: materialized views for hot query patterns, vendor ranking by purchase volume, and sales performance by territory, product, and channel, structured for Power BI consumption.
SQL SERVER · MYSQL · POSTGRESQL · ORACLE · TALEND · SSIS · POWER BI
Machine learning · Capstone grown into a platform

Diabetes Prediction & Analysis Platform

LIVE APP ↗GITHUB ↗DEMO VIDEO ↗
  • Prediction engine: logistic regression at 95.97% accuracy and 0.9587 ROC-AUC on a 100,000-record clinical dataset (glucose, BMI, HbA1c, age, lifestyle factors), with StandardScaler and OneHotEncoder preprocessing and L2 regularization against overfitting.
  • Evaluated five algorithms and wrote the comparison up as a research paper in the repo: SVM and random forest matched or edged the accuracy but were harder to explain, XGBoost's overhead wasn't justified, and logistic regression won on interpretability, which is what matters in a medical context.
  • Streamlit platform with separate prediction, EDA, and educational modules: real-time predictions with visual feedback, cached model loads, and session-state management to stay responsive with multiple visualizations running.
  • Ships with input validation for medical ranges, error handling, and clear disclaimers. A 96% model users can understand beats a 98% black box.
PYTHON · STREAMLIT · SCIKIT-LEARN · PANDAS · NUMPY · PLOTLY
MORE ON GITHUB ↗

Research

Five publications in cardiovascular and medical AI research. Figures are stylized in the site's ink language; the numbers on them are real.

★ ESC YOUNG INVESTIGATOR AWARD · PREVENTIVE CARDIOLOGY 2025
2026 · AMERICAN JOURNAL OF PREVENTIVE CARDIOLOGY

Use of an AI metabolomics score to predict cardiovascular risk among patients with psoriasis in the MGB and UK Biobanks

KOTANIDIS CP, SCHUERMANS A, CHOKSI P, ET AL.

An AI-derived plasma metabolomics score stratifying cardiovascular risk in psoriasis patients, validated across the Mass General Brigham and UK Biobank cohorts. Recognized with the ESC Young Investigator Award in Preventive Cardiology, 2025.

DOI ↗
2026 · MEDRXIV · PREPRINT

Plasma metabolite associations with incident heart failure with reduced and preserved ejection fraction

CHEBROLU B, CHOKSI P, ET AL.

Maps plasma metabolite associations with incident heart failure separately for reduced and preserved ejection fraction, showing which markers track which phenotype. Preprint on medRxiv, 2026.

DOI ↗
2025 · CIRCULATION · AHA SCIENTIFIC SESSIONS

Plasma metabolomics and machine learning identify causal metabolic contributors to incident heart failure

CHEBROLU B, CHOKSI P, ET AL.

Machine learning over plasma metabolomics in 38,628 MGB Biobank participants to isolate causal metabolic contributors to incident heart failure. Presented at AHA Scientific Sessions 2025 and published in Circulation.

DOI ↗
2025 · ANNALS OF THE RHEUMATIC DISEASES · EULAR

Plasma metabolomics and incident major adverse cardiovascular events in patients with rheumatoid arthritis

KOTANIDIS CP, CHOKSI P, ET AL.

Metabolomic signatures predicting major adverse cardiovascular events in patients with rheumatoid arthritis. Published in Annals of the Rheumatic Diseases and presented at EULAR 2025.

DOI ↗
FIRST AUTHOR
2025 · PREPRINTS

A resource-efficient approach to text-conditional chest X-ray generation using latent diffusion models

CHOKSI P

A 151M-parameter latent diffusion model for text-conditional chest X-ray generation, trained on 7,000 images with consumer-grade resources: VAE with 8× compression, BioBERT conditioning through cross-attention, 480 epochs to 0.11 reconstruction MSE. Solo first-author work.

DOI ↗
READING

Bookshelf

Books and pieces of wisdom I've enjoyed reading. Tap any cover for what I actually thought of it.

Currently reading
The shelf