{ "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ "# Python for Data Science\n", "## A Hands-On Workshop for Biomedical Professionals\n", "\n", "**Course**: AI in Medicine — Summer Course 2026 \n", "**Date**: 27 July 2026 | 1400–1715 \n", "**Level**: Beginner to Intermediate \n", "**Environment**: Google Colab (no installation needed) \n", "**Dataset**: [Daily Live Births in Malaysia](https://data.gov.my/data-catalogue/births) (1920–2023)\n", "\n", "---\n", "### What We'll Build Today\n", "\n", "| # | Section | Duration |\n", "|----|---------|----------|\n", "| 1 | Quick Start — Colab & Python Refresher | ~15 min |\n", "| 2 | NumPy Essentials — Arrays & Statistics | ~20 min |\n", "| 3 | Pandas Fundamentals — DataFrames | ~40 min |\n", "| 4 | Data Cleaning — Real-World Data | ~15 min |\n", "| 5 | Basic Statistics — SciPy & Testing | ~15 min |\n", "| 6 | Data Visualization — Matplotlib & Seaborn | ~25 min |\n", "| 7 | Mini Project — Malaysia Births Analysis | ~50 min |\n", "| 8 | Exporting & Sharing Results | ~10 min |\n", "| 9 | Wrap-up — Next Steps | ~5 min |\n", "\n", "---\n", "### Why This Matters for You\n", "\n", "Every skill you learn today maps directly to biomedical workflows:\n", "\n", "- **NumPy** → Process 10,000 patient lab values in one line\n", "- **Pandas** → Clean, filter, and summarize clinical trial data\n", "- **Statistics** → Test if a treatment effect is real or just noise\n", "- **Visualization** → Create publication-ready figures for journals\n", "\n", "> **Rule #1**: Ask questions anytime. If you're lost, someone else probably is too.\n", "> **Rule #2**: You will leave with a working notebook. Every cell runs.\n", "\n", "---\n", "### Before We Start — Colab Quick Tips\n", "- Press **Shift+Enter** to run a cell and move to the next\n", "- Press **Ctrl+Enter** to run a cell and stay put\n", "- If Colab disconnects: **Runtime > Run all** (or re-run from top)\n", "- Save to your Drive: **File > Save a copy in Drive**" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "---\n", "## Section 1: Quick Start — Python Refresher\n", "\n", "You've seen Python before. Here's the 5-minute refresher of what you'll use today. \n", "**Skip if you're comfortable** — jump to the exercise at the bottom.\n", "\n", "### Python in 4 Building Blocks" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "# 1. LISTS — ordered, mutable collections\n", "# Think: a column of patient IDs or lab results\n", "glucose_readings = [98, 105, 87, 112, 94, 130, 88]\n", "print(f'All readings: {glucose_readings}')\n", "print(f'Count: {len(glucose_readings)} readings')\n", "print(f'First: {glucose_readings[0]}, Last: {glucose_readings[-1]}')\n", "\n", "# List comprehension — filter in one line\n", "high = [r for r in glucose_readings if r > 100]\n", "print(f'Readings above 100 mg/dL: {high} ({len(high)} of {len(glucose_readings)})')" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "# 2. DICTIONARIES — key-value pairs\n", "# Think: a single patient record\n", "patient = {\n", " 'id': 'P042',\n", " 'age': 58,\n", " 'glucose_fasting': [98, 105, 87],\n", " 'diagnosis': 'Type 2 Diabetes',\n", " 'on_metformin': True\n", "}\n", "print(f'Patient {patient[\"id\"]}: age {patient[\"age\"]}, {patient[\"diagnosis\"]}')\n", "\n", "# Loop through all fields\n", "for key, value in patient.items():\n", " print(f' {key:20s} → {value}')" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "# 3. FUNCTIONS — reusable logic\n", "# Think: BMI calculator you use on every patient\n", "def compute_bmi(weight_kg, height_m):\n", " return weight_kg / (height_m ** 2)\n", "\n", "def bmi_category(bmi):\n", " if bmi < 18.5:\n", " return 'Underweight'\n", " elif bmi < 25:\n", " return 'Normal'\n", " elif bmi < 30:\n", " return 'Overweight'\n", " else:\n", " return 'Obese'\n", "\n", "patients = [(70, 1.75), (85, 1.60), (60, 1.68), (95, 1.72)]\n", "for w, h in patients:\n", " bmi = compute_bmi(w, h)\n", " print(f' Weight={w}kg Height={h}m: BMI={bmi:.1f} ({bmi_category(bmi)})')" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "# 4. F-STRINGS — readable output\n", "name = 'Dr. Chen'\n", "n_patients = 342\n", "avg_glucose = 108.3\n", "print(f'{name} has {n_patients} patients with average fasting glucose of {avg_glucose:.1f} mg/dL')" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### ✏️ Your Turn (2 min)\n", "Write a function `is_hypertensive(systolic, diastolic)` that returns `True` if systolic ≥ 140 or diastolic ≥ 90. Test it on three synthetic patients." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "# Your solution here\n", "\n", "\n", "\n" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "# 💡 Solution (run this cell to see the answer)\n", "def is_hypertensive(systolic, diastolic):\n", " return systolic >= 140 or diastolic >= 90\n", "\n", "test_cases = [(138, 85, False), (145, 80, True), (120, 95, True)]\n", "for sys_val, dia, expected in test_cases:\n", " result = is_hypertensive(sys_val, dia)\n", " status = '✓' if result == expected else '✗'\n", " print(f' BP {sys_val}/{dia}: Hypertensive={result} {status}')" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "---\n", "### Import Our Core Libraries\n", "Colab has these pre-installed. Just run this cell and you're ready." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "import numpy as np\n", "import pandas as pd\n", "import matplotlib.pyplot as plt\n", "import seaborn as sns\n", "from scipy import stats\n", "\n", "sns.set_style('whitegrid')\n", "%matplotlib inline\n", "\n", "print('NumPy:', np.__version__)\n", "print('Pandas:', pd.__version__)\n", "print('Seaborn:', sns.__version__)\n", "print('All libraries loaded — ready!')" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "> 🎯 **Checkpoint**: You can create lists, dicts, functions, and imports. If anything above didn't make sense, ask now!" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "---\n", "## Section 2: NumPy Essentials\n", "\n", "NumPy gives us **arrays** — fast, vectorized containers for numerical data. \n", "\n", "**Biomedical analogy**: Imagine 10,000 fasting glucose results from a year of patient visits. With NumPy you compute mean, median, std, and percentiles — all in one line each. No loops. No Excel.\n", "\n", "### Creating & Inspecting Arrays" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "# From a Python list\n", "glucose = np.array([5.2, 4.8, 6.1, 5.5, 4.9, 5.0, 7.3, 4.6, 5.8, 5.1])\n", "print(f'Array: {glucose}')\n", "print(f'Shape: {glucose.shape}') # dimensions\n", "print(f'Dtype: {glucose.dtype}') # data type\n", "print(f'Size: {glucose.size} readings')" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "# Creating arrays directly\n", "zeros = np.zeros(5) # five zeros\n", "ones = np.ones(3) # three ones\n", "range_arr = np.arange(0, 100, 10) # 0, 10, 20, ..., 90\n", "random_arr = np.random.randn(5) # 5 random normal values\n", "\n", "print(f'Zeros: {zeros}')\n", "print(f'Range: {range_arr}')\n", "print(f'Random: {np.round(random_arr, 3)}')" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Vectorized Operations — No Loops!\n", "Operations apply to the entire array at once. This is why NumPy is fast." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "# Convert glucose from mmol/L to mg/dL (×18.018)\n", "glucose_mgdl = glucose * 18.018\n", "\n", "# Show side-by-side\n", "print(f'{\"mmol/L\":>10s} {\"mg/dL\":>10s}')\n", "print('-' * 25)\n", "for mmol, mgdl in zip(glucose, glucose_mgdl):\n", " print(f'{mmol:10.1f} {mgdl:10.1f}')" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Statistical Functions\n", "The bread and butter of biomedical data analysis." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "print('Descriptive Statistics for Glucose Readings (mmol/L)')\n", "print('=' * 50)\n", "print(f' Mean: {np.mean(glucose):6.2f}')\n", "print(f' Median: {np.median(glucose):6.2f}')\n", "print(f' Std Deviation: {np.std(glucose, ddof=1):6.2f} (ddof=1 for sample std)')\n", "print(f' Min / Max: {np.min(glucose):6.2f} / {np.max(glucose):6.2f}')\n", "print(f' Range: {np.ptp(glucose):6.2f}')\n", "print(f' 25th percentile: {np.percentile(glucose, 25):6.2f}')\n", "print(f' 75th percentile: {np.percentile(glucose, 75):6.2f}')\n", "print(f' IQR: {np.percentile(glucose, 75) - np.percentile(glucose, 25):6.2f}')" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Boolean Filtering\n", "Select values that meet a condition — like flagging abnormal lab results." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "# Normal fasting glucose: ≤ 5.5 mmol/L (≤ 99 mg/dL)\n", "normal = glucose <= 5.5\n", "elevated = glucose > 5.5\n", "\n", "print(f'Normal readings: {glucose[normal]}')\n", "print(f'Elevated readings: {glucose[elevated]}')\n", "print(f'Count elevated: {np.sum(elevated)} out of {len(glucose)}')\n", "print(f'Percentage elevated: {np.mean(elevated) * 100:.1f}%')" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### ✏️ Your Turn: Heart Rate Analysis (3 min)\n", "\n", "Create an array of 20 synthetic resting heart rates (bpm). Then:\n", "1. Compute mean, median, std\n", "2. What percentage are tachycardic (>100 bpm)?\n", "3. How many patients have bradycardia (<60 bpm)?" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "# Your solution here\n", "\n", "\n", "\n" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "# 💡 Solution\n", "np.random.seed(42) # for reproducible results\n", "hr = np.random.normal(72, 12, 20).astype(int) # mean=72, std=12\n", "hr = np.clip(hr, 45, 130) # clip to realistic range\n", "\n", "print(f'Heart rates: {hr}')\n", "print(f'\\nMean: {np.mean(hr):.0f} bpm')\n", "print(f'Median: {np.median(hr):.0f} bpm')\n", "print(f'Std: {np.std(hr, ddof=1):.0f} bpm')\n", "print(f'\\nTachycardic (>100): {np.sum(hr > 100)} patients ({np.mean(hr > 100)*100:.0f}%)')\n", "print(f'Bradycardic (<60): {np.sum(hr < 60)} patients ({np.mean(hr < 60)*100:.0f}%)')" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Simulating Clinical Data\n", "NumPy's random module is great for generating synthetic data — useful for teaching, testing, and power analysis." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "# Simulate a small clinical trial: 30 patients, two treatment arms\n", "np.random.seed(123)\n", "n = 15 # per group\n", "\n", "# Placebo group: baseline glucose ~6.0 mmol/L, noise ±0.8\n", "placebo = np.random.normal(6.0, 0.8, n)\n", "\n", "# Treatment group: reduced glucose ~5.2 mmol/L, noise ±0.7\n", "treatment = np.random.normal(5.2, 0.7, n)\n", "\n", "print(f'Placebo group: mean={np.mean(placebo):.2f}, std={np.std(placebo, ddof=1):.2f}')\n", "print(f'Treatment group: mean={np.mean(treatment):.2f}, std={np.std(treatment, ddof=1):.2f}')\n", "print(f'Difference: {np.mean(placebo) - np.mean(treatment):.2f} mmol/L reduction')" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "> 🤔 **Think**: Is a 0.8 mmol/L difference clinically meaningful? Statistics tells us if it's *significant* — clinical judgment tells us if it *matters*. We'll revisit this in Section 5.\n", "\n", "> 🎯 **NumPy Takeaway**: Arrays + vectorized operations + built-in stats = the foundation. In biomedical work, this is how you process lab values, vitals, and sensor data efficiently." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "---\n", "## Section 3: Pandas Fundamentals\n", "\n", "Pandas is the workhorse of data analysis. Its centerpiece: the **DataFrame** — a 2D table with named columns, like a programmable spreadsheet.\n", "\n", "**Biomedical analogy**: A DataFrame is your patient registry, lab database, or clinical trial dataset — all in one Python object.\n", "\n", "### Our Dataset: Daily Live Births in Malaysia\n", "- **Source**: National Registration Department (JPN) via [data.gov.my](https://data.gov.my)\n", "- **Columns**: `date`, `state`, `births`\n", "- **Range**: 1920 to 2023 (~38,000 rows)\n", "- **Why biomedical**: Birth demography drives public health planning, maternal health policy, hospital staffing." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "# Load directly from the web — no file download needed\n", "BIRTHS_URL = 'https://storage.data.gov.my/demography/births.csv'\n", "df = pd.read_csv(BIRTHS_URL)\n", "print(f'Loaded {len(df):,} rows × {len(df.columns)} columns')" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "# First look — always start here\n", "df.head(10)" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "# Structure & summary\n", "print('--- Column Info ---')\n", "df.info()\n", "print('\\n--- Summary Statistics ---')\n", "df.describe()" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "# Quick checks\n", "print(f'Shape: {df.shape}')\n", "print(f'Columns: {list(df.columns)}')\n", "print(f'Missing: {df.isnull().sum().sum()} values')\n", "print(f'Duplicates: {df.duplicated().sum()} rows')\n", "print(f'Date range: {df[\"date\"].min()} to {df[\"date\"].max()}')" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Working with Dates\n", "The `date` column is text — convert it to datetime so we can extract year, month, day of week, etc." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "# Convert to datetime & extract components\n", "df['date'] = pd.to_datetime(df['date'])\n", "df['year'] = df['date'].dt.year\n", "df['month'] = df['date'].dt.month\n", "df['day'] = df['date'].dt.day\n", "df['dayofweek'] = df['date'].dt.dayofweek # Mon=0, Sun=6\n", "df['day_name'] = df['date'].dt.day_name()\n", "df['month_name'] = df['date'].dt.month_name()\n", "df['is_weekend'] = df['dayofweek'] >= 5\n", "\n", "df.head()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Filtering Rows\n", "SQL-style `WHERE` clauses — pick rows that meet a condition." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "# After the year 2000\n", "recent = df[df['year'] >= 2000]\n", "print(f'Rows since 2000: {len(recent):,}')\n", "\n", "# Days with more than 2,000 births\n", "high_births = df[df['births'] > 2000]\n", "print(f'Days with >2000 births: {len(high_births)}')\n", "\n", "# Weekends only, since 1990\n", "weekends_modern = df[(df['is_weekend']) & (df['year'] >= 1990)]\n", "print(f'Weekend days since 1990: {len(weekends_modern):,}')" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Grouping & Aggregation\n", "This is where Pandas shines — split data into groups, compute summaries, combine results." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "# Total births by year\n", "yearly = df.groupby('year')['births'].sum().reset_index()\n", "yearly.head(10)" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "# Multiple aggregations at once — the .agg() method\n", "yearly_stats = df.groupby('year').agg(\n", " total_births=('births', 'sum'),\n", " avg_daily=('births', 'mean'),\n", " max_daily=('births', 'max'),\n", " min_daily=('births', 'min'),\n", " days_recorded=('births', 'count')\n", ").reset_index()\n", "yearly_stats.head()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Sorting\n", "Find the highest and lowest values." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "# Top 5 years by total births\n", "yearly_stats_sorted = yearly_stats.sort_values('total_births', ascending=False)\n", "yearly_stats_sorted.head()" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "# The 5 days with the MOST births\n", "df.sort_values('births', ascending=False).head()" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "# The 5 days with the FEWEST births\n", "df.sort_values('births', ascending=True).head()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### The `.apply()` Method\n", "Apply a custom function to every row or column — useful for classifications and transformations." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "# Classify each day's birth volume\n", "def classify_births(count):\n", " if count > 2000:\n", " return 'Very High'\n", " elif count > 1500:\n", " return 'High'\n", " elif count > 1000:\n", " return 'Moderate'\n", " elif count > 500:\n", " return 'Low'\n", " else:\n", " return 'Very Low'\n", "\n", "df['birth_category'] = df['births'].apply(classify_births)\n", "df['birth_category'].value_counts()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### ✏️ Your Turn (5 min)\n", "1. Find the single day with the most births — when was it?\n", "2. Find the single day with the fewest births — when was it?\n", "3. What month (on average) has the most births per day?" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "# Your solution here\n", "\n", "\n", "\n" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "# 💡 Solution\n", "# 1. Day with most births\n", "max_row = df.loc[df['births'].idxmax()]\n", "print(f'Most births: {int(max_row[\"births\"]):,} on {max_row[\"date\"].strftime(\"%d %B %Y\")}')\n", "\n", "# 2. Day with fewest births\n", "min_row = df.loc[df['births'].idxmin()]\n", "print(f'Fewest births: {int(min_row[\"births\"]):,} on {min_row[\"date\"].strftime(\"%d %B %Y\")}')\n", "\n", "# 3. Month with highest average daily births\n", "monthly_avg = df.groupby('month')['births'].mean()\n", "best_month = monthly_avg.idxmax()\n", "print(f'\\nHighest avg daily births: Month {best_month} ({monthly_avg[best_month]:.0f}/day)')\n", "print(f'Lowest avg daily births: Month {monthly_avg.idxmin()} ({monthly_avg.min():.0f}/day)')" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "> 🎯 **Pandas Takeaway**: `read_csv()` → `head()`/`info()`/`describe()` → datetime → filter → `groupby().agg()` → sort. This 7-step pipeline handles 80% of real-world data tasks." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "---\n", "## Section 4: Data Cleaning\n", "\n", "Before we go deeper, let's step back and ask: **how trustworthy is our births dataset?** Real-world data is messy — clinical databases have missing values, duplicate entries, and impossible values. Before any analysis, you must check for problems.\n", "\n", "> **Biomedical reality check**: A 2022 study found that 27% of electronic health record fields contain errors. Data cleaning is the difference between a valid finding and a retracted paper." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Step 1: Audit Our Births Data\n", "Let's check the quality of the dataset we've been working with. Is it clean, or are there hidden problems?" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "# Quality audit of our births data\n", "print('=== Births Data Quality Audit ===\\n')\n", "print(f'Rows: {len(df):,}')\n", "print(f'Columns: {len(df.columns)}')\n", "print(f'Missing values: {df.isnull().sum().sum()}')\n", "print(f'Duplicate rows: {df.duplicated().sum()}')\n", "print(f'\\nBirths range: {df[\"births\"].min():.0f} — {df[\"births\"].max():,.0f}')\n", "print(f'Date range: {df[\"date\"].min().date()} to {df[\"date\"].max().date()}')\n", "print(f'Unique dates: {df[\"date\"].nunique():,}')\n", "print(f'\\n✓ Verdict: This data is remarkably clean — no missing values, no duplicates!')" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Our births data is clean. But in your own work, data rarely arrives this way. So let's **deliberately mess up a copy** of our births data to practice cleaning techniques — without touching the original." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Step 2: Create a Messy Copy\n", "We'll introduce three common problems into a copy of our births data:\n", "1. **Missing values** — set some birth counts to NaN (as if a registration was lost)\n", "2. **Duplicates** — copy a few rows (as if someone double-entered data)\n", "3. **Outliers** — inflate a few birth counts to unrealistic levels (as if a typo added a digit)" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "# Work on a COPY — never modify the original df\n", "df_dirty = df.copy()\n", "\n", "# 1. Introduce missing values (set 5 random birth counts to NaN)\n", "np.random.seed(99)\n", "nan_idx = np.random.choice(df_dirty.index, size=5, replace=False)\n", "df_dirty.loc[nan_idx, 'births'] = np.nan\n", "\n", "# 2. Introduce duplicate rows (copy 3 random rows)\n", "dup_idx = np.random.choice(df_dirty.index, size=3, replace=False)\n", "dupes = df_dirty.loc[dup_idx].copy()\n", "df_dirty = pd.concat([df_dirty, dupes], ignore_index=True)\n", "\n", "# 3. Introduce outliers (multiply 3 random birth counts by 10 — pretend typo)\n", "out_idx = np.random.choice(df_dirty.index[:37000], size=3, replace=False)\n", "df_dirty.loc[out_idx, 'births'] = df_dirty.loc[out_idx, 'births'] * 10\n", "\n", "print(f'Original: {len(df):,} rows')\n", "print(f'Dirty: {len(df_dirty):,} rows (+3 duplicates)')\n", "print(f'Missing births: {df_dirty[\"births\"].isnull().sum()}')\n", "print(f'\\nDirty data sample:')\n", "df_dirty.head()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Step 3: Handle Missing Values\n", "Two strategies: drop rows with missing data, or fill them with a reasonable value." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "# Count missing values\n", "print(f'Rows with missing births: {df_dirty[\"births\"].isnull().sum()}')\n", "print(f'Rows with missing births: {df_dirty[\"births\"].isnull().sum()}')\n", "\n", "# Strategy A: Drop rows with missing births\n", "df_clean_a = df_dirty.dropna(subset=['births'])\n", "print(f'\\nStrategy A (dropna): {len(df_clean_a):,} rows remaining')\n", "\n", "# Strategy B: Fill missing birth counts with the median for that year\n", "df_clean_b = df_dirty.copy()\n", "df_clean_b['births'] = df_clean_b.groupby('year')['births'].transform(\n", " lambda x: x.fillna(x.median())\n", ")\n", "print(f'Strategy B (fill with yearly median): {len(df_clean_b):,} rows (all preserved)')\n", "print(f' Still missing: {df_clean_b[\"births\"].isnull().sum()}')" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "> 🤔 **Which strategy to use?** Drop rows when missing data is rare (<5%) and random. Fill values when you need to preserve sample size, or when missingness follows a pattern (e.g., certain years have gaps). For clinical data, consult a domain expert before filling." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Step 4: Detect & Remove Duplicates" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "# How many duplicate rows did we introduce?\n", "dup_count = df_clean_b.duplicated().sum()\n", "print(f'Duplicate rows found: {dup_count}')\n", "\n", "# Remove duplicates (keep first occurrence)\n", "df_clean_b = df_clean_b.drop_duplicates()\n", "print(f'After dedup: {len(df_clean_b):,} rows (should be close to original {len(df):,})')\n", "\n", "# Verify: duplicates gone?\n", "print(f'Duplicates remaining: {df_clean_b.duplicated().sum()}')" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Step 5: Detect Outliers with IQR\n", "The Interquartile Range (IQR) method: flag values beyond 1.5×IQR from Q1 or Q3.\n", "\n", "Remember — we multiplied 3 random birth counts by 10. Can we find them?" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "# IQR outlier detection on birth counts\n", "births = df_clean_b['births']\n", "Q1 = births.quantile(0.25)\n", "Q3 = births.quantile(0.75)\n", "IQR = Q3 - Q1\n", "lower = Q1 - 1.5 * IQR\n", "upper = Q3 + 1.5 * IQR\n", "\n", "outlier_mask = (births < lower) | (births > upper)\n", "outliers = df_clean_b[outlier_mask]\n", "\n", "print(f'Q1={Q1:.0f}, Q3={Q3:.0f}, IQR={IQR:.0f}')\n", "print(f'Normal range: [{lower:.0f}, {upper:.0f}] births/day')\n", "print(f'\\nOutliers detected: {len(outliers)} rows')\n", "print(f'\\nTop 5 outliers (should include our ×10 inflated values):')\n", "outliers.sort_values('births', ascending=False)[['date', 'births']].head()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "The outliers we injected should appear at the top with birth counts in the tens of thousands — clearly impossible for a single day in Malaysia. In real data analysis, you'd investigate: was this a data entry error? A reporting anomaly? Or a genuine extreme event?\n", "\n", "> 🎯 **Data Cleaning Takeaway**: Every dataset deserves a quick audit — `.isnull().sum()`, `.duplicated().sum()`, and an outlier scan. Our births data passed clean. When you encounter mess, you now know how to fix it — **on a copy**, never the original." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "---\n", "## Section 5: Basic Statistics with SciPy\n", "\n", "Descriptive statistics (NumPy) tell you *what* the data looks like. Inferential statistics (SciPy) tell you *whether patterns are real or just noise*.\n", "\n", "**Why this matters**: A drug that lowers glucose by 0.5 mmol/L — is that a real effect, or random chance? Only statistics can answer that." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "from scipy import stats" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Pearson Correlation\n", "Measures linear relationship between two variables. Range: -1 (perfect negative) to +1 (perfect positive). 0 = no linear correlation." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "# Is yearly total births correlated with average daily births?\n", "r, p = stats.pearsonr(yearly_stats['total_births'], yearly_stats['avg_daily'])\n", "print(f'Pearson r = {r:.4f}')\n", "print(f'p-value = {p:.4f}')\n", "print(f'Interpretation: {\"Strong\" if abs(r) > 0.7 else \"Moderate\" if abs(r) > 0.4 else \"Weak\"} correlation')" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "# Create a correlation matrix for the yearly stats\n", "corr_matrix = yearly_stats[['total_births', 'avg_daily', 'max_daily', 'min_daily']].corr()\n", "corr_matrix" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Independent T-Test\n", "Tests whether two groups have significantly different means.\n", "\n", "**The weekend effect**: Are weekend births genuinely lower than weekday births, or is the difference just random fluctuation?" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "# Split births into weekday vs weekend\n", "weekday_births = df[~df['is_weekend']]['births']\n", "weekend_births = df[df['is_weekend']]['births']\n", "\n", "t_stat, p_val = stats.ttest_ind(weekday_births, weekend_births)\n", "\n", "print(f'Weekday mean: {weekday_births.mean():.0f} births/day')\n", "print(f'Weekend mean: {weekend_births.mean():.0f} births/day')\n", "print(f'Difference: {weekday_births.mean() - weekend_births.mean():.0f} births/day')\n", "print(f'\\nt-statistic: {t_stat:.2f}')\n", "print(f'p-value: {p_val:.2e}')\n", "print(f'\\nSignificant at α=0.05? {\"YES\" if p_val < 0.05 else \"No\"}')" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### ✏️ Your Turn (3 min)\n", "Test whether births in January are significantly different from births in July. Use `stats.ttest_ind()`." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "# Your solution here\n", "\n", "\n", "\n" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "# 💡 Solution\n", "jan = df[df['month'] == 1]['births']\n", "jul = df[df['month'] == 7]['births']\n", "t, p = stats.ttest_ind(jan, jul)\n", "print(f'January mean: {jan.mean():.0f}/day')\n", "print(f'July mean: {jul.mean():.0f}/day')\n", "print(f'p-value: {p:.4f}')\n", "print(f'Significant? {\"YES\" if p < 0.05 else \"No\"}')" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "> 🤔 **Think**: Statistical significance (p < 0.05) is not the same as clinical significance. With 38,000 data points, even tiny differences can be \"significant.\" Always report the effect size (the actual difference), not just the p-value.\n", "\n", "> 🎯 **Statistics Takeaway**: `scipy.stats` gives you `pearsonr`, `ttest_ind`, `chi2_contingency`, ANOVA, and more. For biomedical research, these are the foundation of evidence-based conclusions." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "---\n", "## Section 6: Data Visualization\n", "\n", "**Matplotlib** (low-level, full control) + **Seaborn** (high-level, statistical plots).\n", "\n", "> \"The greatest value of a picture is when it forces us to notice what we never expected to see.\" — John Tukey\n", "\n", "### Line Plot — Births Over Time" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "fig, ax = plt.subplots(figsize=(12, 5))\n", "ax.plot(yearly['year'], yearly['births'], linewidth=1.5, color='steelblue')\n", "ax.set_title('Total Live Births in Malaysia per Year (1920–2023)', fontsize=14, fontweight='bold')\n", "ax.set_xlabel('Year')\n", "ax.set_ylabel('Total Births')\n", "ax.set_ylim(bottom=0)\n", "\n", "# Annotate the WWII dip\n", "ax.annotate('WWII Occupation\\n(1942–1945)', xy=(1943, 100000), xytext=(1955, 150000),\n", " arrowprops=dict(arrowstyle='->', color='coral'),\n", " fontsize=10, color='coral', fontweight='bold')\n", "\n", "plt.tight_layout()\n", "plt.show()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Bar Plot — Births by Month" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "monthly_avg = df.groupby('month')['births'].mean()\n", "month_names = ['Jan', 'Feb', 'Mar', 'Apr', 'May', 'Jun',\n", " 'Jul', 'Aug', 'Sep', 'Oct', 'Nov', 'Dec']\n", "\n", "fig, ax = plt.subplots(figsize=(10, 5))\n", "colors = ['coral' if i == monthly_avg.idxmax() else 'steelblue' for i in range(1, 13)]\n", "bars = ax.bar(month_names, monthly_avg.values, color=colors, edgecolor='white')\n", "ax.set_title('Average Daily Births by Month', fontsize=14, fontweight='bold')\n", "ax.set_xlabel('Month')\n", "ax.set_ylabel('Average Daily Births')\n", "\n", "# Add value labels on bars\n", "for bar, val in zip(bars, monthly_avg.values):\n", " ax.text(bar.get_x() + bar.get_width()/2, bar.get_height() + 5,\n", " f'{val:.0f}', ha='center', fontsize=9, fontweight='bold')\n", "\n", "plt.tight_layout()\n", "plt.show()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Histogram — Distribution of Daily Births" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "fig, ax = plt.subplots(figsize=(10, 5))\n", "ax.hist(df['births'], bins=50, color='steelblue', edgecolor='white', alpha=0.85)\n", "ax.axvline(df['births'].mean(), color='coral', linestyle='--', linewidth=2,\n", " label=f'Mean: {df[\"births\"].mean():.0f}')\n", "ax.axvline(df['births'].median(), color='darkorange', linestyle='-', linewidth=2,\n", " label=f'Median: {df[\"births\"].median():.0f}')\n", "ax.set_title('Distribution of Daily Births in Malaysia (1920–2023)', fontsize=14, fontweight='bold')\n", "ax.set_xlabel('Births per Day')\n", "ax.set_ylabel('Frequency (Number of Days)')\n", "ax.legend()\n", "plt.tight_layout()\n", "plt.show()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Box Plot — Births by Day of Week" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "day_order = ['Monday', 'Tuesday', 'Wednesday', 'Thursday', 'Friday', 'Saturday', 'Sunday']\n", "\n", "fig, ax = plt.subplots(figsize=(10, 5))\n", "sns.boxplot(data=df, x='day_name', y='births', order=day_order,\n", " palette='Blues', ax=ax)\n", "ax.set_title('Daily Births by Day of Week — Clear Weekend Drop', fontsize=14, fontweight='bold')\n", "ax.set_xlabel('Day of Week')\n", "ax.set_ylabel('Number of Births')\n", "plt.tight_layout()\n", "plt.show()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "🤔 The weekend drop is visible. This is a real phenomenon: scheduled C-sections and inductions are planned for weekdays. A biomedical signal in population data!" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Subplots — Multiple Charts in One Figure" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "fig, axes = plt.subplots(2, 2, figsize=(14, 10))\n", "\n", "# 1. Yearly trend (top-left)\n", "axes[0, 0].plot(yearly['year'], yearly['births'], color='steelblue', linewidth=1)\n", "axes[0, 0].set_title('Yearly Total Births')\n", "axes[0, 0].set_xlabel('Year'); axes[0, 0].set_ylabel('Births')\n", "\n", "# 2. By month (top-right)\n", "axes[0, 1].bar(month_names, monthly_avg.values, color='steelblue', edgecolor='white')\n", "axes[0, 1].set_title('Avg Daily Births by Month')\n", "axes[0, 1].set_xlabel('Month')\n", "\n", "# 3. Histogram (bottom-left)\n", "axes[1, 0].hist(df['births'], bins=40, color='coral', edgecolor='white', alpha=0.8)\n", "axes[1, 0].set_title('Distribution of Daily Births')\n", "axes[1, 0].set_xlabel('Births per day'); axes[1, 0].set_ylabel('Frequency')\n", "\n", "# 4. Weekend vs Weekday (bottom-right)\n", "df_box = df[df['year'] >= 2000] # recent data only\n", "sns.boxplot(data=df_box, x='is_weekend', y='births', palette=['steelblue', 'coral'], ax=axes[1, 1])\n", "axes[1, 1].set_title('Weekday vs Weekend (Since 2000)')\n", "axes[1, 1].set_xticklabels(['Weekday', 'Weekend'])\n", "axes[1, 1].set_xlabel('')\n", "\n", "plt.tight_layout()\n", "plt.show()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### ✏️ Your Turn (4 min)\n", "Create a bar chart showing the **top 10 years** with the highest total births. Use the `yearly` DataFrame." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "# Your solution here\n", "\n", "\n", "\n" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "# 💡 Solution\n", "top10 = yearly.sort_values('births', ascending=False).head(10)\n", "\n", "fig, ax = plt.subplots(figsize=(10, 5))\n", "ax.barh(top10['year'].astype(str), top10['births'], color='steelblue', edgecolor='white')\n", "ax.set_title('Top 10 Years — Total Births', fontsize=14, fontweight='bold')\n", "ax.set_xlabel('Total Births')\n", "ax.invert_yaxis()\n", "plt.tight_layout()\n", "plt.show()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "> 🎯 **Visualization Takeaway**: Line plots for trends, bar charts for categories, histograms for distributions, box plots for group comparisons. Add titles, labels, and legends — always. A plot without labels is just a pretty shape." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "---\n", "## Section 7: Mini Project — Malaysia Births Analysis\n", "\n", "Now you'll apply everything — NumPy, Pandas, SciPy, and visualization — to answer **seven** research questions.\n", "\n", "**How this works**: Each question has a markdown prompt, a code cell for you, and (for Q1-Q3) a solution cell. For Q4-Q7, you're on your own — but I'm here to help.\n", "\n", "> 🎯 **Goal**: By the end of this section, you'll have a mini research paper's worth of analysis — all in Python." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Q1: Yearly Trend\n", "\n", "**Why it matters**: Demographers track birth trends to plan healthcare infrastructure, schools, and vaccination programs.\n", "\n", "**Task**: Plot total births per year. Annotate the year with most and fewest births. What historical events might explain the dips and peaks?" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "# Q1 Solution\n", "yearly_births = df.groupby('year')['births'].sum()\n", "\n", "fig, ax = plt.subplots(figsize=(14, 5))\n", "ax.plot(yearly_births.index, yearly_births.values, linewidth=1.5, color='steelblue')\n", "\n", "# Mark peak and trough\n", "max_y = yearly_births.idxmax()\n", "min_y = yearly_births.idxmin()\n", "ax.scatter([max_y, min_y], [yearly_births[max_y], yearly_births[min_y]],\n", " color='coral', s=100, zorder=5)\n", "ax.annotate(f'{int(max_y)}: {int(yearly_births[max_y]):,}', xy=(max_y, yearly_births[max_y]),\n", " xytext=(max_y-20, yearly_births[max_y]+30000), fontsize=10, color='coral', fontweight='bold')\n", "ax.annotate(f'{int(min_y)}: {int(yearly_births[min_y]):,}', xy=(min_y, yearly_births[min_y]),\n", " xytext=(min_y+5, yearly_births[min_y]-40000), fontsize=10, color='coral', fontweight='bold')\n", "\n", "ax.set_title('Total Live Births per Year — Malaysia (1920–2023)', fontsize=15, fontweight='bold')\n", "ax.set_xlabel('Year'); ax.set_ylabel('Total Births')\n", "ax.set_ylim(bottom=0)\n", "plt.tight_layout(); plt.show()\n", "\n", "print(f'Highest: {int(max_y)} — {int(yearly_births[max_y]):,} births')\n", "print(f'Lowest: {int(min_y)} — {int(yearly_births[min_y]):,} births')" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "**Your interpretation**: _What patterns do you see? What historical events might explain the dips and peaks?_" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Q2: Seasonality\n", "\n", "**Why it matters**: Birth seasonality affects hospital staffing and may reveal cultural or environmental influences on conception.\n", "\n", "**Task**: Calculate and plot average births per day for each month. Which month has the most? The fewest? If you count back ~9 months from the peak birth month, what season do conceptions peak in?" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "# Q2 Solution\n", "monthly_avg = df.groupby('month')['births'].mean()\n", "\n", "fig, ax = plt.subplots(figsize=(10, 5))\n", "bars = ax.bar(range(1, 13), monthly_avg.values, color='steelblue', edgecolor='white')\n", "ax.set_xticks(range(1, 13))\n", "ax.set_xticklabels(['Jan','Feb','Mar','Apr','May','Jun','Jul','Aug','Sep','Oct','Nov','Dec'])\n", "ax.set_title('Average Daily Births by Month', fontsize=14, fontweight='bold')\n", "ax.set_xlabel('Month'); ax.set_ylabel('Average Daily Births')\n", "\n", "# Highlight max and min\n", "bars[monthly_avg.idxmax()-1].set_color('coral')\n", "bars[monthly_avg.idxmin()-1].set_color('orange')\n", "\n", "# Add value labels\n", "for bar, val in zip(bars, monthly_avg.values):\n", " ax.text(bar.get_x() + bar.get_width()/2, bar.get_height() + 5,\n", " f'{val:.0f}', ha='center', fontsize=9)\n", "\n", "plt.tight_layout(); plt.show()\n", "\n", "print(f'Highest month: {monthly_avg.idxmax()} ({monthly_avg.max():.0f} births/day avg)')\n", "print(f'Lowest month: {monthly_avg.idxmin()} ({monthly_avg.min():.0f} births/day avg)')\n", "print(f'Conception peak (birth month - 9): month {((monthly_avg.idxmax() - 9 - 1) % 12) + 1}')" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "**Your interpretation**: _Which months stand out? What does this tell you about conception timing in Malaysia?_" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Q3: Weekend Effect\n", "\n", "**Why it matters**: The weekend birth deficit is a well-documented phenomenon in obstetrics — elective C-sections and inductions are scheduled on weekdays.\n", "\n", "**Task**: Calculate the average births on weekends vs weekdays. Run a t-test. Report the percentage difference." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "# Q3 Solution\n", "weekday_avg = df[~df['is_weekend']]['births'].mean()\n", "weekend_avg = df[df['is_weekend']]['births'].mean()\n", "pct_drop = (1 - weekend_avg / weekday_avg) * 100\n", "\n", "t_stat, p_val = stats.ttest_ind(df[~df['is_weekend']]['births'],\n", " df[df['is_weekend']]['births'])\n", "\n", "print(f'Weekday average: {weekday_avg:.0f} births/day')\n", "print(f'Weekend average: {weekend_avg:.0f} births/day')\n", "print(f'Drop: {pct_drop:.1f}%')\n", "print(f't-statistic: {t_stat:.1f}')\n", "print(f'p-value: {p_val:.2e} → {\"SIGNIFICANT\" if p_val < 0.001 else \"not significant\"}')" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "**Your interpretation**: _How big is the weekend drop in Malaysia? What does this tell us about medical intervention in childbirth here?_" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Q4: Top 10 Birthdays\n", "\n", "**Why it matters**: Certain dates consistently have more births — cultural preferences, holiday-conception effects, or data artifacts.\n", "\n", "**Task**: Sum births by month-day combination (ignoring year). Find the top 10 birthdays. Display as a horizontal bar chart." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "# Q4: Your solution here\n", "\n", "\n", "\n" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "# 💡 Solution\n", "df['month_day'] = df['date'].dt.strftime('%B %d')\n", "birthday_totals = df.groupby('month_day')['births'].sum().sort_values(ascending=False)\n", "top10 = birthday_totals.head(10)\n", "\n", "fig, ax = plt.subplots(figsize=(10, 5))\n", "ax.barh(top10.index, top10.values, color='steelblue', edgecolor='white')\n", "ax.set_title('Top 10 Birthdays — Malaysia (1920–2023)', fontsize=14, fontweight='bold')\n", "ax.set_xlabel('Total Births')\n", "ax.invert_yaxis()\n", "plt.tight_layout(); plt.show()\n", "\n", "print('Top 10 birthdays of all time:')\n", "for i, (date, count) in enumerate(top10.items(), 1):\n", " print(f' {i:2d}. {date}: {int(count):,} births')" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Q5: Decade Comparison\n", "\n", "**Why it matters**: Long-term demographic shifts become visible when comparing decades.\n", "\n", "**Task**: Create a decade column (e.g., 1920→1920s). Make a box plot comparing daily births across decades." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "# Q5: Your solution here\n", "\n", "\n", "\n" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "# 💡 Solution\n", "df['decade'] = (df['year'] // 10) * 10\n", "\n", "fig, ax = plt.subplots(figsize=(16, 6))\n", "sns.boxplot(data=df, x='decade', y='births', palette='Blues', ax=ax)\n", "ax.set_title('Daily Births by Decade — Malaysia', fontsize=14, fontweight='bold')\n", "ax.set_xlabel('Decade')\n", "ax.set_ylabel('Births per Day')\n", "plt.xticks(rotation=45)\n", "plt.tight_layout(); plt.show()\n", "\n", "# Which decade had the highest median?\n", "medians = df.groupby('decade')['births'].median()\n", "print(f'Highest median decade: {int(medians.idxmax())}s ({medians.max():.0f} births/day)')" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Q6: Holiday Effect\n", "\n", "**Why it matters**: Fewer births on holidays would strengthen the medical-scheduling hypothesis — elective procedures are rarely scheduled on holidays.\n", "\n", "**Task**: Compare births on January 1st vs January 2nd. Is there a significant drop on New Year's Day?" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "# Q6: Your solution here\n", "\n", "\n", "\n" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "# 💡 Solution\n", "jan1 = df[(df['month'] == 1) & (df['day'] == 1)]['births']\n", "jan2 = df[(df['month'] == 1) & (df['day'] == 2)]['births']\n", "\n", "t_stat, p_val = stats.ttest_ind(jan1, jan2)\n", "print(f'Jan 1 avg: {jan1.mean():.0f} births/day')\n", "print(f'Jan 2 avg: {jan2.mean():.0f} births/day')\n", "pct_drop = (1 - jan1.mean()/jan2.mean()) * 100\n", "print(f'Drop: {pct_drop:.1f}%')\n", "print(f'p-value: {p_val:.4f} → {\"SIGNIFICANT\" if p_val < 0.05 else \"not significant\"}')\n", "\n", "# Bonus: plot Jan 1 vs surrounding days in recent years\n", "recent = df[df['year'] >= 2000]\n", "dec31 = recent[(recent['month'] == 12) & (recent['day'] == 31)]\n", "jan1_r = recent[(recent['month'] == 1) & (recent['day'] == 1)]\n", "jan2_r = recent[(recent['month'] == 1) & (recent['day'] == 2)]\n", "\n", "fig, ax = plt.subplots(figsize=(8, 4))\n", "ax.bar(['Dec 31', 'Jan 1', 'Jan 2'], [dec31['births'].mean(), jan1_r['births'].mean(), jan2_r['births'].mean()],\n", " color=['steelblue', 'coral', 'steelblue'], edgecolor='white')\n", "ax.set_title('New Year Births — Since 2000', fontweight='bold')\n", "ax.set_ylabel('Avg Births/Day')\n", "plt.tight_layout(); plt.show()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Q7: Your Own Question\n", "\n", "**Task**: Ask and answer your own question about this dataset. Examples:\n", "- Are there more births on Fridays? (Scheduled C-sections before the weekend?)\n", "- Does February 29th (leap day) have fewer births? (Do parents avoid it?)\n", "- How did birth rates change during COVID-19 (2020–2021)?\n", "- Is there a lunar cycle effect? (Hint: you'd need external data!)\n", "\n", "Formulate your question, write the code, and share your finding with the group." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "# Q7: Your own question — go explore!\n", "\n", "\n", "\n" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "# 💡 Example: How did birth rates change during COVID-19?\n", "pre_covid = df[(df['year'] >= 2015) & (df['year'] <= 2019)]\n", "covid_era = df[(df['year'] >= 2020) & (df['year'] <= 2021)]\n", "\n", "print(f'Pre-COVID (2015-2019) avg: {pre_covid[\"births\"].mean():.0f}/day')\n", "print(f'COVID era (2020-2021) avg: {covid_era[\"births\"].mean():.0f}/day')\n", "pct_change = (covid_era['births'].mean() / pre_covid['births'].mean() - 1) * 100\n", "print(f'Change: {pct_change:+.1f}%')\n", "\n", "t, p = stats.ttest_ind(pre_covid['births'], covid_era['births'])\n", "print(f'p-value: {p:.4f} → {\"SIGNIFICANT\" if p < 0.05 else \"not significant\"}')\n", "\n", "fig, ax = plt.subplots(figsize=(6, 4))\n", "ax.bar(['Pre-COVID\\n(2015-2019)', 'COVID\\n(2020-2021)'],\n", " [pre_covid['births'].mean(), covid_era['births'].mean()],\n", " color=['steelblue', 'coral'], edgecolor='white')\n", "ax.set_title('Avg Daily Births: Pre-COVID vs COVID', fontweight='bold')\n", "ax.set_ylabel('Avg Births / Day')\n", "plt.tight_layout(); plt.show()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "> 🎯 **Mini Project Takeaway**: You just did what a public health researcher does — loaded raw data, cleaned it, computed statistics, tested hypotheses, visualized findings, and interpreted results. All in Python. All reproducible." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "---\n", "## Section 8: Exporting & Sharing Results\n", "\n", "Your analysis is only useful if you can share it. Here's how to save your data, tables, and figures." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "# Save yearly summary to CSV\n", "yearly_stats.to_csv('births_yearly_summary.csv', index=False)\n", "print('Saved: births_yearly_summary.csv')\n", "\n", "# Save to Excel (multiple sheets!)\n", "with pd.ExcelWriter('births_analysis.xlsx') as writer:\n", " yearly_stats.to_excel(writer, sheet_name='Yearly Summary', index=False)\n", " df.groupby('month')['births'].describe().to_excel(writer, sheet_name='Monthly Stats')\n", "print('Saved: births_analysis.xlsx')" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "# Save a figure to a file\n", "fig, ax = plt.subplots(figsize=(10, 5))\n", "ax.plot(yearly['year'], yearly['births'], color='steelblue', linewidth=1.5)\n", "ax.set_title('Malaysia Live Births 1920–2023', fontsize=14, fontweight='bold')\n", "ax.set_xlabel('Year'); ax.set_ylabel('Total Births')\n", "fig.savefig('births_trend.png', dpi=150, bbox_inches='tight', facecolor='white')\n", "print('Saved: births_trend.png (150 DPI, publication-ready)')" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "# In Colab, download files to your computer\n", "from google.colab import files\n", "files.download('births_yearly_summary.csv')\n", "files.download('births_analysis.xlsx')\n", "files.download('births_trend.png')" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "> 🎯 **Exporting Takeaway**: `.to_csv()`, `.to_excel()`, and `fig.savefig()` cover 95% of sharing needs. Use `dpi=150` or higher for publication-quality figures." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "---\n", "## Section 9: Wrap-Up\n", "\n", "### What You Learned Today\n", "\n", "| Skill | Tool | Biomedical Application |\n", "|-------|------|----------------------|\n", "| Array math & stats | NumPy | Lab values, vitals, sensor data |\n", "| Data loading & cleaning | Pandas | Patient records, clinical databases |\n", "| Filtering & grouping | Pandas | Cohort analysis, subgroup comparisons |\n", "| Date/time handling | Pandas `.dt` | Longitudinal studies, time-series |\n", "| Missing values & outliers | Pandas/NumPy | EHR data quality, lab error detection |\n", "| Statistical testing | SciPy | Treatment effects, group comparisons |\n", "| Publication figures | Matplotlib/Seaborn | Journal papers, conference posters |\n", "| Exporting results | Pandas/Matplotlib | Sharing with non-coding colleagues |\n", "\n", "### Key Pandas Methods Cheat Sheet\n", "\n", "```python\n", "# Reading & Writing\n", "pd.read_csv('file.csv')\n", "pd.read_excel('file.xlsx')\n", "df.to_csv('output.csv', index=False)\n", "df.to_excel('output.xlsx', sheet_name='Data')\n", "\n", "# Exploring\n", "df.head() # First 5 rows\n", "df.info() # Column types, non-null counts\n", "df.describe() # Summary statistics\n", "df.shape # (rows, columns)\n", "df.columns # Column names\n", "df.isnull().sum() # Missing value counts\n", "df.duplicated().sum() # Duplicate row count\n", "\n", "# Filtering\n", "df[df['col'] > value]\n", "df[(df['a'] > 1) & (df['b'] == 'x')]\n", "df[df['col'].isin(['A', 'B', 'C'])]\n", "\n", "# Aggregation\n", "df.groupby('col')['val'].sum()\n", "df.groupby('col').agg({'a': 'mean', 'b': 'max'})\n", "df['col'].value_counts()\n", "\n", "# Dates\n", "pd.to_datetime(df['date'])\n", "df['date'].dt.year / .dt.month / .dt.day\n", "df['date'].dt.day_name()\n", "\n", "# Statistics\n", "from scipy import stats\n", "stats.ttest_ind(group_a, group_b)\n", "stats.pearsonr(x, y)\n", "```\n", "\n", "### Where to Go From Here\n", "\n", "1. **More statistics** — `statsmodels` for regression, ANOVA, survival analysis\n", "2. **Machine learning** — `scikit-learn` for prediction and classification\n", "3. **Bio-specific tools** — `biopython` for sequence analysis, `nilearn` for neuroimaging\n", "4. **More data** — [data.gov.my](https://data.gov.my), [data.moh.gov.my](https://data.moh.gov.my), [WHO GHO](https://www.who.int/data/gho)\n", "5. **Practice** — Try replicating today's analysis with [mortality data](https://data.gov.my/data-catalogue/deaths) or [dengue cases](https://data.gov.my/data-catalogue/dengue)\n", "\n", "---\n", "\n", "### Thank You!\n", "\n", "**You now have a complete, reproducible data analysis pipeline — from raw data to publication-ready figures — entirely in Python.**\n", "\n", "Questions? Ask now, or reach out anytime." ] } ], "metadata": { "kernelspec": { "display_name": "Python 3", "language": "python", "name": "python3" }, "language_info": { "name": "python", "version": "3.10.0" } }, "nbformat": 4, "nbformat_minor": 4 }