Income Classification: A Comparative Study of Predictive Models
This project evaluates whether machine learning can accurately predict income brackets from demographic data, and, more importantly, whether complex models deliver tangible business value over simpler, interpretable alternatives. Using 32,561 census records, I benchmark multiple approaches to determine the optimal balance between predictive power and operational transparency.
A supervised binary classification project on the UCI Census Income dataset. The study encompasses the full ML lifecycle: exploratory data analysis, feature engineering, class imbalance mitigation, hyperparameter tuning (GridSearchCV), and a comparative evaluation of Logistic Regression, Decision Trees, KNN, and a Keras Neural Network, with a final recommendation grounded in performance metrics and business practicality.
Business Context and Problem Framing
Accurate income estimation is a cornerstone of responsible lending, credit risk assessment, and financial inclusion. Institutions rely on these estimates daily to underwrite loans, set credit limits, and comply with fair lending regulations. A robust predictive model can streamline these decisions; a biased or opaque one introduces regulatory exposure and reputational risk. This project directly addresses that tension by prioritizing both accuracy and interpretability.
Objective: predict the binary label income_binary (≤ $50K vs > $50K) from 14 demographic and employment features. The downstream use case is credit risk adjacent: organizations extending credit require reliable income proxies. Achieving high recall on the positive class must be balanced against fair lending considerations.
Experimental design
UCI Adult Census dataset · 32,561 observations · 76/24 class imbalance · evaluation framework: accuracy, F1 score (positive class), and AUC ROC · stratified 80/20 train/test split with a fixed random seed.
Data Profile
The analysis draws on 32,561 individual records from the UCI Adult Census dataset. Each record captures 14 demographic, educational, and occupational attributes. The initial profiling phase focused on identifying data quality issues, redundancies, and potential sources of multicollinearity that could compromise model stability.
Features include age, workclass, fnlwgt, education (categorical and numeric), marital status, occupation, relationship, race, sex, capital gain, capital loss, hours per week, and native country. Initial EDA revealed missing values across age, hours per week, workclass, occupation, and native country, addressed via imputation rather than listwise deletion. Strong collinearity was observed across several categorical variables, requiring feature reduction prior to modeling.
Feature Engineering
Before model development, I conducted a systematic review of feature interdependencies. Several variables exhibited high correlation, effectively encoding the same information twice. Retaining these redundancies would inflate model variance without adding predictive value. I applied statistical measures to identify and remove these overlapping features, streamlining the feature set for the modeling phase.
- Cramer's V analysis revealed strong associations: relationship and sex_selfID (V = 0.65), relationship and marital status (V = 0.49), and marital status and sex_selfID (V = 0.42). To mitigate proxy leakage, relationship was dropped.
- Redundancy check: education (categorical) and education-num (ordinal) encode identical information. The numeric version was retained; the categorical version was excluded.
- Multicollinearity: Variance Inflation Factor scores for all numeric features remained near 1.0, indicating no concerning collinearity among continuous predictors.
- Missing data: imputation strategies were applied to numeric (age, hours per week) and categorical (workclass, occupation, native country) fields to preserve sample size.
Removed: relationship
It overlapped heavily with gender and marital status. Retaining it would let the model use gender by proxy, precisely the kind of hidden bias worth designing out.
Removed: education (text)
The dataset already encodes years of education as a number. Two columns, one fact. The numeric version is more useful for modeling, so the text version was excluded.
Responsible AI and Bias Considerations
The dataset reflects 1990s U.S. census demographics, with inherent skews in race, nationality, and gender representation. A model trained on this historical data risks perpetuating or amplifying those skews if deployed without oversight. While the data cannot be retroactively balanced, the risk can be managed through feature selection (removing proxy variables), rigorous subgroup evaluation, and explicit deployment guardrails.
- Sampling bias: the 1990s census reflects a specific era and labor market; demographic skew (predominantly White, U.S. born) means minority subgroups have thinner support and likely higher error rates.
- Proxy leakage: relationship encodes sex almost directly (Cramer's V 0.65). Removing it reduces, but does not eliminate, proxy pathways; marital status still carries correlated signal.
- Label framing: a fixed $50K threshold from the 1990s is not meaningful today without inflation adjustment, so any deployment claim requires recalibration on current data.
- Representation: the binary sex field excludes nonbinary identities entirely, a data collection limitation worth stating explicitly.
Position
A model like this is appropriate as a decision support tool with human review. It should not be the final authority on whether a person receives credit.
Preprocessing and Pipeline Configuration
Categorical variables were transformed into binary indicator variables, numeric features were standardized to a common scale, and the class imbalance was addressed by resampling the training set only. This ensures the test set remains representative of the true population distribution.
- Encoding: one-hot encoding with OneHotEncoder(drop="first") applied to categoricals; StandardScaler applied to continuous features. Final feature space: 77 dimensions.
- Data partitioning: 80/20 stratified train/test split (random_state=42).
- Imbalance correction: training set majority class downsampled to match the minority class; test set retained at natural distribution to preserve evaluation rigor.
- Target mapping: ≤50K to 0, >50K to 1.
Model Benchmarking and Baseline Comparison
I established a performance baseline by comparing three distinct algorithms: Logistic Regression (a linear, highly interpretable model), Decision Tree (a rule based classifier), and K Nearest Neighbors (a distance based approach). Logistic Regression consistently delivered the strongest performance across accuracy, F1 score, and AUC, demonstrating superior discriminatory power without the complexity of its counterparts. This result held across multiple validation splits.
Baseline evaluation of LogisticRegression, DecisionTreeClassifier, and KNeighborsClassifier on the balanced training set, evaluated against the untouched test set. Logistic Regression led across all metrics; the ranking held across 20/30/40% test splits. GridSearchCV (5 C values, L1/L2, liblinear) confirmed near optimal default parameters: best params C=1, penalty=L2, CV AUC=0.904.
Why no separate validation set?
Hyperparameter tuning didn't need a separate validation set: GridSearchCV already cross-validates internally, testing each candidate setting on five different held-out slices of the training data and averaging the results. Adding a standalone validation split on top would have just repeated what cross-validation already does, while shrinking an already modest training set.
GridSearchCV's 5-fold CV serves the validation role directly, scoring every hyperparameter candidate across five rotating held-out folds of the training set rather than one static split. A separate validation set would be redundant for tuning. The one place a stricter three-way split would matter: using a validation set, not the test set, to choose the winning algorithm among Logistic Regression, Decision Tree, and KNN, so the test set is reserved purely for a single final read.
How did you check for memorization?
There's also a direct check for memorization in the results themselves: the cross-validation score used to pick the best settings (90.4% AUC) came out almost identical to the model's score on the untouched test set (90.5%). If the model had simply memorized the training data, those two numbers would typically pull apart, not land a tenth of a point from each other.
This also functions as an informal overfitting check: the CV AUC used for model selection (0.904) and the held-out test AUC (0.905) are within 0.001 of each other, the kind of agreement that indicates generalization rather than memorization. Note the Decision Tree was left unpruned (default max_depth), so its clear underperformance likely reflects overfitting on its part rather than a purely algorithmic gap.
Performance Deep Dive and Feature Influence
The model achieves an 84.5% true positive rate for high earners, with a 20.4% false positive rate among lower earners. This trade off is a direct consequence of training on a balanced sample. The most influential predictors align with economic expectations: marital status, capital gains, and executive occupations correlate positively with high income, while certain occupational categories and nationalities show negative coefficients that warrant further fairness review.
Row normalized confusion matrix and top logistic regression coefficients. Positive class recall is robust (84.5%) at the cost of a 20.4% false positive rate. This trade off can be adjusted via threshold tuning based on a specific business cost function.
Classification outcomes (row %)
| Predicted ≤50K | Predicted >50K | |
|---|---|---|
| Actual ≤50K | 79.6% | 20.4% |
| Actual >50K | 15.5% | 84.5% |
Key predictive drivers
Positive coefficients: marital status married civ spouse (+2.21), capital gain (+0.87), occupation exec managerial (+0.79). Negative coefficients: occupation private household service (−1.61), farming and fishing (−1.27). Several nationality based coefficients carry large magnitudes but sit on thin support, suggesting instability.
Largest coefficients: marital-status_Married-civ-spouse (+2.21), Married-AF-spouse (+2.03), capital-gain (+0.87), occupation_Exec-managerial (+0.79). Negative: occupation_Priv-house-serv (−1.61), Farming-fishing (−1.27). Native country coefficients should be interpreted with caution due to low sample sizes.
Advanced Modeling: Neural Network Exploration
To determine whether a more complex architecture could unlock additional predictive lift, I developed a sequential neural network. The model was trained over 50 epochs, with validation loss monitored to detect overfitting. While the network learned efficiently in the early epochs, performance plateaued at a level nearly identical to the simpler logistic regression model.
Keras Sequential architecture: input (77 features) to Dense(32, ReLU) to Dense(16, ReLU) to Dense(1, sigmoid). Optimizer: SGD (lr=0.01), loss: binary cross entropy, 50 epochs, validation_split=0.2, training time about 55s. Validation loss plateaued between epochs 15 and 30 with slight upward drift thereafter, indicating diminishing returns.
Recommendation and Business Implications
The neural network achieved an accuracy of 81.3%, only 0.6 percentage points higher than the logistic regression model's 80.7%. AUC, which measures how well a model ranks high earners above low earners across every possible decision threshold rather than just one fixed cutoff, came out nearly identical for both models at around 90.5%. That is a meaningful result on its own: it means the two models distinguish between income groups almost equally well, and the small accuracy edge the neural network shows is really about where it happens to draw its decision line, not about understanding the data any better. Given that, and the critical regulatory requirement for explainability in financial services, logistic regression is the recommended production model. This outcome reinforces a fundamental principle: model complexity does not inherently equate to business value.
Neural Network test performance: accuracy 0.813, F1 0.687, AUC 0.906, vs tuned Logistic Regression at 0.807 / 0.679 / 0.905. The AUC gap (about 0.001) is an order of magnitude smaller than the accuracy or F1 gap, indicating the two models achieve almost identical ranking quality across all thresholds; the neural network's marginal edge in accuracy and F1 reflects threshold placement at the 0.5 cutoff rather than a meaningfully different decision boundary. This does not justify the interpretability loss, tuning overhead, or serving cost in a regulated credit context. Recommendation: deploy logistic regression; revisit deep architectures only if the feature space expands significantly or a specific performance threshold requires it.
What this means for a business
- The model is most likely to predict a high income for someone who is married, has capital gains, and works in an executive or managerial role. For a lender, that means these applicants will usually be approved automatically.
- The model currently flags 20.4% of lower income applicants as high income (false positives). Those people may receive credit they cannot afford. At the same time, it misses 15.5% of true high earners (false negatives), which means lost lending opportunities.
- Because logistic regression is simple and transparent, you can adjust the decision threshold. Raising the threshold would reduce false positives (fewer risky loans) but increase false negatives (more missed good customers). The right setting depends on the specific cost of defaults versus lost revenue.
A concrete example
Imagine two loan applicants.
Applicant A: married, works as a senior manager, reports $15,000 in capital gains. The model gives them a high‑income prediction. Applicant A may get fast‑tracked for a credit line.
Applicant B: single, works in retail, no capital gains. The model predicts low income. Applicant B may be flagged for manual review.
Now, the model also flags about 1 in 5 lower income applicants as high income (false positives). Those people may be over‑credited, increasing default risk. At the same time, it misses about 1 in 7 true high earners (false negatives), meaning potential good customers are sent to manual review or rejected.
With a transparent model, the lender can adjust the threshold to tighten it to reduce false positives (protect against bad debt) or loosen it to capture more true high earners (capture more revenue). The optimal setting depends entirely on how the business weighs the cost of a default against the profit from an extra loan.
What this means for deployment
- The logistic regression coefficients show that marital status Married civ spouse (+2.21), capital gain (+0.87), and occupation Exec managerial (+0.79) are the strongest positive predictors. These translate to the largest increases in log‑odds for the high income class, dominating the model's decision surface.
- On the test set, the model achieves an 84.5% true positive rate (recall) for high earners, but at the cost of a 20.4% false positive rate among lower income individuals. This means 15.5% of true high earners are missed, while one in five lower income applicants would be incorrectly flagged – a direct consequence of training on a balanced sample.
- Because logistic regression outputs a well‑calibrated probability, the decision threshold can be shifted along the ROC curve to prioritise either lower false positives or higher recall. This trade‑off is governed by a business cost function (e.g., cost of default vs. cost of missed opportunity). The model's transparency makes this calibration straightforward, unlike black‑box neural networks, where such adjustments are far harder to interpret and justify.
A concrete example
Imagine two loan applicants.
Applicant A: married, works as a senior manager, reports $15,000 in capital gains. The model gives them a high‑income prediction. Applicant A may get fast‑tracked for a credit line.
Applicant B: single, works in retail, no capital gains. The model predicts low income. Applicant B may be flagged for manual review.
Now, the model also flags about 1 in 5 lower income applicants as high income (false positives). Those people may be over‑credited, increasing default risk. At the same time, it misses about 1 in 7 true high earners (false negatives), meaning potential good customers are sent to manual review or rejected.
With a transparent model, the lender can adjust the threshold to tighten it to reduce false positives (protect against bad debt) or loosen it to capture more true high earners (capture more revenue). The optimal setting depends entirely on how the business weighs the cost of a default against the profit from an extra loan.
Key Learnings and Future Directions
Primary challenge
Feature selection and bias mitigation required the most strategic judgment. Defining defensible thresholds for removing correlated variables and addressing historical data skew demanded more careful consideration than model training itself.
EDA and feature triage. Defending each drop decision with statistical rigor (Cramer's V thresholds, redundancy criteria) required greater discernment than hyperparameter tuning.
Critical insight
The complexity and performance plateau. The marginal lift from a neural network over logistic regression was negligible, reinforcing that more advanced is not the same as more effective in production settings.
The clear trade off between balanced training and confusion matrix outcomes: high recall was achieved at the direct expense of an elevated false positive rate.
Next steps
Conduct subgroup fairness audits to evaluate performance parity across demographic groups, calibrate the decision threshold against a specific business cost function, and validate model stability on contemporary income data.
Per subgroup error analysis (sex, race) for fairness auditing, threshold tuning with an explicit cost function, calibration curve assessment, and gradient boosting as a potential middle ground between Logistic Regression and the Neural Network.
Development approach
I leveraged Claude to accelerate implementation and debugging, while maintaining full ownership of strategic decisions, data transformations, and final model selection. Every step was reviewed for business relevance and technical soundness.