Hyperparameter Optimizer
A lightweight Python library for automated hyperparameter tuning using Particle Swarm Optimization and Pattern Search — designed as a drop-in replacement for grid and random search with any scikit-learn estimator.
2,000+
PyPI Downloads by ML Practitioners Worldwide
PSO + PS
Dual Metaheuristic Algorithms — Global + Local Search
PyPI
Published Package — MIT Licensed, Production-Ready
The Problem
Grid search and random search are the default tools for hyperparameter tuning — but both are fundamentally unintelligent about where they look next
Grid search scales exponentially with the number of parameters and evaluates redundant configurations that contribute nothing to finding the optimum. Random search improves on this but converges slowly — each trial is independent, with no memory of past evaluations and no directional intelligence guiding the search. Both methods treat the hyperparameter space as something to be exhaustively sampled rather than intelligently explored. For practitioners working with complex models and large parameter spaces, this translates directly into wasted computation, long iteration cycles, and suboptimal final performance — with no guarantee that the best configuration was ever evaluated at all.
The Solution
A drop-in replacement that brings swarm intelligence and systematic neighborhood search to scikit-learn hyperparameter tuning
The Hyperparameter Optimizer replaces grid and random search with two metaheuristic algorithms that actively learn from previous evaluations and use that information to guide the search toward better configurations. Particle Swarm Optimization simulates a swarm of candidate solutions — each particle adjusts its position based on its own best-known result and the best result found by any particle in the swarm, balancing global exploration with local exploitation as the swarm converges. Pattern Search takes a different approach: it performs systematic neighborhood exploration around the current best solution, expanding the search mesh when improvements are found and contracting it when they are not — efficiently refining the solution through a sequence of structured moves. Both algorithms are exposed through a single unified API that is fully compatible with any scikit-learn estimator. Swapping from GridSearchCV to the optimizer requires changing only the optimizer call — no changes to model code, no new dependencies beyond the package itself.
Key Outcome
A published Python package — downloaded over 2,000 times by ML practitioners worldwide — that replaces grid and random search with two complementary metaheuristic algorithms: PSO's swarm intelligence for fast global convergence and Pattern Search's systematic neighborhood exploration for precise local refinement, both delivered through a single scikit-learn-compatible API with zero integration overhead.
Technical Deep Dive
Architecture & Design
Optimization Pipeline
Stage 1 — Unified Initialization
Shared API · HyperparameterOptimizer()
Optimizer Initialization
obj_func · params · scoring · opt_type · cv · verbose
Stage 2 — Algorithm Execution · Choose Both
Path A · Swarm Intelligence
optimizePSO()
Step 1
Swarm Initialization
nParticles placed randomly across bounds
Step 2
Velocity Update
Inertia (w) · cognitive (c1) · social (c2)
Step 3
Gbest Convergence
Global best tracked across all iterations
Path B · Systematic Search
optimizePS()
Step 1
Mesh Initialization
Search mesh sized via mesh_size_coeff
Step 2
Poll Step
GPS or MADS directions evaluated
Step 3
Mesh Expand / Contract
acc_coeff expands · contr_coeff contracts
Stage 3 — Shared Output
Results · Both Algorithms Return Identical Output Structure
Best Params · Best Score · Convergence History
Gbest_pos — optimal hyperparameter configuration · Gbest_score — cross-validated metric · Gbest_history — score per iteration
Stage 1
Unified Initialization
HyperparameterOptimizer() accepts the scikit-learn estimator, parameter space dictionary, scoring metric, optimization direction, and cross-validation folds in a single constructor call. This shared initialization layer means both PSO and Pattern Search operate on exactly the same model binding and evaluation protocol — the algorithm choice affects only the search strategy, not how the model is evaluated.
Stage 2A
Particle Swarm Optimization
optimizePSO() initializes a swarm of nParticles candidate configurations placed randomly across the parameter bounds. At each iteration, every particle's velocity is updated using three forces: inertia (w) preserving its current direction, cognitive pull (c1) toward its own historical best, and social pull (c2) toward the global best found by any particle. Discrete hyperparameters are handled via mutation probability, allowing the swarm to explore categorical and integer spaces effectively.
Stage 2B
Pattern Search
optimizePS() works by constructing a search mesh around the current best configuration and polling a set of structured directions — either GPS (Generalized Pattern Search) or MADS (Mesh Adaptive Direct Search). If polling finds an improvement, the algorithm moves to that position and expands the mesh by acc_coeff. If no improvement is found, the mesh contracts by contr_coeff, focusing the search more tightly around the current best — down to min_mesh_ratio as the lower bound.
Stage 3
Shared Output Structure
Both algorithms return the same output structure: Gbest_pos containing the optimal hyperparameter configuration, Gbest_score containing the best cross-validated metric achieved, and Gbest_history containing the best score at every iteration — enabling convergence analysis and comparison between runs. PSO additionally returns the full particles array for deeper swarm behavior inspection. The identical output interface means downstream code is algorithm-agnostic.
Key Design Decisions
One API, two fundamentally different search strategies
PSO and Pattern Search approach the hyperparameter space in opposite ways — PSO explores broadly through swarm dynamics while Pattern Search refines locally through systematic mesh polling. By exposing both through a single HyperparameterOptimizer class with a consistent constructor, the library lets practitioners run both algorithms and compare convergence histories without changing any model or evaluation code. The interface decision makes both methods first-class options rather than treating one as the fallback.
Metaheuristic intelligence replaces blind sampling
Grid search evaluates every configuration in a fixed grid regardless of quality — PSO and Pattern Search use information from previous evaluations to direct the search toward better regions. This means the optimizer reaches competitive or superior configurations in significantly fewer evaluations, reducing total computation time without sacrificing final model quality. The Gbest_history output makes this convergence advantage directly visible and comparable across runs.
Scikit-learn compatibility as a first-class constraint
The library is designed to work with any scikit-learn-compatible estimator without modification — including custom pipelines and third-party models that implement the fit/predict interface. Cross-validation is built into the evaluation loop, using the same protocol as GridSearchCV and RandomizedSearchCV. The parameter space definition mirrors the sklearn convention for hyperparameter bounds. This compatibility means the library integrates into existing ML workflows with zero friction — no refactoring, no new abstractions, no changes to evaluation logic.
Tech Stack
| Technology | Purpose |
|---|---|
| Python | Core language and package implementation |
| scikit-learn | Estimator compatibility and cross-validation support |
| Particle Swarm Optimization (PSO) | Swarm intelligence for fast global convergence across the parameter space |
| Pattern Search (PS) | Derivative-free systematic neighborhood search for precise local refinement |
| PyPI | Package distribution — pip install hyperparameter-optimizer |
Results & Metrics
What the package delivers
2,000+
PyPI Downloads
Adopted by ML practitioners and researchers worldwide since public release
PSO + PS
Dual Algorithms
Global swarm intelligence paired with systematic local neighborhood search
PyPI
Published & MIT Licensed
Installable in one command · Open source · Free for any use
Faster convergence than grid and random search
Grid search evaluates every configuration in a fixed grid regardless of quality — PSO and Pattern Search use information from previous evaluations to direct the search toward better regions. This means the optimizer reaches competitive or superior configurations in significantly fewer evaluations, reducing total computation time without sacrificing final model quality. The Gbest_history output makes this convergence advantage directly visible and comparable across runs.
Drop-in replacement for GridSearchCV and RandomizedSearchCV
The library is built around scikit-learn's estimator interface — any model that implements fit and predict works without modification. The constructor mirrors the familiar sklearn search API: pass the estimator, the parameter space, the scoring metric, and the cross-validation folds, then call the optimizer. Existing ML pipelines require no structural changes to adopt metaheuristic tuning — only the search call itself is replaced.
Handles continuous, integer, and categorical hyperparameters natively
The bounds parameter accepts (min, max) tuples for continuous and integer ranges — if either bound is non-integer the algorithm treats the dimension as continuous, otherwise as integer. Categorical hyperparameters are passed as lists of choices. PSO handles categorical dimensions through a mutation mechanism; Pattern Search handles them through discrete polling directions. Mixed parameter spaces — combining floats, integers, and categoricals — are supported in a single optimizer call without any special configuration.
Convergence history enables transparent algorithm comparison
Both optimizePSO() and optimizePS() return Gbest_history — the best score found at every iteration — alongside the final best configuration and score. This makes it straightforward to plot convergence curves, compare how quickly each algorithm improves, and diagnose whether the search is still making progress or has effectively plateaued. Running both algorithms on the same parameter space and comparing their histories provides a practical basis for algorithm selection on a given problem.
Applied in production — powering the OptiTree benchmarking framework
The package is used directly in OptiTree — a multi-model tree-based benchmarking framework that compares seven classifiers (Decision Tree, Random Forest, AdaBoost, Gradient Boosting, LightGBM, XGBoost, CatBoost) under PSO-tuned hyperparameters. OptiTree demonstrates the library's practical value: PSO-optimized configurations consistently outperformed default settings across all seven models, with all experiments logged in MLflow and best models serialized with joblib for downstream deployment.