1. Parameters versus hyperparameters
Weights are learned from training data. Choices such as tree depth, regularization strength, learning rate and number of layers are usually set outside the inner training loop. These are hyperparameters.
We choose them using validation evidence.
2. Grid search can waste experiments
If only 2 of 10 hyperparameters strongly matter, a dense grid spends most runs changing unimportant dimensions.
Random search samples combinations more broadly and often discovers strong regions faster in high-dimensional search spaces.
Bayesian/optimization methods can use results from earlier trials to choose promising later trials.
3. Search on useful scales
Learning rates such as should often be explored on a logarithmic scale. Sampling uniformly from 0.00001 to 0.1 heavily concentrates trials near the large end.
4. The hidden overfitting problem
Run 10,000 experiments and choose the best validation score: you have adapted to the validation set. It is no longer fully independent evidence.
The final test set must remain untouched. In serious work, repeated validation and nested procedures may be needed.
5. Record the experiment
Every trial should store configuration, code/data version, seed and metrics. Otherwise hyperparameter search becomes irreproducible guesswork.
What to Remember
Hyperparameter tuning is itself a learning process and can overfit. Search deliberately, log everything, and protect the test set.
Next → Cross Validation: when data is scarce, can every example take a turn as validation evidence without contaminating training?