### Install Sphinx and Theme for Documentation

Source: https://github.com/educationaltestingservice/skll/blob/main/CONTRIBUTING.md

Install Sphinx and the Read the Docs theme for building project documentation. Ensure you are using a conda environment.

```bash
$ conda install 'sphinx<6' sphinx_rtd_theme==1.2.0
```

--------------------------------

### Install SKLL using pip

Source: https://github.com/educationaltestingservice/skll/blob/main/doc/getting_started.md

Use this command to install SKLL via pip. Ensure pip is up-to-date.

```default
pip install skll
```

--------------------------------

### Example Class Map Configuration

Source: https://github.com/educationaltestingservice/skll/blob/main/doc/run_experiment.md

Use class_map to collapse or modify labels. This example shows how to group 'beagle' and 'dachsund' into a 'dog' class.

```python
{
'dog': ['beagle', 'dachsund']
}
```

--------------------------------

### Test TestPyPI Package Installation

Source: https://github.com/educationaltestingservice/skll/blob/main/doc/internal/release.md

Install SKLL from TestPyPI with a fallback to the main PyPI. This command is used to test the package before a full release.

```bash
pip install --index-url https://test.pypi.org/simple/ --extra-index-url https://pypi.org/simple skll
```

--------------------------------

### Install Pre-commit Hooks

Source: https://github.com/educationaltestingservice/skll/blob/main/CONTRIBUTING.md

Install pre-commit for automatically running git commit hooks, such as ruff and mypy, to maintain code quality.

```bash
$ pre-commit install
```

--------------------------------

### Example SKLL Experiment Results

Source: https://github.com/educationaltestingservice/skll/blob/main/doc/tutorial.md

This is an example of a human-readable results file generated by SKLL after running an experiment. It includes details about the experiment setup, model parameters, performance metrics, and confusion matrix.

```default
Experiment Name: Titanic_Evaluate_Tuned
SKLL Version: 2.1
Training Set: train
Training Set Size: 569
Test Set: dev
Test Set Size: 143
Shuffle: False
Feature Set: ["family.csv", "misc.csv", "socioeconomic.csv", "vitals.csv"]
Learner: RandomForestClassifier
Task: evaluate
Feature Scaling: none
Grid Search: True
Grid Search Folds: 3
Grid Objective Function: accuracy
Additional Evaluation Metrics: ['roc_auc']
Scikit-learn Version: 0.22.2.post1
Start Timestamp: 10 Mar 2020 14:25:23.595787
End Timestamp: 10 Mar 2020 14:25:28.175375
Total Time: 0:00:04.579588


Fold:
Model Parameters: {"bootstrap": true, "ccp_alpha": 0.0, "class_weight": null, "criterion": "gini", "max_depth": 5, "max_features": "auto", "max_leaf_nodes": null, "max_samples": null, "min_impurity_decrease": 0.0, "min_impurity_split": null, "min_samples_leaf": 1, "min_samples_split": 2, "min_weight_fraction_leaf": 0.0, "n_estimators": 500, "n_jobs": null, "oob_score": false, "random_state": 123456789, "verbose": 0, "warm_start": false}
Grid Objective Score (Train) = 0.797874315418175
+----+------+------+-------------+----------+-------------+
|    |    0 |    1 |   Precision |   Recall |   F-measure |
+====+======+======+=============+==========+=============+
|  0 | [79] |    8 |       0.849 |    0.908 |       0.878 |
+----+------+------+-------------+----------+-------------+
|  1 |   14 | [42] |       0.840 |    0.750 |       0.792 |
+----+------+------+-------------+----------+-------------+
(row = reference; column = predicted)
Accuracy = 0.8461538461538461
Objective Function Score (Test) = 0.8461538461538461

Additional Evaluation Metrics (Test):
 roc_auc = 0.9224137931034483
```

--------------------------------

### Build PyPI Source Distribution

Source: https://github.com/educationaltestingservice/skll/blob/main/doc/internal/release.md

Use this command to build the source distribution for PyPI. Ensure the 'build' package is installed.

```bash
python -m build
```

--------------------------------

### Reader.read Method

Source: https://github.com/educationaltestingservice/skll/blob/main/doc/api/data.md

Loads examples from various supported file formats into a FeatureSet instance.

```APIDOC
## read()

### Description
Load examples from various file formats. The following formats are supported: `.arff`, `.csv`, `.jsonlines`, `.libsvm`, `.ndj`, or `.tsv` formats.

### Returns
A `FeatureSet` instance representing the input file.

### Return type
[`skll.data.featureset.FeatureSet`](#skll.data.featureset.FeatureSet)

### Raises
* **ValueError** – If `ids_to_floats` is True, but IDs cannot be converted.
* **ValueError** – If no features are found.
* **ValueError** – If the example IDs are not unique.
```

--------------------------------

### predict(examples, prediction_prefix=None, append=False, class_labels=True)

Source: https://github.com/educationaltestingservice/skll/blob/main/doc/api/learner.md

Generate predictions for given examples using the learner model, with options to write predictions to a file.

```APIDOC
## predict examples, prediction_prefix=None, append=False, class_labels=True

### Description
Generate predictions for the given examples using the learner model.

Return, and optionally, write out predictions on a given `FeatureSet` to a file. For regressors, the returned and written-out predictions are identical. However, for classifiers:
- if `class_labels` is `True`, class labels are returned as well as written out.
- if `class_labels` is `False` and the classifier is probabilistic (i.e., `self..probability` is `True`), class probabilities are returned as well as written out.
- if `class_labels` is `False` and the classifier is non-probabilistic (i.e., `self..probability` is `False`), class indices are returned and class labels are written out.

TL;DR: for regressors, just ignore `class_labels`. For classfiers, set it to `True` to get class labels and `False` to get class probabilities.

### Parameters
#### Path Parameters
- **examples** (skll.data.featureset.FeatureSet) - Required - The `FeatureSet` instance to predict labels for.
- **prediction_prefix** (Optional[str]) - Optional - If not `None`, predictions will also be written out to a file with the name `<prediction_prefix>_predictions.tsv`. For classifiers, the predictions written out are class labels unless the learner is probabilistic AND `class_labels` is set to `False`. Note that this prefix can also contain a path.
- **append** (bool) - Optional - Should we append the current predictions to the file if it exists? Default: False.
- **class_labels** (bool) - Optional - If `False`, return either the class probabilities (probabilistic classifiers) or the class indices (non-probabilistic ones). If `True`, return the class labels no matter what. Ignored for regressors. Default: True.

### Returns
The predictions returned by the `Learner` instance.

### Return type
numpy.ndarray

### Raises
- **AssertionError** - If invalid predictions are being returned or written out.
- **MemoryError** - If process runs out of memory when converting to dense.
- **RuntimeError** - If there is a mismatch between the learner vectorizer and the test set vectorizer.
```

--------------------------------

### LibSVM Data Format Example

Source: https://github.com/educationaltestingservice/skll/blob/main/doc/run_experiment.md

Illustrates the LibSVM format with optional metadata for IDs, labels, and feature names. Metadata is provided in comments at the end of each line, separated by '#'.

```default
2 1:2.0 3:8.1 # Example1 | 2=ClassY | 1=FeatureA 3=FeatureC
1 5:7.0 6:19.1 # Example2 | 1=ClassX | 5=FeatureE 6=FeatureF
```

--------------------------------

### Upload to TestPyPI

Source: https://github.com/educationaltestingservice/skll/blob/main/doc/internal/release.md

Upload the built source distribution to TestPyPI. Requires 'twine' to be installed and '$HOME/.pypirc' configured.

```bash
twine upload --repository testpypi dist/*
```

--------------------------------

### Test SKLL Conda Package Installation

Source: https://github.com/educationaltestingservice/skll/blob/main/conda-recipe/README.md

Create a new conda environment to test the installation of the SKLL package. This command prioritizes the 'ets' channel to ensure the latest release is installed, even if a slightly older version exists in conda-forge.

```bash
conda create -n foobar -c https://conda.anaconda.org/ets -c conda-forge python=3.11 skll
```

--------------------------------

### Install SKLL in Editable Mode

Source: https://github.com/educationaltestingservice/skll/blob/main/CONTRIBUTING.md

Install SKLL into the active conda environment in editable mode for development purposes.

```bash
$ pip install -e .
```

--------------------------------

### Example SKLL Experiment Output

Source: https://github.com/educationaltestingservice/skll/blob/main/doc/tutorial.md

This is an example of the output generated when running an SKLL experiment. It includes timestamps, model information, training progress, and evaluation results.

```text
2020-03-10 14:25:23,596 - Titanic_Evaluate_Tuned_family.csv+misc.csv+socioeconomic.csv+vitals.csv_RandomForestClassifier - INFO - Task: evaluate
2020-03-10 14:25:23,596 - Titanic_Evaluate_Tuned_family.csv+misc.csv+socioeconomic.csv+vitals.csv_RandomForestClassifier - INFO - Training on train, Test on dev, feature set ['family.csv', 'misc.csv', 'socioeconomic.csv', 'vitals.csv'] ...
Loading /Users/nmadnani/work/skll/examples/titanic/train/family.csv...           done
Loading /Users/nmadnani/work/skll/examples/titanic/train/misc.csv...           done
Loading /Users/nmadnani/work/skll/examples/titanic/train/socioeconomic.csv...           done
Loading /Users/nmadnani/work/skll/examples/titanic/train/vitals.csv...           done
Loading /Users/nmadnani/work/skll/examples/titanic/dev/family.csv...           done
Loading /Users/nmadnani/work/skll/examples/titanic/dev/misc.csv...           done
Loading /Users/nmadnani/work/skll/examples/titanic/dev/socioeconomic.csv...           done
Loading /Users/nmadnani/work/skll/examples/titanic/dev/vitals.csv...           done
2020-03-10 14:25:23,662 - Titanic_Evaluate_Tuned_family.csv+misc.csv+socioeconomic.csv+vitals.csv_RandomForestClassifier - INFO - Featurizing and training new RandomForestClassifier model
2020-03-10 14:25:23,663 - Titanic_Evaluate_Tuned_family.csv+misc.csv+socioeconomic.csv+vitals.csv_RandomForestClassifier - WARNING - Training data will be shuffled to randomize grid search folds.  Shuffling may yield different results compared to scikit-learn.
2020-03-10 14:25:28,129 - Titanic_Evaluate_Tuned_family.csv+misc.csv+socioeconomic.csv+vitals.csv_RandomForestClassifier - INFO - Best accuracy grid search score: 0.798
2020-03-10 14:25:28,130 - Titanic_Evaluate_Tuned_family.csv+misc.csv+socioeconomic.csv+vitals.csv_RandomForestClassifier - INFO - Hyperparameters: bootstrap: True, ccp_alpha: 0.0, class_weight: None, criterion: gini, max_depth: 5, max_features: auto, max_leaf_nodes: None, max_samples: None, min_impurity_decrease: 0.0, min_impurity_split: None, min_samples_leaf: 1, min_samples_split: 2, min_weight_fraction_leaf: 0.0, n_estimators: 500, n_jobs: None, oob_score: False, random_state: 123456789, verbose: 0, warm_start: False
2020-03-10 14:25:28,130 - Titanic_Evaluate_Tuned_family.csv+misc.csv+socioeconomic.csv+vitals.csv_RandomForestClassifier - INFO - Evaluating predictions
2020-03-10 14:25:28,172 - Titanic_Evaluate_Tuned_family.csv+misc.csv+socioeconomic.csv+vitals.csv_RandomForestClassifier - INFO - using probabilities for the positive class to compute "roc_auc" for evaluation.
2020-03-10 14:25:28,178 - Titanic_Evaluate_Tuned_family.csv+misc.csv+socioeconomic.csv+vitals.csv_DecisionTreeClassifier - INFO - Task: evaluate
2020-03-10 14:25:28,178 - Titanic_Evaluate_Tuned_family.csv+misc.csv+socioeconomic.csv+vitals.csv_DecisionTreeClassifier - INFO - Training on train, Test on dev, feature set ['family.csv', 'misc.csv', 'socioeconomic.csv', 'vitals.csv'] ...
Loading /Users/nmadnani/work/skll/examples/titanic/train/family.csv...           done
Loading /Users/nmadnani/work/skll/examples/titanic/train/misc.csv...           done
Loading /Users/nmadnani/work/skll/examples/titanic/train/socioeconomic.csv...           done
Loading /Users/nmadnani/work/skll/examples/titanic/train/vitals.csv...           done
Loading /Users/nmadnani/work/skll/examples/titanic/dev/family.csv...           done
Loading /Users/nmadnani/work/skll/examples/titanic/dev/misc.csv...           done
Loading /Users/nmadnani/work/skll/examples/titanic/dev/socioeconomic.csv...           done
Loading /Users/nmadnani/work/skll/examples/titanic/dev/vitals.csv...           done
2020-03-10 14:25:28,226 - Titanic_Evaluate_Tuned_family.csv+misc.csv+socioeconomic.csv+vitals.csv_DecisionTreeClassifier - INFO - Featurizing and training new DecisionTreeClassifier model
2020-03-10 14:25:28,226 - Titanic_Evaluate_Tuned_family.csv+misc.csv+socioeconomic.csv+vitals.csv_DecisionTreeClassifier - WARNING - Training data will be shuffled to randomize grid search folds.  Shuffling may yield different results compared to scikit-learn.
2020-03-10 14:25:28,269 - Titanic_Evaluate_Tuned_family.csv+misc.csv+socioeconomic.csv+vitals.csv_DecisionTreeClassifier - INFO - Best accuracy grid search score: 0.754
```

--------------------------------

### Example SKLL Cross-Validation Configuration

Source: https://github.com/educationaltestingservice/skll/blob/main/examples/Tutorial.ipynb

This configuration file sets up a cross-validation experiment for the IRIS dataset using multiple learners and hyperparameter tuning.

```ini
[General]
experiment_name = Iris_CV
task = cross_validate

[Input]
# this could also be an absolute path instead (and must be if you're not
# running things in local mode)
train_directory = train
featuresets = [["example_iris_features"]]
# there is only set of features to try with one feature file in it here.
featureset_names = ["example_iris"]
learners = ["RandomForestClassifier", "SVC", "LogisticRegression", "MultinomialNB"]
suffix = .jsonlines

[Tuning]
grid_search = true
objectives = ['f1_score_micro']

[Output]
# again, these can be absolute paths
save_cv_models = true
models = output
results = output
logs = output
predictions = output

```

--------------------------------

### LibSVMReader

Source: https://github.com/educationaltestingservice/skll/blob/main/doc/api/data.md

Create a FeatureSet instance from a LibSVM/LibLinear/SVMLight file. We use a specially formatted comment for storing example IDs, class names, and feature names, which are normally not supported by the format. The comment is not mandatory, but without it, your labels and features will not have names. The comment is structured as follows:

```default
ExampleID | 1=FirstClass | 1=FirstFeature 2=SecondFeature
```

```APIDOC
## LibSVMReader

### Description
Create a `FeatureSet` instance from a LibSVM/LibLinear/SVMLight file. We use a specially formatted comment for storing example IDs, class names, and feature names, which are normally not supported by the format. The comment is not mandatory, but without it, your labels and features will not have names. The comment is structured as follows:

```default
ExampleID | 1=FirstClass | 1=FirstFeature 2=SecondFeature
```

### Parameters
* **path_or_list** (Union[[`skll.types.PathOrStr`](types.md#skll.types.PathOrStr), List[Dict[str, Any]]) – Path or a list of example dictionaries.
* **quiet** (*bool* *,* *default=True*) – Do not print “Loading…” status message to stderr.
* **ids_to_floats** (*bool* *,* *default=False*) – Convert IDs to float to save memory. Will raise error
  if we encounter an a non-numeric ID.
* **label_col** (*Optional* *[**str* *]* *,* *default='y'*) – Name of the column which contains the class labels
  for ARFF/CSV/TSV files. If no column with that name
  exists, or `None` is specified, the data is
  considered to be unlabelled.
* **id_col** (*str* *,* *default='id'*) – Name of the column which contains the instance IDs.
  If no column with that name exists, or `None` is
  specified, example IDs will be automatically generated.
* **class_map** (Optional[[`skll.types.ClassMap`](types.md#skll.types.ClassMap)], default=None) – Mapping from original class labels to new ones. This is
  mainly used for collapsing multiple labels into a single
  class. Anything not in the mapping will be kept the same.
  The keys are the new labels and the list of values for each
  key is the labels to be collapsed to said new label.
* **sparse** (*bool* *,* *default=True*) – Whether or not to store the features in a numpy CSR
  matrix when using a DictVectorizer to vectorize the
  features.
* **feature_hasher** (*bool* *,* *default=False*) – Whether or not a FeatureHasher should be used to
  vectorize the features.
* **num_features** (*Optional* *[**int* *]* *,* *default=None*) – If using a FeatureHasher, how many features should the
  resulting matrix have?  You should set this to a power
  of 2 greater than the actual number of features to
  avoid collisions.
* **logger** (*Optional* *[**logging.Logger* *]* *,* *default=None*) – A logger instance to use to log messages instead of creating
  a new one by default.
```

--------------------------------

### Safe Float Conversion Example

Source: https://github.com/educationaltestingservice/skll/blob/main/doc/run_experiment.md

Demonstrates the internal use of safe_float for label conversion, showing how string labels that can be converted to floats are processed.

```python
import numpy as np
from skll.data.readers import safe_float
np.array([safe_float(x) for x in ["2", "2.2", "2.21"]])
```

--------------------------------

### Create and Train Linear SVM Learner

Source: https://github.com/educationaltestingservice/skll/blob/main/examples/Tutorial.ipynb

Instantiate a LinearSVC learner with custom `max_iter` and train it on provided examples. The training process performs a grid search to find the best model based on accuracy.

```python
new_learner = Learner('LinearSVC', model_kwargs = {'max_iter': 1000})
best_objective_value, grid_search_results = learner.train(train_examples, grid_objective='accuracy')
best_objective_value
```

--------------------------------

### Create Conda Virtual Environment with SKLL

Source: https://github.com/educationaltestingservice/skll/blob/main/doc/tutorial.md

Use conda to create a new virtual environment named 'skllenv' with Python 3.11 and SKLL installed. Activate this environment before proceeding with the tutorial. Deactivate using 'conda deactivate'.

```default
conda create -n skllenv -c conda-forge -c ets python=3.11 skll
```

--------------------------------

### Learner.from_file

Source: https://github.com/educationaltestingservice/skll/blob/main/doc/api.md

Loads a Learner from a file.

```APIDOC
## Learner.from_file

### Description
Loads a Learner from a file.

### Method
`from_file(path)`

### Parameters
* **path** (str) - Required - Path to the file containing the saved Learner.

### Returns
Learner: The loaded Learner object.
```

--------------------------------

### cross_validate

Source: https://github.com/educationaltestingservice/skll/blob/main/doc/api/learner.md

Cross-validate the meta-estimator on the given examples. This method splits examples into training and testing folds, then trains and evaluates the learner on each fold.

```APIDOC
## cross_validate(examples, stratified=True, cv_folds=10, cv_seed=123456789, grid_search=True, grid_search_folds=5, grid_jobs=None, grid_objective=None, output_metrics=[], param_grid_list=None, shuffle=False, save_cv_folds=True, save_cv_models=False, individual_predictions=False, use_custom_folds_for_grid_search=True)

### Description
Cross-validate the meta-estimator on the given examples. We follow essentially the same methodology as in `Learner.cross_validate()` - split the examples into training and testing folds, and then call `self.train()` on the training folds and then `self.evaluate()` on the test fold. Note that this means that underlying estimators with different hyperparameters may be used for each fold, as is the case with `Learner.cross_validate()`.

### Parameters
#### Path Parameters
* **examples** (skll.data.featureset.FeatureSet) - Required - The FeatureSet instance to cross-validate learner performance on.

#### Query Parameters
* **stratified** (bool) - Optional - Default: True - Should we stratify the folds to ensure an even distribution of labels for each fold?
* **cv_folds** (Union[int, skll.types.FoldMapping]) - Optional - Default: 10 - The number of folds to use for cross-validation, or a mapping from example IDs to folds.
* **cv_seed** (int) - Optional - Default: 123456789 - The value for seeding the random number generator used to create the random folds. Note that this seed is *only* used if either `grid_search` or `shuffle` are set to `True`.
* **grid_search** (bool) - Optional - Default: True - Should we do grid search when training each fold? Note: This will make this take *much* longer.
* **grid_search_folds** (Union[int, skll.types.FoldMapping]) - Optional - Default: 5 - The number of folds to use when doing the grid search, or a mapping from example IDs to folds.
* **grid_jobs** (Optional[int]) - Optional - Default: None - The number of jobs to run in parallel when doing the grid search. If `None` or 0, the number of grid search folds will be used.
* **grid_objective** (Optional[str]) - Optional - Default: None - The name of the objective function to use when doing the grid search. Must be specified if `grid_search` is `True`.
* **output_metrics** (Optional[List[str]]) - Optional - Default: [] - List of additional metric names to compute in addition to the metric used for grid search.
* **prediction_prefix** (Optional[str]) - Optional - Default: None - If saving the predictions, this is the prefix that will be used for the filename. It will be followed by `"_predictions.tsv"`.
* **param_grid_list** (Optional[List[Dict[str, Any]]]) - Optional - Default: None - The list of parameters grid to search through for grid search, one for each underlying learner. The order of the dictionaries should correspond to the order If `None`, the default parameter grids will be used for the underlying estimators.
* **shuffle** (bool) - Optional - Default: False - Shuffle examples before splitting into folds for CV.
* **save_cv_folds** (bool) - Optional - Default: True - Whether to save the cv fold ids or not?
* **save_cv_models** (bool) - Optional - Default: False - Whether to save the cv models or not?
* **individual_predictions** (bool) - Optional - Default: False - Write out the cross-validated predictions from each underlying learner as well.
* **use_custom_folds_for_grid_search** (bool) - Optional - Default: True - If `cv_folds` is a custom dictionary, but `grid_search_folds` is not, perhaps due to user oversight, should the same custom dictionary automatically be used for the inner grid-search cross-validation?

### Returns
A 3-tuple containing the following:
> List[skll.types.EvaluateTaskResults]: the confusion matrix, overall accuracy, per-label PRFs, model parameters, objective function score, and evaluation metrics (if any) for each fold.

> Optional[skll.types.FoldMapping]: dictionary containing the test-fold number for each id if `save_cv_folds` is `True`, otherwise `None`.

> Optional[List[skll.learner.voting.VotingLearner]]: list of voting learners, one for each fold if `save_cv_models` is `True`, otherwise `None`.

### Return type
skll.types.CrossValidateTaskResults

### Raises
* **ValueError** – If classification labels are not properly encoded as strings.
* **ValueError** – If `grid_search` is `True` but `grid_objective` is `None`.
```

--------------------------------

### Upload Source and Wheel Packages to PyPI

Source: https://github.com/educationaltestingservice/skll/blob/main/doc/internal/release.md

Use these commands to upload the source and wheel packages to PyPI. This is typically done after successful testing on TestPyPI.

```bash
python setup.py sdist upload
```

```bash
python setup.py bdist_wheel upload
```

--------------------------------

### filtered_iter

Source: https://github.com/educationaltestingservice/skll/blob/main/doc/api/data.md

Retains only the specified features and/or examples from the output. This method returns a generator yielding example IDs, labels, and feature dictionaries.

```APIDOC
## filtered_iter(ids=None, labels=None, features=None, inverse=False)

### Description
Retain only the specified features and/or examples from the output.

### Parameters
#### Path Parameters
- **ids** (Optional[List[[`skll.types.IdType`](types.md#skll.types.IdType)]], default=None) – Examples to keep in the `FeatureSet`. If `None`, no ID filtering takes place.
- **labels** (Optional[List[[`skll.types.LabelType`](types.md#skll.types.LabelType)]], default=None) – Labels that we want to retain examples for. If `None`, no label filtering takes place.
- **features** (*Optional* *[**Collection* *[**str* *]* *]* * *,* *default=None*) – Features to keep in the `FeatureSet`. To help with filtering string-valued features that were converted to sequences of boolean features when read in, any features in the `FeatureSet` that contain a = will be split on the first occurrence and the prefix will be checked to see if it is in `features`. If None, no feature filtering takes place. Cannot be used if `FeatureSet` uses a FeatureHasher for vectorization.
- **inverse** (*bool* *,* *default=False*) – Instead of keeping features and/or examples in lists, remove them.

### Returns
A generator that yields 3-tuples containing:
- [`skll.types.IdType`](types.md#skll.types.IdType) - The ID of the example.
- [`skll.types.LabelType`](types.md#skll.types.LabelType) - The label of the example.
- [`skll.types.FeatureDict`](types.md#skll.types.FeatureDict) - The feature dictionary, with feature name as the key and example value as the value.

### Return type
[`skll.types.FeatGenerator`](types.md#skll.types.FeatGenerator)

### Raises
* **ValueError** – If the vectorizer is not a `DictVectorizer`.
* **ValueError** – If any of the “labels”, “features”, or “vectorizer” attribute is `None`.
```

--------------------------------

### experiments.run_configuration

Source: https://github.com/educationaltestingservice/skll/blob/main/doc/api.md

Runs an experiment based on a configuration.

```APIDOC
## experiments.run_configuration

### Description
Runs an experiment based on a configuration.

### Method
`run_configuration(config_path, results_path)`

### Parameters
* **config_path** (str) - Required - Path to the configuration file.
* **results_path** (str) - Required - Path to the directory where results will be saved.

### Returns
None
```

--------------------------------

### Install SKLL using conda

Source: https://github.com/educationaltestingservice/skll/blob/main/doc/getting_started.md

Use this command to install SKLL via conda, specifying the conda-forge and ets channels. This is recommended for managing complex dependencies.

```default
conda install -c conda-forge -c ets skll
```

--------------------------------

### Learner.load

Source: https://github.com/educationaltestingservice/skll/blob/main/doc/api.md

Loads a Learner from a file.

```APIDOC
## Learner.load

### Description
Loads a Learner from a file.

### Method
`load(path)`

### Parameters
* **path** (str) - Required - Path to the file containing the saved Learner.

### Returns
Learner: The loaded Learner object.
```

--------------------------------

### Run SKLL Experiment

Source: https://github.com/educationaltestingservice/skll/blob/main/examples/Tutorial.ipynb

Execute an SKLL experiment using a configuration file. This command initiates the training and evaluation process defined in the specified .cfg file. Ensure the configuration file path is correct.

```bash
!run_experiment iris/cross_val.cfg
```

--------------------------------

### skll.config.load_cv_folds

Source: https://github.com/educationaltestingservice/skll/blob/main/doc/api/config.md

Loads cross-validation folds from a CSV file. The CSV must contain example ID and fold ID columns with a header. It returns a dictionary mapping example IDs to fold IDs.

```APIDOC
## skll.config.load_cv_folds(folds_file, ids_to_floats=False)

### Description
Load cross-validation folds from a CSV file.

The CSV file must contain two columns: example ID and fold ID (and a header).

### Parameters
#### Path Parameters
- **folds_file** (skll.types.PathOrStr) - Required - The path to a folds file to read.
- **ids_to_floats** (bool) - Optional - Whether to convert IDs to floats. Defaults to False.

### Returns
- **skll.types.FoldMapping** - Dictionary with example IDs as the keys and fold IDs as the values. If ids_to_floats is set to True, the example IDs are floats but otherwise they are strings.

### Raises
- **ValueError** - If example IDs cannot be converted to floats and ids_to_floats is True.
```

--------------------------------

### Run All Pre-commit Checks

Source: https://github.com/educationaltestingservice/skll/blob/main/CONTRIBUTING.md

Execute all pre-commit hooks on all files in the repository to ensure comprehensive code quality checks.

```bash
$ pre-commit run --all-files
```

--------------------------------

### Learner.get_feature_names_out

Source: https://github.com/educationaltestingservice/skll/blob/main/doc/api.md

Gets the feature names out after transformation.

```APIDOC
## Learner.get_feature_names_out

### Description
Gets the feature names out after transformation.

### Method
`get_feature_names_out(input_features=None)`

### Parameters
* **input_features** (list or None) - Optional - List of input feature names. Defaults to None.

### Returns
list: A list of output feature names.
```

--------------------------------

### DictListReader

Source: https://github.com/educationaltestingservice/skll/blob/main/doc/api/data.md

Facilitate programmatic use of methods that take FeatureSet as input. Support Learner.predict() and other methods that take FeatureSet objects as input. It iterates over examples in the same way as other Reader classes, but uses a list of example dictionaries instead of a path to a file.

```APIDOC
## DictListReader

### Description
Facilitate programmatic use of methods that take `FeatureSet` as input.

Support `Learner.predict()` and other methods that take `FeatureSet`
objects as input. It iterates over examples in the same way as other
`Reader` classes, but uses a list of example dictionaries instead of
a path to a file.

### Parameters
* **path_or_list** (Union[[`skll.types.PathOrStr`](types.md#skll.types.PathOrStr), List[Dict[str, Any]]) – Path or a list of example dictionaries.
* **quiet** (*bool* *,* *default=True*) – Do not print “Loading…” status message to stderr.
* **ids_to_floats** (*bool* *,* *default=False*) – Convert IDs to float to save memory. Will raise error
  if we encounter an a non-numeric ID.
* **label_col** (*Optional* *[**str* *]* *,* *default='y'*) – Name of the column which contains the class labels
  for ARFF/CSV/TSV files. If no column with that name
  exists, or `None` is specified, the data is
  considered to be unlabelled.
* **id_col** (*str* *,* *default='id'*) – Name of the column which contains the instance IDs.
  If no column with that name exists, or `None` is
  specified, example IDs will be automatically generated.
* **class_map** (Optional[[`skll.types.ClassMap`](types.md#skll.types.ClassMap)], default=None) – Mapping from original class labels to new ones. This is
  mainly used for collapsing multiple labels into a single
  class. Anything not in the mapping will be kept the same.
  The keys are the new labels and the list of values for each
  key is the labels to be collapsed to said new label.
* **sparse** (*bool* *,* *default=True*) – Whether or not to store the features in a numpy CSR
  matrix when using a DictVectorizer to vectorize the
  features.
* **feature_hasher** (*bool* *,* *default=False*) – Whether or not a FeatureHasher should be used to
  vectorize the features.
* **num_features** (*Optional* *[**int* *]* *,* *default=None*) – If using a FeatureHasher, how many features should the
  resulting matrix have?  You should set this to a power
  of 2 greater than the actual number of features to
  avoid collisions.
* **logger** (*Optional* *[**logging.Logger* *]* *,* *default=None*) – A logger instance to use to log messages instead of creating
  a new one by default.
```

--------------------------------

### model_type property

Source: https://github.com/educationaltestingservice/skll/blob/main/doc/api/learner.md

Get the class name of the underlying scikit-learn model.

```APIDOC
## model_type property

### Description
Return the model type (i.e., the class).

### Return type
[Model class name]
```

--------------------------------

### Display First 5 Lines of JSON Lines File

Source: https://github.com/educationaltestingservice/skll/blob/main/examples/Tutorial.ipynb

Shows the first 5 lines of a '.jsonlines' file, illustrating the structure of SKLL's preferred data format.

```bash
!head -5 iris/train/example_iris_features.jsonlines
```

--------------------------------

### model_params property

Source: https://github.com/educationaltestingservice/skll/blob/main/doc/api/learner.md

Get the model parameters (weights and intercepts) for supported model types.

```APIDOC
## model_params property

### Description
Return model parameters (i.e., weights).

Return the weights for a `LinearModel` (e.g., `Ridge`), regression, and liblinear models. If the model was trained using feature hashing, then names of the form hashed_feature_XX are used instead.

### Returns
- **res** (Dict[str, Any]) - A dictionary of labeled weights.
- **intercept** (Dict[str, Any]) - A dictionary of intercept(s).

### Raises
- **ValueError** - If the instance does not support model parameters.
```

--------------------------------

### Run SKLL Experiment Configuration

Source: https://github.com/educationaltestingservice/skll/blob/main/doc/tutorial.md

Execute an SKLL experiment by providing the path to your configuration file. Ensure the 'skllenv' environment is activated before running this command.

```bash
$ run_experiment titanic/evaluate_tuned.cfg
```

--------------------------------

### Display SKLL Configuration File

Source: https://github.com/educationaltestingservice/skll/blob/main/examples/Tutorial.ipynb

Use this command to display the content of an SKLL experiment configuration file.

```bash
!cat iris/cross_val.cfg
```

--------------------------------

### load(learner_path)

Source: https://github.com/educationaltestingservice/skll/blob/main/doc/api/learner.md

Replace the current learner instance with a saved learner from a specified file path.

```APIDOC
## load learner_path

### Description
Replace the current learner instance with a saved learner.

### Parameters
#### Path Parameters
- **learner_path** (skll.types.PathOrStr) - Required - The path to a saved learner object file to load.

### Return type
None
```

--------------------------------

### cross_validate

Source: https://github.com/educationaltestingservice/skll/blob/main/doc/api/learner.md

Performs cross-validation on a learner using the provided examples. It supports various options for stratification, grid search, shuffling, and saving results.

```APIDOC
## cross_validate

### Description
Cross-validate the learner on the given training examples.

### Method Signature
`cross_validate(examples, stratified=True, cv_folds=10, cv_seed=123456789, grid_search=True, grid_search_folds=5, grid_jobs=None, grid_objective=None, output_metrics=[], prediction_prefix=None, param_grid=None, shuffle=False, save_cv_folds=True, save_cv_models=False, use_custom_folds_for_grid_search=True)`

### Parameters
#### Path Parameters
None

#### Query Parameters
None

#### Request Body
* **examples** (`skll.data.featureset.FeatureSet`) - Required - The `FeatureSet` instance to cross-validate learner performance on.
* **stratified** (*bool*) - Optional - Should we stratify the folds to ensure an even distribution of labels for each fold? (default: True)
* **cv_folds** (Union[int, `skll.types.FoldMapping`]) - Optional - The number of folds to use for cross-validation, or a mapping from example IDs to folds. (default: 10)
* **cv_seed** (*int*) - Optional - The value for seeding the random number generator used to create the random folds. Note that this seed is *only* used if either `grid_search` or `shuffle` are set to `True`. (default: 123456789)
* **grid_search** (*bool*) - Optional - Should we do grid search when training each fold? Note: This will make this take *much* longer. (default: True)
* **grid_search_folds** (Union[int, `skll.types.FoldMapping`]) - Optional - The number of folds to use when doing the grid search, or a mapping from example IDs to folds. (default: 5)
* **grid_jobs** (*Optional* *[**int* *]*) - Optional - The number of jobs to run in parallel when doing the grid search. If `None` or 0, the number of grid search folds will be used. (default: None)
* **grid_objective** (*Optional* *[**str* *]*) - Optional - The name of the objective function to use when doing the grid search. Must be specified if `grid_search` is `True`. (default: None)
* **output_metrics** (*List* *[**str* *]*) - Optional - List of additional metric names to compute in addition to the metric used for grid search. (default: [])
* **prediction_prefix** (*Optional* *[**str* *]*) - Optional - If saving the predictions, this is the prefix that will be used for the filename. It will be followed by `"_predictions.tsv"`. (default: None)
* **param_grid** (*Optional* *[**Dict* *[**str* *,* *Any* *]*) - Optional - The parameter grid to search. (default: None)
* **shuffle** (*bool*) - Optional - Shuffle examples before splitting into folds for CV. (default: False)
* **save_cv_folds** (*bool*) - Optional - Whether to save the cv fold ids or not? (default: True)
* **save_cv_models** (*bool*) - Optional - Whether to save the cv models or not? (default: False)
* **use_custom_folds_for_grid_search** (*bool*) - Optional - If `cv_folds` is a custom dictionary, but `grid_search_folds` is not, perhaps due to user oversight, should the same custom dictionary automatically be used for the inner grid-search cross-validation? (default: True)

### Request Example
None

### Response
#### Success Response (200)
* **Return type:** `skll.types.CrossValidateTaskResults`
  A 5-tuple containing the following:
  > List[[`skll.types.EvaluateTaskResults`](types.md#skll.types.EvaluateTaskResults)]: the confusion matrix, overall accuracy, per-label PRFs, model parameters, objective function score, and evaluation metrics (if any) for each fold.
  > List[float]: the grid search scores for each fold.
  > List[Dict[str, Any]]: list of dictionaries of grid search CV results, one per fold, with keys such as “params”, “mean_test_score”, etc, that are mapped to lists of values associated with each hyperparameter set combination.
  > Optional[[`skll.types.FoldMapping`](types.md#skll.types.FoldMapping)]: dictionary containing the test-fold number for each id if `save_cv_folds` is `True`, otherwise `None`.
  > Optional[List[[`skll.learner.Learner`](#skll.learner.Learner)]]: list of learners, one for each fold if `save_cv_models` is `True`, otherwise `None`.

#### Response Example
None

### Error Handling
* **ValueError** – If classification labels are not properly encoded as strings.
* **ValueError** – If `grid_search` is `True` but `grid_objective` is `None`.
```

--------------------------------

### Run Specific Pre-commit Hook on File

Source: https://github.com/educationaltestingservice/skll/blob/main/CONTRIBUTING.md

Run a specific pre-commit hook, such as 'ruff', on a designated file.

```bash
$ pre-commit run ruff --files <file-path>
```

--------------------------------

### Define Custom Metric Function

Source: https://github.com/educationaltestingservice/skll/blob/main/doc/custom_metrics.md

Define a custom metric function that takes true and predicted values. This example uses `fbeta_score` with beta=0.75.

```python
from sklearn.metrics import fbeta_score

def f075(y_true, y_pred):
    return fbeta_score(y_true, y_pred, beta=0.75)
```

--------------------------------

### save(learner_path)

Source: https://github.com/educationaltestingservice/skll/blob/main/doc/api/learner.md

Save the current Learner instance to a file at the specified path.

```APIDOC
## save learner_path

### Description
Save the `Learner` instance to a file.

### Parameters
#### Path Parameters
- **learner_path** (skll.types.PathOrStr) - Required - The path to save the `Learner` instance to.

### Return type
None
```

--------------------------------

### learning_curve

Source: https://github.com/educationaltestingservice/skll/blob/main/doc/api/learner.md

Generate learning curves for the learner using the provided examples. This method uses cross-validation to assess model performance at different training set sizes.

```APIDOC
## learning_curve(examples, metric, cv_folds=10, train_sizes=array([0.1, 0.325, 0.55, 0.775, 1.]), override_minimum=False)

### Description
Generate learning curves for the learner using the examples.

The learning curves are generated on the training examples
via cross-validation. Adapted from the scikit-learn code for learning
curve generation (cf.
`sklearn.model_selection.learning_curve`).

* **Parameters:**
  * **examples** ([`skll.data.featureset.FeatureSet`](data.md#skll.data.featureset.FeatureSet)) – The `FeatureSet` instance to generate the learning curve on.
  * **cv_folds** (Union[int, [`skll.types.FoldMapping`](types.md#skll.types.FoldMapping)], default=10) – The number of folds to use for cross-validation, or a mapping from
    example IDs to folds.
  * **metric** (*str*) – The name of the metric function to use when computing the train
    and test scores for the learning curve.
  * **train_sizes** ([`skll.types.LearningCurveSizes`](types.md#skll.types.LearningCurveSizes), default= `numpy.linspace()` with start=0.1, stop=1.0, num=5) – Relative or absolute numbers of training examples
    that will be used to generate the learning curve.
    If the type is float, it is regarded as a fraction
    of the maximum size of the training set (that is
    determined by the selected validation method), i.e. it has to be within (0, 1]. Otherwise it
    is interpreted as absolute sizes of the training
    sets. Note that for classification the number of
    samples usually have to be big enough to contain
    at least one sample from each class.
  * **override_minimum** (*bool* *,* *default=False*) – Learning curves can be unreliable for very small sizes
    esp. for > 2 labels. If this option is set to `True`,
    the learning curve would be generated even if the number
    of example is less 500 along with a warning. If `False`,
    the curve is not generated and an exception is raised instead.
* **Returns:**
  * **train_scores** (*List[float]*) – The scores for the training set.
  * **test_scores** (*List[float]*) – The scores on the test set.
  * **fit_times** (*List[float]*) – The average times taken to fit each model.
  * **num_examples** (*List[int]*) – The numbers of training examples used to generate the curve.
* **Raises:**
  **ValueError** – If the number of examples is less than 500.
* **Return type:**
  *Tuple*[*List*[float], *List*[float], *List*[float], *List*[int]]
```

--------------------------------

### Configure Evaluation Metrics and Output Directories

Source: https://github.com/educationaltestingservice/skll/blob/main/doc/tutorial.md

Define additional evaluation metrics and specify output directories for results, logs, predictions, and trained models.

```python
metrics = ['roc_auc']
probability = true
logs = output
results = output
predictions = output
models = output
```

--------------------------------

### Learner.save

Source: https://github.com/educationaltestingservice/skll/blob/main/doc/api.md

Saves the Learner to a file.

```APIDOC
## Learner.save

### Description
Saves the Learner to a file.

### Method
`save(path)`

### Parameters
* **path** (str) - Required - Path to the file where the Learner will be saved.

### Returns
None
```

--------------------------------

### Run Specific Pre-commit Hook

Source: https://github.com/educationaltestingservice/skll/blob/main/CONTRIBUTING.md

Execute a specific pre-commit hook, like 'ruff', on changed files.

```bash
$ pre-commit run ruff
```

--------------------------------

### Define Custom Logistic Regression Wrapper

Source: https://github.com/educationaltestingservice/skll/blob/main/tests/other/custom_learner.txt

This snippet defines a new class `CustomLogisticRegressionWrapper` that inherits directly from `sklearn.linear_model.LogisticRegression`. This is a minimal example for testing custom learner functionality without adding any new methods or attributes.

```python
from sklearn.linear_model import LogisticRegression


class CustomLogisticRegressionWrapper(LogisticRegression):
    pass
```

--------------------------------

### utils Package

Source: https://github.com/educationaltestingservice/skll/blob/main/doc/api.md

Contains utility functions and constants for the SKLL library.

```APIDOC
## utils Package

### Description
Contains utility functions and constants for the SKLL library.

### Constants
- `CLASSIFICATION_ONLY_METRICS`: A list of metrics applicable only to classification tasks.
- `CORRELATION_METRICS`: A list of correlation-based metrics.
- `PROBABILISTIC_METRICS`: A list of metrics that handle probabilistic outputs.
- `REGRESSION_ONLY_METRICS`: A list of metrics applicable only to regression tasks.
- `UNWEIGHTED_KAPPA_METRICS`: A list of unweighted kappa metrics.
- `WEIGHTED_KAPPA_METRICS`: A list of weighted kappa metrics.

### Functions
- `get_skll_logger()`: Retrieves the SKLL logger instance.
```