SOCR ≫ DSPA ≫ Topics ≫

In the next several chapters we will concentrate of various progressively advanced machine learning, classification and clustering techniques. There are two categories of machine learning techniques - unsupervised and supervised (human-guided). In general, supervised classification methods aim to identify or predict predefined classes and label new objects as members of specific classes. Whereas, unsupervised clustering approaches attempt to group objects into subsets, without knowing a priori labels, and determine relationships between objects.

In the context of machine learning, classification is supervised learning and clustering is unsupervised learning.

Unsupervised classification refers to methods where the outcomes (groupings with common characteristics) are automatically derived based on intrinsic affinities and associations in the data without human indication of clustering. Unsupervised learning is purely based on input data (\(X\)) without corresponding output labels. The goal is to model the underlying structure, affinities, or distribution in the data in order to learn more about its intrinsic characteristics. It is called unsupervised learning because there are no a priori correct answers and there is no human guidance. Algorithms are left to their own devises to discover and present the interesting structure in the data. Clustering (discover the inherent groupings in the data) and association (discover association rules that describe the data) represent the core unsupervised learning problems. The k-means clustering and the Apriori association rule provide solutions to unsupervised learning problems.

Unsupervised Clustering Approaches
Bayesian
Hierarchical
Partitioning Based
Decision Based Non-parametric Divisive (Top-Down) Agglomerative (Bottom-Up) Spectral K-Means / Centroid Graph-Theoretic Model Based
Bayesian Classifier has high computational requirements. As there are a priori given labels, some prior is needed/specified to train the classifier, which is in turn used to label new data. Then, the newly labeled samples are subsequently used to train a new (supervised) classifier, i.e.,
decision-directed unsupervised learning. If the initial classifier is not appropriate, the process may diverge
Chinese restaurant process (CRP), infinite Hidden Markov Model (HMM) Principal Direction Divisive Partitioning (PDDP) Start with each case in a separate cluster and repeatedly join the closest pairs into clusters, until a stopping criterion is matched: (1) there is only a single cluster for all cases, (2) predetermined number of clusters is reached, (3) the distance between the closest clusters exceeds a threshold SpecC kmeans, hkmeans Growing Neural Gas mclust

Supervised classification methods utilize user provided labels representative of specific classes associated with concrete observations, cases or units. These training classes/outcomes are used as references for the classification. Many problems can be addressed by decision-support systems utilizing combinations of supervised and unsupervised classification processes. Supervised learning involves input variables (\(X\)) and an outcome variable (\(Y\)) to learn mapping functions from the input to the output: \(Y = f(X)\). The goal is to approximate the mapping function so that when it is applied to new (validation) data (\(Z\)) it (accurately) predicts the (expected) outcome variables (\(Y\)). It is called supervised learning because the learning process is supervised by initial training labels guiding and correcting the learning until the algorithm achieves an acceptable level of performance.

Regression (output variable is a real value) and classification (output variable is a category) problems represent the two types of supervised learning. Examples of supervised machine learning algorithms include Linear regression and Random forest, both provide solutions for regression-type problems, but Random forest also provides solutions to classification problems.

Just like categorization of exploratory data analytics, Chapter 3, is challenging, so is systematic codification of machine learning techniques. The table below attempts to provide a rough representation of common machine learning methods. However, it is not really intended to be a gold-standard protocol for choosing the best analytical method. Before you settle on a specific strategy for data analysis, you should always review the data characteristics in light of the assumptions of each technique and assess the potential to gain new knowledge or extract valid information from applying a specific technique.

Inference Outcome Supervised Unsupervised
Classification & Prediction Binary Classification-Rules, OneR, kNN, NaiveBayes, Decision-Tree, C5.0, AdaBoost, XGBoost, LDA/QDA, Logit/Poisson, SVM Apriori, Association-Rules, k-Means, NaiveBayes
Classification & Prediction Categorical Regression Modeling & Forecasting Apriori, Association-Rules, k-Means, NaiveBayes
Regression Modeling Real Quantitative (MLR) Regression Modeling, LDA/QDA, SVM, Decision-Tree, NeuralNet Regression Modeling Tree, Apriori/Association-Rules

Many of these will be discussed in later chapters. In this chapter, we will present step-by-step the k-nearest neighbor (kNN) algorithm. Specifically, we will demonstrate (1) data retrieval and normalization, (2) splitting the data into training and testing sets, (3) fitting models on the training data, (4) evaluating model performance on testing data, (5) improving model performance, and (6) determining optimal values of \(k\).

In Chapter 13, we will present detailed strategies, and evaluation metrics, to assess the performance of all clustering and classification methods.

1 Motivation

Classification tasks could be very difficult when the features and target classes are numerous, complicated or extremely difficult to understand. In those scenarios, where the items of similar class type tend to be homogeneous, nearest neighbor classification may be appropriate because assigning unlabeled cases to their most similar labeled neighbors may be fairly easy to accomplish.

Such classification methods can help us understand the story behind the unlabeled data using known data and avoiding analyzing those complicated features and target classes. This is because these techniques have no prior distribution assumptions. However, this non-parametric approach makes the methods rely heavy on the training instances, which explains their lazy algorithms designation.

2 The kNN algorithm Overview

The KNN algorithm involves the following steps:

  1. Create a training dataset that has classified examples labeled by nominal variables and different features in ordinal or numerical variables.
  2. Create a testing dataset containing unlabeled examples with similar features as the training data.
  3. Given a predetermined number \(k\), match each test case with the \(k\) closest training records that are “nearest” to the test case, according to a certain similarity or distance measure.
  4. Assign a test case class label according to the majority vote of the \(k\) nearest training cases.

Mathematically, for a given \(k\), a specific similarity metric \(d\), and a new testsing case \(x\), the kNN classifier performs two steps (\(k\) is typically odd to avoid ties):

  • Runs through the whole training dataset (\(y\)) computing \(d(x,y)\). Let \(A\) represent the \(k\) closest points to \(x\) in the training data \(y\).

  • Estimates the conditional probability for each class, which corresponds to the fraction of points in \(A\) with that given class label. If \(I(z)\) is an indicator function \(I(z) = \begin{cases} 1 & z= true \\ 0 & otherwise \end{cases}\), then the testing data input \(x\) gets assigned to the class with the largest probability, \(P(y=j|X=x)\):

\[P(y=j|X=x) =\frac{1}{k} \sum_{i\in A}{I(y^{(i)}=j)}.\]

2.1 Distance Function and Dummy coding

How to measure the similarity between records? We can think of similarity measures as the distance metrics between the two records or cases. There are many distance functions to choose from. Traditionally, we use Euclidean distance as our similarity metric.

If we use a line to connect the two points representing the testing and the training records in \(n\) dimensional space, the length of the line is the Euclidean distance. If \(a, b\) both have \(n\) features, the coordinates for them are \((a_1, a_2, ..., a_n)\) and \((b_1, b_2, ..., b_n)\). Our distance could be:

\[dist(a, b)=\sqrt{(a_1-b_1)^2+(a_2-b_2)^2+...+(a_n-b_n)^2}.\]

When we have nominal features, it requires a little trick to apply the Euclidean distance formula. We could create dummy variables as indicators of all the nominal feature levels. The dummy variable would equal to one when we have the feature and zero otherwise. We show two examples:

\[ Gender= \left\{ \begin{array}{ll} 0 & X=male \\ 1 & X=female \\ \end{array} \right. \] \[ Cold= \left\{ \begin{array}{ll} 0 & Temp \geq 37F \\ 1 & Temp < 37F \\ \end{array} \right. \]

This allows only binary expressions. If we have multiple nominal categories, just make each one as a dummy variable and apply the Euclidean distance.

2.2 Ways to Determine k

The parameter k could not be too large or too small. If our k is too large, the test record tends to be classified as the most popular class in the training records, rather than the most similar one. On the other hand, if the k is too small, outliers, noise, or mislabeled training data cases might lead to errors in predictions.

The common practice is to calculate the square root of the number of training cases and use that number as (initial) estimate of k.

A more robust way would be to choose several k values and select the one with the optimal (best) classifying performance.

2.3 Rescaling of the features

Different features might have different scales. For example, we can have a measure of pain scaling from 1 to 10 or 1 to 100. some similarity or distances measures assume the same measuring unit in all feature dimensions. This requires that the data may need to be transferred into the same scale. Re-scaling can make each feature contribute to the distance in a relatively equal manner, avoiding potential bias.

2.4 Rescaling Formulas

There are many alternative strategies to rescale the data.

2.4.1 min-max normalization

\[X_{new}=\frac{X-min(X)}{max(X)-min(X)}.\]

After re-scaling the data, \(X_{new}\) would range from 0 to 1, representing the distance between each value and its minimum as a percentage. Larger values indicate further distance from the minimum, 100% means that the value is at the maximum.

2.4.2 z-score standardization

\[X_{new}=\frac{X-\mu}{\sigma}=\frac{X-Mean(X)}{SD(X)}\]

This is based on the properties of normal distribution that we have talked about in Chapter 2. After z-score standardization, the re-scaled feature will have unbounded range. This is different from the min-max normalization which always have a finite range from 0 to 1. Following z-score standardization, the transformed features are uniteless and may resemble standard normal distribution.

3 Case Study: Youth Development

3.1 Step 1: Collecting Data

The data we are using for this case study is the “Boys Town Study of Youth Development”, which is the second case study, CaseStudy02_Boystown_Data.csv.

Variables:

  • ID: Case subject identifier
  • Sex: dichotomous variable (1=male, 2=female)
  • GPA: Interval-level variable with range of 0-5 (0-“A” average, 1- “B” average, 2- “C” average, 3- “D” average, 4-“E”, 5-“F”")
  • Alcohol use: Interval level variable from 0-11 (drink everyday - never drinked)
  • Attitudes on drinking in the household: Alcatt- Interval level variable from 0-6 (totally approve - totally disapprove)
  • DadJob: 1-yes, dad has a job and 2- no
  • MomJob: 1-yes and 2-no
  • Parent closeness (example: In your opinion, does your mother make you feel close to her?)
    • Dadclose: Interval level variable 0-7 (usually-never)
    • Momclose: interval level variable 0-7 (usually-never)
  • Delinquency:
    • larceny (how many times have you taken things >$50?): Interval level data 0-4 (never - many times),
    • vandalism: Interval level data 0-7 (never - many times)

3.2 Step 2: Exploring and preparing the data

First, we need to load and do some data manipulation. We are using the Euclidean distance so dummy variable should be used. The following codes transferred sex, dadjob and momjob into dummy variables.

boystown<-read.csv("https://umich.instructure.com/files/399119/download?download_frd=1", sep=" ")
boystown$sex<-boystown$sex-1
boystown$dadjob<--1*(boystown$dadjob-2)
boystown$momjob<--1*(boystown$momjob-2)
str(boystown)
## 'data.frame':    200 obs. of  11 variables:
##  $ id        : int  1 2 3 4 5 6 7 8 9 10 ...
##  $ sex       : num  0 0 0 0 1 1 0 0 1 1 ...
##  $ gpa       : int  5 0 3 2 3 3 1 5 1 3 ...
##  $ Alcoholuse: int  2 4 2 2 6 3 2 6 5 2 ...
##  $ alcatt    : int  3 2 3 1 2 0 0 3 0 1 ...
##  $ dadjob    : num  1 1 1 1 1 1 1 1 1 1 ...
##  $ momjob    : num  0 0 0 0 1 0 0 0 1 1 ...
##  $ dadclose  : int  1 3 2 1 2 1 3 6 3 1 ...
##  $ momclose  : int  1 4 2 2 1 2 1 2 3 2 ...
##  $ larceny   : int  1 0 0 3 1 0 0 0 1 1 ...
##  $ vandalism : int  3 0 2 2 2 0 5 1 4 0 ...

The str() function tells that we have 200 observations and 11 variables. However, the ID variable is not important in this case study so we can delete it. The variable of most interest is the GPA variable. We can classify it into two categories. Whoever gets a “C” or higher will be classified into the “above average” category; Students who have average score below “C” will be in the “average or below” category. These two are the classes of interest for this case study.

boystown<-boystown[, -1]
table(boystown$gpa)
## 
##  0  1  2  3  4  5 
## 30 50 54 40 14 12
boystown$grade<-boystown$gpa %in% c(3, 4, 5)
boystown$grade<-factor(boystown$grade, levels=c(F, T), labels = c("above_avg", "avg_or_below"))
table(boystown$grade)
## 
##    above_avg avg_or_below 
##          134           66

Let’s look at the proportions for the two categorizes.

round(prop.table(table(boystown$grade))*100, digits=1)
## 
##    above_avg avg_or_below 
##           67           33

We can see that most of the students are above average (67%).

The remaining 10 features are all numeric but with different scales. If we use these features directly, the ones with larger scale will have a greater impact on the classification performance. Therefore, re-scaling is needed in this scenario.

summary(boystown[c("Alcoholuse", "larceny", "vandalism")])
##    Alcoholuse       larceny       vandalism  
##  Min.   : 0.00   Min.   :0.00   Min.   :0.0  
##  1st Qu.: 2.00   1st Qu.:0.00   1st Qu.:1.0  
##  Median : 4.00   Median :1.00   Median :2.0  
##  Mean   : 3.87   Mean   :0.92   Mean   :1.9  
##  3rd Qu.: 5.00   3rd Qu.:1.00   3rd Qu.:3.0  
##  Max.   :11.00   Max.   :4.00   Max.   :7.0

3.3 Normalizing Data

First let’s create a function of our own using the min-max normalization formula. We can check the function using some trial vectors.

normalize<-function(x){
  return((x-min(x))/(max(x)-min(x)))
}

# some test examples:
normalize(c(1, 2, 3, 4, 5))
## [1] 0.00 0.25 0.50 0.75 1.00
normalize(c(1, 3, 6, 7, 9))
## [1] 0.000 0.250 0.625 0.750 1.000

After confirming the function definition, we can use lapply() to apply the normalization transformation to each element in a “list”. First, we need to make our dataset into a list. The as.data.frame() function convert our data into a data frame, which is a list of equal-length column vectors. Thus, each feature is an element in the list that we can apply the normalization function to.

boystown_n<-as.data.frame(lapply(boystown[-11], normalize))

Let’s examine one of the features that were normalized.

summary(boystown_n$Alcoholuse)
##    Min. 1st Qu.  Median    Mean 3rd Qu.    Max. 
##  0.0000  0.1818  0.3636  0.3518  0.4545  1.0000

This looks great! Now we can move to the next step.

3.4 Data preparation - creating training and test datasets

We have 200 observations in this dataset. The more data we use to train the the algorithm, the more precise the prediction would be. We can use \(3/4\) of the data for training and the remaining \(1/4\) for testing. For simplicity, we will just take the first cases as training and the remaining as testing, however, later we will see how to use a better randomization strategy to split the data into testing and training sets.

# may want to use random split of the raw data into training and testing
# subset_int <- sample(nrow(boystown_n),floor(nrow(boystown_n)*0.8))  
# 80% training + 20% testing
# bt_train<- boystown_n [subset_int, ]; bt_test<-boystown_n[-subset_int, ] 

bt_train<-boystown_n[1:150, ]
bt_test<-boystown_n[151:200, ]

Then let’s extract the labels or classes (column=11, Delinquency in terms of reoccurring vandalism) for our two subsets.

bt_train_labels<-boystown[1:150, 11]
bt_test_labels<-boystown[151:200, 11]

3.5 Step 3 - Training a model on the data

We are using the class package for the KNN algorithm in R.

#install.packages('class', repos = "http://cran.us.r-project.org")
library(class)

The function knn() has following components:

p<-knn(train, test, class, k)

  • train: data frame containing numeric training data (features)
  • test: data frame containing numeric testing data (features)
  • class/cl: class for each observation in the training data
  • k: predetermined integer indication the number of nearest neighbors

The first k we chose is the square root of our number of observations: \(\sqrt{200}\approx 14\).

bt_test_pred<-knn(train=bt_train, test=bt_test, 
                  cl=bt_train_labels, k=14)

3.6 Step 4 - Evaluating model performance

We utilize the CrossTable() function in Chapter 2 to evaluate the KNN model. We have two classes in this example. The goal is to create a \(2\times 2\) table that shows the matched true and predicted classes as well as the unmatched ones. However chi-square values are not needed so we use option `prop.chisq=False’ to get rid of it.

# install.packages("gmodels", repos="http://cran.us.r-project.org")
library(gmodels)
CrossTable(x=bt_test_labels, y=bt_test_pred, prop.chisq = F)
## 
##  
##    Cell Contents
## |-------------------------|
## |                       N |
## |           N / Row Total |
## |           N / Col Total |
## |         N / Table Total |
## |-------------------------|
## 
##  
## Total Observations in Table:  50 
## 
##  
##                | bt_test_pred 
## bt_test_labels |    above_avg | avg_or_below |    Row Total | 
## ---------------|--------------|--------------|--------------|
##      above_avg |           30 |            0 |           30 | 
##                |        1.000 |        0.000 |        0.600 | 
##                |        0.769 |        0.000 |              | 
##                |        0.600 |        0.000 |              | 
## ---------------|--------------|--------------|--------------|
##   avg_or_below |            9 |           11 |           20 | 
##                |        0.450 |        0.550 |        0.400 | 
##                |        0.231 |        1.000 |              | 
##                |        0.180 |        0.220 |              | 
## ---------------|--------------|--------------|--------------|
##   Column Total |           39 |           11 |           50 | 
##                |        0.780 |        0.220 |              | 
## ---------------|--------------|--------------|--------------|
## 
## 

In this table, the cells in the first row-first column and the second row-second column contain the number for cases that have predicted classes matching the true class labels. The other two cells are the counts for unmatched cases. The accuracy in this classifier is calculated by:\(\frac{cell[1, 1]+cell[2, 2]}{total}=\frac{40}{50}=0.8.\) Note that this value may slightly fluctuate each time you run the classifier, due to the stochastic nature of the algorithm.

3.7 Step 5 - Improving model performance

The above Normalization may be sub-optimal. We can try an alternative standardization method - standard Z-score centralization and normalization (via scale() method). Let’s give it a try:

bt_z<-as.data.frame(scale(boystown[, -11]))
summary(bt_z$Alcoholuse)
##     Min.  1st Qu.   Median     Mean  3rd Qu.     Max. 
## -2.04795 -0.98958  0.06879  0.00000  0.59798  3.77309

The summary() shows the re-scaling is working properly. Then, we can proceed to next steps (retraining the kNN, predicting and assessing the accuracy of the results):

bt_train<-bt_z[1:150, ]
bt_test<-bt_z[151:200, ]
bt_train_labels<-boystown[1:150, 11]
bt_test_labels<-boystown[151:200, 11]
bt_test_pred<-knn(train=bt_train, test=bt_test, 
                  cl=bt_train_labels, k=14)
CrossTable(x=bt_test_labels, y=bt_test_pred, prop.chisq = F)
## 
##  
##    Cell Contents
## |-------------------------|
## |                       N |
## |           N / Row Total |
## |           N / Col Total |
## |         N / Table Total |
## |-------------------------|
## 
##  
## Total Observations in Table:  50 
## 
##  
##                | bt_test_pred 
## bt_test_labels |    above_avg | avg_or_below |    Row Total | 
## ---------------|--------------|--------------|--------------|
##      above_avg |           30 |            0 |           30 | 
##                |        1.000 |        0.000 |        0.600 | 
##                |        0.750 |        0.000 |              | 
##                |        0.600 |        0.000 |              | 
## ---------------|--------------|--------------|--------------|
##   avg_or_below |           10 |           10 |           20 | 
##                |        0.500 |        0.500 |        0.400 | 
##                |        0.250 |        1.000 |              | 
##                |        0.200 |        0.200 |              | 
## ---------------|--------------|--------------|--------------|
##   Column Total |           40 |           10 |           50 | 
##                |        0.800 |        0.200 |              | 
## ---------------|--------------|--------------|--------------|
## 
## 

Under the z-score method, the prediction result is similar to previous result using the normalization strategy. Albeit, in general, there may be marginal differences, e.g., a few more cases may be correctly labeled based on one of the standardization or normalization approaches.

3.8 Testing alternative values of k

Originally, we used the square root of 200 as our k. However, this might not be the best k in this study. We can test different k’s for their predicting performances.

bt_train<-boystown_n[1:150, ]
bt_test<-boystown_n[151:200, ]
bt_train_labels<-boystown[1:150, 11]
bt_test_labels<-boystown[151:200, 11]
bt_test_pred1<-knn(train=bt_train, test=bt_test, 
                  cl=bt_train_labels, k=1)
bt_test_pred5<-knn(train=bt_train, test=bt_test, 
                  cl=bt_train_labels, k=5)
bt_test_pred11<-knn(train=bt_train, test=bt_test, 
                  cl=bt_train_labels, k=11)
bt_test_pred21<-knn(train=bt_train, test=bt_test, 
                  cl=bt_train_labels, k=21)
bt_test_pred27<-knn(train=bt_train, test=bt_test, 
                  cl=bt_train_labels, k=27)
ct_1<-CrossTable(x=bt_test_labels, y=bt_test_pred1, 
           prop.chisq = F)
## 
##  
##    Cell Contents
## |-------------------------|
## |                       N |
## |           N / Row Total |
## |           N / Col Total |
## |         N / Table Total |
## |-------------------------|
## 
##  
## Total Observations in Table:  50 
## 
##  
##                | bt_test_pred1 
## bt_test_labels |    above_avg | avg_or_below |    Row Total | 
## ---------------|--------------|--------------|--------------|
##      above_avg |           27 |            3 |           30 | 
##                |        0.900 |        0.100 |        0.600 | 
##                |        0.818 |        0.176 |              | 
##                |        0.540 |        0.060 |              | 
## ---------------|--------------|--------------|--------------|
##   avg_or_below |            6 |           14 |           20 | 
##                |        0.300 |        0.700 |        0.400 | 
##                |        0.182 |        0.824 |              | 
##                |        0.120 |        0.280 |              | 
## ---------------|--------------|--------------|--------------|
##   Column Total |           33 |           17 |           50 | 
##                |        0.660 |        0.340 |              | 
## ---------------|--------------|--------------|--------------|
## 
## 
ct_5<-CrossTable(x=bt_test_labels, y=bt_test_pred5, 
           prop.chisq = F)
## 
##  
##    Cell Contents
## |-------------------------|
## |                       N |
## |           N / Row Total |
## |           N / Col Total |
## |         N / Table Total |
## |-------------------------|
## 
##  
## Total Observations in Table:  50 
## 
##  
##                | bt_test_pred5 
## bt_test_labels |    above_avg | avg_or_below |    Row Total | 
## ---------------|--------------|--------------|--------------|
##      above_avg |           30 |            0 |           30 | 
##                |        1.000 |        0.000 |        0.600 | 
##                |        0.857 |        0.000 |              | 
##                |        0.600 |        0.000 |              | 
## ---------------|--------------|--------------|--------------|
##   avg_or_below |            5 |           15 |           20 | 
##                |        0.250 |        0.750 |        0.400 | 
##                |        0.143 |        1.000 |              | 
##                |        0.100 |        0.300 |              | 
## ---------------|--------------|--------------|--------------|
##   Column Total |           35 |           15 |           50 | 
##                |        0.700 |        0.300 |              | 
## ---------------|--------------|--------------|--------------|
## 
## 
ct_11<-CrossTable(x=bt_test_labels, y=bt_test_pred11, 
           prop.chisq = F)
## 
##  
##    Cell Contents
## |-------------------------|
## |                       N |
## |           N / Row Total |
## |           N / Col Total |
## |         N / Table Total |
## |-------------------------|
## 
##  
## Total Observations in Table:  50 
## 
##  
##                | bt_test_pred11 
## bt_test_labels |    above_avg | avg_or_below |    Row Total | 
## ---------------|--------------|--------------|--------------|
##      above_avg |           30 |            0 |           30 | 
##                |        1.000 |        0.000 |        0.600 | 
##                |        0.769 |        0.000 |              | 
##                |        0.600 |        0.000 |              | 
## ---------------|--------------|--------------|--------------|
##   avg_or_below |            9 |           11 |           20 | 
##                |        0.450 |        0.550 |        0.400 | 
##                |        0.231 |        1.000 |              | 
##                |        0.180 |        0.220 |              | 
## ---------------|--------------|--------------|--------------|
##   Column Total |           39 |           11 |           50 | 
##                |        0.780 |        0.220 |              | 
## ---------------|--------------|--------------|--------------|
## 
## 
ct_21<-CrossTable(x=bt_test_labels, y=bt_test_pred21, 
           prop.chisq = F)
## 
##  
##    Cell Contents
## |-------------------------|
## |                       N |
## |           N / Row Total |
## |           N / Col Total |
## |         N / Table Total |
## |-------------------------|
## 
##  
## Total Observations in Table:  50 
## 
##  
##                | bt_test_pred21 
## bt_test_labels |    above_avg | avg_or_below |    Row Total | 
## ---------------|--------------|--------------|--------------|
##      above_avg |           30 |            0 |           30 | 
##                |        1.000 |        0.000 |        0.600 | 
##                |        0.714 |        0.000 |              | 
##                |        0.600 |        0.000 |              | 
## ---------------|--------------|--------------|--------------|
##   avg_or_below |           12 |            8 |           20 | 
##                |        0.600 |        0.400 |        0.400 | 
##                |        0.286 |        1.000 |              | 
##                |        0.240 |        0.160 |              | 
## ---------------|--------------|--------------|--------------|
##   Column Total |           42 |            8 |           50 | 
##                |        0.840 |        0.160 |              | 
## ---------------|--------------|--------------|--------------|
## 
## 
ct_27<-CrossTable(x=bt_test_labels, y=bt_test_pred27, 
           prop.chisq = F)
## 
##  
##    Cell Contents
## |-------------------------|
## |                       N |
## |           N / Row Total |
## |           N / Col Total |
## |         N / Table Total |
## |-------------------------|
## 
##  
## Total Observations in Table:  50 
## 
##  
##                | bt_test_pred27 
## bt_test_labels |    above_avg | avg_or_below |    Row Total | 
## ---------------|--------------|--------------|--------------|
##      above_avg |           30 |            0 |           30 | 
##                |        1.000 |        0.000 |        0.600 | 
##                |        0.682 |        0.000 |              | 
##                |        0.600 |        0.000 |              | 
## ---------------|--------------|--------------|--------------|
##   avg_or_below |           14 |            6 |           20 | 
##                |        0.700 |        0.300 |        0.400 | 
##                |        0.318 |        1.000 |              | 
##                |        0.280 |        0.120 |              | 
## ---------------|--------------|--------------|--------------|
##   Column Total |           44 |            6 |           50 | 
##                |        0.880 |        0.120 |              | 
## ---------------|--------------|--------------|--------------|
## 
## 

The choice of \(k\) in KNN clustering is very important and it can be fine-tuned using the e1071::tune.knn() method.

# install.packages("e1071")
library(e1071)
knntuning = tune.knn(x= bt_train, y = bt_train_labels, k = 1:30)
knntuning
## 
## Parameter tuning of 'knn.wrapper':
## 
## - sampling method: 10-fold cross validation 
## 
## - best parameters:
##  k
##  5
## 
## - best performance: 0.1666667
summary(knntuning)
## 
## Parameter tuning of 'knn.wrapper':
## 
## - sampling method: 10-fold cross validation 
## 
## - best parameters:
##  k
##  5
## 
## - best performance: 0.1666667 
## 
## - Detailed performance results:
##     k     error dispersion
## 1   1 0.2266667 0.07166451
## 2   2 0.2666667 0.13333333
## 3   3 0.2066667 0.10159226
## 4   4 0.2133333 0.10795518
## 5   5 0.1666667 0.06478835
## 6   6 0.1666667 0.08461970
## 7   7 0.1733333 0.08432740
## 8   8 0.2000000 0.07027284
## 9   9 0.1933333 0.06629526
## 10 10 0.1800000 0.08344437
## 11 11 0.1866667 0.07568616
## 12 12 0.2133333 0.08777075
## 13 13 0.2000000 0.06285394
## 14 14 0.1933333 0.04919099
## 15 15 0.2066667 0.06629526
## 16 16 0.2133333 0.06126244
## 17 17 0.2133333