Machining is becoming increasingly advanced to satisfy industrial precision standards of the components. With accuracy being analogous to quality, the study of tool wear is growing in importance as wear can notably affect the tool life. In smart manufacturing, machining processes incorporate artificial intelligence in machine vision to improve key process functions through metrology, improving dimensional accuracy and surface integrity of products. By enhancing machine vision or other suitable forms of optical metrology, it is possible to identify different wear types and their corresponding causative mechanisms. In this study, the flank wear region was identified and classified using deep learning features and transfer learning by pre-trained CNN models. Tool wear mechanism’s features extracted by Alexnet and VggNet16, have been used to train a Support Vector Machine (SVM) for classification. A comparison of fine-tuned CNN and CNN-SVM models has been made. The fine-tuned models employed backpropagation (BP) and stochastic gradient descent (SGD) to optimise the weights for performance improvement. The fine-tuned CNN models accomplished a higher accuracy compared to a CNN-SVM with an average validation accuracy of at least 95%. Fine-tuned Alexnet had a better performance than VggNet16 with an average validation accuracy of 96.43% and classification time of 0.244 s. With high accuracy and minimum time complexity, fine-tuned Alexnet can be used for online tool wear mechanisms classification. This implies that fine-tuned CNNs are effective and reliable in performing classification of tool wear.