Note: This is my first ever blog post, so my apologies in advance if it's not up to par. Also, since this is my first post, I decided to not use any AI for polishing, etc., so that the reader can get a better feel for who I am as a writer. Introduction The purpose of this blog post is to document my journey, as a current non-professional or hobbyist, into the world of Artificial Intelligence, more specifically, into Machine Learning. This journey includes a learning phase, where I both learn the concepts and the math behind Machine Learning, and an implementation phase, where I learn a new programming language and endeavor to implement the math that I learned about in code, both on the CPU and the GPU, and then endeavor to make the functions I'd written as fast and accurate, and the code as readable, as I knew how. I also discuss my reasoning behind my choice of programming language as well as why I chose to implement a Multi-Layer Perceptron (MLP) instead of a more advanced Neural Network (NN), as well as certain other design choices. A follow-up blog post, which I will publish soon, will detail my experiences with data augmentation for image-based datasets and will include descriptions of my implementation of various image processing functions such as rotations and distortions. The Learning Phase A few years ago, I became aware of the existence of a field of study known as Machine Learning. It immediately piqued my interest and seeing its potential and that it was probably the wave of the future I wanted to learn more about it. Consequently, I soon enrolled in a couple of Nanodegree programs through the online learning platform Udacity. The first Nanodegree I enrolled in was called AI Programming with Python, an Introductory Nanodegree, and the second one was called Deep Learning, an Intermediate Nanodegree. I completed these programs successfully with the intention of completing an Advanced Nanodegree but was unable to for financial reasons. However, I wasn't really satisfied with my understanding of the math behind these technologies after having completed these programs, so I took it upon myself to work out much of the math for the standard ML algorithm, both for the Feed Forward (FF) phase as well as the Back-Propagation (BP) phase, and then I implemented this math in code both on the CPU and the GPU in a fairly new high level programming language called Julia. (As an aside, these Nanodegrees were my only experience with Python, most of which I've forgotten.) Now, I'm not saying I was able to figure out from scratch the entire algorithm for a MLP/FFN. Far from it. Numerous researchers over a period of decades who had a deeper understanding of certain branches of Mathematics (one of which was Optimization, I believe) were able to do that. But what I was able to do, after some starts and stops, is figure out how to compute all the derivatives necessary for the BP step, for both a batch size of 1 and a batch size > 1. (Of course, I also worked out the math for the feed forward step, but that was somewhat trivial by comparison.)¹ After having chosen the language I would use to implement an MLP in code I got to work learning it. Essentially, I was both learning how to correctly work out the math for an MLP and learning a new programming language at the same time. Initially, I made the decision to only implement in code what I could workout in Math. I don't currently operate by that constraint, but it's the main reason why I never implemented a Convolutional Neural Network (CNN) in code, because I could never figure out what the derivative of the convolutional operation was. The Implementation Phase But I'd mostly figured out the math for an MLP, so I set out implementing the various derivatives for BP for an MLP as functions, implementing the FF portion as a function, and trying various ways to organize the code to make it more efficient and readable. I soon discovered that training an MLP (or any NN, I'd wager) requires a fair amount of patience, since it often takes a good long while to find the right set of hyperparameters to get good results (especially when one is not using any optimizers, which is the case for me). I have spent countless hours and hundreds or even thousands of training cycles trying to achieve the goals I initially set for my NN, which was 99% accuracy on the training dataset with a training time not to exceed 5 minutes. While I eventually surpassed those goals, there were times when I got frustrated and discouraged at my lack of progress, and in fact there was a period of several months where I'd just given up and decided that it was too much for me and that I would never reach my goals. But at some point I decided to try again, and so I approached the problem with new enthusiasm and renewed energy. Since I was almost obsessed with reaching the goals I'd set, and since I hadn't quite reached them, I was constantly reading about recent or well-established methods of improving the performance and generalization of neural networks. The two technologies I learned about that had the greatest impact on the performance of my NN are Data Augmentation and Momentum, especially DA. I also tried to implement other technologies like Batch Normalization and Dropout, but they didn't seem to have much of an effect on my MLP, so I did not keep them. Perhaps they are best used for CNNs and other more advanced neural networks. Long story short, after coming back to the problem after my brief period of giving up, and after having created my own image distortion method, implementing it in code, and using it for data augmentation, I finally reached the goals I'd set of at least 99% accuracy on the training dataset and no greater than 5 minutes of training time. In fact, I didn't just meet those goals, I surpassed them. Not only did I achieve 99% accuracy on the training set (and validation set, I might add) I eventually achieved at least 99% accuracy on the MNIST Test set for a final accuracy of 99.46%, and I did that in a training time of about 2 minutes, which is more than twice as fast as my initial goal.² Explanation of Certain Choices Many reading this blog may be thinking to themselves, "Why was this guy working on an MLP when the world has moved way beyond MLP's to LLM's, etc.?" Well, apart from the issue I mentioned earlier about CNNs, the purpose for working on an MLP is because I felt that it was one of the bedrocks of ML, and so the concepts learned working on an MLP, especially back-propagation and gradient descent, also apply to more advanced NNs. So, it was intended as a way to learn about the fundamentals of ML. Furthermore, I worked on it because it involved Math that I was more or less already familiar with, though I was by no means an expert at any of it. However, I also had to learn some new math that I hadn't been exposed to before, in particular, how to take the derivative of a matrix wrt another matrix, and similar operations. In short, I worked on it because it involved math I could handle. (The same might also be true of LLM's, but I don't know that for sure since I haven't studied them yet.) Finally, I worked on it out of curiosity, since I'd read somewhere about a theorem that says that a neural network can theoretically compute any function (though I might have misunderstood that to imply that MLPs can compute any function). Well, that inspired me to try to push my MLP as far as I could push it and see just how high an accuracy I could get on the MNIST Test dataset. I feel like I did pretty well, since I maxed out at a respectable 99.46% accuracy.³ Another question someone might be asking is, "Why Julia? Julia is a niche language that's primarily used for scientific computing and research. It doesn't have a very large user base for ML." There are 5 main reasons why I chose Julia over the more popular Python for implementing an MLP: It is a high-level language but it has near the performance of a low-level language for the types of problems I would be working on, and it doesn't require importing a C compiled library in order to get high performance. It has a (mostly) pleasing syntax, and its design felt more natural to me, especially multiple dispatch. It has the most mature GPU programming support, especially for CUDA, of any language I'm aware of outside of C/C++ (GPU programming is one of my main interests). I wasn't aware at the time that it was a niche language that wasn't used very much in the world of ML, though I did know that it was a new language and had a small userbase. However, I was motivated primarily by both performance and ease of use, not popularity. The niche for which Julia is known for is the one I'm most interested in, namely scientific computing and high-performance computing. (I'm also interested in ML, of course, so I still hold out hopes that Julia will gain market share in this field as well, but it's not really a requirement for me to appreciate it and use it.) Finally, one of my overarching goals for this project was to use as few libraries as possible. That way, I would get more experience with pure Julia rather than relying on library functions to do something for me. So in general, whatever I needed a function to accomplish, I coded. With that said, here is a full list of my imports: using Base.Threads; using Images; using LinearAlgebra; using Plots; using Random; using BenchmarkTools; using CUDA; using IntelVectorMath; # Used for faster processing on the CPU of trig. and other functions with vector inputs using MLDatasets; # Handles setting up the various training datasets such as MNIST and CIFAR10 using ProgressMeter; # Used for pretty printing of training performance metrics So, the core augmentation and ML processing, as well as almost all supporting functions, was done using my own code⁴ (including bilinear interpolation for image transformations, but that's the subject of another blog post that is soon to be published). Performance Characteristics As was stated earlier, I achieved a high of 99.46% accuracy on the MNIST Test dataset. But this required two technologies: Data Augmentation, which is done before training, and Momentum, which is used during training. During the data preparation phase, I perform 16 random augmentations (which for now consists solely of random distortions using my own image distortion method) for a total of 1,020,000 images, which is divided into 940,032 images for training and 79,968 images for validation. I do not perform augmentation on the Test dataset. I train on these augmented images on a 1660ti MaxQ laptop GPU for 15 epochs, which takes about 2 minutes to complete. This equates to about 8 s per epoch for all 1,030,000 images (1,020,000 for training and validation + 10,000 for testing), which further equates to about 7.8 μs per image. A further breakdown of the performance of my code is as follows: using what I call the "high performance" settings, which were hyperparameters chosen expressly for the purpose of seeing how fast my code could complete the MLP algorithm on the un-augmented MNIST dataset (Hidden Layer size 128, Batch Size 512, relu), my code can complete the entire ML algorithm for all 70000 28x28 images in roughly 75 ms, which equates to about 1.25 μs per image; but for the hyperparameters that I used to get 99.46% accuracy (HL size 2048, BS 128, silu), i.e., the "high quality" settings, it takes about 470 ms to do the same thing, which, again, equates to about 7.8 μs per image. Again, this was done on a 6-year-old 1660ti MaxQ laptop GPU. Wrap-Up You probably noticed that I emphasized "I" a lot or "my code" a lot. Well, that's by design, because the reality is, as a current hobbyist and having never completed a degree or gotten a job as a programmer, I don't have years of professional experience working on projects, solving problems, and shipping product, so I feel like I have to emphasize what I coded and what I accomplished in my spare time as a hobbyist, since I don't have a resume highlighting these things. Finally, I hope this wasn't too much of a slog to read through and that someone gained something by reading it. Again, this was my first ever blog post, so if anyone has any suggestions or critiques on how I can improve, please let me know. ... But in a respectful way, please. Take care. Notes ¹ It was particularly challenging to figure out how the various derivatives for a batch size > 1 needed to be multiplied together and implemented in code, but I eventually figured it out, after a lot of trial and error, and then eventually wrote functions that would allow my code to do this as efficiently as I knew how. One of the errors I referenced was an error in my early program in the math for one of the derivatives. I wasn't aware at first that I'd made an error in the math, but as I tested my program, I'd noticed that I wasn't getting a very high accuracy for a subset of MNIST that contained only 0s and 1s. I'd read somewhere on one of the Stanford University tutorials or notes pages that one should be getting 100% accuracy for this subset, but I was not getting that high an accuracy. I finally tracked the discrepancy down to an error I'd made in one of the derivatives for back-propagation, and after I corrected this error, my NN finally performed the way the literature said it was supposed to perform on this subset. Again, this was an early version of my program before I'd figured everything out and well before I'd implemented DA or momentum. ² As an aside, I have seen 100% accuracy for both training and validation, but such a high accuracy doesn't necessarily equate to a similar accuracy on the Test dataset. So, it appears that a neural network can overfit on both the training set as well as the validation set, though I suspect there's a better theoretical way to describe what's happening in this case other than "overfitting". ³ For comparison, the highest known accuracy for an MLP for MNIST is 99.69% which was achieved by a team at IDSIA (Istituto Dalle Molle di Studi sull'Intelligenza Artificiale) in Manno-Lugano, Switzerland, specifically Dan Cireşan, Ueli Meier, Luca Gambardella, and Jürgen Schmidhuber, in their 2011 paper "Handwritten Digit Recognition with a Committee of Deep Neural Nets on GPUs." Without the committee of deep neural nets their highest achieved accuracy was 99.65%, which was part of a prior 2010 paper. Also, their network architecture was as follows: 29^2 → 2500 → 2000 → 1500 → 1000 → 500 → 10, whereas mine is just 28^2 → 2048 → 10. So, I think that 99.46% accuracy is not too shabby a result for a non-professional self-taught tinkerer. :) ⁴ As a side project, to gain even more experience with writing Julia functions, I decided to write my own versions of functions that I knew already existed, such as broadcast and zeros. This was initially done to simply get practice writing GPU kernels, then to add multi-threading capability to built-in functions, but it was also done out of curiosity. I discovered that I could often write functions that have a built-in counterpart that can often outperform the built-in counterpart, sometimes by a wide margin and even without using multithreading, though this wasn't always the case. More often than not, the functions I wrote that had a built-in counterpart were merely on par with the built-in counterpart. However, built-in functions in Julia (as far as I know) are not multi-threaded, so it's nice to know that one can get a bit more performance out of these functions by creating multi-threaded counterparts.