3.2 Images && Arrays with ML5

What is Machine Learning?

Machine learning is an application of artificial intelligence (AI) that provides a system the ability to automatically learn and improve from experience without being explicitly programmed.

Rather than coding millions of conditionals, programmers set up a model that allow machines to learn from data. After they have started learning, machines will learn from all new data that they are given.

There are lots of examples of machine learning at play that you may not even realize are driven by machine learning:

• Your e-mail learns what spam looks like once you start marking messages as spam and filters them automatically. (It will get better and better at doing this over time!)
• Spotify learns what kind of music you listen to and then predicts things you might enjoy. (Netflix does this, too!)
• Pinterest, Amazon, and other websites also do this when they suggest similar things you like/looked at/have purchased.
• Siri, Alexa, and other personal assistance may use machine learning to adapt to the sound of your voice, the types of requests you are making, etc.

How do Machines Learn?

Machines must first be taught to learn. A simplified process might look something like this:

1. A human creates a model, which is a prediction that the machine will use based on the data they have RIGHT NOW.
2. Teach the model to the computer.
3. Give the computer LOTS AND LOTS AND LOTS OF DATA. SO MUCH DATA. (This is called the 'training set')
4. The model learns from this data and adjusts the model to fit what it is learning so it can make better predictions than with what the human initially told it. (It uses math to make these predictions better!)
5. Repeat steps 3 && 4
6. Keep repeating and adding new data until the computer has a high confidence in making its predictions.
7. The machine has learned && keeps learning!

How This Looks In a Basic Model

Let's imagine that it's the first day of school. A student walks in, and the teacher immediately hands her a test. The student has no idea what to do, but instead of teaching, the teacher just hands the student a page of answers. The student assumes they are correct and keeps filling them in.

Each day, the student comes in and the teacher repeats the pattern of giving a test and then a page of answers that the student fills in. Even though the student is filling in the given answers, she is still reading the questions and answer choices and learning from that. Eventually, the teacher could give her a blank test and she might remember enough to start filling in the answers on her own.

This is the basis of how a neural network, the backbone of ml5, is created, except the teacher is a programmer, and the student is the machine that is being taught to identify or answer something very specific. In the machine learning model, this test/answer page setup would happen thousands of times before the computer is ready to start answering on its own with confidence.

This sounds like a beautiful system, but it can be flawed. The problem with this is that we assume the teacher is giving her student the 'right answers,' and that she is learning correctly from them. What if the teacher was giving the student a times table each day, and every day was giving her answers so that 3x7=37? The student would learn incorrectly, and when asked to answer questions on their own would only know how to give the wrong answer. This can happen to machines, too, and they will then give (at best) inaccurate results, and (at worst) biased or racist results.

Working with a basic machine learning model

We are going to look at a basic machine learning example that was taught to identify what is in a picture. Usually, when computers see pictures, they just see a series of bits that tell it things like what colors go where, but the computer has no idea what's in the picture. (And even if it does know what the picture looks like, it doesn't know what to do with that information.) Our code uses ml5, a library like p5 that makes machine learning a little easier for everyone to use.

This works off of MobileNet, a model that was trained on thousands and thousands of pictures that exist in the public domain. Most of these pictures were of animals, so that is what we will test it on. Based on the confidence of the model and it's predictions, let's see if we can predict which animals it does best on!

To get to ml5 (so it can talk with our model), we need to make sure our HTML points to the ml5 library. If you've looked in the p5 library, you've seen something similar for p5!

All of this code can be found here - you can click FORK and it will make a copy that you can work into on your own.

Once we've established all our libraries in the HTML of our repl.it, we can dive back into the Javascript. There is a lot that goes into machine learning, and we aren't expecting you to know all of it - we are just going to focus on the p5 parts that we know how to use.

Look at the javascript below. What looks familiar to p5? What is a new part of the ml5 library? Do we have any theories about what is happening? Once you've read through the code, click run (on repl.it) and see what happens.

Adding Arrays And A Slider To Our Model

Right now, our model is pretty boring - it's just one picture, and there is no way for a user to interact. We want to make this more exciting, so we are going to do the following:

• Create an array of different images
• Create a slider so that the user can pick an image from the array
• Make the image appear on the screen and the model make a prediction about what it is seeing.

First, let's create an array that can hold all of our images. The initial repl.it has some images preloaded, and we will use these for now - eventually, you can go back and add in your own set of images and replace them in the array.

To add images to an array, we can do something like this, ideally at the top of our program where it has a global scope and can be used anywhere:

var listOfImg = ["cat.jpg", "dog.jpg", "mouse.jpg", "robin.jpg"]

This is great, but we haven't done anything with this array, nor do we have a way to call specific items from the array. While we could make an image randomly appear, that won't be super useful if we want to pull up information about a specific image. So, instead, let's create a slider that will pick an image for us. It might look something like the code below.

Notice that we have also used an array method to find the length of our array, rather than hard-coding in a number. This is useful if we want to add more images later - we don't have to worry about updating our code.

A final version of this can be found here.

Hypothesis On The Model

Your ml5 creation is currently running off of MobileNet - this is a machine learning model built by a person and trained on hundreds and thousands of free images that are in the public domain online. (Lots and lots of animal photos!)

This means that the things it has learned to identify have some bias - in this case, based on the images available and how it was taught to recognize them. There are lots of examples of machines inheriting the bias of their creators based on the model they are trained on - some example articles can be found in the resources section.

It's time for you to make a prediction about this model - for example, you might say, "I predict that based on how it was trained, this machine will recognize types of birds with greater accuracy and higher level of confidence then it will predict types of dogs." or "I predict that this model will be better able to recognize people with long hair than with short hair."

Test your Hypothesis on the Model && Justify your Conclusion

To test your hypothesis, you will need to find multiple images that you will load into your program and put into the array. You will probably want to find examples and non-examples for your model - if we are testing if the model is better at recognizing birds than at recognizing dogs, I will want to find images of both.

Remember, the model likes things without a busy background, so keep that in mind for accurate results.

Once you've updated your program and tested your model, answer the following:

• Summarize your results!
• Was your hypothesis accurate? Why do you think this? (Use data from your predictions to back up your answer!)
• How could this model be retrained to give better results?