Numbers recognition

[JosselinSomerville]

Hi !
I've tried to implement the same neural network library as in the videos
However I have some problems

I am coding in Python, in Processing. Just to warn you, I am not really familiar with Python and Processing, that's why i have everything in one file ( I wasn't able to separate it in defferent files and then import them , if you know how to do it please tell me)

So the problem i am getting, is that the neural network is getting stuck. In fact I rarely go over 85% accuracy with is quite bad I think. And it doesn't matter how long i let it run, arround 85% , the accuracy just randomly changes. I know that my code is not really clear and messy , but could someone try to find what's going on ? Just so you know, I've stored, all the images in 60 000 files (1 per image ) , each containing a number between 0 and 1 for the pixel color

I've also added the possibility to have several hidden layers in the neural network
However, iam not sure if I am doing the backpropagation and the gradient descent correctly, could anyone check and tell me ?

Thanks a lot, I am new to GitHub so sorry if I did things wrong
( and by the way, I am French, that's why my english might be bad )

[JosselinSomerville]

import math

class Matrix:
    
    def __init__(self, nb_lignes = 1, nb_colonnes = 1, value = 0):
        self.n = nb_lignes
        self.m = nb_colonnes
        self.mat = []
        
        for i in range(0,self.n):
            self.mat.append([])
            for j in range(0,self.m):
                self.mat[-1].append(value)
    
        
        
    def show(self):
        maxLongueur = 1
        for ligne in self.mat:
            for elt in ligne:
                if(len(str(elt)) > maxLongueur):
                    maxLongueur = len(str(elt))
    
        
        for ligne in self.mat:
            string = "( "
    
            for elt in ligne:
                string += str(elt) + " "*(maxLongueur-len(str(elt)) + 1) + "| "
    
            string += ")"
            print(string)
            
    def __add__(self, B):
        """Additionne 2 matrices A et B de meme dimensions ( n x m )
        A une complexite en O(n x m) operations elementaires"""
        
        if(type(B) == type(self)):
            n = len(self.mat)
            m = len(self.mat[0])
        
            R = Matrix(n, m) # Matrice que l'on va renvoyer
        
            for i in range(n): # Pour chaque ligne
                for j in range(m): # Pour chaque case de la ligne
                    R.mat[i][j] = self.mat[i][j] + B.mat[i][j]
        
            return R
        elif(type(B) == type(1)):
            n = len(self.mat)
            m = len(self.mat[0])
            
            R = Matrix(n, m, 0) # Matrice que l'on va renvoyer
            
            for i in range(n): # Pour chaque ligne
                for j in range(m): # Pour chaque case de la ligne
                    R.mat[i][j] = self.mat[i][j] + B
                    
            return R
        else:
            return NotImplemented
        
        
    def __sub__(self, B):
        return self + B * (-1)
    
    
    def __mul__(self, B):
        """Calcule le produit A x B avec A de taille n x m et B de taille m x p ( AB de taille n x p )
        A une complexite en O(n x m * p) operations elementaires"""
        
        if(type(B) == type(self)):
            if(self.m == B.n):
                n = self.n
                m = B.n
                p = B.m
                #print("matrix(" + str(self.n) + "," + str(self.m) + ") * matrix(" + str(B.n) + "," + str(B.m) + ")")
            
                R = Matrix(n, p, 0) # Matrice que l'on va renvoyer
            
                for i in range(n): # Pour chaque ligne
                    for j in range(p): # Pour chaque case de la ligne
                        somme = 0
                        for k in range(m):
                            somme += self.mat[i][k] * B.mat[k][j]
                        R.mat[i][j] = somme
            
                return R
            elif((self.n == B.n) and (self.m == B.m)):
                #print("matrix x matrix (same dimensions)")
                R = Matrix(self.n, self.m, 0) # Matrice que l'on va renvoyer
            
                for i in range(self.n): # Pour chaque ligne
                    for j in range(self.m): # Pour chaque case de la ligne
                        R.mat[i][j] = self.mat[i][j] * B.mat[i][j]
            
                return R
            
        elif((type(B) == type(1.0)) or (type(B) == type(1))):
            #print("matrix * number")
            n = len(self.mat)
            m = len(self.mat[0])
            
            R = Matrix(n, m, 0) # Matrice que l'on va renvoyer
            
            for i in range(n): # Pour chaque ligne
                for j in range(m): # Pour chaque case de la ligne
                    R.mat[i][j] = self.mat[i][j] * B
                    
            return R
        else:
            print("not implemented")
            print(type(B), type(1))
            return NotImplemented
        
    
    def map_function(A, f):
        n = len(A.mat)
        m = len(A.mat[0])
            
        R = Matrix(n, m, 0) # Matrice que l'on va renvoyer
            
        for i in range(n): # Pour chaque ligne
            for j in range(m): # Pour chaque case de la ligne
                R.mat[i][j] = f(A.mat[i][j])
                    
        return R
    
    def randomize(self):
        n = len(self.mat)
        m = len(self.mat[0])
        
        for i in range(n): # Pour chaque ligne
            for j in range(m): # Pour chaque case de la ligne
                self.mat[i][j] = random(2) - 1
                
    def fromArray(self, arr):
        n = len(arr)
        m = 1
        A = Matrix(n,m)
        
        for i in range(n):
            A.mat[i][0] = arr[i]
        
        return A
            
    def toArray(self):
        arr = []
        for i in range(self.n): # Pour chaque ligne
            for j in range(self.m): # Pour chaque case de la ligne
                arr.append(self.mat[i][j])
        return arr
    
    
    def transpose(A):
        n = len(A.mat)
        m = len(A.mat[0])
            
        R = Matrix(m, n, 0) # Matrice que l'on va renvoyer
        for i in range(n): # Pour chaque ligne
            for j in range(m): # Pour chaque case de la ligne
                R.mat[j][i] = A.mat[i][j]
                
        return R


class NeuralNetwork:
    
    def sigmoid(x):
        return 1 / (1 + exp(-x))
    
    def derivative_sigmoid(y):
        return y * ( 1 - y)
    
    
    def __init__(self, nb_inputs, nb_layers, nb_nodes_per_hidden, nb_outputs, function = sigmoid, derivative = derivative_sigmoid):
        self.input_nodes = nb_inputs
        self.layers = nb_layers
        self.hidden_nodes = nb_nodes_per_hidden
        self.output_nodes = nb_outputs
        
        self.activation = function
        self.derivative_activation = derivative
        self.nb_train = 0
        self.learningRate = 0.1
        
        
        self.weights = []
        self.bias = []
        self.nb_char_weights = 15
        
        # On ajoute l'entree
        self.weights.append(Matrix(self.hidden_nodes, self.input_nodes))
        self.weights[-1].randomize()
        self.bias.append(Matrix(self.hidden_nodes, 1))
        self.bias[-1].randomize()
        
        # On ajoute les couches cachees
        for i in range(max(0, self.layers - 2)):
            self.weights.append(Matrix(self.hidden_nodes, self.hidden_nodes))
            self.weights[-1].randomize()
            self.bias.append(Matrix(self.hidden_nodes, 1))
            self.bias[-1].randomize()
        
        # On ajoute la sortie
        self.weights.append(Matrix(self.output_nodes, self.hidden_nodes))
        self.weights[-1].randomize()
        self.bias.append(Matrix(self.output_nodes, 1))
        self.bias[-1].randomize()

        
        
    def feed(self, input_array):
        results = []
        for i in range(self.layers + 1):
            results.append(Matrix())
            
        # On 'rentre' l entree
        results[0] = Matrix().fromArray(input_array)
        
        # On genere les resultats
        for i in range(self.layers):
            results[i+1] = self.weights[i] * results[i]
            results[i+1] = results[i+1] + self.bias[i]
            results[i+1] = Matrix.map_function(results[i+1], self.activation)
        
        return results[-1].toArray()
    
    
    def train(self, input_array, target_array):
        self.nb_train += 1
        
        # ---------- Feed function ------------- #
        results = []
        for i in range(self.layers + 1):
            results.append(Matrix())
            
        # On 'rentre' l entree
        results[0] = Matrix().fromArray(input_array)
        
        # On genere les resultats
        for i in range(self.layers):
            results[i+1] = self.weights[i] * results[i]
            results[i+1] = results[i+1] + self.bias[i]
            results[i+1] = Matrix.map_function(results[i+1], self.activation)
        
        
        
        
        # ---------- Computing errors ----------#
        targets = Matrix().fromArray(target_array)
        
        errors = []
        for i in range(self.layers):
            errors.append(Matrix())
            
        # On connait la dernière erreur
        errors[-1] = targets - results[-1]
        
        # On calcule les autres erreurs
        for i in range(self.layers - 2, -1, -1):
            weights_t = Matrix.transpose(self.weights[i+1])
            errors[i] = weights_t * errors[i+1]
        
    
        
        
        # ---------- Computing gradients --------#
        gradients = []
        for i in range(self.layers):
            gradients.append(Matrix())
            gradients[i] = Matrix.map_function(results[i+1], self.derivative_activation)
            gradients[i] = errors[i] * gradients[i]
            gradients[i] = gradients[i] * self.learningRate
            
        
        # ---------- Computing deltas ----------#
        deltas = []
        for i in range(self.layers):
            deltas.append(Matrix())
            results_t = Matrix.transpose(results[i])
            deltas[i] = gradients[i] * results_t
            
        
        # ----------- Adjusting weights ---------#
        for i in range(self.layers):
            self.weights[i] = self.weights[i] + deltas[i]
            self.bias[i] = self.bias[i] + gradients[i]
            
            
            
            
    
    
    def saveWeights(self, name_file, folder):
        file = open("C:/Users/josse/Documents/Processing/" + folder + "/data/weights/" + name_file, "w")
        string = str(self.nb_char_weights)+ "/"
        for matrix in self.weights:
            for ligne in matrix.mat:
                for elt in ligne:
                    l_round = self.nb_char_weights -(1 + len(str(int(elt))))
                    if(elt < 0):
                        l_round -= 1
                    l_zeros = self.nb_char_weights - len(str(round(elt, l_round)))
                    string += str(round(elt, l_round)) + "0"*l_zeros + "," 
                    
                    
        for matrix in self.bias:
            for ligne in matrix.mat:
                for elt in ligne:
                    l_round = self.nb_char_weights -(1 + len(str(int(elt))))
                    if(elt < 0):
                        l_round -= 1
                    l_zeros = self.nb_char_weights - len(str(round(elt, l_round)))
                    string += str(round(elt, l_round)) + "0"*l_zeros + "," 
                    
        #print(string)
        file.write(string)
        file.close()
        
    
    def loadWeights(self, name_file):
        file = open("C:/Users/josse/Documents/Processing/" + folder + "/data/weights/" + name_file, "rb")
        nb_char = int(file.read(2))
        file.read(1) # Skip "/"
        
        weights = []
        for matrix in self.weights:
            weights.append(Matrix(matrix.n, matrix.m))
            for i in range(matrix.n):
                for j in range(matrix.m):
                    weights[-1].mat[i][j] = float(file.read(nb_char))
                    file.read(1) # Skip ","
        
        bias = []
        for matrix in self.bias:
            bias.append(Matrix(matrix.n, matrix.m))
            for i in range(matrix.n):
                for j in range(matrix.m):
                    bias[-1].mat[i][j] = float(file.read(nb_char))
                    file.read(1) # Skip ","
                    
        self.weights = weights
        self.bias = bias
                    
            
        file.close()
            
        
        
        
            
               

    
    
    
file_weights = "weights_4_16.txt"
#file_weights = "weights_2_16.txt"
folder = "Neural_chiffres"
loadWeights = True
   
nn = NeuralNetwork(784,3,50,10)

nb_train = 60000
nb_test = 10000

nb_feed_par_test = 100
decrement_learning_rate = 0.05
save_tick = 1000
test_tick = 250


last_test = 0
last_saved = [hour(), minute(), second()]
last_predicted = 0
last_probabilite = 0


canvas = 0
zoom = 10

def setup():
    global  canvas
    canvas = []
    for i in range(784):
        canvas.append(0)
        
    if(loadWeights):
        nn.loadWeights(file_weights)
        
    size(84*zoom,28*zoom)
    print("Done")
    frameRate(30)
    nn.learningRate = 0.0065
    
    
def draw():
    global nb_train, nb_test, nn, canvas, last_saved, last_test, last_predicted, file_weights, test_tick, save_tick, folder
    
    clear()
    noStroke()
    textAlign(CENTER)
    

    # ============ Canvas ============= #
    for i in range(784):
        fill(canvas[i]*255)
        rect(i%28 * zoom, i // 28 * zoom, zoom, zoom)
    
    if(mousePressed):
        if(mouseX < 28*zoom):
            x = int(mouseX / zoom)
            y = int(mouseY / zoom)
            canvas[y*28 + x] = 1
            
            if(x > 0):
                canvas[y*28 + x -1] = 1
            if(x < 27):
                canvas[y*28 + x +1] = 1
            if(y > 0):
                canvas[(y-1)*28 + x] = 1
            if(y <27):
                canvas[(y+1)*28 + x] = 1
                
    
    # ============= Training =========== #
    index = int(random(nb_train))
    n,liste = read(index, "train")
    
    result = []
    for i in range(10):
        if(i == n):
            result.append(1)
        else:
            result.append(0)
                
    
    
    if((nn.nb_train % save_tick == 0) and (nn.nb_train >0)):
        nn.saveWeights(file_weights, folder)
        print("Saved")
        last_saved = [hour(), minute(), second()]

    if((nn.nb_train % test_tick == 0)):
        pourcentage = test(nb_feed_par_test)
        last_test = pourcentage
        #nn.learningRate = round((100 - pourcentage)**1 / 100.0,7) * decrement_learning_rate
        print("LearningRate = " + str(nn.learningRate))
        
    nn.train(liste, result)
    drawImage(liste)
            
                

        
        
    
    # Affichage infos
    fill(255)
    text("Nombre d'entrainements : " + str(nn.nb_train),42*zoom, 2*zoom)
    text("Vitesse : " + str(int(frameRate + 0.5)) + " entrainements / s", 42*zoom, 3.5*zoom)
    text("Derniere sauvegarde : ",42*zoom, 8.5*zoom)
    text(str(last_saved[0]) + "h " + str(last_saved[1]) + "m " + str(last_saved[2]) + "s",42*zoom, 10*zoom)
    text("Dernier test de " + str(nb_feed_par_test) + " : " ,42*zoom, 13.5*zoom)
    text("Taux de reussite : " + str(last_test) + " %",42*zoom, 15*zoom)
    text("Learning rate : " + str(nn.learningRate),42*zoom, 18.5*zoom)
    text("Prediction du dessin :",42*zoom, 25*zoom)
    text("C'est un : " + str(last_predicted) + " ( " + str(last_probabilite) + "% )",42*zoom, 26.5*zoom)
   
            

def drawImage(liste):
    global zoom
    img = createImage(28,28,RGB)
    
    for i in range(784):
        img.pixels[i] = color(liste[i]*255)
    
    img.resize(28*zoom, 28*zoom)
    image(img, 56*zoom,0)
        
    
            
    

def read(index, type):
    global folder
    file = open("C:/Users/josse/Documents/Processing/" + folder + "/data/rc/" + type + "/" + str(index) + ".txt", "rb")
    nb_char = int(file.read(1))
    num = int(file.read(1))
    liste = []
    
    for i in range(784):
        liste.append(float(file.read(nb_char)))
        file.read(1) # La virgule
        
    return num, liste
    

def keyPressed():
    global canvas, nn, last_saved, last_test, last_predicted, last_probabilite, file_weights, folder
    #print(key)
    if(key == " "):
        for i in range(784):
            canvas[i] = 0
    elif(key == "\n"):
        liste = []
        for i in range(len(canvas)):
            liste.append(canvas[i] * 2.0 - 1)
            
        outputs = nn.feed(liste)
        maximum = 0
        for i in range(len(outputs)):
            if(outputs[i] > outputs[maximum]):
                maximum = i
                
        print("C'est un : " + str(maximum))
        last_predicted = maximum
        last_probabilite = int(100 * outputs[maximum])
        print(outputs)
    elif(key == "t"):
        pourcentage = test(nb_feed_par_test)
        last_test = pourcentage
        #nn.learningRate = round((100 - pourcentage)**1 / 100.0,7) * decrement_learning_rate
        print("LearningRate = " + str(nn.learningRate))
        print("\nPourcentage = " + str(pourcentage) + "\n")
    elif(key == "g"):
        loop()
    elif(key == "s"):
        nn.saveWeights(file_weights, folder)
        print("Saved")
        last_saved = [hour(), minute(), second()]




def test(nb):
    global  nn, nb_test, nb_train        
        
    reussite = 0.0
    
    for i in range(nb):
        index = int(random(nb_train))
        n,liste = read(index, "train")
                
        outputs = nn.feed(liste)
        maximum = 0
        for i in range(len(outputs)):
            if(outputs[i] > outputs[maximum]):
                maximum = i
                
        if(maximum == n):
            reussite += 1
            
    return float(reussite/nb) * 100

[Versatilus]

Because Python uses formatting and white space for structure, I need you to edit your comment on GitHub website and put three backtick characters (`) and the word "python" on the line above your code and another three backtick characters on the line after your code. That will make GitHub do syntax highlighting for your code and preserve all of the white space.

```python
print "Countdown..."

for i in [3, 2, 1, 0]:
if i > 0:
print str(i) + "..."
else:
print "BLAST OFF!"
```

becomes

print "Countdown..."

for i in [3, 2, 1, 0]:
    if i > 0:
        print str(i) + "..."
    else:
        print "BLAST OFF!"

[JosselinSomerville]

Of course, I didn't know how to do it !
Here you go, I hope you find what's wrong

[JosselinSomerville]

By the way tell me if you need the code that creates the files for the input data

[Versatilus]

I'm still not done yet. I haven't forgotten about this. Every time I started working on this over the weekend I would fall asleep. It wasn't boring. I was just very sleepy all weekend!

[JosselinSomerville]

Oh thanks, I was a bit worried ! No problem, tell me if you need any precisions !

[tobiasvonarx]

I'm not sure about the error, you can find my version of a similar neural network here if it helps you. It's for number recognition as well and has a similar structure.

[tobiasvonarx]

To import python scripts (e.g. nn.py) from the same directory into another python file (e.g. main.py) you can do:

# import relevant classes from nn.py into main.py
from nn import Matrix, NeuralNetwork

with the folder structure looking like this:

/ nn.py
/ main.py

If you want the file in a different directory you will have to reference the folder:

from folder.nn import Matrix, NeuralNetwork

where the folder structure looks like this

/ main.py
    / folder
        / nn.py

Note that you might need to have a __init__.py file for this to work (to be included as a package), so you can create an empty one in the directory of the file you want to import (nn.py in this case). I hope I could help you 😉

[JosselinSomerville]

Thanks a lot ! I'm a bit tired right now, but I'll definitely check it out tomorrow.
I'll make sure to send you my feed back
Thanks !

[JosselinSomerville]

Hi ! I wanted to try your code but I couldn't because I don't know which files were needed and where to place them, could you tell me please or upload them on GitHub ?
Thanks !

[tobiasvonarx]

Hi, you can find my code by clicking here. You can either download the repository in its current state or fork it, whichever you prefer. To run it, you can run
python NeuralNetwork.py
from the root directory inside the repository folder. Make sure to use Python 3.x to run the script, otherwise errors will occur.

The python script NeuralNetwork.py contains both the NeuralNetwork class definition and the code for the MNIST classification in one file (i'll seperate them into seperate files some day). When running NeuralNetwork.py, it will look for pre-trained weight matrices in the data/ subdirectory. Make sure not to delete them, as the training process has 10 epochs and is very long-winded. When run, the code will query the NeuralNetwork about your own pictures, found in the my_own_images subdirectory, and output what it thinks in the console output. If you want to add your own, create a 28x28 png (without transparency and b/w works best) and rename it 2828_my_own_label, where label is the number your image is supposed to represent.

If you want to look into the code, I'd give you a tip and tell you that all code before line 90 belongs to the Neural Network itself, whereas everything after line 90 is specific to MNIST and number recognition.

[JosselinSomerville]

Yeah sure I'll try, it's just that I do not have the mnist.csv file. I have one mnist file but it's not à .csv file. So where did you download it ? Could you upload it maybe ?
Thanks again for your help !

[tobiasvonarx]

The one that I use can be found here, it isn't the full length version, as that exceeded GitHub's file size limit of 100MB. The original CSV can be downloaded from here

[JosselinSomerville]

Thanks ! And how accurate did you get with this setup ?
I'll try it tonight

[JosselinSomerville]

Hi ! I've tried your code and the results are amazing !
I've mosty understood you code, however, they are some parts that I don't really get :

self.who += self.lr * numpy.dot((output_errors * final_outputs * (1.0 - final_outputs)),
                                        numpy.transpose(hidden_outputs))

What is : (1.0 - final_outputs) ?

And now in the training process, you're doing some image treatment but, I don't really understand what you are doing with +10.0 and -1.0 ( could you explain please ? )

inputs = (numpy.asfarray(all_values[1:]) / 255.0 * 0.99) + 0.01
inputs_plus10 = rotate(inputs.reshape(28,28), 10.0, cval=0.01, order=1, reshape=False).reshape(784)
inputs_minus10 = rotate(inputs.reshape(28,28), -10.0, cval=0.01, order=1, reshape=False).reshape(784)
# create the target output values (all 0.01, except the desired label which is 0.99)
targets = numpy.zeros(output_nodes) + 0.01
# all_values[0] is the target label for this record
targets[int(all_values[0])] = 0.99
neuralNetwork.train(inputs, targets)
neuralNetwork.train(inputs_plus10, targets)
neuralNetwork.train(inputs_minus10, targets)

Anyway thanks a lot for sharing your code ! You're awesome !

[Versatilus]

final_outputs * (1.0 - final_outputs) is the derivative of the cost function, as the first step in back propagation. I haven't read his full source code, but I think that's to change the data up a little before training. Some research has shown improved accuracy on this dataset by skewing the images.

I still haven't given up on looking at your code, but every time I get settled down to go over it someone or something distracts me.

[tobiasvonarx]

@Versatilus summarized it perfectly, thank you for that. In addition, i'd like to clarify that for every image in the mnist dataset the code skews the picture, effectively rotating the picture 10 degrees in either direction (+10 and 10) and resulting in much more training data. Thank you for the positive feedback!

[Versatilus]

@JosselinSomerville I'm sorry that it's been almost a month and I still haven't finished looking at your code. I haven't forgotten about it. I've just been very busy, which is no real excuse. If you're still interested, I'll try to get to it soon. Assuming I don't nap too much, I'll try to get to it this weekend.