In this article we will talk about the usage of a Genetic Algorithm approach to optimize Keras Neural Network that may use 2 types of Hidden Layers (Dense and/or Dropout) mixed.
To accomplish that goal, we work to optimize a Neural Network created in my previous article: https://www.theobservator.net/neural-network-for-open-ai-cartpole-v1-challenge-with-keras/
To learn and understand more about Genetic Algorithm, click here.
If you are anxious for the code, click > https://github.com/guibacellar/OpenAi/blob/master/CartPole-GA.py
I strongly recommend you to read the previous links about the Genetic Algorithm (in case you don’t have any idea about Genetic Algorithm) and the link to the CartPole-V1 challenge.
Otherwise, this article may be confuse for who don’t have the correct background.
OK, let’s dive into code
Genetic Algorithm Parameters
# Define Hyperparameters for NN
HIDDEN_LAYER_COUNT = [1, 2, 3, 4, 5, 6, 7, 8, 9, 10]
HIDDEN_LAYER_NEURONS = [8, 16, 24, 32, 64, 128, 256, 512]
HIDDEN_LAYER_RATE = [0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 0.9]
HIDDEN_LAYER_ACTIVATIONS = ['tanh', 'relu', 'sigmoid', 'linear', 'softmax']
HIDDEN_LAYER_TYPE = ['dense', 'dropout']
MODEL_OPTIMIZER = ['adam', 'rmsprop']
# Define Generic Algorithm Parameters
MAX_GENERATIONS = 50 # Max Number of Generations to Apply the Generic Algorithm
POPULATION_SIZE = 20 # Max Number of Individuals in Each Population
BEST_CANDIDATES_COUNT = 4 # Number of Best Candidates to Use
RANDOM_CANDIDATES_COUNT = 2 # Number of Random Candidates (From Entire Population of Generation) to Next Population
OPTIMIZER_MUTATION_PROBABILITY = 0.1 # 10% of Probability to Apply Mutation on Optimizer Parameter
HIDDEN_LAYER_MUTATION_PROBABILITY = 0.1 # 10% of Probability to Apply Mutation on Hidden Layer Quantity
First of all, let’s define our parameters range to use for you Neural Network.
The parameter range consist in range of all possible values for each parameters that we want to evaluate for the Neural Network topology. In this example case we defines the Hidden Layers Count, Layer Size, Layer Neuron Count and Activation Function (Only for Dense Layer), Layer Dropout rate (only for Dropout Layer) and whole model optimization strategy.
Then, we have all definitions for the Genetic Algorithm processing it self. That contains the Number of Generations to Run, Population Size, Candidates Selection Count (Best and Random) and Mutation Probabilities Rate.
Chromosome and Hidden Layer Layout
class LayerLayout:
def __init__(self, layer_type):
self.neurons = None
self.activation = None
self.rate = None
self.layer_type = layer_type
class Chromosome:
def __init__(self, layer_layout, optimizer, specie):
self.layer_layout = layer_layout
self.optimizer = optimizer
self.result_worst = None
self.result_best = None
self.result_avg = None
self.result_sum = None
self.specie = specie
self.ml_model = None
def safe_get_hidden_layer_node(self, index=0):
if len(self.layer_layout) > index:
return self.layer_layout[index]
return None
Using 2 new classes, one for Hidden Layer Specification (LayerLayout) and other one for the Chromosome (Chromosome).
This 2 classes holds all information that’s our algorithm needs to run, evaluate, evolve, mutate and run it again, again, again, and again over the Generations.
Main Process
def main():
# Play Random (Initial) Games to create Test and Training Data
df = play_random_games(games=10000)
print("\n********** Reference Network Model **********")
# Creates a Reference NN Model bases on CartPole.py Sample
ml_model = generate_reference_ml(df)
# Play Games with Reference NN Model
print("[+] Playing Games with Reference NN Model \t>", end='', flush=True)
ref_worst, ref_best, ref_avg, ref_sum = play_game(ml_model=ml_model, games=100)
print(f"\tWorst Score:{ref_worst} | Average Score:{ref_avg} | Best Score:{ref_best} | Total Score:{ref_sum}")
# >>>>>> Genetic Algorithm Section <<<<<<
print("\n********** Genetic Algorithm **********")
population = generate_first_population_randomly(
population_size=POPULATION_SIZE
)
# Run Each Generation
for current_generation in range(MAX_GENERATIONS):
print(f"[+] Generation {current_generation+1} of {MAX_GENERATIONS}")
i = 0
# >>>>>> Training Phase <<<<<<
print(f"\tTraining Models: ", end='', flush=True)
training_start = time.time()
# Train all Models in Population
for individual in population:
generate_model_from_chromosome(df, individual)
training_stop = time.time()
print(f"Done > Takes {training_stop - training_start} sec")
# >>>>>> Evaluation Phase <<<<<<
print(f"\tEvaluating Population: ", end='', flush=True)
evaluation_start = time.time()
for individual in population:
# Play the Games
score_worst, score_best, score_avg, score_sum = play_game(
ml_model=individual.ml_model,
games=100,
model_name=individual.specie
)
# Update Chromosome Results
individual.result_worst = score_worst
individual.result_best = score_best
individual.result_avg = score_avg
individual.result_sum = score_sum
# Update Indexer
i += 1
evaluation_stop = time.time()
print(f"Done > Takes {evaluation_stop - evaluation_start} sec")
# Sort Candidates by Sum of Results
population.sort(key=lambda x: x.result_sum, reverse=True)
# Compute Generation Metrics
gen_score_avg = np.average([item.result_avg for item in population])
gen_score_min = np.min([item.result_worst for item in population])
gen_score_max = np.max([item.result_best for item in population])
gen_score_sum = np.sum([item.result_sum for item in population])
print(f"\tWorst Score:{gen_score_min} | Average Score:{gen_score_avg} | Best Score:{gen_score_max} | Total Score:{gen_score_sum}")
# >>>>>> Genetic Selection, Children Creation and Mutation <<<<<<
population = evolve_population(population)
Our main process consist into execute the Generic Algorithm steps:
- Generate First Population
- Evaluate Generations
- Training Population
- Evaluate Individuals
- Select Parents
- Select Random Individuals
- Create Children;s
- Mutate new Children’s
Generating the First Population
def generate_first_population_randomly(population_size=10):
result = []
for current in range(population_size):
# Choose Hidden Layer Count
hidden_layer_count = HIDDEN_LAYER_COUNT[random.randint(0, len(HIDDEN_LAYER_COUNT)-1)]
hidden_layer_layout = []
# Define Layer Structure
for current_layer in range(hidden_layer_count):
hidden_layer_layout.append(create_random_layer())
chromosome = Chromosome(
layer_layout=hidden_layer_layout,
optimizer=MODEL_OPTIMIZER[random.randint(0, len(MODEL_OPTIMIZER)-1)],
specie=f"I {current}"
)
result.append(chromosome)
return result
def create_random_layer():
layer_layout = LayerLayout(
layer_type=HIDDEN_LAYER_TYPE[random.randint(0, len(HIDDEN_LAYER_TYPE) - 1)]
)
if layer_layout.layer_type == 'dense':
layer_layout.neurons = HIDDEN_LAYER_NEURONS[random.randint(0, len(HIDDEN_LAYER_NEURONS) - 1)]
layer_layout.activation = HIDDEN_LAYER_ACTIVATIONS[random.randint(0, len(HIDDEN_LAYER_ACTIVATIONS) - 1)]
elif layer_layout.layer_type == 'dropout':
layer_layout.rate = HIDDEN_LAYER_RATE[random.randint(0, len(HIDDEN_LAYER_RATE) - 1)]
return layer_layout
Initially, we generate a completely random population. Each individual with a random number of Layers, random Optimizer and random configuration for each Hidden Layer.
Converting Layer Layout and Chromosome into a Neural Network Model
def generate_model_from_chromosome(df, chromosome):
# Define Neural Network Topology
m_model = Sequential()
# Define Input Layer (Fixed)
m_model.add(InputLayer(input_shape=(4,)))
# Add Hidden Layers
for layer in chromosome.layer_layout:
if layer.layer_type == 'dense':
m_model.add(
Dense(
layer.neurons,
activation=layer.activation
)
)
elif layer.layer_type == 'dropout':
m_model.add(
Dropout(rate=layer.rate)
)
# Define Output Layer (Fixed)
m_model.add(Dense(2, activation='sigmoid'))
# Compile Neural Network
m_model.compile(optimizer=chromosome.optimizer, loss='categorical_crossentropy')
# Fit Model with Data
m_model.fit(
df[['cart_position', 'cart_velocity', 'pole_angle', 'pole_velocity_at_tip']],
df[['action_to_left', 'action_to_right']],
epochs=20,
verbose=0
)
# Update Model into Chromosome
chromosome.ml_model = m_model
Now, it’s time to convert our definition models (LayerLayout and Chromosome) into a valid Keras Neural Network Model. During this procedure, we build, compile and fit the NN model.
Children Generation
After each generation fitness and evaluation we need to generate a fresh new children from selected parents. Again, it is time to use some randomness in the process.
def generate_children(mother, father):
# Layer Layout
c_layer_layout = []
layers_counts = len(mother.layer_layout) if random.randint(0, 1) == 0 else len(father.layer_layout)
for ix in range(layers_counts):
c_layer_layout.append(
mother.safe_get_hidden_layer_node(ix) if random.randint(0, 1) == 0 else father.safe_get_hidden_layer_node(ix)
)
# Remove all Nones on Layers Layout
c_layer_layout = [item for item in c_layer_layout if item is not None]
# Optimizer
c_optimizer = mother.optimizer if random.randint(0, 1) == 0 else father.optimizer
chromosome = Chromosome(
layer_layout=c_layer_layout,
optimizer=c_optimizer,
specie=""
)
return chromosome
Out algorithm select characteristics from one of the parents randomly.
Population Evolve Process
That’s are our primary process select the parents (The N Best and Y Random), create and evolve the next population of individuals based on selected parents and apply some mutations.
def evolve_population(population):
# Clear Graphs from Keras e TensorFlow
K.clear_session()
tf.reset_default_graph()
# Select N Best Candidates + Y Random Candidates. Kill the Rest of Chromosomes
parents = []
parents.extend(population[0:BEST_CANDIDATES_COUNT]) # N Best Candidates
for rn in range(RANDOM_CANDIDATES_COUNT):
parents.append(population[random.randint(0, POPULATION_SIZE - 1)]) # Y Random Candidate
# Create New Population Through Crossover
new_population = []
new_population.extend(parents)
# Fill Population with new Random Children with Mutation
while len(new_population) < POPULATION_SIZE:
parent_a = random.randint(0, len(parents) - 1)
parent_b = random.randint(0, len(parents) - 1)
while parent_a == parent_b:
parent_b = random.randint(0, len(parents) - 1)
new_population.append(
mutate_chromosome(
generate_children(
mother=parents[parent_a],
father=parents[parent_b]
)
)
)
return new_population
Mutation
It’s time to apply mutation. That’s are very import into the process to ensure that we have some chance to evolve and keep variations from the parent chromosomes definition. Otherwise, our algorithm are unable to find new combinations and other alternatives to the problem they are trying to resolve.
def mutate_chromosome(chromosome):
# Apply Mutation on Optimizer
if random.random() <= OPTIMIZER_MUTATION_PROBABILITY:
chromosome.optimizer = MODEL_OPTIMIZER[random.randint(0, len(MODEL_OPTIMIZER)-1)]
# Apply Mutation on Hidden Layer Size
if random.random() <= HIDDEN_LAYER_MUTATION_PROBABILITY:
new_hl_size = HIDDEN_LAYER_COUNT[random.randint(0, len(HIDDEN_LAYER_COUNT)-1)]
# Check if Need to Expand or Reduce Layer Count
if new_hl_size > len(chromosome.layer_layout):
# Increase Layer Count
while len(chromosome.layer_layout) < new_hl_size:
chromosome.layer_layout.append(
create_random_layer()
)
elif new_hl_size < len(chromosome.layer_layout):
# Reduce Layers Count
chromosome.layer_layout = chromosome.layer_layout[0: new_hl_size]
else:
pass # Do not Change Layer Size
return chromosome
Access the code at https://github.com/guibacellar/OpenAi/blob/master/CartPole-GA.py
I hope I explained in a simple way. Until the next article!