danlog.txt

## let's add some parameter sweeps, as per reviewer request

## reviewer 2 request that we tinker with the tuning parameters

## the parameters that were not swept in our original efforts are:

## parameters available to explore are:

endophytism
ws
endodisp
decompdisp
leafdisp
leaffall
numdecomp
numendo
endoloss
newwood
woodfreq
width
kappa
sigma
mu
nuke

## nuke is not really a parameter of interest
## neither is width
## the ones that are already explored in the notebooks are:
endophytism
ws
endodisp
decompdisp
endoloss
newwood


## so we need to explore
leafdisp
leaffall
numdecomp
numendo
woodfreq
width
kappa
sigma
mu

## so just work through these, one-by-one
## start by playing with leafdisp

## how do we do a graphical run again?

## our runs usually went under this script, which I think I 
## just kept modifying for each new use I wanted to peek at.

python3 ./forestrun.py

## had to update some syntax, but seems to work okay. So what are our settings 
## for sweeping leafdisp? And where are these kept?

## all kept in FAmodel.py. Defaults are update with notebook.

## if we want to run using these defaults:

python3 runFA.py -sims 1

python3 runFA.py -sims 3 -steps 3

## how can we save and graph the results?

python3
import pickle
import os
import pandas as pd
import numpy as np
import matplotlib.pyplot as plt

runs = pd.read_pickle('run.2020-03-18_15.28.58.317963.p')

runs = pd.read_pickle('run.2020-03-18_15.43.02.290763.p')

## this is a list of dataframes:
runs[0].head(5)

runs[0]

lvls = sorted(list(runs.keys()))

## we made a function to process these lists of dataframes a bit:

import plotparsers as pr

pr.pldata(runs[0],lvl=3)

pr.pldata_single(runs[0])

pr.pldata()

## not working. What's different? Let's look at one of our old results...

aa = pd.read_pickle('sweeps/results/onefungusdisp/onefungusdisp.p')

type(aa[1])

len(aa[1])

## ah. each level of the parameter that is being swept has its own key.
## 

aa[1][0]

## the value that is contained for each key is a list. The length of 
## the list is determined by how many simulations were done, usually
## 100. 

## Each element of that list is a dataframe with a row for each time
## step, usually fifty. 

## so for us, we need to run 100 simulations, 50 time steps each, 
## for each parameter of interest. We can set up one and see how 
## long it takes...seems precarious to run from ssh.

## we can try...

ssh danthomas@10.4.2.6 "ls -la"

python3 runFA.py -sims 3 -steps 3

ssh danthomas@10.4.2.6 "python3 /Users/danthomas/Documents/FA_ABM/runFA.py -sims 1 -steps 1"

## nope. what about a shell script?

############################################################

#!/usr/bin/env sh

cd /Users/danthomas/Documents/FA_ABM
/usr/local/bin/python3 /Users/danthomas/Documents/FA_ABM/runFA.py -sims 1 -steps 1

############################################################

## from home:

ssh danthomas@10.4.2.6 sh "/Users/danthomas/Documents/FA_ABM/testScript.sh"

## works.

## we want to sweep through a set of values....

## to explore leafdisp, what makes sense? I think it makes sense to 
## do as was done with the endophyte loss sweep - give the endophytes
## a slight advantage, and see what it takes to erode or maintain that
## advantage. 

## leaf dispersal is set at a default of 4. We can sweep from 0 to 8, seems
## reasonable. 

## how to code that? Mesa has a builtin batch job script, but I think 
## our custon runFA.py is probably up for the job, too:

## in our other sweeps, we did 100 simulation runs, 50 steps each

## looks like the lab mac has 4 logical cores, so we can safely send 
## it 3 jobs?

## so maybe 3 bash scripts that runs through 3 levels each (0,1,2), 
## (3,4,5) and (6,7,8)?

## so...


#########################################
#!/usr/bin/env bash

cd /Users/danthomas/Documents/FA_ABM

for i in {0..2}
do 
echo $i
/usr/local/bin/python3 /Users/danthomas/Documents/FA_ABM/runFA.py \
    -sims 100 \
    -steps 50 \
    -endodisp 3.0 \
    -leafdisp $i \
    -fileout "leafdisp"$i 
done 

#########################################

## try it:

ssh danthomas@10.4.2.6 "/bin/bash /Users/danthomas/Documents/scriptsABM/leafdisp0-2.sh"

## seems to be working... what happens if we give it another to chew on?

########################################

#!/usr/bin/env bash

cd /Users/danthomas/Documents/FA_ABM

for i in {3..5}
do 
/usr/local/bin/python3 /Users/danthomas/Documents/FA_ABM/runFA.py \
    -sims 100 \
    -steps 50 \
    -endodisp 3.0 \
    -leafdisp $i \
    -fileout "leafdisp"$i 
done 

########################################

ssh danthomas@10.4.2.6 "/bin/bash /Users/danthomas/Documents/scriptsABM/leafdisp3-5.sh"

## seems to work... so setup the last installment, cross fingers, go to bed. 

ssh danthomas@10.4.2.6 "/bin/bash /Users/danthomas/Documents/scriptsABM/leafdisp6-8.sh"

## tomorrow - remember to clean up git, pull/push everything. 
## oh, wait. Didn't work. The loop breaks if I close my shell on this side. 
## fuck. Why is it so hard to do this? 

## I guess I can leave my computer on...

## what is left, then? edit the scripts to exclude what's already done...

## can't stay up so let's just pile it on and hope we don't freeze up the computer:

#ssh danthomas@10.4.2.6 "/bin/bash /Users/danthomas/Documents/scriptsABM/leafdisp0-2.sh"
#ssh danthomas@10.4.2.6 "/bin/bash /Users/danthomas/Documents/scriptsABM/leafdisp3-5.sh"
#ssh danthomas@10.4.2.6 "/bin/bash /Users/danthomas/Documents/scriptsABM/leafdisp6-8.sh"

## that last run failed. redo?

python3 /Users/danthomas/Documents/FA_ABM/runFA.py \
    -sims 100 \
    -steps 50 \
    -endodisp 3.0 \
    -leafdisp 8.0 \
    -fileout "leafdisp8" 

## otherwise, worked well, I think. Can we make a graphic out of one of these, 
## and make sure?

ls /home/daniel/Documents/submissions/FAabm/FA_ABM/sweeps/results/leafdisp
## we have all but one run done... let's 

python3
import pickle
import os
import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
import plotparsers as pr
plt.ion()

os.chdir("/home/daniel/Documents/submissions/FAabm/FA_ABM/sweeps/results/leafdisp")

## how can we combine all of these files into a dictionary with levels as keys?

## for the next round of simulations, let's put an easy way to divide up the 
## strings. For this one, I guess we may have to do this manually. We want 
## a dataframe for each run, a list of runs for each level, and a dictionary 
## of all the lists as values, with their level as the key

levels = []
for i in os.listdir():
    if ".p" in i: 
        print (i)
        i=i.split('2020')[0].split('leafdisp')[1]
        i=int(i)
        print(i)
        levels.append(i)

levels.sort()
## looks like we have doubles of 4, 5, 7

levels = list(set(levels))

## check logs...all of these completed to 100 sims, might as well use the 
## extra data

## combine the doubles if we have them

aa = pd.read_pickle("leafdisp42020-03-18_23.24.26.009182.p")
bb = pd.read_pickle("leafdisp42020-03-18_23.31.24.077112.p")
aa.extend(bb)

## now write it back out?

pickle.dump(aa, open('leafdisp42020-combined.p', 'wb'))

aa = pd.read_pickle('leafdisp42020-combined.p') ## works

## repeat this with 5 and 7:
aa = pd.read_pickle("leafdisp52020-03-19_00.24.35.275498.p")
bb = pd.read_pickle("leafdisp52020-03-19_00.39.06.402233.p")
aa.extend(bb)
pickle.dump(aa, open('leafdisp52020-combined.p', 'wb'))

aa = pd.read_pickle("leafdisp72020-03-18_23.25.23.043032.p")
bb = pd.read_pickle("leafdisp72020-03-19_00.12.20.460393.p")
aa.extend(bb)
pickle.dump(aa, open('leafdisp72020-combined.p', 'wb'))


## so can we fill a dictionary while reading in the files?

os.chdir("/home/daniel/Documents/submissions/FAabm/FA_ABM/sweeps/results/leafdisp")
leafDispDic = {}
levels = []
for i in os.listdir():
    if ".p" in i: 
        filename = i
        i=i.split('2020')[0].split('leafdisp')[1]
        i=int(i)
        print(i)
        print(filename)
        leafDispDic[i] = pd.read_pickle(filename)

## did that work? looks like it:

leafDispDic.keys()

type(leafDispDic[6])

len(leafDispDic[8])


## can we now use our plotter functions?

runs = leafDispDic
lvls = sorted(list(runs.keys())) ## lvls of the run, from large to small

runplotdata = {}
for i in lvls:
    runplotdata[i] = pr.pldata(runs, i) 

## make a plotting function:

def plotComp(ax, runs, level, title=None, titletextsize=20, legendtextsize=20):
    ax.set_ylim(0,50)
    ax.plot(x, runs[level]['decomp_means'],
            '-',
           c='red',
            label=('decomposer'),
                )
    ax.fill_between(x, runs[level]['decomp_updev'],
                    runs[level]['decomp_downdev'],
                    linewidth=0,
                    facecolor='red',
                    alpha = 0.3,
                    )
    ax.plot(x, runs[level]['endo_means'],
            '-',
           c='blue',
            label=('endophyte'),
                )
    ax.fill_between(x, runs[level]['endo_updev'],
                    runs[level]['endo_downdev'],
                    linewidth=0,
                    facecolor='blue',
                    alpha = 0.3,
                    )
    handles, labels = ax.get_legend_handles_labels()
    ax.legend(handles, labels, prop={'size':legendtextsize})
    ax.set_ylabel("Substrates occupied")
    ax.set_xlabel("Timestep")
    ax.set_title('{}={}'.format(title,level), {'fontsize':titletextsize})

fig,axes = plt.subplots(3,3, 
    figsize=(25, 40), 
    sharex=True, 
    sharey=True
    )

for i in runplotdata.keys():
    aa=plotComp(axes.flatten()[i], runplotdata, i, "leafdisp", 15, 10)

plt.tight_layout()

## well, that's cool. The function, not the results.
## results are pretty much expected. 


#######################

## let's set up another sweep

## let's compare what happens with a more 
## seasonal system, where leaf fall
## occurs in a single "autumn" time
## step. 

## what would this look like? 

## 100 sims of the default and ...
## 100 sims of leaffall = 4?

## actually, we need to review the code a little...confirm
## that leaffall works how we think it does...

## yes. if leaffall = 4, then on timesteps that are
## multiples of 4, leaf drop occurs. So leaffall
## is the number of days between leaf fall, and 
## leaffall=4 is equivalent to one season per year of 
## leaf drop.

## but we also want to beef up leaf dispersal a bit, 
## because during leaf fall at this time, many more
## leaves fall, increasing the chance of incoulation
## via leaf, say to d=6

## I think this would look like:

python3 /Users/danthomas/Documents/FA_ABM/runFA.py \
    -sims 100 \
    -steps 50 \
    -leaffall 4 \
    -leafdisp 6 \
    -fileout "leaffall4__"

## but let's also try leaf dispersal at 4 and 5, so a minisweep

#ssh danthomas@10.4.2.6 "/bin/bash /Users/danthomas/Documents/scriptsABM/leaffall4.sh"


############################################

## priority effects

## what sweep is next? 

## what we call priority effects test - does have multiple 
## additional fungi on your side to begin with help you 
## win competition experiments?

## using the default settings otherwise, sweep 
## the number of additional initial fungi. This will
## be two sweeps, one for decomps, one for viaphytes

##  

for i in 1 2 3 4 5 10 15; do
python3 /Users/danthomas/Documents/FA_ABM/runFA.py \
    -sims 100 \
    -steps 50 \
    -numdecomp i \
    -fileout "numdecomp"$i"__"
done

## and the same for viaphytes

ssh danthomas@10.4.2.6 "/bin/bash /Users/danthomas/Documents/scriptsABM/priorityDecomp.sh"

ssh danthomas@10.4.2.6 "/bin/bash /Users/danthomas/Documents/scriptsABM/priorityViaphyte.sh"

## these are going to be in the home of directory on cooley comp
## forgot to change directory

############################################

## woodfreq

## let's sweep through woodfreq,
## because fuck why not

cd /Users/danthomas/Documents/FA_ABM

for i in {1..4}; do
/usr/local/bin/python3 /Users/danthomas/Documents/FA_ABM/runFA.py \
    -sims 100 \
    -steps 50 \
    -woodfreq $i \
    -fileout "woodfreq"$i"__"
done


ssh danthomas@10.4.2.6 "/bin/bash /Users/danthomas/Documents/scriptsABM/woodfreq.sh"

############################################

## tomorrow - incorporate the leafdisp=8 into the 
## leafdisp sweep

## look at leaffall result, if it makes sense, 
## "sweep" two other settings at leaffal=4 - leafdisp={4,5}

## if results look good from all available drafts, 
## set up remaining sweeps?: 

width

## not going to do width, we'll keep the spatial 
## scale the same

kappa
sigma
mu

## not sure how to play with these, these are somewhat 
## like