Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Various fixes to reproducible benchmarks #1800

Merged
merged 120 commits into from
Sep 11, 2023
Merged
Changes from 6 commits
Commits
Show all changes
120 commits
Select commit Hold shift + click to select a range
bd738ec
ANN-benchmarks: switch to use gbench
achirkin Aug 9, 2023
7473c62
Disable NVTX if the nvtx3 headers are missing
achirkin Aug 9, 2023
aa10d7c
Merge branch 'branch-23.10' into enh-google-benchmarks
achirkin Aug 10, 2023
bed126c
Merge branch 'branch-23.10' into enh-google-benchmarks
achirkin Aug 10, 2023
09ea7a7
Merge remote-tracking branch 'upstream/branch-23.10' into python-ann-…
divyegala Aug 11, 2023
2917886
try to run gbench executable
divyegala Aug 12, 2023
49732b1
Allow to compile ANN_BENCH without CUDA
achirkin Aug 17, 2023
76cfb40
Merge remote-tracking branch 'rapidsai/branch-23.10' into enh-google-…
achirkin Aug 17, 2023
9b588af
Fix style
achirkin Aug 17, 2023
6d6c17d
Adapt ANN benchmark python scripts
achirkin Aug 17, 2023
b89b27d
Make the default behavior to produce one executable per benchmark
achirkin Aug 17, 2023
163a40c
Fix style problems / pre-commit
achirkin Aug 17, 2023
0bb51a3
Merge branch 'branch-23.10' into enh-google-benchmarks
achirkin Aug 22, 2023
2b9f649
Merge remote-tracking branch 'rapidsai/branch-23.10' into enh-google-…
achirkin Aug 23, 2023
9728f7e
Merge branch 'branch-23.10' into enh-google-benchmarks
achirkin Aug 24, 2023
7b1bf01
Merge remote-tracking branch 'origin/branch-23.10' into enh-google-be…
cjnolet Aug 24, 2023
1daf2bf
Adding k and batch-size options to run.py
cjnolet Aug 24, 2023
4e0a53e
Merge branch 'branch-23.10' - CONFIGS ONLY - dataset_memtype follows …
achirkin Aug 24, 2023
04893c9
Add dataset_memory_type/query_memory_type as build/search parameters
achirkin Aug 24, 2023
b24fcf7
middle of merge, not building
divyegala Aug 24, 2023
30f7467
Tuning guide
cjnolet Aug 24, 2023
3e35121
Merge remote-tracking branch 'artem/enh-google-benchmarks' into enh-g…
cjnolet Aug 24, 2023
f927f69
compiling, index building successful, search failing
divyegala Aug 24, 2023
404cd10
Merge remote-tracking branch 'corey/enh-google-benchmarks' into pytho…
divyegala Aug 24, 2023
2f19c44
FEA first commit rebasing changes on gbench branch
dantegd Aug 25, 2023
e0586de
FIX fixing straggling changes from rebase
dantegd Aug 25, 2023
0eaa7e0
Fix FAISS using a destroyed stream from previous benchmark case
achirkin Aug 25, 2023
9896963
Merge remote-tracking branch 'artem/enh-google-benchmarks' into enh-g…
cjnolet Aug 25, 2023
4062d6f
Fixing issue in conf file and stubbing out parameter tuning guide
cjnolet Aug 25, 2023
7141c21
Adding CAGRA to tuning guide
cjnolet Aug 25, 2023
7c42a78
Adding ivf-flat description to tuning guide
cjnolet Aug 25, 2023
92a37a8
Updating ivf-flat and ivf-pq
cjnolet Aug 25, 2023
3982840
Adding tuning guide tables for ivf-flat and ivf-pq for faiss and raft
cjnolet Aug 25, 2023
d2bfc11
Reatio is not required
cjnolet Aug 25, 2023
80482fb
Merge remote-tracking branch 'corey/enh-google-benchmarks' into pytho…
divyegala Aug 25, 2023
31594e7
FIX changes that got lost during rebasing
dantegd Aug 25, 2023
82f195e
write build,search results
divyegala Aug 25, 2023
be6eb56
FIX PEP8 fixes
dantegd Aug 25, 2023
0cf1c6f
CLeaning up a couple configs
cjnolet Aug 25, 2023
f5bf15a
FIX typo in cmake conditional
dantegd Aug 25, 2023
617c60f
add tuning guide for cagra, modify build param
divyegala Aug 25, 2023
3948f0c
Merge remote-tracking branch 'corey/enh-google-benchmarks' into pytho…
divyegala Aug 26, 2023
74c9a1b
remove data_export, use gbench csvs to plot
divyegala Aug 26, 2023
902f9f4
fix typo in docs path for results
divyegala Aug 26, 2023
9b82f85
Merge pull request #2 from divyegala/python-ann-bench-use-gbench
cjnolet Aug 26, 2023
1198e1a
for plotting, pick up recall/qps from anywhere in the csv columns
divyegala Aug 26, 2023
24c1619
Merge remote-tracking branch 'divye/python-ann-bench-use-gbench' into…
cjnolet Aug 26, 2023
3f647c3
add output-filepath for plot.py
divyegala Aug 26, 2023
354287d
fix typo in docs
divyegala Aug 26, 2023
e0dfbab
Reverting changes to deep-100M
cjnolet Aug 26, 2023
16e233b
FIX typo in build.sh
dantegd Aug 27, 2023
cac89d0
DBG Make cmake verbose
dantegd Aug 27, 2023
bb3a194
Merge branch 'branch-23.10' into enh-google-benchmarks
achirkin Aug 28, 2023
7d8ee13
FAISS refinement
cjnolet Aug 28, 2023
1720e11
Merge branch 'enh-google-benchmarks' of github.com:cjnolet/raft into …
cjnolet Aug 28, 2023
c0ee323
FIX typo in build.sh
dantegd Aug 28, 2023
aa608d2
DBG single commit of changes
dantegd Aug 28, 2023
697ab89
FIX Add openmp changes from main branch
dantegd Aug 28, 2023
8b0c4c2
FIX recipe env variables
dantegd Aug 28, 2023
49fd31d
adding build time plot
divyegala Aug 28, 2023
be3da1a
merging corey's upstream
divyegala Aug 28, 2023
e92827a
FIX flag in the wrong conditional in build.sh
dantegd Aug 28, 2023
b9e7771
Merge pull request #4 from divyegala/python-ann-bench-use-gbench
cjnolet Aug 28, 2023
8e5ab5d
Merge remote-tracking branch 'artem/enh-google-benchmarks' into enh-g…
cjnolet Aug 29, 2023
f331a94
Merge branch 'enh-google-benchmarks' of github.com:cjnolet/raft into …
cjnolet Aug 29, 2023
b9defb7
FIX remove accidentally deleted file
dantegd Aug 29, 2023
b569861
ENH merge changes from debug PR
dantegd Aug 29, 2023
cdd8d6b
ENH merge changes from enh-google-benchmark branch and create package…
dantegd Aug 29, 2023
b9e9ea6
FIX build.sh flag that was deleted in a bad merge
dantegd Aug 29, 2023
e420593
Merge branch 'branch-23.10' into enh-google-benchmarks
achirkin Aug 29, 2023
913dec2
Move the 'dump_parameters' earlier in the benchmarks to have higher c…
achirkin Aug 29, 2023
8861fc8
Implementing some of the review feedback
cjnolet Aug 29, 2023
2f52b02
Bench ann
cjnolet Aug 29, 2023
c28326c
Merge remote-tracking branch 'artem/enh-google-benchmarks' into enh-g…
cjnolet Aug 29, 2023
521b696
Fixing a couple potential merge artifacts
cjnolet Aug 29, 2023
94296ca
FIX multiple fixes
dantegd Aug 29, 2023
0a35608
FIX multiple fixes
dantegd Aug 30, 2023
d11043c
Merge branch 'enh-google-benchmarks' into dev-enh-google-benchmarks
cjnolet Aug 30, 2023
d6757c1
Merge branch 'branch-23.10' into dev-enh-google-benchmarks
cjnolet Aug 30, 2023
78356aa
Merging python (will need some more fixes later but this will work fo…
cjnolet Aug 30, 2023
9ce6ce0
Merge branch 'branch-23.10' into dev-enh-google-benchmarks
cjnolet Aug 30, 2023
184c46d
FIX merge conflicts from fork and local
dantegd Aug 31, 2023
0dc3ce4
FIX many improvements and small fixes
dantegd Aug 31, 2023
5dd7db2
FIX small fixes from minor errors in prior merges
dantegd Aug 31, 2023
14bcb92
ANN-bench: more flexible cuda_stub.hpp
achirkin Aug 31, 2023
50c9fe2
Make dlopen more flexible looking for the cudart version to link.
achirkin Aug 31, 2023
c6df11a
Fixing style
cjnolet Aug 31, 2023
aaaa182
Merge remote-tracking branch 'artem/enh-ann-bench-flexible-stub' into…
cjnolet Aug 31, 2023
9ffd68e
Fixing omp error
cjnolet Aug 31, 2023
c947004
Merge branch 'branch-23.10' into enh-ann-bench-flexible-stub
achirkin Aug 31, 2023
11f353b
FIxing a couple thing in conf files
cjnolet Aug 31, 2023
858d0a5
Merge branch 'branch-23.10' into dev-enh-google-benchmarks
dantegd Sep 1, 2023
d236090
Adding data_export
cjnolet Sep 1, 2023
c47a1bf
Merge branch 'dev-enh-google-benchmarks' of github.com:dantegd/raft i…
cjnolet Sep 1, 2023
615807a
Merge branch 'branch-23.10' into enh-ann-bench-flexible-stub
achirkin Sep 1, 2023
feef4f3
Merge branch 'branch-23.10' into dev-enh-google-benchmarks
cjnolet Sep 1, 2023
fb2140f
Merge remote-tracking branch 'artem/enh-ann-bench-flexible-stub' into…
cjnolet Sep 1, 2023
998bf48
fix dask pinnings in raft-dask recipe
divyegala Sep 1, 2023
be85537
FIX PR review feedback and readme updates
dantegd Sep 1, 2023
abb4f69
Merge branch 'branch-23.10' into dev-enh-google-benchmarks
dantegd Sep 1, 2023
076d2de
DOC doc updates
dantegd Sep 1, 2023
c6014a9
FIX pep8
dantegd Sep 1, 2023
5a12ce3
FIX docs and plot datasets path
dantegd Sep 1, 2023
d0c150b
Doing some cleanup of ggnn, expanding param tuning docs, fixing hnswl…
cjnolet Sep 1, 2023
03b5e4f
Merge branch 'branch-23.10' into imp-google-benchmarks
cjnolet Sep 1, 2023
fbdc1fa
FIX found typo in cmake
dantegd Sep 1, 2023
954aa87
FIX missing parameter in python
dantegd Sep 1, 2023
15b0dc0
FIX correct conditional
dantegd Sep 1, 2023
d863ce6
FIX for single gpu arch detection in CMake
dantegd Sep 2, 2023
c271a4e
Merge branch 'dev-enh-google-benchmarks' into imp-google-benchmarks
cjnolet Sep 2, 2023
0d60c56
FIX PR review fixes and a {yea}
dantegd Sep 2, 2023
0193607
More fixes
cjnolet Sep 2, 2023
fcc158a
Merge remote-tracking branch 'origin/branch-23.10' into dev-enh-googl…
cjnolet Sep 2, 2023
047e941
Merge branch 'dev-enh-google-benchmarks' into imp-google-benchmarks
cjnolet Sep 2, 2023
b7a6d9a
Merge branch 'branch-23.10' into imp-google-benchmarks
cjnolet Sep 5, 2023
9244674
Merge remote-tracking branch 'origin/branch-23.10' into imp-google-be…
cjnolet Sep 5, 2023
432fa45
Merge branch 'imp-google-benchmarks' of github.com:cjnolet/raft into …
cjnolet Sep 5, 2023
732b923
Update cpp/bench/ann/src/common/util.hpp
cjnolet Sep 7, 2023
ef112d0
Merge branch 'branch-23.10' into imp-google-benchmarks
cjnolet Sep 8, 2023
be6cd5c
Merge branch 'branch-23.10' into imp-google-benchmarks
cjnolet Sep 11, 2023
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
221 changes: 177 additions & 44 deletions bench/ann/plot.py
Original file line number Diff line number Diff line change
@@ -23,9 +23,11 @@

mpl.use("Agg") # noqa
import argparse
from collections import OrderedDict
import itertools
import matplotlib.pyplot as plt
import numpy as np
import pandas as pd
import os


@@ -42,6 +44,16 @@
}
}

def positive_int(input_str: str) -> int:
try:
i = int(input_str)
if i < 1:
raise ValueError
except ValueError:
raise argparse.ArgumentTypeError(f"{input_str} is not a positive integer")

return i


def generate_n_colors(n):
vs = np.linspace(0.3, 0.9, 7)
@@ -76,42 +88,35 @@ def get_left_right(metric):
return "right"


def get_plot_label(xm, ym):
template = "%(xlabel)s-%(ylabel)s tradeoff - %(updown)s and" " to the %(leftright)s is better"
return template % {
"xlabel": xm["description"],
"ylabel": ym["description"],
"updown": get_up_down(ym),
"leftright": get_left_right(xm),
}


def create_pointset(data, xn, yn):
xm, ym = (metrics[xn], metrics[yn])
rev_y = -1 if ym["worst"] < 0 else 1
rev_x = -1 if xm["worst"] < 0 else 1
data.sort(key=lambda t: (rev_y * t[-1], rev_x * t[-2]))

axs, ays, als = [], [], []
axs, ays, als, aidxs = [], [], [], []
# Generate Pareto frontier
xs, ys, ls = [], [], []
xs, ys, ls, idxs = [], [], [], []
last_x = xm["worst"]
comparator = (lambda xv, lx: xv > lx) if last_x < 0 else (lambda xv, lx: xv < lx)
for algo_name, xv, yv in data:
for algo_name, index_name, xv, yv in data:
if not xv or not yv:
continue
axs.append(xv)
ays.append(yv)
als.append(algo_name)
aidxs.append(algo_name)
if comparator(xv, last_x):
last_x = xv
xs.append(xv)
ys.append(yv)
ls.append(algo_name)
return xs, ys, ls, axs, ays, als
idxs.append(index_name)
return xs, ys, ls, idxs, axs, ays, als, aidxs


def create_plot(all_data, raw, x_scale, y_scale, fn_out, linestyles):
def create_plot_search(all_data, raw, x_scale, y_scale, fn_out, linestyles,
dataset, k, batch_size):
xn = "k-nn"
yn = "qps"
xm, ym = (metrics[xn], metrics[yn])
@@ -122,13 +127,13 @@ def create_plot(all_data, raw, x_scale, y_scale, fn_out, linestyles):

# Sorting by mean y-value helps aligning plots with labels
def mean_y(algo):
xs, ys, ls, axs, ays, als = create_pointset(all_data[algo], xn, yn)
xs, ys, ls, idxs, axs, ays, als, aidxs = create_pointset(all_data[algo], xn, yn)
return -np.log(np.array(ys)).mean()

# Find range for logit x-scale
min_x, max_x = 1, 0
for algo in sorted(all_data.keys(), key=mean_y):
xs, ys, ls, axs, ays, als = create_pointset(all_data[algo], xn, yn)
xs, ys, ls, idxs, axs, ays, als, aidxs = create_pointset(all_data[algo], xn, yn)
min_x = min([min_x] + [x for x in xs if x > 0])
max_x = max([max_x] + [x for x in xs if x < 1])
color, faded, linestyle, marker = linestyles[algo]
@@ -169,7 +174,7 @@ def inv_fun(x):
else:
ax.set_xscale(x_scale)
ax.set_yscale(y_scale)
ax.set_title(get_plot_label(xm, ym))
ax.set_title(f"{dataset} k={k} batch_size={batch_size}")
plt.gca().get_position()
# plt.gca().set_position([box.x0, box.y0, box.width * 0.8, box.height])
ax.legend(handles, labels, loc="center left", bbox_to_anchor=(1, 0.5), prop={"size": 9})
@@ -188,39 +193,125 @@ def inv_fun(x):
# Workaround for bug https://github.com/matplotlib/matplotlib/issues/6789
ax.spines["bottom"]._adjust_location()

print(f"writing search output to {fn_out}")
plt.savefig(fn_out, bbox_inches="tight")
plt.close()


def load_all_results(dataset_path):
def create_plot_build(build_results, search_results, linestyles, fn_out,
dataset, k, batch_size):
xn = "k-nn"
yn = "qps"

recall_85 = [-1] * len(linestyles)
qps_85 = [-1] * len(linestyles)
bt_85 = [0] * len(linestyles)
i_85 = [-1] * len(linestyles)
recall_90 = [-1] * len(linestyles)
qps_90 = [-1] * len(linestyles)
bt_90 = [0] * len(linestyles)
i_90 = [-1] * len(linestyles)
recall_95 = [-1] * len(linestyles)
qps_95 = [-1] * len(linestyles)
bt_95 = [0] * len(linestyles)
i_95 = [-1] * len(linestyles)
data = OrderedDict()
colors = OrderedDict()

# Sorting by mean y-value helps aligning plots with labels
def mean_y(algo):
xs, ys, ls, idxs, axs, ays, als, aidxs = create_pointset(search_results[algo], xn, yn)
return -np.log(np.array(ys)).mean()

for pos, algo in enumerate(sorted(search_results.keys(), key=mean_y)):
xs, ys, ls, idxs, axs, ays, als, aidxs = create_pointset(search_results[algo], xn, yn)
# x is recall, y is qps, ls is algo_name, idxs is index_name
for i in range(len(xs)):
if xs[i] >= 0.85 and xs[i] < 0.9 and ys[i] > qps_85[pos]:
qps_85[pos] = ys[i]
bt_85[pos] = build_results[(ls[i], idxs[i])][0][2]
i_85[pos] = idxs[i]
elif xs[i] >= 0.9 and xs[i] < 0.95 and ys[i] > qps_90[pos]:
qps_90[pos] = ys[i]
bt_90[pos] = build_results[(ls[i], idxs[i])][0][2]
i_90[pos] = idxs[i]
elif xs[i] >= 0.95 and ys[i] > qps_95[pos]:
qps_95[pos] = ys[i]
bt_95[pos] = build_results[(ls[i], idxs[i])][0][2]
i_95[pos] = idxs[i]
data[algo] = [bt_85[pos], bt_90[pos], bt_95[pos]]
colors[algo] = linestyles[algo][0]

index = ['@85% Recall', '@90% Recall', '@95% Recall']

df = pd.DataFrame(data, index=index)
plt.figure(figsize=(12, 9))
ax = df.plot.bar(rot=0, color=colors)
fig = ax.get_figure()
print(f"writing search output to {fn_out}")
plt.title("Build Time for Highest QPS")
plt.suptitle(f"{dataset} k={k} batch_size={batch_size}")
plt.ylabel("Build Time (s)")
fig.savefig(fn_out)


def load_lines(results_path, result_files, method, index_key):
results = dict()
results_path = os.path.join(dataset_path, "result", "search")
for result_filepath in os.listdir(results_path):
with open(os.path.join(results_path, result_filepath), 'r') as f:

linebreaker = "name,iterations"

for result_filename in result_files:
with open(os.path.join(results_path, result_filename), 'r') as f:
lines = f.readlines()
lines = lines[:-1] if lines[-1] == "\n" else lines
idx = 0
for pos, line in enumerate(lines):
if "QPS" in line:
if linebreaker in line:
idx = pos
break

keys = lines[idx].split(',')
recall_idx = -1
qps_idx = -1
for pos, key in enumerate(keys):
if "Recall" in key:
recall_idx = pos
if "QPS" in key:
qps_idx = pos
if method == "build":
if "hnswlib" in result_filename:
key_idx = [2]
else:
key_idx = [10]
elif method == "search":
if "hnswlib" in result_filename:
key_idx = [10, 6]
else:
key_idx = [12, 10]

for line in lines[idx+1:]:
split_lines = line.split(',')

algo_name = split_lines[0].split('.')[0].strip("\"")
if algo_name not in results:
results[algo_name] = []
results[algo_name].append([algo_name, float(split_lines[recall_idx]),
float(split_lines[qps_idx])])
index_name = split_lines[0].split('/')[0].strip("\"")

if index_key == "algo":
dict_key = algo_name
elif index_key == "index":
dict_key = (algo_name, index_name)
if dict_key not in results:
results[dict_key] = []
to_add = [algo_name, index_name]
for key_i in key_idx:
to_add.append(float(split_lines[key_i]))
results[dict_key].append(to_add)

return results


def load_all_results(dataset_path, algorithms, k, batch_size, method, index_key):
results_path = os.path.join(dataset_path, "result", method)
result_files = os.listdir(results_path)
result_files = [result_filename for result_filename in result_files \
if f"{k}-{batch_size}" in result_filename]
if len(algorithms) > 0:
result_files = [result_filename for result_filename in result_files if \
result_filename.split('-')[0] in algorithms]

results = load_lines(results_path, result_files, method, index_key)

return results


@@ -232,8 +323,27 @@ def main():
parser.add_argument("--dataset-path", help="path to dataset folder",
default=os.path.join(os.getenv("RAFT_HOME"),
"bench", "ann", "data"))
parser.add_argument("--output-filename",
default="plot.png")
parser.add_argument("--output-filepath",
help="directory for PNG to be saved",
default=os.getcwd())
parser.add_argument("--algorithms",
help="plot only comma separated list of named \
algorithms",
default=None)
parser.add_argument(
"-k", "--count", default=10, type=positive_int, help="the number of nearest neighbors to search for"
)
parser.add_argument(
"-bs", "--batch-size", default=10000, type=positive_int, help="number of query vectors to use in each query trial"
)
parser.add_argument(
"--build",
action="store_true"
)
parser.add_argument(
"--search",
action="store_true"
)
parser.add_argument(
"--x-scale",
help="Scale to use when drawing the X-axis. \
@@ -249,15 +359,38 @@ def main():
parser.add_argument(
"--raw", help="Show raw results (not just Pareto frontier) in faded colours", action="store_true"
)
args = parser.parse_args()

output_filepath = os.path.join(args.dataset_path, args.dataset, args.output_filename)
print(f"writing output to {output_filepath}")

results = load_all_results(os.path.join(args.dataset_path, args.dataset))
linestyles = create_linestyles(sorted(results.keys()))
args = parser.parse_args()

create_plot(results, args.raw, args.x_scale, args.y_scale, output_filepath, linestyles)
if args.algorithms:
algorithms = args.algorithms.split(',')
else:
algorithms = []
k = args.count
batch_size = args.batch_size
if not args.build and not args.search:
build = True
search = True
else:
build = args.build
search = args.search

search_output_filepath = os.path.join(args.output_filepath, f"search-{args.dataset}-{k}-{batch_size}.png")
build_output_filepath = os.path.join(args.output_filepath, f"build-{args.dataset}-{k}-{batch_size}.png")

search_results = load_all_results(
os.path.join(args.dataset_path, args.dataset),
algorithms, k, batch_size, "search", "algo")
linestyles = create_linestyles(sorted(search_results.keys()))
if search:
create_plot_search(search_results, args.raw, args.x_scale, args.y_scale,
search_output_filepath, linestyles, args.dataset, k, batch_size)
if build:
build_results = load_all_results(
os.path.join(args.dataset_path, args.dataset),
algorithms, k, batch_size, "build", "index")
create_plot_build(build_results, search_results, linestyles, build_output_filepath,
args.dataset, k, batch_size)


if __name__ == "__main__":
12 changes: 6 additions & 6 deletions bench/ann/run.py
Original file line number Diff line number Diff line change
@@ -35,15 +35,15 @@ def validate_algorithm(algos_conf, algo):
return algo in algos_conf_keys and not algos_conf[algo]["disabled"]


def find_executable(algos_conf, algo):
def find_executable(algos_conf, algo, k, batch_size):
executable = algos_conf[algo]["executable"]
conda_path = os.path.join(os.getenv("CONDA_PREFIX"), "bin", "ann",
executable)
build_path = os.path.join(os.getenv("RAFT_HOME"), "cpp", "build", executable)
if os.path.exists(conda_path):
return (executable, conda_path, algo)
return (executable, conda_path, f"{algo}-{k}-{batch_size}")
elif os.path.exists(build_path):
return (executable, build_path, algo)
return (executable, build_path, f"{algo}-{k}-{batch_size}")
else:
raise FileNotFoundError(executable)

@@ -198,7 +198,7 @@ def main():
curr_algo = index["algo"]
if index["name"] in indices and \
validate_algorithm(algos_conf, curr_algo):
executable_path = find_executable(algos_conf, curr_algo)
executable_path = find_executable(algos_conf, curr_algo, k, batch_size)
if executable_path not in executables_to_run:
executables_to_run[executable_path] = {"index": []}
executables_to_run[executable_path]["index"].append(index)
@@ -212,7 +212,7 @@ def main():
curr_algo = index["algo"]
if curr_algo in algorithms and \
validate_algorithm(algos_conf, curr_algo):
executable_path = find_executable(algos_conf, curr_algo)
executable_path = find_executable(algos_conf, curr_algo, k, batch_size)
if executable_path not in executables_to_run:
executables_to_run[executable_path] = {"index": []}
executables_to_run[executable_path]["index"].append(index)
@@ -222,7 +222,7 @@ def main():
for index in conf_file["index"]:
curr_algo = index["algo"]
if validate_algorithm(algos_conf, curr_algo):
executable_path = find_executable(algos_conf, curr_algo)
executable_path = find_executable(algos_conf, curr_algo, k, batch_size)
if executable_path not in executables_to_run:
executables_to_run[executable_path] = {"index": []}
executables_to_run[executable_path]["index"].append(index)
5 changes: 3 additions & 2 deletions docs/source/raft_ann_benchmarks.md
Original file line number Diff line number Diff line change
@@ -180,14 +180,15 @@ CSV file in `<dataset-path/<dataset>/search/result/<algo.csv>.

The usage of this script is:
```bash
usage: plot.py [-h] [--dataset DATASET] [--dataset-path DATASET_PATH] [--output-filename OUTPUT_FILENAME] [--x-scale X_SCALE] [--y-scale {linear,log,symlog,logit}] [--raw]
usage: plot.py [-h] [--dataset DATASET] [--dataset-path DATASET_PATH] [--output-filepath OUTPUT_FILEPATH] [--x-scale X_SCALE] [--y-scale {linear,log,symlog,logit}] [--raw]

options:
-h, --help show this help message and exit
--dataset DATASET dataset to download (default: glove-100-inner)
--dataset-path DATASET_PATH
path to dataset folder (default: ${RAFT_HOME}/bench/ann/data)
--output-filename OUTPUT_FILENAME
--output-filepath OUTPUT_FILEPATH
directory for PNG to be saved (default: os.getcwd())
--x-scale X_SCALE Scale to use when drawing the X-axis. Typically linear, logit or a2 (default: linear)
--y-scale {linear,log,symlog,logit}
Scale to use when drawing the Y-axis (default: linear)