Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Quick patch to check that Dataset Keys contain non-None Values #1228

Merged
merged 5 commits into from
May 22, 2024
Merged

Quick patch to check that Dataset Keys contain non-None Values #1228

Changes from all commits
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
3fc0d89
quick patch
KuuCi May 22, 2024
dda8e11
also seperately check for local path
KuuCi May 22, 2024
d04036d
typo
KuuCi May 22, 2024
27304c4
typo
KuuCi May 22, 2024
79b0fd8
Merge branch 'main' into dataset-empty-key-patch
dakinggg May 22, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 6 additions & 2 deletions llmfoundry/utils/config_utils.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -589,7 +589,7 @@ def _process_data_source(
('uc_volume', source_dataset_path[len('dbfs:'):], true_split),
)
# Check for HF path
elif 'hf_name' in dataset:
elif 'hf_name' in dataset and dataset['hf_name']:
hf_path = dataset['hf_name']
backend, _, _ = parse_uri(hf_path)
if backend:
Expand All @@ -600,7 +600,7 @@ def _process_data_source(
else:
data_paths.append(('hf', hf_path, true_split))
# Check for remote path
elif 'remote' in dataset:
elif 'remote' in dataset and dataset['remote']:
remote_path = dataset['remote']
backend, _, _ = parse_uri(remote_path)
if backend:
Expand All @@ -610,7 +610,11 @@ def _process_data_source(
) if cfg_split else remote_path
data_paths.append((backend, remote_path, true_split))
else:
# No backend detected so assume local path
data_paths.append(('local', remote_path, true_split))
# Check for local path
elif 'local' in dataset and dataset['local']:
data_paths.append(('local', dataset['local'], true_split))
else:
log.warning('DataSource Not Found.')

Expand Down
Loading