Allow for subdirectory structure within data directory

[PfisterAn]

• recursively read the data directory
  • @TODO/SUGGEST - may be worth to think about using the sub directory information for enriching the index and/or use it for access control (e.g., subdirectory names may be named like the Azure AD Group for access control)

Purpose

• Ability to bring subfolder structure into data directory for better arranging the data, at least on the directory level for now

Does this introduce a breaking change?

[ ] Yes
[ x] No

Pull Request Type

What kind of change does this Pull Request introduce?

[ ] Bugfix
[ x] Feature
[ ] Code style update (formatting, local variables)
[ ] Refactoring (no functional changes, no api changes)
[ ] Documentation content changes
[ ] Other... Please describe:

How to Test

• Run prepdocs on the data directory where content is arranged within subdirectories

git clone [repo-address]
cd [repo-name]
git checkout [branch-name]
npm install

• Test the code

What to Check

Verify that the following are valid

• ...

Other Information

[pamelafox]

Thanks, this seems useful!

[pamelafox]

Please check the CI results and see CONTRIBUTING.md for instructions on running linters locally.

[PfisterAn]

Its in the main body, it was already there before, thats why you dont see it in the Review.

azure-search-openai-demo/scripts/prepdocs.py

Lines 397 to 405 in 36c14b6

	if args.removeall:
	remove_blobs(None)
	remove_from_index(None)
	else:
	if not args.remove:
	create_search_index()
	print("Processing files...")
	read_files(args.files, use_vectors)

[pamelafox]

CI/CD is still failing - https://github.com/Azure-Samples/azure-search-openai-demo/actions/runs/6037247433/job/16381542344?pr=569
Please see CONTRIBUTING for details on running linters

[PfisterAn]

Missed to pass the filename towards the split_text function. split_text was setup to utilise the global variable filename, which doesnt exist anymore after introducing the read_files function.

[pamelafox]

Merged, thanks for the changes!