Sweep: change the /convert endpoint so if there is a user which is not yet in the database then create new folders for that user

[IvoLeist]

Details

The new folders to be created should be inside the folder output and uploads.
The folder name should be the keycloak userid hash

Checklist

• Modify convertPheno_server/server/apis/submission.py ✓ e913362 Edit
• Running GitHub Actions for convertPheno_server/server/apis/submission.py ✓ Edit

[sweep-ai]

Here's the PR! #10. See Sweep's process at dashboard.

⚡ Sweep Basic Tier: I'm using GPT-4. You have 4 GPT-4 tickets left for the month and 3 for the day. (tracking ID: cc35936943)

For more GPT-4 tickets, visit our payment portal. For a one week free trial, try Sweep Pro (unlimited GPT-4 tickets).

---

Actions (click)

• ↻ Restart Sweep

Sandbox Execution ✓

Here are the sandbox execution logs prior to making any changes:

Sandbox logs for c888244

Checking convertPheno_server/server/apis/submission.py for syntax errors... ✅ convertPheno_server/server/apis/submission.py has no syntax errors! 1/1 ✓

Checking convertPheno_server/server/apis/submission.py for syntax errors...
✅ convertPheno_server/server/apis/submission.py has no syntax errors!

Sandbox passed on the latest main, so sandbox checks will be enabled for this issue.

---

Step 1: 🔎 Searching

I found the following snippets in your repository. I will now analyze these snippets and come up with a plan.

Some code snippets I think are relevant in decreasing order of relevance (click to expand). If some file is missing from here, you can mention the path in the ticket description.

convert-pheno-ui/convertPheno_server/server/config/general_config.py

Lines 15 to 49 in c888244

	class GeneralConfig:
	"""
	config for convertPheno_server
	"""
	# postgres
	SQLALCHEMY_TRACK_MODIFICATIONS = False
	# TODO
	# there should be a way to check if the expected folders exist
	# and if not throw an error
	# flask
	FLASK_UPLOAD_DIR = Path("data/uploads/")
	FLASK_EXAMPLE_DIR = Path("data/example_in/").resolve()
	FLASK_OUT_DIR = Path("data/output/").resolve()
	ALLOWED_EXTENSIONS = ["tsv", "csv", "txt", "json"]
	# convert-pheno
	CONVERT_PHENO_PATH = "/usr/share/convert-pheno"
	DATA_DICTIONARY = "redcap_dictionary.csv"
	REDCAP_IN_FOLDER = f"{CONVERT_PHENO_PATH}/t/redcap2bff/in"
	RCD = f"{REDCAP_IN_FOLDER}/{DATA_DICTIONARY}"
	MAPPING_FILE = f"{REDCAP_IN_FOLDER}/redcap_mapping.yaml"
	CP_EXECUTABLE_PATH = f"{CONVERT_PHENO_PATH}/bin/convert-pheno"
	IN_OUT_DIR = "/home"
	UPLOAD_DIR = f"{IN_OUT_DIR}/uploads/"
	EXAMPLE_DIR = f"{IN_OUT_DIR}/example_in/"
	OUT_DIR = f"{IN_OUT_DIR}/output/"
	LOG_FILE_NAME = "convert-pheno-log.json"

convert-pheno-ui/convertPheno_server/server/apis/submission.py

Lines 437 to 506 in c888244

	@ns.route("/download", methods=("POST",))
	class DownloadFile(Resource):
	"""
	API to download the converted file
	"""
	@login(login_required)
	@api.expect(parser)
	def post(self, userid):
	"""
	Flask send_file
	"""
	user = db.session.query(User).filter_by(name=userid).one_or_none()
	if user is None:
	return {"message": "User does not exist"}, 404
	data = request.get_json()
	job_id = data["jobId"]
	job = (
	db.session.query(Job).filter_by(job_id=job_id, owner=user.id).one_or_none()
	)
	if job is None:
	return {"message": "Job does not exist"}, 404
	if data.get("downloadAllFiles"):
	mem_zip = downloadAllFiles(data, job.id)
	return send_file(mem_zip, mimetype="application/zip")
	temp_filename = data["tempFilename"]
	clinical_format = temp_filename.split(".")[1]
	output = (
	db.session.query(Output)
	.filter_by(target_format=clinical_format, job_id=job.id)
	.with_entities(Output.data)
	.one_or_none()
	)
	if output is None:
	return {"message": "clinical data not found"}, 404
	return send_file(
	BytesIO(json.dumps(output[0]).encode()),
	mimetype="application/json",
	)
	@ns.route("/download/example", methods=("POST",))
	class DownloadExampleFile(Resource):
	"""
	API to download example input
	"""
	@login(login_required)
	@api.expect(parser)
	def post(self, userid):
	"""
	Flask send_file
	"""
	data = request.get_json()
	input_format = data["inputFormat"]
	example_file = cfg["EXAMPLE_FILES"][input_format]
	file_path = cfg["FLASK_EXAMPLE_DIR"] / example_file
	return send_file(file_path, as_attachment=True)
	ns.add_resource(
	DownloadExampleFile, "/download/example", endpoint="download_example_file"
	)
	ns.add_resource(UploadFile, "/upload", endpoint="upload_file")
	ns.add_resource(ConvertFile, "/convert", endpoint="convert_file")

---

Step 2: ⌨️ Coding

• Modify convertPheno_server/server/apis/submission.py ✓ e913362 Edit

Modify convertPheno_server/server/apis/submission.py with contents:
• Import the os and Path modules at the top of the file to handle directory creation and path manipulation.
• In the post method of the ConvertFile class, before the user is queried from the database, retrieve the Keycloak userid hash from the request data.
• Still in the post method, after the user is queried from the database, add a condition to check if the user is None.
• If the user is None, use the os module to create new directories in the 'output' and 'uploads' directories with the name of the Keycloak userid hash. Use the Path module to join the base directory paths from the GeneralConfig class with the Keycloak userid hash to form the new directory paths.
• After the new directories are created, proceed with the rest of the method as before.

---

+++
@@ -13,6 +13,9 @@
"""
For submitting a to be converted file
"""
+import os
+from pathlib import Path
+
from copy import deepcopy
from io import BytesIO
import gzip
@@ -254,6 +257,20 @@
if runExample:
ns.logger.info("run /w example data")
user_id = get_or_create_user(userid).id
+

•    # Retrieve the Keycloak userid hash from the request data
  

•    user_keycloak_hash = data.get("KeycloakUserIdHash")
  

•    # Check if the user exists in the database based on the Keycloak hash
  

•    user = db.session.query(User).filter_by(keycloak_hash=user_keycloak_hash).one_or_none()
  

•    if user is None:
  

•        # Create new directories in the "output" and "uploads" directories with the Keycloak userid hash
  

•        upload_dir = GeneralConfig.FLASK_UPLOAD_DIR / user_keycloak_hash
  

•        output_dir = GeneralConfig.FLASK_OUT_DIR / user_keycloak_hash
  

•        os.makedirs(upload_dir, exist_ok=True)
  

•        os.makedirs(output_dir, exist_ok=True)
  

•    # Proceed with getting or creating the user as before
  

•    else:
         ns.logger.info("run /w uploaded data")
         user = (
  

• Running GitHub Actions for convertPheno_server/server/apis/submission.py ✓ Edit

Check convertPheno_server/server/apis/submission.py with contents:

Ran GitHub Actions for e9133625019b0d7938efef29faed434bbd95e107:

---

Step 3: 🔁 Code Review

I have finished reviewing the code for completeness. I did not find errors for sweep/issue-convert-endpoint-user-folders.

---

🎉 Latest improvements to Sweep:

• We just released a dashboard to track Sweep's progress on your issue in real-time, showing every stage of the process – from search to planning and coding.
• Sweep uses OpenAI's latest Assistant API to plan code changes and modify code! This is 3x faster and significantly more reliable as it allows Sweep to edit code and validate the changes in tight iterations, the same way as a human would.

---

💡 To recreate the pull request edit the issue title or description. To tweak the pull request, leave a comment on the pull request.
^{Join Our Discord}

[IvoLeist]

Sweep: There is no need for adding an extra endpoint rather modify this one:
class ConvertFile(Resource):

Further, there is no need to modify any files in convertPheno_client.

[IvoLeist]

Took inspiration by sweep's suggestion and the implemented it on my own 🚀