diff --git a/docs/ert/reference/configuration/keywords.rst b/docs/ert/reference/configuration/keywords.rst index e07a436c0ae..5ff65068b85 100644 --- a/docs/ert/reference/configuration/keywords.rst +++ b/docs/ert/reference/configuration/keywords.rst @@ -1825,9 +1825,8 @@ in :ref:`queue-system-chapter`. In brief, the queue systems have the following o ``BHIST_CMD``, ``SUBMIT_SLEEP``, ``PROJECT_CODE``, ``EXCLUDE_HOST``, ``MAX_RUNNING`` * :ref:`TORQUE ` — ``QSUB_CMD``, ``QSTAT_CMD``, ``QDEL_CMD``, - ``QSTAT_OPTIONS``, ``QUEUE``, ``CLUSTER_LABEL``, ``MAX_RUNNING``, ``NUM_NODES``, - ``NUM_CPUS_PER_NODE``, ``MEMORY_PER_JOB``, ``KEEP_QSUB_OUTPUT``, ``SUBMIT_SLEEP``, - ``QUEUE_QUERY_TIMEOUT`` + ``QUEUE``, ``CLUSTER_LABEL``, ``MAX_RUNNING``, ``KEEP_QSUB_OUTPUT``, + ``SUBMIT_SLEEP`` * :ref:`SLURM ` — ``SBATCH``, ``SCANCEL``, ``SCONTROL``, ``SACCT``, ``SQUEUE``, ``PARTITION``, ``SQUEUE_TIMEOUT``, ``MAX_RUNTIME``, ``INCLUDE_HOST``, ``EXCLUDE_HOST``, ``MAX_RUNNING`` diff --git a/docs/ert/reference/configuration/queue.rst b/docs/ert/reference/configuration/queue.rst index 94ea02ac7ab..f584982c5a2 100644 --- a/docs/ert/reference/configuration/queue.rst +++ b/docs/ert/reference/configuration/queue.rst @@ -251,12 +251,6 @@ The following is a list of all queue-specific configuration options: QUEUE_OPTION TORQUE QSTAT_CMD /path/to/my/qstat QUEUE_OPTION TORQUE QDEL_CMD /path/to/my/qdel -.. _torque_qstat_options: -.. topic:: QSTAT_OPTIONS - - Options to be supplied to the ``qstat`` command. This defaults to :code:`-x`, - which tells the ``qstat`` command to include exited processes. - .. _torque_queue: .. topic:: QUEUE @@ -283,37 +277,6 @@ The following is a list of all queue-specific configuration options: If ``n`` is zero (the default), then it is set to the number of realizations. -.. _torque_nodes_cpus: -.. topic:: NUM_NODES, NUM_CPUS_PER_NODE - - The support for running a job over multiple nodes is deprecated in Ert, - but was previously accomplished by setting NUM_NODES to a number larger - than 1. - - NUM_CPUS_PER_NODE is deprecated, instead please use NUM_CPU to specify the - number of CPU cores to reserve on a single compute node. - -.. _torque_memory_per_job: -.. topic:: MEMORY_PER_JOB - - You can specify the amount of memory you will need for running your - job. This will ensure that not too many jobs will run on a single - shared memory node at once, possibly crashing the compute node if it - runs out of memory. - - You can get an indication of the memory requirement by watching the - course of a local run using the ``htop`` utility. Whether you should set - the peak memory usage as your requirement or a lower figure depends on - how simultaneously each job will run. - - The option to be supplied will be used as a string in the ``qsub`` - argument. You must specify the unit, either ``gb`` or ``mb`` as in - the example:: - - QUEUE_OPTION TORQUE MEMORY_PER_JOB 16gb - - By default, this value is not set. - .. _torque_keep_qsub_output: .. topic:: KEEP_QSUB_OUTPUT @@ -332,23 +295,6 @@ The following is a list of all queue-specific configuration options: QUEUE_OPTION TORQUE SUBMIT_SLEEP 0.5 -.. _torque_queue_query_timeout: -.. topic:: QUEUE_QUERY_TIMEOUT - - The driver allows the backend TORQUE/PBS system to be flaky, i.e. it may - intermittently not respond and give error messages when submitting jobs - or asking for job statuses. The timeout (in seconds) determines how long - ERT will wait before it will give up. Applies to job submission (``qsub``) - and job status queries (``qstat``). Default is 126 seconds. - - ERT will do exponential sleeps, starting at 2 seconds, and the provided - timeout is a maximum. Let the timeout be sums of series like 2+4+8+16+32+64 - in order to be explicit about the number of retries. Set to zero to disallow - flakyness, setting it to 2 will allow for one re-attempt, and 6 will give two - re-attempts. Example allowing six retries:: - - QUEUE_OPTION TORQUE QUEUE_QUERY_TIMEOUT 254 - .. _torque_project_code: .. topic:: PROJECT_CODE diff --git a/docs/everest/config_generated.rst b/docs/everest/config_generated.rst index 190cbf8361a..93d6ff6db4c 100644 --- a/docs/everest/config_generated.rst +++ b/docs/everest/config_generated.rst @@ -1037,27 +1037,12 @@ Simulation settings The kill command -**qstat_options (optional)** - Type: *Optional[str]* - - Options to be supplied to the qstat command. This defaults to -x, which tells the qstat command to include exited processes. - - **cluster_label (optional)** Type: *Optional[str]* The name of the cluster you are running simulations in. -**memory_per_job (optional)** - Type: *Optional[str]* - - You can specify the amount of memory you will need for running your job. This will ensure that not too many jobs will run on a single shared memory node at once, possibly crashing the compute node if it runs out of memory. - You can get an indication of the memory requirement by watching the course of a local run using the htop utility. Whether you should set the peak memory usage as your requirement or a lower figure depends on how simultaneously each job will run. - The option to be supplied will be used as a string in the qsub argument. You must specify the unit, either gb or mb. - - - **keep_qsub_output (optional)** Type: *Optional[int]* @@ -1070,15 +1055,6 @@ Simulation settings To avoid stressing the TORQUE/PBS system you can instruct the driver to sleep for every submit request. The argument to the SUBMIT_SLEEP is the number of seconds to sleep for every submit, which can be a fraction like 0.5 -**queue_query_timeout (optional)** - Type: *Optional[int]* - - - The driver allows the backend TORQUE/PBS system to be flaky, i.e. it may intermittently not respond and give error messages when submitting jobs or asking for job statuses. The timeout (in seconds) determines how long ERT will wait before it will give up. Applies to job submission (qsub) and job status queries (qstat). Default is 126 seconds. - ERT will do exponential sleeps, starting at 2 seconds, and the provided timeout is a maximum. Let the timeout be sums of series like 2+4+8+16+32+64 in order to be explicit about the number of retries. Set to zero to disallow flakyness, setting it to 2 will allow for one re-attempt, and 6 will give two re-attempts. Example allowing six retries: - - - **project_code (optional)** Type: *Optional[str]* diff --git a/src/ert/config/parsing/config_schema_deprecations.py b/src/ert/config/parsing/config_schema_deprecations.py index a97cbe89075..5198f5da06f 100644 --- a/src/ert/config/parsing/config_schema_deprecations.py +++ b/src/ert/config/parsing/config_schema_deprecations.py @@ -181,36 +181,6 @@ "for the Ensemble Smoother update algorithm. " "Please use ENKF_ALPHA and STD_CUTOFF keywords instead.", ), - DeprecationInfo( - keyword="QUEUE_OPTION", - message="QUEUE_QUERY_TIMEOUT as QUEUE_OPTION is ignored. " - "Please remove the line.", - check=lambda line: "QUEUE_QUERY_TIMEOUT" in line, - ), - DeprecationInfo( - keyword="QUEUE_OPTION", - message="QSTAT_OPTIONS as QUEUE_OPTION to the TORQUE is ignored. " - "Please remove the line.", - check=lambda line: "QSTAT_OPTIONS" in line, - ), - DeprecationInfo( - keyword="QUEUE_OPTION", - message="NUM_CPUS_PER_NODE as QUEUE_OPTION to Torque is deprecated and will removed in " - "the future. Replace by NUM_CPU.", - check=lambda line: "NUM_CPUS_PER_NODE" in line, - ), - DeprecationInfo( - keyword="QUEUE_OPTION", - message="NUM_NODES as QUEUE_OPTION to Torque is deprecated and will removed in " - "the future. Replace by NUM_CPU on a single compute node.", - check=lambda line: "NUM_NODES" in line, - ), - DeprecationInfo( - keyword="QUEUE_OPTION", - message="MEMORY_PER_JOB as QUEUE_OPTION to TORQUE is deprecated and will be removed in " - "the future. Replace by REALIZATION_MEMORY.", - check=lambda line: "MEMORY_PER_JOB" in line, - ), DeprecationInfo( keyword="QUEUE_OPTION", message="Memory requirements in LSF should now be set using REALIZATION_MEMORY and not" diff --git a/src/ert/config/queue_config.py b/src/ert/config/queue_config.py index ae0855226d7..a6795e3daff 100644 --- a/src/ert/config/queue_config.py +++ b/src/ert/config/queue_config.py @@ -126,16 +126,10 @@ class TorqueQueueOptions(QueueOptions): qstat_cmd: NonEmptyString | None = None qdel_cmd: NonEmptyString | None = None queue: NonEmptyString | None = None - memory_per_job: NonEmptyString | None = None - num_cpus_per_node: pydantic.PositiveInt = 1 - num_nodes: pydantic.PositiveInt = 1 cluster_label: NonEmptyString | None = None job_prefix: NonEmptyString | None = None keep_qsub_output: bool = False - qstat_options: str | None = pydantic.Field(default=None, deprecated=True) - queue_query_timeout: str | None = pydantic.Field(default=None, deprecated=True) - @property def driver_options(self) -> dict[str, Any]: driver_dict = asdict(self) @@ -143,17 +137,8 @@ def driver_options(self) -> dict[str, Any]: driver_dict["queue_name"] = driver_dict.pop("queue") driver_dict.pop("max_running") driver_dict.pop("submit_sleep") - driver_dict.pop("qstat_options") - driver_dict.pop("queue_query_timeout") return driver_dict - @pydantic.field_validator("memory_per_job") - @classmethod - def check_memory_per_job(cls, value: str | None) -> str | None: - if not torque_memory_usage_format.validate(value): - raise ValueError("wrong memory format") - return value - @pydantic.dataclasses.dataclass class SlurmQueueOptions(QueueOptions): @@ -323,23 +308,6 @@ def from_dict(cls, config_dict: ConfigDict) -> QueueConfig: if tags: queue_options.project_code = "+".join(tags) - if selected_queue_system == QueueSystem.TORQUE: - _check_num_cpu_requirement( - config_dict.get("NUM_CPU", 1), queue_options, raw_queue_options - ) - - for _queue_vals in all_validated_queue_options.values(): - if ( - isinstance(_queue_vals, TorqueQueueOptions) - and _queue_vals.memory_per_job - and realization_memory - ): - _throw_error_or_warning( - "Do not specify both REALIZATION_MEMORY and TORQUE option MEMORY_PER_JOB", - "MEMORY_PER_JOB", - selected_queue_system == QueueSystem.TORQUE, - ) - return QueueConfig( job_script, realization_memory, @@ -372,22 +340,6 @@ def submit_sleep(self) -> float: return self.queue_options.submit_sleep -def _check_num_cpu_requirement( - num_cpu: int, torque_options: TorqueQueueOptions, raw_queue_options: list[list[str]] -) -> None: - flattened_raw_options = [item for line in raw_queue_options for item in line] - if ( - "NUM_NODES" not in flattened_raw_options - and "NUM_CPUS_PER_NODE" not in flattened_raw_options - ): - return - if num_cpu != torque_options.num_nodes * torque_options.num_cpus_per_node: - raise ConfigValidationError( - f"When NUM_CPU is {num_cpu}, then the product of NUM_NODES ({torque_options.num_nodes}) " - f"and NUM_CPUS_PER_NODE ({torque_options.num_cpus_per_node}) must be equal." - ) - - def _parse_realization_memory_str(realization_memory_str: str) -> int: if "-" in realization_memory_str: raise ConfigValidationError( diff --git a/src/ert/scheduler/openpbs_driver.py b/src/ert/scheduler/openpbs_driver.py index 9cf0f97703b..0c4c0352017 100644 --- a/src/ert/scheduler/openpbs_driver.py +++ b/src/ert/scheduler/openpbs_driver.py @@ -124,9 +124,6 @@ def __init__( queue_name: str | None = None, project_code: str | None = None, keep_qsub_output: bool | None = None, - memory_per_job: str | None = None, - num_nodes: int | None = None, - num_cpus_per_node: int | None = None, cluster_label: str | None = None, job_prefix: str | None = None, qsub_cmd: str | None = None, @@ -139,9 +136,6 @@ def __init__( self._queue_name = queue_name self._project_code = project_code self._keep_qsub_output = keep_qsub_output - self._memory_per_job = memory_per_job - self._num_nodes: int | None = num_nodes - self._num_cpus_per_node: int | None = num_cpus_per_node self._cluster_label: str | None = cluster_label self._job_prefix = job_prefix self._max_pbs_cmd_attempts = 10 @@ -158,45 +152,15 @@ def __init__( self._finished_job_ids: set[str] = set() self._finished_iens: set[int] = set() - if self._num_nodes is not None and self._num_nodes > 1: - logger.warning( - "OpenPBSDriver initialized with num_nodes > 1, " - "this behaviour is deprecated and will be removed" - ) - - if self._num_cpus_per_node is not None and self._num_cpus_per_node > 1: - logger.warning( - "OpenPBSDriver initialized with num_cpus_per_node, " - "this behaviour is deprecated and will be removed. " - "Use NUM_CPU in the config instead." - ) - def _build_resource_string( self, num_cpu: int = 1, realization_memory: int = 0 ) -> list[str]: resource_specifiers: list[str] = [] cpu_resources: list[str] = [] - if self._num_nodes is not None: - cpu_resources += [f"select={self._num_nodes}"] - if self._num_cpus_per_node is not None: - num_nodes = self._num_nodes or 1 - if num_cpu != self._num_cpus_per_node * num_nodes: - raise ValueError( - f"NUM_CPUS_PER_NODE ({self._num_cpus_per_node}) must be equal " - f"to NUM_CPU ({num_cpu}). " - "Please remove NUM_CPUS_PER_NODE from the configuration" - ) if num_cpu > 1: cpu_resources += [f"ncpus={num_cpu}"] - if self._memory_per_job is not None and realization_memory > 0: - raise ValueError( - "Overspecified memory pr job. " - "Do not specify both memory_per_job and realization_memory" - ) - if self._memory_per_job is not None: - cpu_resources += [f"mem={self._memory_per_job}"] - elif realization_memory > 0: + if realization_memory > 0: cpu_resources += [f"mem={realization_memory // 1024**2 }mb"] if cpu_resources: resource_specifiers.append(":".join(cpu_resources)) diff --git a/src/everest/config/simulator_config.py b/src/everest/config/simulator_config.py index 9f02948d670..44e54e51301 100644 --- a/src/everest/config/simulator_config.py +++ b/src/everest/config/simulator_config.py @@ -119,21 +119,10 @@ class SimulatorConfig(BaseModel, HasErtQueueOptions, extra="forbid"): # type: i qsub_cmd: str | None = Field(default="qsub", description="The submit command") qstat_cmd: str | None = Field(default="qstat", description="The query command") qdel_cmd: str | None = Field(default="qdel", description="The kill command") - qstat_options: str | None = Field( - default="-x", - description="Options to be supplied to the qstat command. This defaults to -x, which tells the qstat command to include exited processes.", - ) cluster_label: str | None = Field( default=None, description="The name of the cluster you are running simulations in.", ) - memory_per_job: str | None = Field( - default=None, - description="""You can specify the amount of memory you will need for running your job. This will ensure that not too many jobs will run on a single shared memory node at once, possibly crashing the compute node if it runs out of memory. - You can get an indication of the memory requirement by watching the course of a local run using the htop utility. Whether you should set the peak memory usage as your requirement or a lower figure depends on how simultaneously each job will run. - The option to be supplied will be used as a string in the qsub argument. You must specify the unit, either gb or mb. - """, - ) keep_qsub_output: int | None = Field( default=0, description="Set to 1 to keep error messages from qsub. Usually only to be used if somethign is seriously wrong with the queue environment/setup.", @@ -142,13 +131,6 @@ class SimulatorConfig(BaseModel, HasErtQueueOptions, extra="forbid"): # type: i default=0.5, description="To avoid stressing the TORQUE/PBS system you can instruct the driver to sleep for every submit request. The argument to the SUBMIT_SLEEP is the number of seconds to sleep for every submit, which can be a fraction like 0.5", ) - queue_query_timeout: int | None = Field( - default=126, - description=""" - The driver allows the backend TORQUE/PBS system to be flaky, i.e. it may intermittently not respond and give error messages when submitting jobs or asking for job statuses. The timeout (in seconds) determines how long ERT will wait before it will give up. Applies to job submission (qsub) and job status queries (qstat). Default is 126 seconds. - ERT will do exponential sleeps, starting at 2 seconds, and the provided timeout is a maximum. Let the timeout be sums of series like 2+4+8+16+32+64 in order to be explicit about the number of retries. Set to zero to disallow flakyness, setting it to 2 will allow for one re-attempt, and 6 will give two re-attempts. Example allowing six retries: - """, - ) project_code: str | None = Field( default=None, description="String identifier used to map hardware resource usage to a project or account. The project or account does not have to exist.", diff --git a/src/everest/config_keys.py b/src/everest/config_keys.py index 15c950547b9..791fb8c631f 100644 --- a/src/everest/config_keys.py +++ b/src/everest/config_keys.py @@ -123,7 +123,6 @@ class ConfigKeys: TORQUE_QDEL_CMD = "qdel_cmd" TORQUE_QUEUE_NAME = "name" TORQUE_CLUSTER_LABEL = "cluster_label" - TORQUE_MEMORY_PER_JOB = "memory_per_job" TORQUE_KEEP_QSUB_OUTPUT = "keep_qsub_output" TORQUE_SUBMIT_SLEEP = "submit_sleep" TORQUE_PROJECT_CODE = "project_code" diff --git a/src/everest/queue_driver/queue_driver.py b/src/everest/queue_driver/queue_driver.py index f41a9d2e1c9..d33f7601b7d 100644 --- a/src/everest/queue_driver/queue_driver.py +++ b/src/everest/queue_driver/queue_driver.py @@ -32,8 +32,6 @@ (ConfigKeys.TORQUE_QDEL_CMD, "QDEL_CMD"), (ConfigKeys.TORQUE_QUEUE_NAME, "QUEUE"), (ConfigKeys.TORQUE_CLUSTER_LABEL, "CLUSTER_LABEL"), - (ConfigKeys.CORES_PER_NODE, "NUM_CPUS_PER_NODE"), - (ConfigKeys.TORQUE_MEMORY_PER_JOB, "MEMORY_PER_JOB"), (ConfigKeys.TORQUE_KEEP_QSUB_OUTPUT, "KEEP_QSUB_OUTPUT"), (ConfigKeys.TORQUE_SUBMIT_SLEEP, "SUBMIT_SLEEP"), (ConfigKeys.TORQUE_PROJECT_CODE, "PROJECT_CODE"), diff --git a/tests/ert/unit_tests/config/config_dict_generator.py b/tests/ert/unit_tests/config/config_dict_generator.py index 9acedff897d..14db1d48649 100644 --- a/tests/ert/unit_tests/config/config_dict_generator.py +++ b/tests/ert/unit_tests/config/config_dict_generator.py @@ -167,8 +167,6 @@ def valid_queue_values(option_name, queue_system): elif option_name in queue_options_by_type["posfloat"][queue_system]: return small_floats.map(str) elif option_name in queue_options_by_type["posint"][queue_system]: - if option_name in {"NUM_NODES", "NUM_CPUS_PER_NODE"}: - return st.just("1") return positives.map(str) elif option_name in queue_options_by_type["bool"][queue_system]: return booleans.map(str) diff --git a/tests/ert/unit_tests/config/test_ert_config.py b/tests/ert/unit_tests/config/test_ert_config.py index 2438849e03f..97179024ca3 100644 --- a/tests/ert/unit_tests/config/test_ert_config.py +++ b/tests/ert/unit_tests/config/test_ert_config.py @@ -11,7 +11,7 @@ from unittest.mock import MagicMock import pytest -from hypothesis import HealthCheck, assume, example, given, settings +from hypothesis import HealthCheck, assume, given, settings from hypothesis import strategies as st from pydantic import RootModel, TypeAdapter @@ -566,50 +566,6 @@ def test_negative_std_cutoff_raises_validation_error(): ) -@pytest.mark.filterwarnings("ignore::ert.config.ConfigWarning") -@settings(max_examples=10) -@given(st.integers(min_value=0), st.integers(min_value=0), st.integers(min_value=0)) -@example(1, 1, 2) # error -@example(2, 1, 2) # no error -@example(2, 1, 0) # no error -@example(2, 0, 1) # no error -def test_num_cpu_vs_torque_queue_cpu_configuration( - num_cpu_int, num_nodes_int, num_cpus_per_node_int -): - # Only strictly positive ints are valid configuration values, - # zero values are used to represent the "not configured" scenario: - num_cpu = f"NUM_CPU {num_cpu_int}" if num_cpu_int else "" - num_nodes = ( - f"QUEUE_OPTION TORQUE NUM_NODES {num_nodes_int}" if num_nodes_int else "" - ) - num_cpus_per_node = ( - f"QUEUE_OPTION TORQUE NUM_CPUS_PER_NODE {num_cpus_per_node_int}" - if num_cpus_per_node_int - else "" - ) - - test_config_contents = dedent( - f"""\ - NUM_REALIZATIONS 1 - QUEUE_SYSTEM TORQUE - {num_cpu} - {num_nodes} - {num_cpus_per_node} - """ - ) - - if (num_nodes or num_cpus_per_node) and ( - (num_cpu_int or 1) != (num_nodes_int or 1) * (num_cpus_per_node_int or 1) - ): - with pytest.raises( - expected_exception=ConfigValidationError, - match="product .* must be equal", - ): - ErtConfig.from_file_contents(test_config_contents) - else: - ErtConfig.from_file_contents(test_config_contents) - - def test_that_non_existent_job_directory_gives_config_validation_error(): with pytest.raises( expected_exception=ConfigValidationError, diff --git a/tests/ert/unit_tests/config/test_queue_config.py b/tests/ert/unit_tests/config/test_queue_config.py index 7ddc985da81..363bd25589a 100644 --- a/tests/ert/unit_tests/config/test_queue_config.py +++ b/tests/ert/unit_tests/config/test_queue_config.py @@ -154,44 +154,6 @@ def test_invalid_realization_memory(invalid_memory_spec: str): ) -def test_conflicting_realization_openpbs_memory_per_job(): - with ( - pytest.raises(ConfigValidationError), - pytest.warns(ConfigWarning, match="deprecated"), - ): - ErtConfig.from_file_contents( - "NUM_REALIZATIONS 1\n" - "REALIZATION_MEMORY 10Mb\n" - "QUEUE_SYSTEM TORQUE\n" - "QUEUE_OPTION TORQUE MEMORY_PER_JOB 20mb\n" - ) - - -def test_conflicting_realization_openpbs_memory_per_job_but_slurm_activated_only_warns(): - with pytest.warns(ConfigWarning): - ErtConfig.from_file_contents( - "NUM_REALIZATIONS 1\n" - "REALIZATION_MEMORY 10Mb\n" - "QUEUE_SYSTEM SLURM\n" - "QUEUE_OPTION TORQUE MEMORY_PER_JOB 20mb\n" - ) - - -@pytest.mark.parametrize("torque_memory_with_unit_str", ["gb", "mb", "1 gb"]) -def test_that_invalid_memory_pr_job_raises_validation_error( - torque_memory_with_unit_str, -): - with ( - pytest.raises(ConfigValidationError), - pytest.warns(ConfigWarning, match="deprecated"), - ): - ErtConfig.from_file_contents( - "NUM_REALIZATIONS 1\n" - "QUEUE_SYSTEM TORQUE\n" - f"QUEUE_OPTION TORQUE MEMORY_PER_JOB {torque_memory_with_unit_str}" - ) - - @pytest.mark.parametrize( "queue_system, queue_system_option", [("LSF", "LSF_QUEUE"), ("SLURM", "SQUEUE"), ("TORQUE", "QUEUE")], @@ -244,7 +206,6 @@ def test_initializing_empty_config_queue_options_resets_to_default_value( "queue_system, queue_option, queue_value, err_msg", [ ("SLURM", "SQUEUE_TIMEOUT", "5a", "should be a valid number"), - ("TORQUE", "NUM_NODES", "3.5", "should be a valid integer"), ], ) def test_wrong_config_option_types(queue_system, queue_option, queue_value, err_msg): @@ -255,11 +216,7 @@ def test_wrong_config_option_types(queue_system, queue_option, queue_value, err_ ) with pytest.raises(ConfigValidationError, match=err_msg): - if queue_system == "TORQUE": - with pytest.warns(ConfigWarning, match="deprecated"): - ErtConfig.from_file_contents(file_contents) - else: - ErtConfig.from_file_contents(file_contents) + ErtConfig.from_file_contents(file_contents) def test_that_configuring_another_queue_system_gives_warning(): @@ -271,44 +228,6 @@ def test_that_configuring_another_queue_system_gives_warning(): ) -@pytest.mark.parametrize( - "mem_per_job", - ["5gb", "5mb", "5kb"], -) -def test_that_valid_torque_queue_mem_options_are_ok(mem_per_job): - with pytest.warns(ConfigWarning, match="deprecated"): - ErtConfig.from_file_contents( - "NUM_REALIZATIONS 1\n" - "QUEUE_SYSTEM SLURM\n" - f"QUEUE_OPTION TORQUE MEMORY_PER_JOB {mem_per_job}\n" - ) - - -@pytest.mark.parametrize( - "mem_per_job", - ["5", "5g"], -) -def test_that_torque_queue_mem_options_are_corrected(mem_per_job: str): - with ( - pytest.raises(ConfigValidationError) as e, - pytest.warns(ConfigWarning, match="deprecated"), - ): - ErtConfig.from_file_contents( - "NUM_REALIZATIONS 1\n" - "QUEUE_SYSTEM TORQUE\n" - f"QUEUE_OPTION TORQUE MEMORY_PER_JOB {mem_per_job}\n" - ) - - info = e.value.errors[0] - - assert ( - f"Value error, wrong memory format. Got input '{mem_per_job}'." in info.message - ) - assert info.line == 3 - assert info.column == 36 - assert info.end_column == info.column + len(mem_per_job) - - def test_max_running_property(): config = ErtConfig.from_file_contents( "NUM_REALIZATIONS 1\n" diff --git a/tests/ert/unit_tests/scheduler/test_openpbs_driver.py b/tests/ert/unit_tests/scheduler/test_openpbs_driver.py index e3a35e3a3c3..1a01609279f 100644 --- a/tests/ert/unit_tests/scheduler/test_openpbs_driver.py +++ b/tests/ert/unit_tests/scheduler/test_openpbs_driver.py @@ -137,25 +137,18 @@ def parse_resource_string(qsub_args: str) -> dict[str, str]: @pytest.mark.usefixtures("capturing_qsub") -async def test_memory_per_job(): - driver = OpenPBSDriver(memory_per_job="10gb") - await driver.submit(0, "sleep") - assert " -l mem=10gb" in Path("captured_qsub_args").read_text(encoding="utf-8") - - -@pytest.mark.usefixtures("capturing_qsub") -async def test_no_default_memory_per_job(): +async def test_no_default_realization_memory(): driver = OpenPBSDriver() await driver.submit(0, "sleep") assert " -l " not in Path("captured_qsub_args").read_text(encoding="utf-8") @pytest.mark.usefixtures("capturing_qsub") -async def test_no_validation_of_memory_per_job(): +async def test_realization_memory_not_negative(): # Validation will happen during config parsing - driver = OpenPBSDriver(memory_per_job="a_lot") - await driver.submit(0, "sleep") - assert " -l mem=a_lot" in Path("captured_qsub_args").read_text(encoding="utf-8") + driver = OpenPBSDriver() + await driver.submit(0, "sleep", realization_memory=-1) + assert " -l " not in Path("captured_qsub_args").read_text(encoding="utf-8") @pytest.mark.usefixtures("capturing_qsub") @@ -179,47 +172,6 @@ async def test_job_name_with_prefix(): assert " -Npre_sleepy " in Path("captured_qsub_args").read_text(encoding="utf-8") -@pytest.mark.usefixtures("capturing_qsub") -async def test_num_nodes(): - driver = OpenPBSDriver(num_nodes=2) - await driver.submit(0, "sleep") - resources = parse_resource_string( - Path("captured_qsub_args").read_text(encoding="utf-8") - ) - assert resources["select"] == "2" - - -@pytest.mark.usefixtures("capturing_qsub") -async def test_num_nodes_default(): - # This is the driver default, Ert can fancy a different default - driver = OpenPBSDriver() - await driver.submit(0, "sleep") - resources = parse_resource_string( - Path("captured_qsub_args").read_text(encoding="utf-8") - ) - assert "nodes" not in resources - - -@pytest.mark.usefixtures("capturing_qsub") -async def test_num_cpus_per_node(): - driver = OpenPBSDriver(num_cpus_per_node=2) - await driver.submit(0, "sleep", num_cpu=2) - resources = parse_resource_string( - Path("captured_qsub_args").read_text(encoding="utf-8") - ) - assert resources["ncpus"] == "2" - - -@pytest.mark.usefixtures("capturing_qsub") -async def test_num_cpus_per_node_default(): - driver = OpenPBSDriver() - await driver.submit(0, "sleep") - resources = parse_resource_string( - Path("captured_qsub_args").read_text(encoding="utf-8") - ) - assert "ncpus" not in resources - - @pytest.mark.usefixtures("capturing_qsub") async def test_cluster_label(): driver = OpenPBSDriver(cluster_label="foobar") @@ -309,76 +261,24 @@ async def started(iens): assert was_started == started_expected -@given(words, st.integers(min_value=1), words) +@given(st.integers(), st.integers(min_value=1), words) @pytest.mark.usefixtures("capturing_qsub") -async def test_full_resource_string(memory_per_job, num_cpu, cluster_label): +async def test_full_resource_string(realization_memory, num_cpu, cluster_label): driver = OpenPBSDriver( - memory_per_job=memory_per_job if memory_per_job else None, cluster_label=cluster_label if cluster_label else None, ) - await driver.submit(0, "sleep", num_cpu=num_cpu) - resources = parse_resource_string( - Path("captured_qsub_args").read_text(encoding="utf-8") + await driver.submit( + 0, "sleep", num_cpu=num_cpu, realization_memory=realization_memory ) - assert resources.get("mem", "") == memory_per_job - assert resources.get("select", "1") == "1" - assert resources.get("ncpus", "1") == str(num_cpu) - - if cluster_label: - # cluster_label is not a key-value thing in the resource list, - # the parser in this test handles that specially - assert resources.get(cluster_label) == "_present_" - - assert len(resources) == sum( - [ - bool(memory_per_job), - num_cpu > 1, - bool(cluster_label), - ] - ), "Unknown or missing resources in resource string" - - -@pytest.mark.usefixtures("capturing_qsub") -async def test_submit_with_realization_memory(): - driver = OpenPBSDriver() - await driver.submit(0, "sleep", realization_memory=1024**2) resources = parse_resource_string( Path("captured_qsub_args").read_text(encoding="utf-8") ) - assert resources.get("mem", "") == "1mb" - - -@pytest.mark.usefixtures("capturing_qsub") -async def test_submit_with_realization_memory_and_memory_per_job(): - driver = OpenPBSDriver(memory_per_job="1") - with pytest.raises(ValueError, match="Overspecified memory"): - await driver.submit(0, "sleep", realization_memory=1) - - -@given(words, st.integers(min_value=0), st.integers(min_value=0), words) -@pytest.mark.usefixtures("capturing_qsub") -async def test_full_resource_string_deprecated_options( - memory_per_job, num_nodes, num_cpus_per_node, cluster_label -): - """Test deprecated queue options to the driver. - - Remove this test function altogether when NUM_CPUS_PER_NODE and NUM_NODES - support is removed""" - driver = OpenPBSDriver( - memory_per_job=memory_per_job if memory_per_job else None, - num_nodes=num_nodes if num_nodes > 0 else None, - num_cpus_per_node=num_cpus_per_node if num_cpus_per_node > 0 else None, - cluster_label=cluster_label if cluster_label else None, - ) - num_cpu = (num_nodes or 1) * (num_cpus_per_node or 1) - await driver.submit(0, "sleep", num_cpu=num_cpu) - resources = parse_resource_string( - Path("captured_qsub_args").read_text(encoding="utf-8") + assert resources.get("mem", "") == ( + f"{realization_memory // 1024**2 }mb" if realization_memory > 0 else "" ) - assert resources.get("mem", "") == memory_per_job - assert resources.get("select", "0") == str(num_nodes) + assert resources.get("select", "1") == "1" assert resources.get("ncpus", "1") == str(num_cpu) - # "0" here is a test implementation detail, not related to driver + if cluster_label: # cluster_label is not a key-value thing in the resource list, # the parser in this test handles that specially @@ -386,9 +286,8 @@ async def test_full_resource_string_deprecated_options( assert len(resources) == sum( [ - bool(memory_per_job), + realization_memory > 0, num_cpu > 1, - bool(num_nodes), bool(cluster_label), ] ), "Unknown or missing resources in resource string" diff --git a/tests/ert/unit_tests/scheduler/test_scheduler.py b/tests/ert/unit_tests/scheduler/test_scheduler.py index ba8df1e97f8..7736dbb2998 100644 --- a/tests/ert/unit_tests/scheduler/test_scheduler.py +++ b/tests/ert/unit_tests/scheduler/test_scheduler.py @@ -647,9 +647,6 @@ def test_scheduler_create_lsf_driver(): def test_scheduler_create_openpbs_driver(): queue_name = "foo_queue" keep_qsub_output = "True" - memory_per_job = "13gb" - num_nodes = 1 - num_cpus_per_node = 1 cluster_label = "bar_cluster_label" job_prefix = "foo_job_prefix" qsub_cmd = "bar_qsub_cmd" @@ -661,9 +658,6 @@ def test_scheduler_create_openpbs_driver(): "QUEUE_OPTION": [ ("TORQUE", "QUEUE", queue_name), ("TORQUE", "KEEP_QSUB_OUTPUT", keep_qsub_output), - ("TORQUE", "MEMORY_PER_JOB", memory_per_job), - ("TORQUE", "NUM_NODES", str(num_nodes)), - ("TORQUE", "NUM_CPUS_PER_NODE", str(num_cpus_per_node)), ("TORQUE", "CLUSTER_LABEL", cluster_label), ("TORQUE", "JOB_PREFIX", job_prefix), ("TORQUE", "QSUB_CMD", qsub_cmd), @@ -676,9 +670,6 @@ def test_scheduler_create_openpbs_driver(): assert isinstance(driver, OpenPBSDriver) assert driver._queue_name == queue_name assert driver._keep_qsub_output == True if keep_qsub_output == "True" else False - assert driver._memory_per_job == memory_per_job - assert driver._num_nodes == num_nodes - assert driver._num_cpus_per_node == num_cpus_per_node assert driver._cluster_label == cluster_label assert driver._job_prefix == job_prefix assert str(driver._qsub_cmd) == qsub_cmd diff --git a/tests/everest/test_res_initialization.py b/tests/everest/test_res_initialization.py index bc7e989971d..920f1d78a9e 100644 --- a/tests/everest/test_res_initialization.py +++ b/tests/everest/test_res_initialization.py @@ -43,8 +43,6 @@ "qsub_cmd": "qsub", "qstat_cmd": "qstat", "qdel_cmd": "qdel", - "num_cpus_per_node": 3, - "num_nodes": 1, "keep_qsub_output": True, "queue_name": "permanent_8", },