diff --git a/activitysim/core/configuration/top.py b/activitysim/core/configuration/top.py index ffb57db17..825283126 100644 --- a/activitysim/core/configuration/top.py +++ b/activitysim/core/configuration/top.py @@ -790,7 +790,7 @@ def _check_store_skims_in_shm(self): .. versionadded:: 1.6 """ - fraction_of_failed_choices_allowed: float = 0.1 + fraction_of_failed_choices_allowed: float = 0.001 """ Threshold for the fraction of households that can be skipped before failing the model run, used in conjunction with `skip_failed_choices`. diff --git a/docs/dev-guide/changes.md b/docs/dev-guide/changes.md index b1934bae2..7d02826c5 100644 --- a/docs/dev-guide/changes.md +++ b/docs/dev-guide/changes.md @@ -14,6 +14,31 @@ branch (i.e., the main branch on GitHub), but not yet released in a stable versi of ActivitySim. See below under the various version headings for changes in released versions. +### Skipping Failed Choices + +A new feature is introduced to ActivitySim to skip failed choices during model execution. +By default, ActivitySim will skip any failed choices during a model run, i.e., `skip_failed_choices` is set to `True`. +A failed choice occurs when the computed utilities for all alternatives are zero, or infinite, or nan, which can happen due to +data issues or model specification problems. A warning message is logged when a failed choice is encountered, +and the corresponding household (along with its persons, vehicles, tours, trips, etc) will be excluded from further model steps. +At the end of the model run, a summary of all skipped households is provided in the log file for user reference. Users can set +a threshold for the maximum allowed percentage of skipped households `fraction_of_failed_choices_allowed`, +and if the percentage of skipped households exceeds this threshold, +the model run will be terminated with an error. The default threshold is set to `0.001`, 0.1% of households. + +This feature helps to ensure that the model can continue running even in the presence of data or specification issues, +while also providing visibility into such issues that need to be addressed. +See more information in ActivitySim's users guide "Skip Failed Choices" and code updates in [PR #1023](https://github.com/ActivitySim/activitysim/pull/1023) + +##### Potential Impact on Existing Model Runs: + +With `skip_failed_choices` defaulted to `True` and an allowed failure threshold defaulted as 0.1% of households, +failures that were previously silently masked will now generate warnings, be explicitly skipped, and counted toward the threshold. +As a result, model runs that previously completed "successfully" may now fail earlier, +surfacing underlying specification or data issues that require attention rather than being silently absorbed. If an agency does not +want to address the underlying issues immediately, they can set `skip_failed_choices` to `False` to maintain the previous model results, +but it is recommended to review the log file for any warnings about failed choices and address them as soon as possible to ensure model quality. + ### Shadow Price Zones Reopening We fixed an issue with reopening work and school zones during iterative shadow pricing using the diff --git a/docs/users-guide/ways_to_run.rst b/docs/users-guide/ways_to_run.rst index dda69f889..35a9692e5 100644 --- a/docs/users-guide/ways_to_run.rst +++ b/docs/users-guide/ways_to_run.rst @@ -312,6 +312,11 @@ For interaction_simulate models, users can manually specify trace IDs to perform Users can configure ActivitySim to not skip failed choices by setting the ``skip_failed_choices`` option to ``False`` in the settings file. When this option is disabled, the system will fall back to using the legacy ``overflow_protection`` mechanism to handle such cases. Specifically, if the computed utilities lead to zero or infinite exponentiated values, -overflow protection will adjust the utilities to prevent numerical overflow during exponentiation and arbitarily making a choice. -No loggings will be made for these cases. When ``skip_failed_choices`` is enabled, -ActivitySim will not use the legacy ``overflow_protection`` mechanism to handle failed choices. +the legacy ``overflow_protection`` will adjust the utilities to prevent numerical overflow during exponentiation and arbitarily making a choice, however, no loggings will be made for these cases. +When ``skip_failed_choices`` is enabled, ActivitySim will not use the legacy ``overflow_protection`` mechanism to handle failed choices. + +.. note:: + When an agency turns on ``skip_failed_choices`` for the first time in an existing ActivitySim implementation, + warning messages may appear in the log for failed choices that were previously handled silently by the legacy ``overflow_protection`` mechanism. + These warnings are expected and reflect the change in behavior: failed choices are now explicitly skipped and traced rather than silently handled. + In some cases, model runs may crash if the number of skipped failed choices exceeds the configured ``fraction_of_failed_choices_allowed`` threshold. \ No newline at end of file