`--config-file` incorrectly expects `[tool.pytest]` instead of `[pytest]` in TOML config on pytest 9

[francis-clairicia]

Description

After a migration from a custom config file pytest-benchmark.ini:

[pytest]
python_files = bench_*.py
python_classes = Bench
python_functions = bench_*
addopts =
    --strict-markers
    --benchmark-warmup=on
    --benchmark-warmup-iterations=10000
    --benchmark-timer=time.process_time
    --benchmark-time-unit=us
    --benchmark-min-time=0.000002
    --benchmark-sort=fullname
    --benchmark-disable-gc
minversion = 7.4
testpaths = benchmarks

... to pytest-benchmark.toml with the help of the documentation:

[pytest]
minversion = "9.0"
python_files = ["bench_*.py"]
python_classes = ["Bench"]
python_functions = ["bench_*"]
strict_config = true
strict_markers = true
strict_parametrization_ids = true
addopts = [
    "--benchmark-warmup=on",
    "--benchmark-warmup-iterations=10000",
    "--benchmark-timer=time.process_time",
    "--benchmark-time-unit=us",
    "--benchmark-min-time=0.000002",
    "--benchmark-sort=fullname",
    "--benchmark-disable-gc",
]
testpaths = ["benchmarks"]

... launching pytest -c pytest-benchmark.toml --collect-only ignores the benchmark files defined and uses the default configuration values.

I tried to put the configuration in [tool.pytest] and it worked:

- [pytest]
+ [tool.pytest]
minversion = "9.0"
python_files = ["bench_*.py"]
python_classes = ["Bench"]
python_functions = ["bench_*"]
strict_config = true
strict_markers = true
strict_parametrization_ids = true
addopts = [
    "--benchmark-warmup=on",
    "--benchmark-warmup-iterations=10000",
    "--benchmark-timer=time.process_time",
    "--benchmark-time-unit=us",
    "--benchmark-min-time=0.000002",
    "--benchmark-sort=fullname",
    "--benchmark-disable-gc",
]
testpaths = ["benchmarks"]

Environment

• pytest: 9.1.1
• OS: Linux

[The-Compiler]

Good catch, indeed the code hardcodes the names for [pytest] headers:

pytest/src/_pytest/config/findpaths.py

Lines 93 to 161 in 67a174f

	# '.toml' files are considered if they contain a [tool.pytest] table (toml mode)
	# or [tool.pytest.ini_options] table (ini mode) for pyproject.toml,
	# or [pytest] table (toml mode) for pytest.toml/.pytest.toml.
	elif filepath.suffix == ".toml":
	if sys.version_info >= (3, 11):
	import tomllib
	else:
	import tomli as tomllib
	toml_text = filepath.read_text(encoding="utf-8")
	try:
	config = tomllib.loads(toml_text)
	except tomllib.TOMLDecodeError as exc:
	raise UsageError(f"{filepath}: {exc}") from exc
	# pytest.toml and .pytest.toml use [pytest] table directly.
	if filepath.name in ("pytest.toml", ".pytest.toml"):
	if "pytest" in config:
	# TOML mode - preserve native TOML types.
	return {
	k: ConfigValue(v, origin="file", mode="toml")
	for k, v in config["pytest"].items()
	}
	top_level_options = [
	key for key, value in config.items() if not isinstance(value, dict)
	]
	if top_level_options:
	raise UsageError(
	f"{filepath}: pytest configuration must be under a "
	f"[pytest] table (found top-level options: "
	f"{', '.join(top_level_options)})"
	)
	# "pytest.toml" files are always the source of configuration, even if empty.
	return {}
	# pyproject.toml uses [tool.pytest] or [tool.pytest.ini_options].
	else:
	tool_pytest = config.get("tool", {}).get("pytest", {})
	# Check for toml mode config: [tool.pytest] with content outside of ini_options.
	toml_config = {k: v for k, v in tool_pytest.items() if k != "ini_options"}
	# Check for ini mode config: [tool.pytest.ini_options].
	ini_config = tool_pytest.get("ini_options", None)
	if toml_config and ini_config:
	raise UsageError(
	f"{filepath}: Cannot use both [tool.pytest] (native TOML types) and "
	"[tool.pytest.ini_options] (string-based INI format) simultaneously. "
	"Please use [tool.pytest] with native TOML types (recommended) "
	"or [tool.pytest.ini_options] for backwards compatibility."
	)
	if toml_config:
	# TOML mode - preserve native TOML types.
	return {
	k: ConfigValue(v, origin="file", mode="toml")
	for k, v in toml_config.items()
	}
	elif ini_config is not None:
	# INI mode - TOML supports richer data types than INI files, but we need to
	# convert all scalar values to str for compatibility with the INI system.
	def make_scalar(v: object) -> str | list[str]:
	return v if isinstance(v, list) else str(v)
	return {
	k: ConfigValue(make_scalar(v), origin="file", mode="ini")
	for k, v in ini_config.items()
	}

The comments sound a bit like this is intentional, but I'm not entirely sure it is. In any case, fixing it in a backwards-compatible way is unfortunately going to make that logic even more complex, most likely it would need to consider both [tool.pytest] and [pytest] in any custom TOML files.

[francis-clairicia]

I don't think the fix needs to be backward-compatible because:

1. the behavior has never been documented in this way,
2. even with the ini format, the section [tool:pytest] is read only with the setup.cfg file.

Until now, the "tool" section has always been dedicated to the project file.

[The-Compiler]

From a cursory GitHub code search, I suppose it might indeed be fine. Still a corner case though.

[molloyzak13]

reply_target: #14705
The comment at line 128 names pyproject.toml only (it says the else branch handles "pyproject.toml" and reads from [tool.pytest]) but the condition at line 109 checks for pytest.toml and .pytest.toml. A custom TOML file passed with --config-file falls into that else branch and reads the wrong section. The comments document a three-way split but the branching only implements two.

PR #14707 has the right approach, keeping the [tool.pytest] lookup for pyproject.toml and routing custom TOML files to [pytest].