Skip to content

perf(avro): cache Avro-to-Iceberg schema conversion#3663

Open
vishnuprakaz wants to merge 1 commit into
apache:mainfrom
vishnuprakaz:perf/cache-avro-schema-conversion
Open

perf(avro): cache Avro-to-Iceberg schema conversion#3663
vishnuprakaz wants to merge 1 commit into
apache:mainfrom
vishnuprakaz:perf/cache-avro-schema-conversion

Conversation

@vishnuprakaz

Copy link
Copy Markdown
Contributor

Closes #3662

Rationale for this change

AvroFileHeader.get_schema() re-parses the embedded Avro schema and converts it to an Iceberg schema on every file
open. All manifests in a table share the same schema, so during scan planning we redo that identical conversion
once per manifest for no reason.

Caching it (keyed on the schema string) means each distinct schema is converted only once.

I timed scan().plan_files() on unpartitioned tables of a few sizes, main vs this branch (median of 10 warm runs):

manifests before after speedup
10 5.6 ms 3.1 ms 1.8x
50 29.4 ms 15.6 ms 1.9x
150 86.2 ms 48.0 ms 1.8x
400 233.9 ms 128.8 ms 1.8x

Roughly 1.8x throughout, and since the wasted work is per-manifest, larger tables save more. Results are identical.
This is a narrow, unpartitioned table; a partitioned one has a wider manifest schema and would gain at least as
much.

Uses functools.lru_cache, like the existing _cached_resolve_s3_region in io/pyarrow.py, so no new dependency.
Schema is immutable and already shared across scan threads, so sharing one cached instance is safe.

Are these changes tested?

Yes. A unit test in tests/avro/test_file.py checks the cached schema is correct, that the same schema string
returns the same object, and that a different string returns a different schema. There's also a benchmark in
tests/benchmark/test_avro_schema_cache_benchmark.py (run with -m benchmark) for the numbers above. Existing
avro/table/manifest tests pass.

Are there any user-facing changes?

No, same behavior, just faster scan planning on tables with many manifests.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Avro schema is re-converted to an Iceberg schema on every manifest read during scan planning

1 participant