Skip to content

[To dev/1.3] Optimize pipe tablet memory accounting#18220

Open
Caideyipi wants to merge 1 commit into
apache:dev/1.3from
Caideyipi:backport/18211-dev-1.3
Open

[To dev/1.3] Optimize pipe tablet memory accounting#18220
Caideyipi wants to merge 1 commit into
apache:dev/1.3from
Caideyipi:backport/18211-dev-1.3

Conversation

@Caideyipi

Copy link
Copy Markdown
Collaborator

Description

Backport #18211 to dev/1.3.

  • Avoid creating an identity deduplication set for single InsertNode leaves and single-child composite nodes in InsertNodeMemoryEstimator.
  • Adapt the single-raw-tablet string-intern-pool optimization to the Batch V1 request path used by dev/1.3.
  • Preserve shared string interning for batches containing multiple raw tablets.
  • Add opt-in manual performance UTs for the 100,000-column paths.

Tests

  • mvn -o -pl iotdb-core/datanode spotless:apply
    • BUILD SUCCESS
  • mvn -o -pl iotdb-core/datanode -Dcheckstyle.skip=true -Dspotless.skip=true -Dtest=InsertNodeMemoryEstimatorTest,PipeDataNodeThriftRequestTest test
    • Tests run: 25, Failures: 0, Errors: 0, Skipped: 0
  • mvn -o -pl iotdb-core/datanode -Dcheckstyle.skip=true -Dspotless.skip=true -Dtest=InsertNodeMemoryEstimatorPerformanceTest,PipeTransferTabletBatchReqPerformanceTest -Diotdb.pipe.insert.node.memory.estimator.perf.enabled=true -Diotdb.pipe.insert.node.memory.estimator.perf.iterations=3 -Diotdb.pipe.insert.node.memory.estimator.perf.warmup.iterations=1 -Diotdb.pipe.tablet.batch.deserialize.perf.enabled=true -Diotdb.pipe.tablet.batch.deserialize.perf.iterations=3 -Diotdb.pipe.tablet.batch.deserialize.perf.warmup.iterations=1 test
    • Tests run: 2, Failures: 0, Errors: 0, Skipped: 0
    • InsertNode memory estimate benchmark: 100000 columns, 2.439-3.155 ms/op
    • Batch V1 single raw tablet deserialization benchmark: 100000 columns, 15.224-17.135 ms/op

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant