Skip to content

Conversation

@gpshead
Copy link
Member

@gpshead gpshead commented Sep 16, 2022

Further reduce quote_from_bytes memory consumption on large input values. Based on Dennis Sweeney's chunking idea.

on large input values.  Based on Dennis Sweeney's chunking idea.
@gpshead gpshead requested a review from sweeneyde September 16, 2022 07:55
@gpshead gpshead added type-feature A feature request or enhancement performance Performance or resource usage stdlib Standard Library Python modules in the Lib/ directory labels Sep 16, 2022
@gpshead
Copy link
Member Author

gpshead commented Sep 16, 2022

On large values, this consumes ~half the memory.

@sweeneyde
Copy link
Member

sweeneyde commented Sep 16, 2022

This looks like a correct, comprehensible change, my only wonder would be whether some sort of regex-based approach would be more efficient for this quote direction as well, i.e., have a regex recognize strings of safe characters.

@sweeneyde
Copy link
Member

It may also depend on what sorts of strings are typically quoted/unquoted. Different texts/corpuses probably have different performance characteristics, and I am not all that familiar with how this gets used in the wild.

@gpshead
Copy link
Member Author

gpshead commented Sep 19, 2022

there are likely further optimization opportunities. but as this does use notably less memory already, lets go ahead and do it. I don't like introducing regexes when possible that could be tested and considered as a separate change if anyone ever cares here.

@gpshead gpshead merged commit e61ca22 into python:main Sep 19, 2022
@gpshead gpshead deleted the gh/95865/quote_mem_use branch September 19, 2022 23:06
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

performance Performance or resource usage stdlib Standard Library Python modules in the Lib/ directory type-feature A feature request or enhancement

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants