Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Using binary format for COPY #2286

Open
joto opened this issue Jan 6, 2025 · 1 comment
Open

Using binary format for COPY #2286

joto opened this issue Jan 6, 2025 · 1 comment

Comments

@joto
Copy link
Collaborator

joto commented Jan 6, 2025

As part of #2110 we found that using the binary format for COPY promises some performance benefits. Lets think about what changes we need for that.

The binary format seems simple enough to implement and it appears to be pretty stable. But there is no documentation for all the types, instead the PostgreSQL docs say you should look into the code. There are probably also some client implementations out there we could find that implement this.

Text fields are just length + content and ints are ints in network byte order. But I don't know about more complex types such as arrays and json. This could be especially problematic for tables that the user creates, because they can use any type they want, even one where we might not know the internal representation (see also #2274). But maybe we can still use the text representation and PostgreSQL converts on import as it does with the COPY text format? If not we probably can't use the binary format for user tables, but we could still use it for middle tables.

@lonvia
Copy link
Collaborator

lonvia commented Jan 6, 2025

Psycopg, which supports binary copy, states in its documentation that Postgesql does not apply any conversions. So custom types are indeed not possible without the user taking care to produce a binary format. That sounds complicated.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants