English is currently available.
Go DISCUSSION
My exporter builds a CSV report by doing out += line + "\n" inside a loop over about 100,000 records. It takes many seconds, and the memory profile shows an enormous total allocation figure even though the final string is only around 5 MB.
I understand that strings are immutable in Go, but I do not see why the cost grows so badly. What is the idiomatic way to build a big string, and does it matter for small cases such as joining three or four pieces?
Because strings are immutable, out += line cannot extend the string in place. Each iteration allocates a new string of the combined length and copies everything accumulated so far plus the new piece. The total copied is 50 + 100 + 150 + ... bytes, which is quadratic.
With 100,000 lines of about 50 bytes the final string is 5 MB, but the bytes copied add up to roughly 50 × 100,000² / 2 = 2.5 × 10^11, about 250 GB of memory traffic, and every intermediate string becomes garbage. That is what your profile shows. A builder that appends into a growing buffer copies each byte only a small constant number of times.
The standard tool is strings.Builder, available since Go 1.10: var sb strings.Builder, then sb.WriteString(line) and sb.WriteByte('\n') in the loop, and out := sb.String() at the end. It appends into an internal byte slice that grows geometrically, and String() returns the result without copying the buffer again.
If you can estimate the size, call sb.Grow(len(records) * 50) first and the regrowth copies disappear as well. Formatted pieces can be written into it directly with fmt.Fprintf(&sb, "%d,%.2f\n", id, value), since *strings.Builder implements io.Writer. Do not copy a Builder by value once it has been written to; it detects that and panics.
If the pieces are already in a slice, strings.Join(lines, "\n") is the simplest and is optimal: it computes the total length, allocates once and copies each piece once.
If the output is going to a file or a socket anyway, do not build the string at all. Wrap the destination in bufio.NewWriter(f), write each line with w.WriteString or fmt.Fprintf(w, ...), and call w.Flush() at the end, checking its error. Memory use is then the buffer, 4096 bytes by default, instead of the whole report. For CSV specifically, the encoding/csv writer also quotes fields that contain commas, quotes or newlines, which naive concatenation gets wrong.
For small cases it does not matter. s := a + ", " + b + "!" in a single expression is compiled into one runtime call that adds up the lengths and allocates once, so it is as good as a builder and clearer to read. fmt.Sprintf is slower than + because it interprets the format string at run time, but in code that runs a few times per request that is irrelevant; choose for readability.
For number conversion in a hot path, strconv.Itoa(n) or strconv.AppendInt(buf, int64(n), 10) avoid the fmt machinery. To see the difference for yourself, write a benchmark and run go test -bench . -benchmem; the allocations-per-operation column separates the += loop from the builder at a glance.
Back to Go community
