22 — I/O, Files & the io Package

// io.Pipe creates a synchronous pipe: a Reader and Writer pair where
// writes block until reads consume (and vice versa). It's a channel
// for byte streams.

func pipeDemo() {
    r, w := io.Pipe()

    // Writer goroutine:
    go func() {
        defer w.Close()  // signal EOF to the reader
        json.NewEncoder(w).Encode(map[string]int{"x": 1, "y": 2})
    }()

    // Reader (blocks until writer writes, then blocks again for more):
    var m map[string]int
    if err := json.NewDecoder(r).Decode(&m); err != nil {
        log.Fatal(err)
    }
    fmt.Println(m)  // map[x:1 y:2]
}

// Use case: stream a large JSON response to S3 without buffering it all
// in memory. The HTTP body reader writes to the pipe; S3 upload reads
// from it concurrently.

💡 Tips & Tricks

• Idiom: use io.Copy(dst, src) for all data transfer — it handles buffering, EOF, and short reads. Don't manually loop Read + Write — io.Copy is tested and optimized.
• Idiom: use os.ReadFile/os.WriteFile (Go 1.16+) for whole-file operations — they replace ioutil.ReadFile/WriteFile (deprecated). For streaming, use os.Open + io.Copy.
• Idiom: use bufio.Scanner for line-by-line reading — increase the buffer with scanner.Buffer(make([]byte, 0, 64*1024), maxSize) for long lines (default max is 64KB).
• Performance: io.Copy uses a 32KB internal buffer — optimal for most cases. For very large transfers, io.CopyBuffer lets you reuse a buffer (zero allocation).
• Safety: for write durability, call f.Sync() before f.Close() — Sync flushes kernel buffers to disk. Without it, a crash after Close may lose data (the OS buffer is lost).
• Idiom: check scanner.Err() after the scanner.Scan() loop — the loop exits on EOF (normal) or error; Err() tells you which.

⚠️ Edge Cases & Gotchas

• Read may return n < len(p) with err=nil: that's normal — don't assume the buffer is full. Always use p[:n] for the actual data.
• bufio.Scanner default max line is 64KB: lines longer than 64KB cause scanner.Err() to return bufio.ErrTooLong. Increase with scanner.Buffer(...).
• defer f.Close() discards the error: Close can fail (flush error, network write). For writes, call Close explicitly and check the error. For reads, the discarded error is usually fine.
• os.WriteFile doesn't sync: it writes and closes, but doesn't call Sync. For durability, open with O_SYNC or call Sync explicitly.
• io.ReadAll reads everything into memory: for large streams, this causes OOM. Use io.Copy to stream to a file or json.NewDecoder for streaming decode.
• http.Response.Body must be closed: defer resp.Body.Close() — leaking it leaks the TCP connection. Even on error, close it.
• io.Pipe is synchronous: writes block until reads consume. If the reader is slow, the writer blocks. Use a buffered channel or bytes.Buffer for async buffering.
• Short writes: Write may write fewer bytes than requested (n < len(p)). io.Copy handles this; manual code must loop until all bytes are written.

🧠 Quick Quiz

func readConfig(path string) ([]byte, error) {
    f, err := os.Open(path)
    if err != nil {
        return nil, err
    }
    defer f.Close()
    return io.ReadAll(f)
}