Code Snippet

Just another Code Snippet site

[Unix] Split large text file

$ split --help
Usage: split [OPTION] [INPUT [PREFIX]]
Output fixed-size pieces of INPUT to PREFIXaa, PREFIXab, ...; default
size is 1000 lines, and default PREFIX is `x'.  With no INPUT, or when INPUT
is -, read standard input.

Mandatory arguments to long options are mandatory for short options too.
  -a, --suffix-length=N   use suffixes of length N (default 2)
  -b, --bytes=SIZE        put SIZE bytes per output file
  -C, --line-bytes=SIZE   put at most SIZE bytes of lines per output file
  -d, --numeric-suffixes  use numeric suffixes instead of alphabetic
  -l, --lines=NUMBER      put NUMBER lines per output file
      --verbose           print a diagnostic to standard error just
                            before each output file is opened
      --help     display this help and exit
      --version  output version information and exit

example :

split -d -l 250000 A.log

Will generate x00, x01, x02, etc.


One thought on “[Unix] Split large text file

  • Content Split (csplit) instead of size split:

    csplit -s -z MY_INPUT_FILE.data /REGEXP/ '{*}' --prefix='my_sub_file-' --suffix-format='%02d.json'
    

    output will be:
    my_sub_file-00.json
    my_sub_file-01.json
    my_sub_file-02.json
    ….

Leave a Reply

Your email address will not be published. Required fields are marked *