NAME

uniq - report or filter out repeated lines in a file

 SYNOPSIS

uniq [-c|-d|-u][-f fields][-s char][input_file [output_file]]

uniq [-c|-d|-u][-n][+m][input_file [output_file]]

 DESCRIPTION

The uniq utility will read an input file comparing adjacent lines, and write one copy of each input line on the output. The second and succeeding copies of repeated adjacent input lines will not be written.

Repeated lines in the input will not be detected if they are not adjacent.

 OPTIONS

The uniq utility supports the XBD specification, Utility Syntax Guidelines  ; the obsolescent version does not, as one of the options begins with "+" and the -m and +n options do not have option letters.

The following options are supported:

-c

Precede each output line with a count of the number of times the line occurred in the input.

-d

Suppress the writing of lines that are not repeated in the input.

-f fields

Ignore the first fields fields on each input line when doing comparisons, where fields is a positive decimal integer. A field is the maximal string matched by the basic regular expression:

[[:blank:]]*[^[:blank:]]*

If the fields option-argument specifies more fields than appear on an input line, a null string will be used for comparison.

-s chars

Ignore the first chars characters when doing comparisons, where chars is a positive decimal integer. If specified in conjunction with the -f option, the first chars characters after the first fields fields will be ignored. If the chars option-argument specifies more characters than remain on an input line, a null string will be used for comparison.

-u

Suppress the writing of lines that are repeated in the input.

-n

Equivalent to -f fields with fields set to n.

+m

Equivalent to -s chars with chars set to m.

 OPERANDS

The following operands are supported:

input_file

A pathname of the input file. If the input_file operand is not specified, or if the input_file is "-", the standard input will be used.

output_file

A pathname of the output file. If the output_file operand is not specified, the standard output will be used. The results are unspecified if the file named by output_file is the file named by input_file.

 STDIN

The standard input will be used only if no input_file operand is specified or if input_file is "-". See the INPUT FILES section.

 INPUT FILES

The input file must be a text file.

 ENVIRONMENT VARIABLES

The following environment variables affect the execution of uniq:

LANG

Provide a default value for the internationalisation variables that are unset or null. If LANG is unset or null, the corresponding value from the implementation-dependent default locale will be used. If any of the internationalisation variables contains an invalid setting, the utility will behave as if none of the variables had been defined.

LC_ALL

If set to a non-empty string value, override the values of all the other internationalisation variables.

LC_CTYPE

Determine the locale for the interpretation of sequences of bytes of text data as characters (for example, single- as opposed to multi-byte characters in arguments and input files) which characters constitute a blank character in the current locale.

LC_MESSAGES

Determine the locale that should be used to affect the format and contents of diagnostic messages written to standard error.

NLSPATH

Determine the location of message catalogues for the processing of LC_MESSAGES .

 ASYNCHRONOUS EVENTS

Default.

 STDOUT

The standard output will be used only if no output_file operand is specified. See the OUTPUT FILES section.

 STDERR

Used only for diagnostic messages.

 OUTPUT FILES

If the -c option is specified, the output file must be empty or each line must be of the form:

"%d %s", <number of duplicates>, <line>

otherwise, the output file must be empty or each line must be of the form:

"%s", <line>

 EXTENDED DESCRIPTION

None.

 EXIT STATUS

The following exit values are returned:

0

The utility executed successfully.

>0

An error occurred.

 CONSEQUENCES OF ERRORS

Default.

 APPLICATION USAGE

The sort utility can be used to cause repeated lines to be adjacent in the input file.

 EXAMPLES

The following input file data (but flushed left) was used for a test series on uniq:

#01 foo0 bar0 foo1 bar1
#02 bar0 foo1 bar1 foo1
#03 foo0 bar0 foo1 bar1
#04
#05 foo0 bar0 foo1 bar1
#06 foo0 bar0 foo1 bar1
#07 bar0 foo1 bar1 foo0

What follows is a series of test invocations of the uniq utility that use a mixture of uniq options against the input file data. These tests verify the meaning of adjacent . The uniq utility views the input data as a sequence of strings delimited by \n. Accordingly, for the fields th member of the sequence, uniq interprets unique or repreated adjacent lines strictly relative to the fields +1th member.

1. This first example tests the line counting option, comparing each line of the input file data starting from the second field:

   
   uniq -c -f 1 uniq_0I.t
     1 #01 foo0 bar0 foo1 bar1
     1 #02 bar0 foo1 bar1 foo0
     1 #03 foo0 bar0 foo1 bar1
     1 #04
     2 #05 foo0 bar0 foo1 bar1
     1 #07 bar0 foo1 bar1 foo0
   
   

   The number 2, prefixing the fifth line of output, signifies that the uniq utility detected a pair of repeated lines. Given the input data, this can only be true when uniq is run using the -f 1 option (which causes uniq to ignore the first field on each input line).

2. The second example tests the option to suppress unique lines, comparing each line of the input file data starting from the second field:

   
   uniq -d -f 1 uniq_0I.t
   #05 foo0 bar0 foo1 bar1
   
   

3. This test suppresses repeated lines, comparing each line of the input file data starting from the second field:

   
   uniq -u -f 1 uniq_0I.t
   #01 foo0 bar0 foo1 bar1
   #02 bar0 foo1 bar1 foo1
   #03 foo0 bar0 foo1 bar1
   #04
   #07 bar0 foo1 bar1 foo0
   
   

   
   

4. This suppresses unique lines, comparing each line of the input file data starting from the third character:

   
   uniq -d -s 2 uniq_0I.t
   
   

   In the last example, the uniq utility found no input matching the above criteria.

 FUTURE DIRECTIONS

None.

 SEE ALSO

comm, sort.