Communities

Writing
Writing
Codidact Meta
Codidact Meta
The Great Outdoors
The Great Outdoors
Photography & Video
Photography & Video
Scientific Speculation
Scientific Speculation
Cooking
Cooking
Electrical Engineering
Electrical Engineering
Judaism
Judaism
Languages & Linguistics
Languages & Linguistics
Software Development
Software Development
Mathematics
Mathematics
Christianity
Christianity
Code Golf
Code Golf
Music
Music
Physics
Physics
Linux Systems
Linux Systems
Power Users
Power Users
Tabletop RPGs
Tabletop RPGs
Community Proposals
Community Proposals
tag:snake search within a tag
answers:0 unanswered questions
user:xxxx search by author id
score:0.5 posts with 0.5+ score
"snake oil" exact phrase
votes:4 posts with 4+ votes
created:<1w created < 1 week ago
post_type:xxxx type of post
Search help
Notifications
Mark all as read See all your notifications »
Q&A

Welcome to Software Development on Codidact!

Will you help us build our independent community of developers helping developers? We're small and trying to grow. We welcome questions about all aspects of software development, from design to code to QA and more. Got questions? Got answers? Got code you'd like someone to review? Please join us.

What are field separators in operating-programming languages (such as Bash)?

+7
−0

The concept of field separator has some private cases in different operating system shells and their utilities (IFS on Bourne shell and derivates, RS on AWK and perhaps more) but I am having trouble understanding what is the purpose of the concept in general or which problem it was first aimed to solve.

I have personally understood that a "field separator" is any indentation character and line break character such as the following but I am not sure that's correct:

  • Indentation characters:
    • Whitespace character
    • Tabulation character
    • Other, less standard, indentation characters
  • Line break characters:
    • (non-carriage-return) Line Feed (LF)
    • Carriage Return Line Feed (CRLF)
    • Other, less standard, line break characters
History
Why does this post require moderator attention?
You might want to add some details to your flag.
Why should this post be closed?

1 comment thread

`RS` in awk is the Record Separator, not the `Field Separator`. In awk `FS` is the input field separa... (1 comment)

1 answer

+9
−0

In Bash, IFS is an internal variable and it stands for "Internal Field Separator" <- according to this link, it "determines how Bash recognizes fields, or word boundaries, when it interprets character strings".

Its default value is a "whitespace" (space, tab, and newline), but you can change it to whatever you need.

To give an example, using the default value, the following commands:

text="a:b c-d e/f"
for word in $text; do echo "Word: $word"; done

would ouput:

Word: a:b
Word: c-d
Word: e/f

Note that the spaces were used to split the string into fields/"words", so each iteration of the for loop gets one part of the split results.

But if we change IFS:

IFS=':-/'
text="a:b c-d e/f"
for word in $text; do echo "Word: $word"; done

Now the output is:

Word: a
Word: b c
Word: d e
Word: f

By setting IFS=':-/', I'm saying that :, - and / should be the characters used to determine a field/"word" boundaries, thus the result is quite different (note that the spaces were "ignored", so b c and d e are considered two fields/"words").

If we change to IFS=':', only the : character will be considered, and the result would be only 2 fields: a and b c-d e/f.


IFS is used by other commands, such as read:

IFS=':'
echo "abc:def" | (read x y; echo "x=$x y=$y")
# output is "x=abc y=def"

And it also affects the output of the special variable $* (which contains all the command line arguments of a script), when printed inside double quotes. Suppose I have this simple script:

#!/bin/bash
echo "Args: $*"

If I run this script: script.sh a b c, the output will be Args: a b c.
But if I change it to:

#!/bin/bash
IFS=':'
echo "Args: $*"

The first character of IFS will be used in the output, and displayed between the fields, so the output will be: Args: a:b:c.


One detail regarding whitespace versus non-whitespace characters: if IFS contains whitespace, a sequence of one or more whitespaces is considered to be a single separator, but a sequence of one or more non-whitespaces isn't. Example:

# text with 4 spaces before "c", and a trailing space in the end
text='a::b    c '
# IFS is just a space
IFS=' '
for word in $text; do echo "Word: [$word]"; done

In this case, IFS is just a space, but a sequence of one or more spaces is considered to be a single separator, so the output is:

Word: [a::b]
Word: [c]

If we set IFS=':', now the field separator is a non-whitespace, so a sequence of one or more is not considered a single separator, and the output would change to:

Word: [a]
Word: []
Word: [b    c ]

The second field is an empty string, because each : is another separator, and :: is considered "an empty string between two :".


But anyway, the field separator can be whatever you need, not limited to the ones defined in the question.

History
Why does this post require moderator attention?
You might want to add some details to your flag.

1 comment thread

and to learn about Field Separators in awk see https://www.gnu.org/software/gawk/manual/gawk.html#Fie... (1 comment)

Sign up to answer this question »