Communities

Writing
Writing
Codidact Meta
Codidact Meta
The Great Outdoors
The Great Outdoors
Photography & Video
Photography & Video
Scientific Speculation
Scientific Speculation
Cooking
Cooking
Electrical Engineering
Electrical Engineering
Judaism
Judaism
Languages & Linguistics
Languages & Linguistics
Software Development
Software Development
Mathematics
Mathematics
Christianity
Christianity
Code Golf
Code Golf
Music
Music
Physics
Physics
Linux Systems
Linux Systems
Power Users
Power Users
Tabletop RPGs
Tabletop RPGs
Notifications
Mark all as read
Q&A

How to delete contents of a specific field, if it matches a pattern and there is nothing else in the field

+8
−0

How do I delete contents of a specific field, if it matches a pattern, and there is nothing else in the field? I have a several GB tsv file, and I am interested in a specific field (72). If it contains hyphens, and only hyphens, then I want the hyphens deleted, leaving a blank field. I am using Ubuntu 20.04, with GNU awk v5. I've tried something like this:

awk 'BEGIN{FS=OFS="\t"}{gsub(/-/,"",$72)}1' file.tsv

But that also deletes the hyphens if there are other characters in the field too, which I do not want. E.g.

blah------

becomes

blah

but I want to leave it unchanged; but change

---------------

to nothing.

Why does this post require moderator attention?
You might want to add some details to your flag.
Why should this post be closed?

0 comment threads

3 answers

+6
−0

You're matching the regex pattern of /-/, so it just matches every individual hyphen, regardless of where. You want to match the entire entry if it's only hyphens, or /^-+$/.

^ – Beginning of line
-+ – One or more hyphens
$ – End of line

Putting that in gives

awk 'BEGIN{FS=OFS="\t"}{gsub(/^-+$/,"",$72)}1' file.tsv

which does what you want.

Why does this post require moderator attention?
You might want to add some details to your flag.

1 comment thread

While using `gsub()` won't hurt, aside from maybe a small performance hit and making readers wonder w... (1 comment)
+6
−0

The awk gsub function takes as its first argument a regular expression indicating the substring to be replaced, and replaces a matching substring with the value of the second argument, which is the replacement string.

Since your regular expression, /-/, is not anchored, it will match anywhere in the value that is being matched against. That's why it matches the hyphens that are contained as a part of the value. Here's an example run to illustrate this:

$ (printf '%s\n' 'blah---blah' '---blah---' 'blah------' '---------------') \
> | awk '{gsub(/-/,"",$1)}1'
blahblah
blah
blah

$

(note the empty line)

To get the behavior you are after, you need to anchor the regular expression at both the beginning and the end of the string, and provide a regular expression that matches against that whole string. Since you want to do a replacement only of strings that consist solely of hyphens, the regular expression should instead be /^-+$/ where the ^ anchors at the beginning, the -+ specifies an unbounded but non-zero number of - characters (that's the +), and the $ anchors at the end.

Example:

$ (printf '%s\n' 'blah---blah' '---blah---' 'blah------' '---------------') \
> | awk '{gsub(/^-+$/,"",$1)}1'
blah---blah
---blah---
blah------

$

As you can see, the contents of a field that consists of something other than only hyphens is now preserved, but the all-hyphens field contents is replaced by the empty string.

Why does this post require moderator attention?
You might want to add some details to your flag.

1 comment thread

While using gsub() won't hurt, aside from maybe a small performance hit and making readers wonder wha... (2 comments)
+3
−0

For the regular expression to only match a full field of hyphens, you have, as others already have explained, to put the ^ anchor at the begin of the regular expression and the $ anchor at the end and make it clear that you are interested in a sequence of hyphens by adding a + after the hyphen.

There is, however, one additional aspect that may be of interest for you: Instead of using gsub for both the matching and the replacement you could separate these two aspects in the following way:

awk 'BEGIN{FS=OFS="\t"}$72~/^-+$/{$72=""}1' file.tsv

The expression $72~/^-+$/ will match those lines where field $72 consists of only hyphens, and the action {$72=""} clears field $72. This approach gives you the freedom to choose some different action on those lines than only deleting the field content. This can be of interest if, for example, you do not only want to clear the field, but also get rid of the surrounding field separators or the like.

Why does this post require moderator attention?
You might want to add some details to your flag.

0 comment threads

Sign up to answer this question »

This community is part of the Codidact network. We have other communities too — take a look!

You can also join us in chat!

Want to advertise this community? Use our templates!

Like what we're doing? Support us! Donate