Apache Beam Regex

In a previous post, I showed how to use Beam’s Regex class to split up a string.

In this post, I’m going to going to show some other features of the Regex class.

The Regex class gives you a distributed way to work with strings. I tried to make the interface very familiar to Java developers. The Regex methods mimic the methods in the String class.

Here is a sample of the file this code is running against:

6 Diamond
3 Diamond
4 Club
4 Heart
3 Club
5 Spade

Let’s look at the code.

p.apply(TextIO.Read.from("playing_cards.tsv"))
  .apply(Regex.find("\\d*\\sHeart"))
  .apply(TextIO.Write.to("output/allheart"));

In this code snippet, we’ll be processing the file to only include the ones matching the regular expression. In this case, the regular expression is looking for all numbers followed by whitespace and the word Heart. The result are files with only the Heart lines.

p.apply(TextIO.Read.from("playing_cards.tsv"))
  .apply(Regex.find("(\\d*)\\sHeart", 1))
  .apply(TextIO.Write.to("output/allnumbers"));

Sometimes, you’ll want to get a specific group in the regular expression. This code snippet shows how to specify a group in the regular expression and choose it. By specifying the 1, you are choosing first group.

p.apply(TextIO.Read.from("playing_cards.tsv"))
  .apply(Regex.replaceFirst("Heart", "Hearts"))
  .apply(TextIO.Write.to("output/allhearts"));

The final example shows how to do a distributed search and replace. The dataset says the word Heart. We want to change the word to Hearts. The replaceFirst method takes in a regular expression and the string to replace it with. The result is the entire dataset looking like:

6 Diamond
3 Diamond
4 Club
4 Hearts
3 Club
5 Spade

If you interested in a course on Apache Beam, please sign up to be notified once it’s out.