Improve handling of disfluencies

speechmatics · Nov 29, 2023 · 9083aec · 9083aec
1 parent 55ea02b
commit 9083aec
Show file tree

Hide file tree

Showing 10 changed files with 1,974 additions and 1,817 deletions.
diff --git a/metrics/diarization/Makefile b/metrics/diarization/Makefile
@@ -9,5 +9,5 @@ wheel:
 	(pip install wheel && python3 setup.py bdist_wheel)
 
 .PHONY: install
-install: 
+install:
 	pip install ./dist/*
diff --git a/metrics/diarization/requirements.txt b/metrics/diarization/requirements.txt
@@ -1,4 +1,4 @@
 pyannote.core
 pyannote.database
 docopt
-tabulate
+tabulate
diff --git a/metrics/diarization/sm_diarization_metrics/metrics/errors/segmentation.py b/metrics/diarization/sm_diarization_metrics/metrics/errors/segmentation.py
@@ -35,7 +35,6 @@ def __init__(self):
         super(SegmentationErrorAnalysis, self).__init__()
 
     def __call__(self, reference, hypothesis):
-
         if isinstance(reference, Annotation):
             reference = reference.get_timeline()
 
@@ -47,7 +46,6 @@ def __call__(self, reference, hypothesis):
         prev_r = reference[0]
         intersection = []
         for r, h in reference.co_iter(hypothesis):
-
             if r != prev_r:
                 intersection = sorted(intersection)
                 for _, segment in intersection[:-1]:
@@ -67,7 +65,6 @@ def __call__(self, reference, hypothesis):
         prev_h = hypothesis[0]
         intersection = []
         for h, r in hypothesis.co_iter(reference):
-
             if h != prev_h:
                 intersection = sorted(intersection)
                 for _, segment in intersection[:-1]:

diff --git a/metrics/examples/hypothesis-wer.txt b/metrics/examples/hypothesis-wer.txt
@@ -1 +1 @@
-Gary and Paul O'Donovan from Skibbereen. Olympic silver medallist. How does it feel? It's fantastic. We haven't haven't had too much of a chance yet to come to terms with it. We wanted to win the gold medal and come over with a silver medal. Like we're just so happy. We can't complain with that. Paul Go For me to be as fast as you can and pull like a dog. Fourth at 500. Fourth at 1000. You pulled fairly hard in that last closing stages of that race. We did. Josh We're a little bit disappointed we didn't come away with the gold medal. I think we put it up to the French as best we could and we dived close there at the end. I'd say we were going all over the lane, but we're dreading going home now because Mick Conlan said he boxed the head office if we didn't get to go. So there was only a split second in it. But lads, you've secured Ireland's first ever Olympic rowing medal. Yeah, we're delighted, you know. We set out when we qualified last year, finishing 11th at the World Championships. Our goal was to win the Olympics and we knew we'd have to beat one of the best crews in the world in France. And then they went and made their crew a little bit stronger, putting in a new guy to come away so close to winning the silver medal. You know, we we have to go home happy. And I don't know if you're aware. I don't know if you were aware. Yeah, you deserve it. I saw him come up here. I've been racing the bombing since 2013 and my first world championships in the doubles. So to be on a podium with him, I don't know if you. If you're aware the huge interest this has garnered at home that the fact the whole of Ireland not to mind the whole of Skibbereen is watching. What does it mean after all the years of work, starting off at junior level, working your way up to be here today? That's good. Yeah, we're just happy to do it for the sport, really. And like I was saying, there's a big buzz around rowing now, so just a fantastic sport. So we're just hoping that a lot more people will start to realise that and maybe take it up and give it a go. And you never know. Like there's plenty people out there with two arms and two legs like the two of ourselves. So there could be more Olympic champions to come. Please God. And a wonderful day for the people back in the club, the likes of Dominic Casey, who's who's been in, I believe, involved with Skibbereen for about 30 years. Yeah, probably more since since day one, pretty much. I can't wait to run into him. Yeah. Can't wait to put the silver medal around possible without him really going to put the silver medal around his neck. Do you know he is a young family at home and he spent more than half of this year with us. And we are in training camps and stuff like that. So it's he's given up a lot of his time, too. So the time it's a huge credit to him and everyone else in the rowing club as well. And Joe Martin Espersen as well for for allowing us to do it with our club coach as well. Yeah. Very final question for you. What do you want to say to the people back home? We're hugely honoured and so proud to be representing Ireland and everybody at home in Ireland, and we've been getting such huge support and we've gotten such a big following following some of our interviews. I mean, you know, you've just been asking us questions and we'd have been answering them. People have been enjoying it. So, you know, we're so proud to people. The people are kind of following us and supporting us and that we take great, great pride in that. Well, Gary and Paul, thanks so much for joining us. And well, well done, lads. You've done your country proud. Cheers. Cheers. Chuck it all.
+Gary and Paul O'Donovan from Skibbereen. Olympic silver medallist. How does it feel? It's fantastic. We haven't haven't had too much of a chance yet to come to terms with it. We wanted to win the gold medal and come over with a silver medal. Like we're just so happy. We can't complain with that. Paul Go For me to be as fast as you can and pull like a dog. Fourth at 500. Fourth at 1000. You pulled fairly hard in that last closing stages of that race. We did. Josh We're a little bit disappointed we didn't come away with the gold medal. I think we put it up to the French as best we could and we dived close there at the end. I'd say we were going all over the lane, but we're dreading going home now because Mick Conlan said he boxed the head office if we didn't get to go. So there was only a split second in it. But lads, you've secured Ireland's first ever Olympic rowing medal. Yeah, we're delighted, you know. We set out when we qualified last year, finishing 11th at the World Championships. Our goal was to win the Olympics and we knew we'd have to beat one of the best crews in the world in France. And then they went and made their crew a little bit stronger, putting in a new guy to come away so close to winning the silver medal. You know, we we have to go home happy. And I don't know if you're aware. I don't know if you were aware. Yeah, you deserve it. I saw him come up here. I've been racing the bombing since 2013 and my first world championships in the doubles. So to be on a podium with him, I don't know if you. If you're aware the huge interest this has garnered at home that the fact the whole of Ireland not to mind the whole of Skibbereen is watching. What does it mean after all the years of work, starting off at junior level, working your way up to be here today? That's good. Yeah, we're just happy to do it for the sport, really. And like I was saying, there's a big buzz around rowing now, so just a fantastic sport. So we're just hoping that a lot more people will start to realise that and maybe take it up and give it a go. And you never know. Like there's plenty people out there with two arms and two legs like the two of ourselves. So there could be more Olympic champions to come. Please God. And a wonderful day for the people back in the club, the likes of Dominic Casey, who's who's been in, I believe, involved with Skibbereen for about 30 years. Yeah, probably more since since day one, pretty much. I can't wait to run into him. Yeah. Can't wait to put the silver medal around possible without him really going to put the silver medal around his neck. Do you know he is a young family at home and he spent more than half of this year with us. And we are in training camps and stuff like that. So it's he's given up a lot of his time, too. So the time it's a huge credit to him and everyone else in the rowing club as well. And Joe Martin Espersen as well for for allowing us to do it with our club coach as well. Yeah. Very final question for you. What do you want to say to the people back home? We're hugely honoured and so proud to be representing Ireland and everybody at home in Ireland, and we've been getting such huge support and we've gotten such a big following following some of our interviews. I mean, you know, you've just been asking us questions and we'd have been answering them. People have been enjoying it. So, you know, we're so proud to people. The people are kind of following us and supporting us and that we take great, great pride in that. Well, Gary and Paul, thanks so much for joining us. And well, well done, lads. You've done your country proud. Cheers. Cheers. Chuck it all.
diff --git a/metrics/examples/reference-wer.txt b/metrics/examples/reference-wer.txt
@@ -1 +1 @@
-Gary and Paul O'Donovan from Skibbereen. Olympic silver medallists. How does it feel? It's fantastic. We haven't haven't had too much of a chance yet to come to terms with it. We wanted to win the gold medal and come over with a silver medal, like we're just so happy. We can't complain with that. Paul go from A to B as fast as you can and pull like a dog. Fourth at 500. Fourth at 1000. You pulled fairly hard in that last closing stages of that race. We did. Josh We're a little bit disappointed we didn't come away with the gold medal. I think we put it up to the French as best we could and we dived close there at the end. I'd say we were going all over the lane, but we're dreading going home now because Mick Conlan said he box the head office if we didn't get to go so. There was only a split second in it. But lads, you've secured Ireland's first ever Olympic rowing medal. Yeah, we're delighted, you know. We set out when we qualified last year, finishing 11th at the World Championships. Our goal was to win the Olympics and we knew we'd have to beat one of the best crews in the world in France. And then they went and made their crew a little bit stronger, putting in a new guy to come away so close to winning the silver medal. You know, we we have to go home happy. And I don't know if you're aware. I don't know if you were aware. Yeah, thank you very much. You deserve it. Awesome, well done Pierre. I've been racing the Pierre Houin since 2013 at my first world championships in the doubles. So to be on a podium with him. I don't know if you. If you're aware the huge interest this has garnered at home that the fact the whole of Ireland not to mind the whole of Skibbereen is watching. What does it mean after all the years of work, starting off at junior level, working your way up to be here today? It's good. Yeah, we're just happy to do it for the sport, really. And like I was saying, there's a big buzz around rowing now, so just a fantastic sport. So we're just hoping that a lot more people will start to realise that and maybe take it up and give it a go. And you never know. Like there's plenty people out there with two arms and two legs like the two of ourselves. So there could be more Olympic champions to come. Please God. And a wonderful day for the people back in the club, the likes of Dominic Casey, who's who's been in, I believe, involved with Skibbereen for about 30 years. Yeah, probably more since since day one, pretty much. I can't wait to run into him. Yeah. Can't wait to put the silver medal around. It wouldn't be possible without him really, no. Going to put the silver medal around his neck. Do you know he has a young family at home and he spent more than half of this year with us. And we are in training camps and stuff like that. So it's he's given up a lot of his time, too. So the time it's a huge credit to him and everyone else in the rowing club as well. And do you know Morten Espersen as well for for allowing us to do it with our club coach as well. Yeah. Very final question for you. What do you want to say to the people back home? We're hugely honoured and so proud to be representing Ireland and everybody at home in Ireland, and we've been getting such huge support and we've gotten such a big following following some of our interviews. I mean, you know, you've just been asking us questions and we'd have been answering them. People have been enjoying it. So, you know, we're so proud to people. The people are kind of following us and supporting us and that we take great, great pride in that. Well, Gary and Paul, thanks so much for joining us. And well, well done, lads. You've done your country proud. Cheers. Cheers.
+Gary and Paul O'Donovan from Skibbereen. Olympic silver medallists. How does it feel? It's fantastic. We haven't haven't had too much of a chance yet to come to terms with it. We wanted to win the gold medal and come over with a silver medal, like we're just so happy. We can't complain with that. Paul go from A to B as fast as you can and pull like a dog. Fourth at 500. Fourth at 1000. You pulled fairly hard in that last closing stages of that race. We did. Josh We're a little bit disappointed we didn't come away with the gold medal. I think we put it up to the French as best we could and we dived close there at the end. I'd say we were going all over the lane, but we're dreading going home now because Mick Conlan said he box the head office if we didn't get to go so. There was only a split second in it. But lads, you've secured Ireland's first ever Olympic rowing medal. Yeah, we're delighted, you know. We set out when we qualified last year, finishing 11th at the World Championships. Our goal was to win the Olympics and we knew we'd have to beat one of the best crews in the world in France. And then they went and made their crew a little bit stronger, putting in a new guy to come away so close to winning the silver medal. You know, we we have to go home happy. And I don't know if you're aware. I don't know if you were aware. Yeah, thank you very much. You deserve it. Awesome, well done Pierre. I've been racing the Pierre Houin since 2013 at my first world championships in the doubles. So to be on a podium with him. I don't know if you. If you're aware the huge interest this has garnered at home that the fact the whole of Ireland not to mind the whole of Skibbereen is watching. What does it mean after all the years of work, starting off at junior level, working your way up to be here today? It's good. Yeah, we're just happy to do it for the sport, really. And like I was saying, there's a big buzz around rowing now, so just a fantastic sport. So we're just hoping that a lot more people will start to realise that and maybe take it up and give it a go. And you never know. Like there's plenty people out there with two arms and two legs like the two of ourselves. So there could be more Olympic champions to come. Please God. And a wonderful day for the people back in the club, the likes of Dominic Casey, who's who's been in, I believe, involved with Skibbereen for about 30 years. Yeah, probably more since since day one, pretty much. I can't wait to run into him. Yeah. Can't wait to put the silver medal around. It wouldn't be possible without him really, no. Going to put the silver medal around his neck. Do you know he has a young family at home and he spent more than half of this year with us. And we are in training camps and stuff like that. So it's he's given up a lot of his time, too. So the time it's a huge credit to him and everyone else in the rowing club as well. And do you know Morten Espersen as well for for allowing us to do it with our club coach as well. Yeah. Very final question for you. What do you want to say to the people back home? We're hugely honoured and so proud to be representing Ireland and everybody at home in Ireland, and we've been getting such huge support and we've gotten such a big following following some of our interviews. I mean, you know, you've just been asking us questions and we'd have been answering them. People have been enjoying it. So, you know, we're so proud to people. The people are kind of following us and supporting us and that we take great, great pride in that. Well, Gary and Paul, thanks so much for joining us. And well, well done, lads. You've done your country proud. Cheers. Cheers.
diff --git a/metrics/wer/__main__.py b/metrics/wer/__main__.py
@@ -237,6 +237,11 @@ def get_wer_args(parser: argparse.ArgumentParser) -> argparse.ArgumentParser:
         """,
         action="store_true",
     )
+    parser.add_argument(
+        "--keep-disfluencies",
+        help="Retain disfluencies such as 'uhhh' and 'umm'. Disfluencies will be removed otherwise.",
+        action="store_true",
+    )
     parser.add_argument("--csv", help="Write the results to a CSV", type=Path)
     parser.add_argument("ref_path", help="Path to the reference transcript", type=str)
     parser.add_argument("hyp_path", help="Path to the hypothesis transcript", type=str)
@@ -252,7 +257,12 @@ def main(args: Optional[argparse.Namespace] = None):
         parser = get_wer_args(argparse.ArgumentParser())
         args = parser.parse_args()
 
-    normaliser = BasicTextNormalizer() if args.non_en else EnglishTextNormalizer()
+    if args.non_en:
+        normaliser = BasicTextNormalizer()
+    elif args.keep_disfluencies:
+        normaliser = EnglishTextNormalizer(remove_disfluencies=False)
+    else:
+        normaliser = EnglishTextNormalizer(remove_disfluencies=True)
 
     ref_files, hyp_files = check_paths(args.ref_path, args.hyp_path)
     columns = [
Original file line number	Diff line number	Diff line change
		@@ -1 +1 @@
		Gary and Paul O'Donovan from Skibbereen. Olympic silver medallist. How does it feel? It's fantastic. We haven't haven't had too much of a chance yet to come to terms with it. We wanted to win the gold medal and come over with a silver medal. Like we're just so happy. We can't complain with that. Paul Go For me to be as fast as you can and pull like a dog. Fourth at 500. Fourth at 1000. You pulled fairly hard in that last closing stages of that race. We did. Josh We're a little bit disappointed we didn't come away with the gold medal. I think we put it up to the French as best we could and we dived close there at the end. I'd say we were going all over the lane, but we're dreading going home now because Mick Conlan said he boxed the head office if we didn't get to go. So there was only a split second in it. But lads, you've secured Ireland's first ever Olympic rowing medal. Yeah, we're delighted, you know. We set out when we qualified last year, finishing 11th at the World Championships. Our goal was to win the Olympics and we knew we'd have to beat one of the best crews in the world in France. And then they went and made their crew a little bit stronger, putting in a new guy to come away so close to winning the silver medal. You know, we we have to go home happy. And I don't know if you're aware. I don't know if you were aware. Yeah, you deserve it. I saw him come up here. I've been racing the bombing since 2013 and my first world championships in the doubles. So to be on a podium with him, I don't know if you. If you're aware the huge interest this has garnered at home that the fact the whole of Ireland not to mind the whole of Skibbereen is watching. What does it mean after all the years of work, starting off at junior level, working your way up to be here today? That's good. Yeah, we're just happy to do it for the sport, really. And like I was saying, there's a big buzz around rowing now, so just a fantastic sport. So we're just hoping that a lot more people will start to realise that and maybe take it up and give it a go. And you never know. Like there's plenty people out there with two arms and two legs like the two of ourselves. So there could be more Olympic champions to come. Please God. And a wonderful day for the people back in the club, the likes of Dominic Casey, who's who's been in, I believe, involved with Skibbereen for about 30 years. Yeah, probably more since since day one, pretty much. I can't wait to run into him. Yeah. Can't wait to put the silver medal around possible without him really going to put the silver medal around his neck. Do you know he is a young family at home and he spent more than half of this year with us. And we are in training camps and stuff like that. So it's he's given up a lot of his time, too. So the time it's a huge credit to him and everyone else in the rowing club as well. And Joe Martin Espersen as well for for allowing us to do it with our club coach as well. Yeah. Very final question for you. What do you want to say to the people back home? We're hugely honoured and so proud to be representing Ireland and everybody at home in Ireland, and we've been getting such huge support and we've gotten such a big following following some of our interviews. I mean, you know, you've just been asking us questions and we'd have been answering them. People have been enjoying it. So, you know, we're so proud to people. The people are kind of following us and supporting us and that we take great, great pride in that. Well, Gary and Paul, thanks so much for joining us. And well, well done, lads. You've done your country proud. Cheers. Cheers. Chuck it all.
		Gary and Paul O'Donovan from Skibbereen. Olympic silver medallist. How does it feel? It's fantastic. We haven't haven't had too much of a chance yet to come to terms with it. We wanted to win the gold medal and come over with a silver medal. Like we're just so happy. We can't complain with that. Paul Go For me to be as fast as you can and pull like a dog. Fourth at 500. Fourth at 1000. You pulled fairly hard in that last closing stages of that race. We did. Josh We're a little bit disappointed we didn't come away with the gold medal. I think we put it up to the French as best we could and we dived close there at the end. I'd say we were going all over the lane, but we're dreading going home now because Mick Conlan said he boxed the head office if we didn't get to go. So there was only a split second in it. But lads, you've secured Ireland's first ever Olympic rowing medal. Yeah, we're delighted, you know. We set out when we qualified last year, finishing 11th at the World Championships. Our goal was to win the Olympics and we knew we'd have to beat one of the best crews in the world in France. And then they went and made their crew a little bit stronger, putting in a new guy to come away so close to winning the silver medal. You know, we we have to go home happy. And I don't know if you're aware. I don't know if you were aware. Yeah, you deserve it. I saw him come up here. I've been racing the bombing since 2013 and my first world championships in the doubles. So to be on a podium with him, I don't know if you. If you're aware the huge interest this has garnered at home that the fact the whole of Ireland not to mind the whole of Skibbereen is watching. What does it mean after all the years of work, starting off at junior level, working your way up to be here today? That's good. Yeah, we're just happy to do it for the sport, really. And like I was saying, there's a big buzz around rowing now, so just a fantastic sport. So we're just hoping that a lot more people will start to realise that and maybe take it up and give it a go. And you never know. Like there's plenty people out there with two arms and two legs like the two of ourselves. So there could be more Olympic champions to come. Please God. And a wonderful day for the people back in the club, the likes of Dominic Casey, who's who's been in, I believe, involved with Skibbereen for about 30 years. Yeah, probably more since since day one, pretty much. I can't wait to run into him. Yeah. Can't wait to put the silver medal around possible without him really going to put the silver medal around his neck. Do you know he is a young family at home and he spent more than half of this year with us. And we are in training camps and stuff like that. So it's he's given up a lot of his time, too. So the time it's a huge credit to him and everyone else in the rowing club as well. And Joe Martin Espersen as well for for allowing us to do it with our club coach as well. Yeah. Very final question for you. What do you want to say to the people back home? We're hugely honoured and so proud to be representing Ireland and everybody at home in Ireland, and we've been getting such huge support and we've gotten such a big following following some of our interviews. I mean, you know, you've just been asking us questions and we'd have been answering them. People have been enjoying it. So, you know, we're so proud to people. The people are kind of following us and supporting us and that we take great, great pride in that. Well, Gary and Paul, thanks so much for joining us. And well, well done, lads. You've done your country proud. Cheers. Cheers. Chuck it all.