NUWC Public Affairs, NAVSEA
16 May 2018
NEWPORT, R.I. — Twenty years ago Naval Undersea Warfare Center (NUWC) Division Newport engineer Gary Huntress came across a technical challenge he felt he just had to try. Entrants were tasked with identifying the call sign of a very weak EME (earth-moon-earth) Morse code signal in a zipped 1 minute .wav file. Using MatLab as the primary analysis tool, Huntress became the first person to solve the problem in the contest’s two-year history – and as an added bonus, nabbed the $100 prize in the process.
Reflecting on his experience with the challenge and interest in machine learning, Huntress recently issued a contest of his own to the NUWC Newport Machine Learning Community of Interest.
“A year and a half ago thereabouts I got started with the whole big topic of machine learning,” Huntress said. “I got excited about it and NUWC’s chief technology officer suggested, then endorsed, having a machine learning community of interest and we’ve been having it ever since.
“The intent is to get people excited about the topic and one of the ways that I thought that we could do it is by having a contest.”
The challenge – for those with the expertise – was simple enough: Utilizing your own time and resources, write an image classifier that could distinguish cruise ships from merchant vessels.
Developing a truly effective image classifier was where the difficulty resided, yet in the end Huntress received two submissions that he felt were equally successful and elected to award both Caleb Martin and Robert Bretz $100 each.
“I really loosely defined the ‘expectations’ to leave it open for whatever they produced, and I was really happy with the two submissions. They both worked great,” Huntress said. “The only outcome I was hoping for is that I would learn something and they would learn something and find it interesting – whoever participated, because I had no idea who would.”
As explained by Martin, Bretz and Huntress, machine learning is a data-centric way of computing what is different than the way it has been done for the previous 50 years. Machine learning uses very complicated optimization schemes and curve-fitting routines to go into the raw data to solve the problem.
For image classification, Huntress used the example of classifying what is a cat.
“The traditional way would be someone with domain knowledge would say, ‘cats have ears, I’ll detect ears; cats have eyes, I’ll figure out how to detect eyes.’ Pre-defined fragile features that you lovingly handcraft,” Huntress said. “The new way is to let the machine learning algorithm figure out what features are important.”
According to Martin, about 10 years ago the problem presented by Huntress would not have been computationally solvable.
“It would be just find a bunch of interns, who would go through the images one by one because there was no way of automating that process,” Martin said. “Now, that’s a fairly trivial problem. It’s because we have a better understanding of how do you differentiate that from raw data.”
That said, Huntress’ challenge certainly had its fair share of obstacles. In order to solve a machine learning problem, the basic steps are to collect all the data, clean it up, try your initial model and then tweak the model until it is optimally efficient.
Once Bretz and Martin were satisfied with the search parameters of their neural network, each came to a similar point of variability in the problem.
“What is a merchant ship?” Bretz said. “That was kind of a data science problem.”
For both Bretz and Martin, determining pictures of cruise ships was not as much of an issue, as what they look like is fairly defined.
“It’s a much easier problem to compare two very specific things and then your model can fairly quickly pick up features that will be present in one and not the other, and then use that to divide the two classes,” Martin said. “If your classes are just something vs. a vaguer collection of not that something it’s much harder.
“I tried a few things and I eventually settled on my one class being pictures of cruise ships and my other class being pictures of container ships, oil tankers and freighters as reasonable representation of merchant ships.”
Bretz had a similar classification of merchant ships, but also included roll-on-roll-off ships in his data set. Where the true challenge came into play was in refining their
models – eliminating or adding ships for a number of reasons, including color and shape.
“The first time I ran it, it was better than random. You know you’re on the right track when it’s better than flipping a coin. I ran it first, and it was 70 or 75 percent and was like, alright, now let’s figure out why it’s not 95 percent,” Bretz said. “I started tweaking it and I really had a hard time getting over the 93 to 94 percent rate. I just don’t think with what I was working with that I could have done better than that.
“Basically, I hit 90 percent in the first several minutes of tweaking and then it took the rest of my time that I was working on it. From 75 to 90 percent took probably less time than from 90 percent to 92 percent.”
Both Martin and Bretz said the $100 prize was nice and would be put toward other side-projects on which they are working, but it was not their motivation for the challenge.
“Machine learning has been sort of a casual hobby/side interest of mine for probably two or three years now and I’ve been slowly trying to shoehorn it into my work,” Martin said. “It’s tough because I’m an S&T person here, which means all of my time needs to be devoted to projects that I am funded to do. I can’t just spend half of my day playing around. That’s what I do at home.
“If this leads to increased visibility and being one of the POCs for this kind of stuff that would be great. That’s where I’m looking for this to go.”
While Bretz does not have quite the same experience in machine learning as Huntress and Martin, all three are hoping to raise the profile of the technique and see how it could be applicable to their work at NUWC Newport.
All three also noted that application of machine learning could be years – if not decades – away, but could have relevant uses with unmanned undersea vehicles (UUVs), sonar or tactical simulations.
“We have to resist throwing machine learning at our absolute hardest problem right now and then saying, ‘Oh, it doesn’t work,’” Huntress said. “We need to consider the path over the next five years. What’s beyond the next baby step?”
No comments:
Post a Comment