Removing duplicates from a list
Method reference shortcuts
Here's how to remove duplicates using a custom comparison with Java streams (Claude Sonnet 4.5 on 18 JAN 2026)
// Remove duplicates based on a custom field/expression
list.stream()
.filter(distinctByKey(obj -> obj.getCustomField()))
.collect(Collectors.toList());
// Helper method (add to your class)
private static <T> Predicate<T> distinctByKey(Function<? super T, ?> keyExtractor) {
Set<Object> seen = ConcurrentHashMap.newKeySet();
return t -> seen.add(keyExtractor.apply(t));
}
list.stream()
.collect(Collectors.toMap(
obj -> obj.getCustomField(), // Key: what makes it unique
Function.identity(), // Value: the object itself
(existing, replacement) -> existing // Merge: keep first occurrence
))
.values()
.stream()
.collect(Collectors.toList());
Remove duplicates by field:
// Keep first Person with each unique name
List<Person> unique = people.stream()
.filter(distinctByKey(Person::getName))
.collect(Collectors.toList());
Remove duplicates by composite key:
// Unique by name + age combination
List<Person> unique = people.stream()
.filter(distinctByKey(p -> p.getName() + "|" + p.getAge()))
.collect(Collectors.toList());
Remove duplicates by multiple fields:
// Unique by custom comparison
List<Person> unique = people.stream()
.filter(distinctByKey(p -> Arrays.asList(p.getFirstName(), p.getLastName())))
.collect(Collectors.toList());
The key is that keyExtractor defines what makes two objects "the same" for deduplication purposes, independent of the object's equals() implementation.
private static <T> Predicate<T> distinctByKey(Function<? super T, ?> keyExtractor) {
Set<Object> seen = ConcurrentHashMap.newKeySet();
return t -> seen.add(keyExtractor.apply(t));
}
private static <T>
private: Only accessible within this classstatic: Not tied to any instance; purely functional utility<T>: Generic type parameter - T represents "whatever type your list contains"
Person, PropertyEvaluationOption, String, anythingPredicate<T> (Return Type)
Predicate<T> is a functional interface with one method: boolean test(T t)stream().filter() to decide keep (true) or discard (false)distinctByKey (Method Name)
Function<? super T, ?> keyExtractor (Parameter) - THE COMPLEX PART
Let me break down this parameter type piece by piece:
Function<INPUT, OUTPUT>
OUTPUT apply(INPUT input)? super T (First type parameter - INPUT)
? means "wildcard" - some unknown typesuper T means "T or any superclass of T"Dog, you can pass a function that takes Animal
? (Second type parameter - OUTPUT)
? without bounds means "any type at all"Putting it together:
Function<? super T, ?>
Means: "Give me a function that:
Set<Object> seen = ConcurrentHashMap.newKeySet();
Set<Object>: Set to track keys we've already seenObject? Because keyExtractor returns ? (unknown type)
ObjectConcurrentHashMap.newKeySet(): Thread-safe Set implementation
return t -> seen.add(keyExtractor.apply(t));
This is a lambda that becomes the Predicate<T>. Let's unpack it:
t
filter()keyExtractor.apply(t)
seen.add(...)
Set.add() returns a boolean:
true if element was added (wasn't already in set) → FIRST occurrence → KEEPfalse if element already exists (duplicate) → DUPLICATE → DISCARDSo the lambda returns:
true (keep) for the first occurrence of each keyfalse (discard) for subsequent occurrencesList<Person> people = List.of(
new Person("Alice", 30),
new Person("Bob", 25),
new Person("Alice", 35) // Duplicate name
);
// Remove duplicates by name
List<Person> unique = people.stream()
.filter(distinctByKey(Person::getName))
.collect(Collectors.toList());
Step by step:
First person: Alice, 30
keyExtractor.apply(person) → "Alice"seen.add("Alice") → true (added to empty set)seen = {"Alice"}Second person: Bob, 25
keyExtractor.apply(person) → "Bob"seen.add("Bob") → true (new key)seen = {"Alice", "Bob"}Third person: Alice, 35
keyExtractor.apply(person) → "Alice"seen.add("Alice") → false (already in set!)seen = {"Alice", "Bob"} (unchanged)Result: Only first Alice and Bob are kept
Without bounds (Function<T, Object>):
// Would fail if you have inheritance
class Animal { String getName() {...} }
class Dog extends Animal { }
List<Dog> dogs = ...;
dogs.stream().filter(distinctByKey(Animal::getName)) // ERROR!
// Function<Animal, String> is not Function<Dog, ?>
With contravariant input (Function<? super T, ?>):
// Works! Function can accept Animal which accepts Dog
List<Dog> dogs = ...;
dogs.stream().filter(distinctByKey(Animal::getName)) // OK!
With wildcard output (? instead of Object):
The real magic is the closure over seen:
seen is created once when you call distinctByKey()filter() calls the Predicate, it uses the same seen SetWithout this closure, you'd need external state management or complex workarounds.
@FunctionalInterface
public interface Predicate<T> {
boolean test(T t); // Single abstract method
// Default methods for composition (bonus features)
default Predicate<T> and(Predicate<? super T> other) {...}
default Predicate<T> or(Predicate<? super T> other) {...}
default Predicate<T> negate() {...}
static <T> Predicate<T> isEqual(Object targetRef) {...}
static <T> Predicate<T> not(Predicate<? super T> target) {...}
}
Predicate = "A question that returns true or false"
// Define a predicate
Predicate<Integer> isEven = n -> n % 2 == 0;
// Use it
boolean result = isEven.test(4); // true
boolean result2 = isEven.test(5); // false
1. Stream Filtering
List<Integer> numbers = List.of(1, 2, 3, 4, 5, 6);
// filter() expects Predicate<T>
List<Integer> evens = numbers.stream()
.filter(n -> n % 2 == 0) // Predicate<Integer>
.collect(Collectors.toList());
// Result: [2, 4, 6]
2. Collection Methods
List<String> names = new ArrayList<>(List.of("Alice", "Bob", "Charlie"));
// removeIf() expects Predicate<T>
names.removeIf(name -> name.startsWith("A")); // Predicate<String>
// names is now ["Bob", "Charlie"]
3. Standalone Validation
Predicate<String> isValidEmail = email ->
email != null && email.contains("@") && email.contains(".");
if (isValidEmail.test(userInput)) {
// Process email
}
// Lambda
Predicate<String> isEmpty = s -> s.isEmpty();
// Method reference (equivalent)
Predicate<String> isEmpty = String::isEmpty;
// Instance method
Predicate<Person> isAdult = p -> p.getAge() >= 18;
Predicate<Person> isAdult = Person::isAdult; // if isAdult() method exists
1. and() - Both conditions must be true
Predicate<Integer> isEven = n -> n % 2 == 0;
Predicate<Integer> isPositive = n -> n > 0;
Predicate<Integer> isPositiveEven = isEven.and(isPositive);
isPositiveEven.test(4); // true (even AND positive)
isPositiveEven.test(-4); // false (even but NOT positive)
isPositiveEven.test(3); // false (positive but NOT even)
2. or() - Either condition can be true
Predicate<Integer> isEven = n -> n % 2 == 0;
Predicate<Integer> isNegative = n -> n < 0;
Predicate<Integer> isEvenOrNegative = isEven.or(isNegative);
isEvenOrNegative.test(4); // true (even)
isEvenOrNegative.test(-3); // true (negative)
isEvenOrNegative.test(3); // false (neither)
3. negate() - Flip the result
Predicate<Integer> isEven = n -> n % 2 == 0;
Predicate<Integer> isOdd = isEven.negate();
isOdd.test(3); // true
isOdd.test(4); // false
4. not() - Static method (Java 11+)
Predicate<String> isEmpty = String::isEmpty;
// Old way
Predicate<String> isNotEmpty = isEmpty.negate();
// Java 11+ static method
Predicate<String> isNotEmpty = Predicate.not(String::isEmpty);
Predicate<Person> isAdult = p -> p.getAge() >= 18;
Predicate<Person> hasEmail = p -> p.getEmail() != null;
Predicate<Person> hasVerifiedEmail = p -> p.isEmailVerified();
// Complex condition
Predicate<Person> canReceiveNewsletter =
isAdult.and(hasEmail).and(hasVerifiedEmail);
List<Person> eligiblePeople = people.stream()
.filter(canReceiveNewsletter)
.collect(Collectors.toList());
// Generic filter method that accepts any predicate
public List<Person> filterPeople(List<Person> people, Predicate<Person> condition) {
return people.stream()
.filter(condition)
.collect(Collectors.toList());
}
// Usage - pass different predicates
List<Person> adults = filterPeople(people, p -> p.getAge() >= 18);
List<Person> seniors = filterPeople(people, p -> p.getAge() >= 65);
List<Person> namedJohn = filterPeople(people, p -> "John".equals(p.getName()));
<T> ExplainedPredicate<String> // Tests strings
Predicate<Integer> // Tests integers
Predicate<Person> // Tests Person objects
// Generic method that works with any predicate type
public <T> boolean anyMatch(List<T> items, Predicate<T> condition) {
for (T item : items) {
if (condition.test(item)) {
return true;
}
}
return false;
}
Before Java 8 (ugly):
List<Integer> filterEvens(List<Integer> numbers) {
List<Integer> result = new ArrayList<>();
for (Integer n : numbers) {
if (n % 2 == 0) { // Logic hardcoded
result.add(n);
}
}
return result;
}
After Java 8 (flexible):
List<Integer> filter(List<Integer> numbers, Predicate<Integer> condition) {
return numbers.stream()
.filter(condition) // Logic passed as parameter
.collect(Collectors.toList());
}
// Now you can:
filter(numbers, n -> n % 2 == 0); // Evens
filter(numbers, n -> n > 10); // Greater than 10
filter(numbers, n -> n > 0 && n < 100); // Range
// distinctByKey returns a Predicate
private static <T> Predicate<T> distinctByKey(Function<? super T, ?> keyExtractor) {
Set<Object> seen = ConcurrentHashMap.newKeySet();
return t -> seen.add(keyExtractor.apply(t)); // Returns Predicate<T>
}
// The returned predicate gets used by filter()
list.stream()
.filter(distinctByKey(obj -> obj.getKey())) // Predicate<T> plugs in here
.collect(Collectors.toList());
and(), or(), negate()<T> ensures type consistencyThink of Predicate as: