How to Remove Duplicate Items in Python List?

Python List – Remove Duplicate Items

We know that Python List can contain values of any data type. This also means that a list can contain duplicate items. And during some scenarios, we may not want our list to have duplicate items. So, in this tutorial, we will learn how to remove duplicate items from a Python List.

1. Remove duplicate items from List using Membership Operator

The syntax of membership operator is

x in collection
x not in collection

where x is element, in is Python keyword and collection is any Python collection.

x in collection returns True if x is present in collection, False otherwise.

x not in collection returns True if x is not present in collection, False otherwise.

Following is a step by step process to remove duplicate items from a Python List using membership operator.

1. Take a list of items. We should create a new list with duplicate elements removed from this list.
2. Create a new empty list.
3. For each item in the given original list, check if item is present in the new list.
   1. If the item is not present in the new list, add the item to the new list.

In this example, we will take a list of numbers, with some of them being duplicates. Then, we will apply the above step by step process to discard the duplicates and include only unique items.

list1 = [2, 3, 7, 3, 6, 2, 8, 8]
list2 = []

for item in list1:
    if item not in list2:
        list2.append(item)

print(list2)

Output

[2, 3, 7, 6, 8]

2. Remove duplicate items from List in-place

In the above example, we copied unique elements to a new list. But, what if we would like to remove duplicate items from the original list.

Follow these steps to remove duplicate items from list inplace.

1. Take a list of elements. We are going to remove duplicates present in this list.
2. Initialize index with 1. Since, the first element is going to be unique, we are checking from second position instead of first.
3. Start a while loop with condition that index should be less than the length of given list.
   1. If element at index is present in the sub list, part of the list that is present before this index position, remove the element at index. Otherwise, increment the index and continue with while loop.
4. After the loop is done, we are left with unique elements in the list.

To the find the sub-list present before the index during each while loop iteration, we use Python Slicing technique.

list1 = [2, 3, 7, 3, 6, 2, 8, 8]

index = 1
while index < len(list1):
    if list1[index] in list1[ : index]:
        list1.pop(index)
    else:
        index += 1

print(list1)

Output

[2, 3, 7, 6, 8]

Summary

In this tutorial of Python Examples, we learned how to remove duplicate items from a Python List.