Answers by Sentry

Remove duplicates in a Python list

David Y.

The Problem

How can I remove duplicate elements from a list in Python?

The Solution

The simplest way to do this is to convert our list into a set, a Python data structure that cannot contain duplicates, and then convert it back to a list. For example:

list_with_duplicates = [5, 2, 2, 4, 4, 1, 3, 5]
list_without_duplicates = list(set(list_with_duplicates))
print(list_without_duplicates)
# will print [1, 2, 3, 4, 5]

As we can see, this method does not preserve the order of the list. If we want to remove duplicates while preserving the list order, we can use dict.fromkeys to create a new dictionary using the list items as keys, and then convert that dictionary back into a list:

list_with_duplicates = [5, 2, 2, 4, 4, 1, 3, 5]
# remove duplicates while preserving list order
list_without_duplicates = list(dict.fromkeys(list_with_duplicates))
print(list_without_duplicates)
# will print [5, 2, 4, 1, 3]

This works because keys in a dictionary must be unique, and since Python 3.7, keys are ordered according to when they were inserted. However, it may be considered unclear. A more straightforward alternative would be to use a for loop:

list_with_duplicates = [5, 2, 2, 4, 4, 1, 3, 5]
list_without_duplicates = []
for item in list_with_duplicates:
    if item not in list_without_duplicates:
        list_without_duplicates.append(item)

print(list_without_duplicates)
# will print [5, 2, 4, 1, 3]

This loop adds items from the first list to the second list, skipping any items that are already present.

Get Started With Sentry

Get actionable, code-level insights to resolve Python performance bottlenecks and errors.

  1. Create a free Sentry account

  2. Create a Python project and note your DSN

  3. Grab the Sentry Python SDK

pip install --upgrade sentry-sdk
  1. Configure your DSN
import sentry_sdk

sentry_sdk.init(
    "https://<key>@sentry.io/<project>",

    # Set traces_sample_rate to 1.0 to capture 100%
    # of transactions for performance monitoring.
    # We recommend adjusting this value in production.
    traces_sample_rate=1.0,
)

Loved by over 4 million developers and more than 90,000 organizations worldwide, Sentry provides code-level observability to many of the world’s best-known companies like Disney, Peloton, Cloudflare, Eventbrite, Slack, Supercell, and Rockstar Games. Each month we process billions of exceptions from the most popular products on the internet.

learn more about sentryjoin our discord
Share on Twitter
Bookmark this page
Ask a questionJoin the discussion

Related Answers

A better experience for your users. An easier life for your developers.

Try Sentry For FreeRequest a Demo
    TwitterGitHubDribbbleLinkedinDiscord
© 2024 • Sentry is a registered Trademark
of Functional Software, Inc.