Pandas Removing Duplicates

Pandas Removing Duplicates (Beginner → Advanced)

In real datasets, duplicate rows are very common (API data, CSV files, logs, joins, etc.).

Pandas gives us drop_duplicates() to handle this cleanly.

1. What Are Duplicates?

Duplicates can be:

• Entire row duplicate

• Duplicate based on specific column(s)

• First / last occurrence

• Partial duplicates

 2. Basic Example (Beginner)

Remove duplicate rows:

•  Keeps first occurrence by default

 3. Keeps First / Last Duplicate

Keep first (default):

Keep last:

Remove ALL duplicates:

• Interview favorite

 4. Remove Duplicates Based on One Column

•  Removes duplicate id, keeps first row

 5. Remove Duplicates Based on Multiple Columns

•  Duplicate only if both id & name match

 6. Remove Duplicates In-Place

• df itself is modified
•  Cannot undo easily

 7. Check Duplicate Rows (Before Removing)

Find duplicates:

Output:

See only duplicate rows:

Include first occurrence also:

8. Count Duplicate Rows

•  Very useful for data auditing

9. Remove Duplicates After Sorting (Advanced)

Sometimes you want:

• latest record

• highest value

• newest date

Example:

•  Keeps last sorted record

10. Remove Duplicates with Index Reset

•  Clean index after deletion

11. Case-Insensitive Duplicate Removal

• "Amit" and "amit" treated as same

12. Remove Duplicates in Large Datasets (Performance Tip)

•  Faster
•  Cleaner index

13. Real-World Example (API / CSV Data)

 14. Common Mistakes

•  Forgetting subset
• Using keep=False unintentionally
•  Modifying original data with inplace=True

15. Interview Questions (Must Know)

Q1: Default behavior of drop_duplicates()?
 Keeps first occurrence

Q2: How to delete all duplicates?
keep=False

Q3: How to check duplicates without deleting?
duplicated()

Q4: How to remove duplicates based on multiple columns?
subset=["col1", "col2"]

Quick Cheat Sheet