DadOverflow.com

Musings of a dad with too much time on his hands and not enough to do. Wait. Reverse that.

Month: July 2020

Thinning out your tick labels

July 24, 2020 /

Have you ever rendered a chart with Pandas and/or Matplotlib where one or both of your axes (axises?) rendered as a smear of overlapping, unreadable black text?

https://youtu.be/d9kuDizrBPc?t=70
If you can read this, you don’t need glasses

As an example, let’s create a bar chart of COVID-19 data. [As an aside: I’ve noticed that line charts seem to automatically thin out any overlapping tick labels and tend not to fall prey to this problem.]

Load and clean up the data

After downloading the CSV data, I wrote the following code to load the data and prepare it for visualization:

df_covid_confirmed_us = pd.read_csv('./data/time_series_covid19_confirmed_US_20200720.csv')
df_covid_deaths_us = pd.read_csv('./data/time_series_covid19_deaths_US_20200720.csv')

cols_to_keep1 = [i for i, v in enumerate(df_covid_confirmed_us.columns) if v in ['Admin2', 'Province_State'] or v.endswith('20')]
cols_to_keep2 = [i for i, v in enumerate(df_covid_deaths_us.columns) if v in ['Admin2', 'Province_State'] or v.endswith('20')]
df_covid_confirmed_ohio = df_covid_confirmed_us[df_covid_confirmed_us.Province_State=='Ohio'].iloc[:,cols_to_keep1].copy()
df_covid_deaths_ohio = df_covid_deaths_us[df_covid_deaths_us.Province_State=='Ohio'].iloc[:,cols_to_keep2].copy()

df_covid_confirmed_ohio.head()

Tidy up the dataframes

The data is still a bit untidy, so I wrote this additional code to transform it into a more proper format:

date_cols = df_covid_confirmed_ohio.columns.tolist()[2:]
rename_cols_confirmed = {'variable': 'obs_date', 'value': 'confirmed_cases'}
rename_cols_deaths = {'variable': 'obs_date', 'value': 'deaths'}

df_covid_confirmed_ohio = pd.melt(df_covid_confirmed_ohio.reset_index(), id_vars=['Admin2', 'Province_State'], 
                                  value_vars=date_cols).rename(columns=rename_cols_confirmed)
df_covid_deaths_ohio = pd.melt(df_covid_deaths_ohio.reset_index(), id_vars=['Admin2', 'Province_State'], 
                               value_vars=date_cols).rename(columns=rename_cols_deaths)

df_covid_confirmed_ohio['obs_date'] = pd.to_datetime(df_covid_confirmed_ohio.obs_date)
df_covid_deaths_ohio['obs_date'] = pd.to_datetime(df_covid_deaths_ohio.obs_date)

print(df_covid_confirmed_ohio.head())
print(df_covid_deaths_ohio.head())

Concatenate the two dataframes together

I’d like to do a nice, side-by-side comparison, in bar chart form, of these two datasets. One way to do that is to concatenate both dataframes together and then render your chart from the single result. Here’s the code I wrote to concatenate both datasets together:

df_covid_confirmed_ohio['data_type'] = 'confirmed cases'
df_covid_confirmed_ohio['cnt'] = df_covid_confirmed_ohio.confirmed_cases
df_covid_deaths_ohio['data_type'] = 'deaths'
df_covid_deaths_ohio['cnt'] = df_covid_deaths_ohio.deaths
drop_cols = ['confirmed_cases', 'deaths', 'Admin2', 'Province_State']

df_combined_data = pd.concat([df_covid_confirmed_ohio[df_covid_confirmed_ohio.obs_date>='2020-5-1'], 
               df_covid_deaths_ohio[df_covid_deaths_ohio.obs_date>='2020-5-1']], sort=False).drop(columns=drop_cols)

Now, render the chart

Ok, I’m finally ready to create my chart:

fig, ax = plt.subplots(figsize=(12,8))
_ = df_combined_data.groupby(['obs_date', 'data_type']).sum().unstack().plot(kind='bar', ax=ax)

# draws the tick labels at an angle
fig.autofmt_xdate()

title = 'Number of COVID-19 cases/deaths in Ohio: {0:%d %b %Y} - {1:%d %b %Y}'.format(df_combined_data.obs_date.min(), 
                                                                                      df_combined_data.obs_date.max())
_ = ax.set_title(title)
_ = ax.set_xlabel('Date')
_ = ax.set_ylabel('Count')

# clean up the legend
original_legend = [t.get_text() for t in ax.legend().get_texts()]
new_legend = [t.replace('(cnt, ', '').replace(')', '') for t in original_legend]
_ = ax.legend(new_legend)
Wow! Those dates along the X axis are completely unreadable!

The X axis is a mess! Fortunately, there are a variety of ways to fix this problem: I particularly like the approach mentioned in this solution. Basically, I’m going to thin out the labels at a designated frequency. In my solution, I only show every fourth date/label. So, here’s my new code with my label fix highlighted:

fig, ax = plt.subplots(figsize=(12,8))
_ = df_combined_data.groupby(['obs_date', 'data_type']).sum().unstack().plot(kind='bar', ax=ax)

# draws the tick labels at an angle
fig.autofmt_xdate()

title = 'Number of COVID-19 cases/deaths in Ohio: {0:%d %b %Y} - {1:%d %b %Y}'.format(df_combined_data.obs_date.min(), 
                                                                                     df_combined_data.obs_date.max())
_ = ax.set_title(title)
_ = ax.set_xlabel('Date')
_ = ax.set_ylabel('Count')

# clean up the legend
original_legend = [t.get_text() for t in ax.legend().get_texts()]
new_legend = [t.replace('(cnt, ', '').replace(')', '') for t in original_legend]
_ = ax.legend(new_legend)

# tick label fix
tick_labels = [l.get_text().replace(' 00:00:00', '') for l in ax.get_xticklabels()]
new_tick_labels = [''] * len(tick_labels)
new_tick_labels[::4] = tick_labels[::4]
_ = ax.set_xticklabels(new_tick_labels)
Much better!

That X axis is much more readable now thanks to the power of Python list slicing.

Cleaning up Stacked Bar Charts, Part 3

July 20, 2020 /

In my final mini-series on cleaning up stacked bar charts (Part 1 and Part 2, in case you missed them), let’s talk about how you might order the bars of your chart.

In my last post, each bar in my chart represented a different day of the week and I allowed the bars to be ordered accordingly:

The bars are ordered Monday – Sunday (starting at the bottom left)

Most people would probably expect this sort of ordering. However, what if your groups don’t have an inherent order like day-of-the-week?

For my example, I generated some random email data for five fake email accounts:

import numpy as np
from datetime import date, timedelta
import pandas as pd


# names compliments of: https://frightanic.com/goodies_content/docker-names.php
email_accounts = ['fervent_saha@test.com', 'serene_cori@test.com', 'agitated_pike@test.com', 
                  'cocky_turing@test.com', 'sad_babbage@test.com']
email_data = []

for acct in email_accounts:
    for cat in ['primary', 'promotions', 'social']:
        nbr_of_email = np.random.randint(50, high=100)
        for i in range(0, nbr_of_email):
            email_dt = date(2020, 6, 1) + timedelta(days=np.random.randint(0, high=30))
            email_data.append([email_dt, acct, cat])
            
df_email_accts = pd.DataFrame(email_data, columns=['email_date', 'email_account', 'email_category'])
df_email_accts['email_date'] = pd.to_datetime(df_email_accts.email_date)
df_email_accts.head()
A bunch of random, fake email data

Now, let’s use a stacked bar chart to compare the emails counts, by category, of the five different email accounts:

fig, ax = plt.subplots(figsize=(12,8))
_ = df_email_accts.groupby(['email_account', 'email_category']).count().unstack().plot(kind='barh', stacked=True, ax=ax)

_ = ax.set_title('Email counts by category, June 2020')
_ = ax.set_xlabel('Email Count')
_ = ax.set_ylabel('Email Account')
Bar chart chaos!

Technically, matplotlib has ordered the email accounts alphabetically–from agitated_pike@test.com to serene_cori@test.com–but most folks probably don’t care about that: they’ll likely want the chart ordered either greatest count to least or least count to greatest.

How can you then order your stacked bar chart by the total count? There may be a more elegant way to do this in pandas, but I came up with three lines to code to get the order right.

To start with, take a look at the dataframe we get with my standard groupby and unstack approach:

df_email_accts.groupby(['email_account', 'email_category']).count().unstack()

What I need is a way to total the counts of the three categories–primary, promotions, and social–for each of the five email accounts and then sort the dataframe by that total.

No problem! I can use the pandas sum function with axis=1–meaning, sum across the columns–to get that total:

df_rpt = df_email_accts.groupby(['email_account', 'email_category']).count().unstack()
df_rpt['total'] = df_rpt.sum(axis=1)
df_rpt.head()
The sum function gives me a “total” value I can use for sorting

Putting it all together, then, here’s the code I came up with to nicely sorted my stacked bar chart in a meaningful way:

# two lines of code to provide a "total" column that can be used for sorting
df_rpt = df_email_accts.groupby(['email_account', 'email_category']).count().unstack()
df_rpt['total'] = df_rpt.sum(axis=1)

fig, ax = plt.subplots(figsize=(12,8))

# sort the dataframe by the "total" column, then drop it before rendering the chart
_ = df_rpt.sort_values('total')[df_rpt.columns.tolist()[:-1]].plot(kind='barh', stacked=True, ax=ax)
_ = ax.set_title('Email counts by category, June 2020')
_ = ax.set_xlabel('Email Count')
_ = ax.set_ylabel('Email Account')

# and, of course, clean up the legend
original_legend = [t.get_text() for t in ax.legend().get_texts()]
new_legend = [t.replace('(email_date, ', '').replace(')', '') for t in original_legend]
_ = ax.legend(new_legend, title='Category')
A nicely sorted, stacked bar chart where the high and low counts are immediately apparent

Cleaning up Stacked Bar Charts, Part 2

July 11, 2020 /

Here is the second installment in my mini-series on stacked bar charts.

Grouping in your stacked bar charts can be powerful and insightful. With time series data, grouping by the day of the week, by month, or even by year can provide an interesting perspective on your data.

Considering the email data I used in my previous post, I can use the following code to group my data by day of week:

fig, ax = plt.subplots(figsize=(12,8))
title = 'Email counts by day of week: {0:%d %b %Y} - {1:%d %b %Y}'.format(df_email.email_dt.min(), df_email.email_dt.max())

_ = df_email[['email_dt','category','dow']].groupby(['dow','category']).count().unstack().\
    plot(stacked=True, kind='barh', title=title, ax=ax)
Just what are those numbers in the Y column?

Interesting: I certainly receive more email on days 2 and 3 but…wait…what are days 2 and 3?!

Days 2 and 3 correspond to Wednesday and Thursday, respectively. I know this because I used the pandas dayofweek function to get those values and that’s what those numbers translate to. I may know that, but the average viewer of my chart won’t. So, I need a way to change those labels to ones the viewer can understand. I can do that with the following code (with the most pertinent code highlighted):

fig, ax = plt.subplots(figsize=(12,8))
title = 'Email counts by day of week: {0:%d %b %Y} - {1:%d %b %Y}'.format(df_email.email_dt.min(), df_email.email_dt.max())

df_email[['email_dt','category','dow']].groupby(['dow','category']).count().unstack().\
    plot(stacked=True, kind='barh', ax=ax)

_ = ax.set_title(title)
_ = ax.set_xlabel('Email Count')
_ = ax.set_ylabel('Day of Week')

# clean up the legend
original_legend = [t.get_text() for t in ax.legend().get_texts()]
new_legend = [t.replace('(email_dt, ', '').replace(')', '') for t in original_legend]
_ = ax.legend(new_legend, title='Category')

# now, replace the day numbers with their names
day_labels = {0: 'Monday', 1: 'Tuesday', 2: 'Wednesday', 3: 'Thursday', 4: 'Friday', 5: 'Saturday', 6: 'Sunday'}
curr_ylabels = [t.label.get_text() for t in ax.yaxis.get_major_ticks()]
new_ylabels = [day_labels[int(l)] for l in curr_ylabels]
_ = ax.set_yticklabels(new_ylabels)
Ahhh: much better!

Interestingly, pandas does have a day_name function that returns the name of the day instead of its number. The nice thing about my approach–using the dayofweek numbers and then replacing the numbers with the friendly names–is that matplotlib automatically sorts my bars numerically, so my bars are already in a natural order. In this case: Monday through Sunday. Were I to use the day_name function instead, matplotlib would want to sort the bars alphabetically, from Friday to Wednesday. That would make for an oddly arranged bar chart.

© 2024 DadOverflow.com

Theme by Anders Noren — Up ↑