DadOverflow.com

Musings of a dad with too much time on his hands and not enough to do. Wait. Reverse that.

Tag: tools (Page 27 of 35)

Grouping and counting by date

November 9, 2018 /

Subtitle: In one line

The other day at work, I had a file of a bunch of activities and the dates on which they occurred.  Something like this:

What I wanted to do was to get a count of the number of activities occurring on each day, so I fired up a PowerShell command shell and ran the following one-liner:



1
ipcsv "C:\my_path\test_data.csv"|group event_date|select Name, Count

That line produced these results:



1
2
3
4
5
6
7
8
Name       Count

----       -----

8/23/2018     10

9/19/2018      8

7/3/2018      18

10/13/2018    15

11/2/2018      9

10/1/2018     15

Nice.  I have my counts, but it’d be even better if my dates were in order.  Let’s try to sort those suckers:



1
ipcsv "C:\my_path\test_data.csv"|sort event_date|group event_date|select Name, Count

That produced:



1
2
3
4
5
6
7
8
Name       Count

----       -----

10/1/2018     15

10/13/2018    15

11/2/2018      9

7/3/2018      18

8/23/2018     10

9/19/2018      8

Not the kind of sorting I was looking for.  It sorted the dates like they were strings.  Wait a minute…



1
2
3
4
5
PS C:\WINDOWS\system32> (ipcsv "C:\my_path\test_data.csv")[0].event_date.GetType()



IsPublic IsSerial Name                                     BaseType                                                                                                                           

-------- -------- ----                                     --------                                                                                                                           

True     True     String                                   System.Object

Just as I suspected: PowerShell is importing those dates as strings!  If I want to sort my results by date, I’ll have to cast that date to a proper datetime data type first.  Here’s what I came up with:



1
ipcsv "C:\my_path\test_data.csv"|select -Property @{Name="event_dt"; Expression={[datetime]$_.event_date}}|sort event_dt|group event_dt|select Name, Count

And the results:



1
2
3
4
5
6
7
8
Name                   Count

----                   -----

7/3/2018 12:00:00 AM      18

8/23/2018 12:00:00 AM     10

9/19/2018 12:00:00 AM      8

10/1/2018 12:00:00 AM     15

10/13/2018 12:00:00 AM    15

11/2/2018 12:00:00 AM      9

Nice!  I generated my counts and sorted by date all in one line of code.  Sweet!

Visiting colleges

October 29, 2018 /

I am deep within the season of college visits with my high school senior: actually, I’m nearing the end. As I’m told switching colleges (and majors) can be quite expensive, I want to make sure that wherever my daughter lands, she’s there for good (until she graduates, of course). So, we’ve been visiting colleges. Lots of them. Sometimes repeatedly.

The whole process has been time consuming, but to make the most of our visits, I try to do a fair amount of preparation before our trips. Here’s a checklist I put together to better prepare me for each sojourn:

1. Register for the event

Chances are, your early visits will be in response to pre-arranged “open houses” and similar events. Typically, the college will want you to register for the event. In theory, the more colleges see your child’s name in their registration lists, the more favorably they’ll look upon her during application time. Regardless, I always try to register. Also, don’t wait until the day before: sometimes the registration links will expire a day or two before the event.

2. Map it out

Plan out your route a day or two before. Figure out what building you need to be at, when, and where you can park in close proximity. By mapping out your driving route, say with Google Maps, you can also figure out how early you need to hit the road. Here’s a pro tip: see if the college you’re headed to has a mobile app. Mobile apps can be great for navigating a large campus. Otherwise, try to print out a campus map. Here’s another pro tip: see if the college recommends a particular parking garage. Occasionally, they’ll validate!

3. Plan ahead for weather

We’ve visited campuses in the hot summer sun and on cold, snowy days in January. Check with your favorite weather app ahead of time to know what sort of weather to expect.

4. Dress accordingly

Obviously, dress for the weather including bringing an umbrella if you expect rain. Think about wearing comfortable shoes, particularly if you anticipate a tour of a large campus. Remember that this is a chance to make an impression on the school’s admissions staff, so your child and you should dress decently.

5. Bring questions

Admissions staff always ask attendees if they have questions. Have some. Your child should have some, too. If you can’t think of any yourself, just Google for some. I have my own list that I print out when we go on visits.

6. Bring pen and paper

I always bring at least one notebook and several pens for taking notes. I also bring a clipboard so that it’s a little easier for me to take notes if I find myself on a walking tour. When I get back home, I try to type up all my notes electronically so that I can easily reference them later.

7. Bring other helpful items

I bring a small backpack in which I house my notebook, pens, and clipboard. Since I like to be prepared, I usually bring other items including:

  • Bottled water
  • Snacks and protein bars
  • A small portable battery and cables for charging phones
  • A flashlight…just in case

8. Look through any handout material

Obviously, any handout material you receive should help answer some of your questions. It will include contact information where you can follow up with questions you think of later. Occasionally, colleges may even include coupons for the bookstore and/or local eating establishments.

9. Score the school

You could be doing this whole college visit thing for up to a year or more. By the fourth or fifth school, you may even begin forgetting the earlier ones. Aside from taking decent notes on each school, you should objectively score them against criteria that’s important to you. For example, if graduating in the smallest amount of time is important to you, you might ask yourself: “I have a strong likelihood of graduating in four years from this school.” Then, based on the information you’ve gleaned from you visit, you might score that statement from one to five with five being the highest likelihood. Come up with a variety of similar questions that reflect what’s important to you in a school and put them in a scorecard. Either during the visit or shortly after it’s over, have your child fill it out. Later, record those scores in Google sheets or other spreadsheet software. Eventually, you can tally those scores to see which school scores highest. Here’s a scorecard I’ve used in the past.

10. Pick your visits thoughtfully

Most admissions counselors say–and it’s probably true–that the best time to visit a school is during the school year, when you and your child can see the hustle and bustle of school life and better make a determination if the “vibe” is right for your child. Of course, visiting at these times introduces a few other concerns: 1) you’ll likely be visiting during the work week which means you’ll probably have to burn a vacation day and 2) when the campus is full, the parking lots may be, as well. Also pay attention to the school’s sports schedule: you’ll probably want to avoid a visit to a Big 10 school when their football team is playing.

The goodness of Notepad++

October 4, 2018 /

Notepad++ was #7 on my list of awesome, free tools and for good reason: it just rocks!

The other day, I found myself working through some online training with Pandas dataframes and friends.  Part of the course included working exercises in the site’s web-based Python console.  When I work such exercises, I like to copy off the code I come up with to local Jupyter Notebooks that I can easily reference in the future.  I also like to copy down whatever test data the exercises have me working with, so I can make sure I calculate the same results locally that I do in the exercises.

Here’s the challenge: how do you copy the dataframes from the online exercises to your local Jupyter Notebook?  Usually, when I want to copy off a dataframe, I’ll call the to_csv() function to save the contents of the dataframe to a CSV file that I can easily transport.  That’s not really an option with online exercises, though.  Here’s a thought: what about the to_dict() function to write the contents of the dataframe to the standard out of the online console as a dictionary?  Then, I can copy that dictionary over to my Notebook.  Let’s see what that looks like:

The view from the online training console

Let’s pretend for a moment that we’re in the online training console and we’re working some dataset (for simplicity, I’m using the Iris dataset).  The dataframe in the console might look like this:

Now, we can use to_dict() to write the dataframe out to the console (note: for brevity, I’m only writing out the first 5 records):

Copy the dictionary locally

With the dictionary written out to the online console, let’s copy that output to our clipboard, paste it into a local Notebook, and see if we can now load it into a local dataframe:

Well, waddya know?  That worked!  Wait a minute…where’s Notepad++ in all of this?

Yes, this approach works when you’re copying the dictionary directly into a code cell in a local Jupyter Notebook, but what if, instead, you copy that dictionary into a JSON file that you then load into a dataframe?

That approach doesn’t end well.  The reason is that Pandas doesn’t consider that copied JSON as valid JSON.  Specifically, it wants all the key names to be surrounded by quotation marks.

Incidentally, you might be asking, why would you copy the dictionary to a file when I’ve already demonstrated that you can copy it directly into a local Notebook?  The big reason is size: copying a dictionary of five rows is fine, but what if the dataframe you’re working with has 200 rows?  That becomes a very long dictionary that really muddies up your local Notebook.  To keep things clean, I find it best to copy such dictionaries to a JSON file.

So, how do you format this copied dictionary so Pandas can load it successfully?  Notepad++!  Notepad++ has a great find/replace feature that let’s you use regular expressions.  What I need to do is find all numbers that serve as key names in my dictionary and make sure these keys are surrounded by quotation marks.

For my “find” regular expression, I’ll use this: (\d+):

With this expression, I look for digits that are followed by a colon.  I’ll group those digits so that I can reference them in the “replace”.

For my “replace” expression, I’ll use this: \”\1\”:

The \1 refers to my group of digits.  I’m surrounding that group with quotation marks and then making sure the colon follows.  That yields the following:

And when we load that local JSON file into our local dataframe:

…we get success!  So, just one clever way Notepad++ has really helped me out.

« Older posts Newer posts »

© 2024 DadOverflow.com

Theme by Anders Noren — Up ↑