Analyzing my Google Location History

-
I recently read this article on how to create a heat-map from your Google Location History data. Trying it out myself, I got some amazing results:
The large red circles represent the cities I have spent a significant amount of time. The purple shade in different places represents places I had travelled to or passed through while on a train journey.
My previous phone had some GPS issues, which was leading to my location being shown in Arizona, USA!. Surprisingly (or not?) it even gave a proof of that!
All this was really cool to look at, but I really wanted to dive in and get some more insights into my travelling patterns throughout the years.

Data Pre-Processing

Like most data science problems, data pre-processing was definitely the pain point. The data was in a JSON format where the meaning of different attributes wasn’t very clear.
Data Extraction
{'timestampMs': '1541235389345',
  'latitudeE7': 286648226,
  'longitudeE7': 773296344,
  'accuracy': 22,
  'activity': [{'timestampMs': '1541235388609',
    'activity': [{'type': 'ON_FOOT', 'confidence': 52},
     {'type': 'WALKING', 'confidence': 52},
     {'type': 'UNKNOWN', 'confidence': 21},
     {'type': 'STILL', 'confidence': 7},
     {'type': 'RUNNING', 'confidence': 6},
     {'type': 'IN_VEHICLE', 'confidence': 5},
     {'type': 'ON_BICYCLE', 'confidence': 5},
     {'type': 'IN_ROAD_VEHICLE', 'confidence': 5},
     {'type': 'IN_RAIL_VEHICLE', 'confidence': 5},
     {'type': 'IN_TWO_WHEELER_VEHICLE', 'confidence': 3},
     {'type': 'IN_FOUR_WHEELER_VEHICLE', 'confidence': 3}]}]},
 {'timestampMs': '1541235268590',
  'latitudeE7': 286648329,
  'longitudeE7': 773296322,
  'accuracy': 23,
  'activity': [{'timestampMs': '1541235298515',
    'activity': [{'type': 'TILTING', 'confidence': 100}]}]
After researching a bit, I stumbled upon this article, which cleared a lot of things. However, there are some questions that still remain unanswered:-
  • What does the activity typetilting mean?
  • I assumed confidence to be the probability of each task. However, often they do not add up to 100. If they do not represent probabilities what do they represent?
  • What is the difference between the activity type walking and on foot ?
  • How can Google possibly predict activity type between IN_TWO_WHEELER_VEHICLE vs IN_FOUR_WHEELER_VEHICLE ?!
If anyone has been able to figure it out, please let me know in comments.
Edit: There has been some discussion about these topics on this thread. A paper on Human Activity Recognition using smartphone data can be found here.
Assumptions
As I continued to structure my pre-processing pipeline, I realized I will have to take some assumptions to take into account all the attributes of the data.
  • The GPS is always on (A strong assumption which is later taken care of).
  • The confidence interval is the probability of the activity type. This assumption helps us take into account various possible activity types for a given instance without under-representing or over-representing any particular activity type.
  • Each log has two types of timestamps. (i) Corresponding to position latitude and longitude. (ii) Corresponding to the activity. Since the difference between two timestamps was usually very minute ( < 30 seconds) I safely used the timestamp corresponding to latitude and longitude for our analysis
Data Cleaning
Remember I told how my GPS was giving Arizona, USA as the location? I did not want those data points to significantly differ the results. Using the longitudinal boundaries of India, I filtered out the data points pertaining only to India.
def remove_wrong_data(data):
    degrees_to_radians = np.pi/180.0
    data_new = list()
    for index in range(len(data)):
        longitude = data[index]['longitudeE7']/float(1e7)
        if longitude > 68 and longitude < 93:
            data_new.append(data[index])
    return data_new

Feature Engineering

Cities for each data-point
I wanted to get the corresponding city for each given latitude and longitude. A simple Google search got me the coordinates of the major cities I’ve lived in i.e. Delhi, Goa, Trivandrum and Bangalore.
def get_city(latitude):
    latitude = int(latitude)
    if latitude == 15:
        return 'Goa'
    elif latitude in [12,13]:
        return 'Bangalore'
    elif latitude == 8:
        return 'Trivandrum'
    elif latitude > 27.5 and latitude < 29:
        return 'Delhi'
    else:
        return 'Other'
data_low['city'] = data.latitude.apply(lambda x:get_city(x))
Distance
Logs consist of latitude and longitude. To calculate distance travelled between logs, one has to convert these values to formats that can be used for distance related calculations.
from geopy.distance import vincenty
coord_1 = (latitude_1,longitude_1)
corrd_2 = (longitude_2, longitude_2)
distance = vincenty(coord_1,coord_2)
Normalized Distance
Each log consists of activity. Each activity consists of one or more activity types along with the confidence (called as probability). To take into account the confidence of measurement, I devised a new metric called normalized distance which is simply distance * confidence

Data Analysis

Now comes the interesting part! Before I dive into the insights, let me just brief on some of the data attributes:-
  • accuracy: Estimation of how accurate the data is. An accuracy of less than 800 is generally considered high. We have therefore dropped the columns with accuracy greater than 1000
  • day: Represents the day of the month
  • day_of_week: Represents the day of the week
  • month: Represents the month
  • year: Represents the year
  • distance: Total distance travelled
  • city: City corresponding to that data point
Outlier Detection
There are total 1158736 data points. 99% of the points cover a distance less than 1 mile. The rest 1% are anomalies generated due to poor reception/flight mode.
To avoid the 1% of data causing significant changes in our observations, we’ll split the data into two based on the normalized distance.
This also ensures that we remove the points which do not obey the assumption #1 we made during our analysis
data_low = data[data.normalized_distance < 1]
data_large = data[data.normalized_distance > 1]
Distance travelled with respect to the city
The data for 2018 correctly represents that majority of the time was spent at Bangalore and Trivandrum.
I wondered how distance travelled in Delhi (my hometown) came out to be more than the Goa, where I did my graduation. Then it hit me, I did not have a mobile internet connection for the majority of my college life :).
Travel Patterns in Bangalore and Trivandrum
In June 2018, I completed my internship in my previous organization (at Trivandrum) and joined Nineleaps (at Bangalore). I wanted to know how my habits changed on transitioning from one city to another. I was particularly interested in observing my patterns for two reasons:
  • Since I always had mobile internet while residing in these cities, I expected the representation to be an accurate representation of reality.
  • I’ve spent roughly the same amount of time in two cities, hence the data will not be biased towards any particular city.
Month vs Distance travelled while in Bangalore and Trivandrum
  • Multiple friends and family members visiting Bangalore in the month of October resulted in a huge spike in distance travelled in vehicles.
  • Initially, I was exploring Trivandrum. However, as my focus shifted to securing a full-time data science opportunity, the distance travelled drastically reduced from January to February to March.
  • Vehicle usage is much higher in Bangalore between 20:00–00:00. I guess I am leaving office later in Bangalore.
  • I was walking a lot more in Trivandrum! The difference in walking distance from 10:00–20:00 shows how I was living a healthier lifestyle by taking a walk after every hour or two in office.

Conclusion

There is a lot more (like this and this) to do with your Location history. You can also explore your Twitter/Facebook/Chrome data. Some handy tips when trying to explore your dataset:
  • Spend a significant amount of time pre-processing your data. It’s painful but worth it.
  • When working with large volumes of data, pre-processing can be computationally heavy. Instead of re-running the Jupyter Cells every time, dump the post-preprocessed data into a pickle file and simply import the data when you start again.
  • Initially, you might miserably fail (like me) in finding any patterns. Make a list of your observations and keep exploring the dataset from different angles. If you ever reach a point wondering whether any patterns are even present, ask yourself three questions: (i) Do I have a thorough understanding of the various attributes of data? (ii) Is there anything I can do to improve my pre-processing step? (iii) Have I explored the relationship between all the attributes using all possible visualization / statistical tools?
To get started you can use my Jupyter Notebook here.
If you have any questions/suggestions feel free to post on comments.
You can connect with me on LinkedIn or email me at k.mathur68@gmail.com.

Comments

Post a Comment

Popular posts from this blog

Five Minutes to Your Own Website

-
Image
f Jekyll is a simple static site generator. This means it takes plain text files and renders them as a website that can then be served to the web through a web hosting service. One option for this service is GitHub Pages , free “websites hosted and easily published through GitHub.” With Jekyll + GitHub Pages, the source files for your website live in a GitHub repository that is automatically published to the web. Basically, Jekyll transforms text documents in a repository into a website and then that website is hosted by GitHub. The only thing you have to worry about is writing posts and editing a few of the text files to make things look how you want. As the Jekyll design philosophy states: Jekyll gets out of your way and allows you to concentrate on what truly matters: your content. You can follow this guide on GitHub to set up a new blank website. However, if you’re looking to make a simple blog as quick as possible, a better option is to fork (a GitHub term ...
Read more

15 Websites To Get Creative Commons Music For Free

-
Image
Whether it’s background audio for that upcoming project presentation of yours, or some funky sound effects to be used in the game you’re working on, or just about anything else, having good quality music at your disposal is one of the essential requirements. However, generally, most musical compositions are copyrighted, and can’t be used without being licensed, which often means paying hefty amounts of cash to the copyright holders. Quite an issue, right? Not really. There are numerous online sources where you can get musical compositions (including everything from full tracks to sound effects) released under the Creative Commons license . You can use them freely in any of your projects (commercial or non-commercial), as long as you provide proper credit and follow the attribution rules mentioned in the terms of the license. Intrigued? Let’s take a look at 15 such websites where you can get Creative Commons music for free. Get Creative Commons Music for Free YouTube...
Read more

Object detection and tracking in PyTorch

-
Image
Detecting multiple objects in images and tracking them in videos Chris Fotache Dec 10 In my previous story, I went over how to train an image classifier in PyTorch , with your own images, and then use it for image recognition. Now I’ll show you how to use a pre-trained classifier to detect multiple objects in an image, and later track them across a video. What’s the difference between image classification (recognition) and object detection? In classification, you identify what’s the main object in the image and the entire image is classified by a single class. In detection, multiple objects are identified in the image, classified, and a location is also determined (as a bounding box). Object Detection in Images There are several algorithms for object detection, with YOLO and SSD among the most popular. For this story, I’ll use YOLOv3 . I won’t get into the technical details of how YOLO (You Only Look Once) works — you can read that here  — but...
Read more