UE19CS322: Assignment 1
Analysis of US Road Accident Data using MapReduce
This is the first assignment for the UE19CS322 Big Data Course at PES University. The assignment consists of 2 tasks and focuses on running MapReduce jobs to analyse data recorded from accidents in the USA.
The files required for the assignment can be found here.
Assignment Objectives and Outcomes
- This assignment will help students become familiar with the Map Reduce programming environment and the HDFS.
- At the end of this assignment, the student will be able to write and debug MapReduce code.
Ethical practices
Please submit original code only. You can discuss your approach with your friends but you must write original code. All solutions must be submitted through the portal. We will perform a plagiarism check on the code and you will be penalised if your code is found to be plagiarised.
The Dataset
You will be provided with a link to the dataset on PESU Forum. You will be working with the following set of attributes.
| Key | Type | Description |
|---|---|---|
| Severity | integer | Severity of the accident (between 1 - 4) |
| Start_Time | datetime | Start time of accident in local time zone |
| Start_Lat | float | Latitude as GPS coordinate of the start point |
| Start_Lng | float | Longitude as GPS coordinate of the start point |
| Description | string | Natural language description of the accident |
| Visibility(mi) | float | Visibility (in miles) during the accident |
| Precipitation(in) | float | Precipitation amount in inches, if there is any |
| Weather_Condition | string | Weather condition during the accident - rain, snow, thunderstorm, fog, etc |
| Sunrise_Sunset | String | Shows the period of day (i.e. day or night) during the accident |
Software/Languages to be used:
- Python
3.8.x - Hadoop
v3.2.2only
Marks
Task 1: 2 marks
Task 2: 2 marks
Report: 1 mark
Tasks Overview:
- Load the data into HDFS.
- Create
mapper.pyandreducer.pyfor Task 1 and Task 2 - Run your code on the sample dataset until you get the right answer
- Submit the files to the portal
- Submit one page report based on the template and answer the questions on the report
Submission Date
16th September, 11:59 PM
Submission Guidelines
You will need to make the following changes to your mapper.py and reducer.py scripts to run them on the portal:
-
Include the following
shebangon the first line of your code#!/usr/bin/env python3 -
Convert your files to an executable
chmod +x mapper.py reducer.py -
Convert line breaks in
DOSformat toUnixformat (this is necessary if you are coding on Windows)dos2unix mapper.py reducer.py
Check out a detailed list of submission guidelines here.
Task Specifications
Task 1
Problem Statement
Find record count per hour
Description
Find the number of accidents occurring per hour that satisfy a set of conditions and display them in sorted fashion.
All the following conditions must be satisfied by a record:
| Attribute | Condition |
|---|---|
| Description | Accident should result in either a “lane blocked”, “shoulder blocked” or an “overturned vehicle” |
| Severity | >= 2 |
| Sunrise_Sunset | Night |
| Visibility(mi) | <= 10 |
| Precipitation(in) | >= 0.2 inches |
| Weather_Condition | Should either be “Heavy Snow”, “Thunderstorm”, “Heavy Rain”, “Heavy Rain Showers” or “Blowing Dust” |
Comments
Ignore records which do not satisfy the mentioned conditions. You do not require any command line arguments for this task. Additionally, if any of the required attributes contain NaN, ignore the record.
Recommended module: datetime
Output Format
For each hour that contains accident data satisfying the provided conditions, print the hour followed by the number of accidents on a separate line. For hours that do not contain any accident records, do not print anything.
Example, here is the sample output for the 5% dataset split
3 1
4 1
5 4
6 4
7 1
18 1
19 2
20 1
21 3
22 1
23 1
Example, here is the sample output for the 15% dataset split
0 3
2 1
3 1
4 6
5 11
6 9
7 4
16 2
17 1
18 4
19 4
20 9
21 4
22 5
23 3
Task 2
Problem Statement
Find record count per city and state
Description
Find the number of accidents occurring per city and state where the distance between the start coordinates and a given pair of coordinates (LATITUDE, LONGITUDE) is within D using Euclidean Distance.
For each record, make a POST request to the course’s reverse geocoding API to obtain the city and state information. The API accepts only POST requests, and expects a JSON payload containing a pair of start coordinates in the following format:
{
"latitude": Start_Lat,
"longitude": Start_Lng
}
The API sends back a response containing a JSON payload with the city and state information in the following format:
{
"city": ...,
"state": ...
}
Note: The API server used for this task is no longer running.
Comments
Ignore records which do not satisfy the required distance condition. Do not round off any numerical values as this may lead to erroneous results. Additionally, if any of the required attributes contain NaN, ignore the record.
You are required to take 3 command line arguments in your mapper.py script:
LATITUDE LONGITUDE D
Recommended modules: datetime, requests (to be installed via pip3)
You will not be allowed to install any other libraries or use any other APIs to execute your code.
Output Format
For each state, you will first have to display the name of the state. Following this, you will have to determine the number of accidents that occur in each city in that state, and display each city’s count on a separate line. You do not have to display cities where the count is zero. Finally, display the state again and the total number of accidents for that entire state.
Example, for the 5% data split with LATITUDE = 40, LONGITUDE = -66 and D = 5.3
Taking the last state ME as an example, the counts for the cities Ellsworth, Hope and Trenton are determined to be 3, 1 and 1 respectively. Hence, the total count for the state is 5.
MA
Brewster 1
Buzzards Bay 1
Carver 2
Cotuit 1
Duxbury 13
East Falmouth 2
East Freetown 1
East Sandwich 1
East Wareham 1
Eastham 1
Halifax 7
Hyannis 4
Kingston 1
Mashpee 4
Middleboro 1
New Bedford 3
North Dartmouth 2
Orleans 1
Osterville 1
Plymouth 4
Plympton 2
Rochester 1
Sandwich 2
South Dennis 2
Teaticket 1
Truro 2
Wareham 3
West Barnstable 1
West Wareham 1
MA 67
ME
Ellsworth 3
Hope 1
Trenton 1
ME 5
Example, for the 15% data split with LATITUDE = 26, LONGITUDE = -80 and D = 1
FL
Aventura 26
Bay Harbor Islands 4
Belle Glade 6
Boca Raton 308
Boynton Beach 196
Coconut Creek 5
Conservation 19
Coral Gables 14
Coral Springs 17
Coral Springs-Margate 1
Cutler Bay 78
Dania 95
Davie 39
Deerfield Beach 124
Delray Beach 194
Doral 36
El Portal 1
Florida City 7
Fort Lauderdale 1709
Glades 1
Golden Beach 1
Greenacres 3
Hallandale Beach 14
Hallandale 234
Haverhill 1
Hialeah 923
Hobe Sound 1
Hollywood 599
Homestead 316
Jupiter 72
Key Biscayne 10
Key Largo 22
Lake Worth Beach 17
Lake Worth 267
Lantana 1
Lauderdale Lakes 3
Lauderhill 4
Loxahatchee 12
Margate 5
Medley 3
Miami Beach 64
Miami Gardens 139
Miami Lakes 34
Miami Shores 9
Miami Springs 7
Miami 7340
Miramar 4
North Bay Village 4
North Lauderdale 1
North Miami Beach 109
North Miami 190
North Palm Beach 4
North Westside 2
Oakland Park 19
Ochopee 9
Opa Locka 301
Palm Beach Gardens 114
Palmetto Bay 21
Parkland 1
Pembroke Pines 46
Plantation 5
Pompano Beach 357
South Bay 3
South Miami 12
Sunny Isles Beach 14
Sunrise 9
Surfside 3
Sweetwater 2
Tamarac 4
Upper Keys 1
Virginia Gardens 1
Wellington 11
West Miami 4
West Palm Beach 629
West Park 1
Weston 2
Wilton Manors 6
FL 14870
Helpful Commands
Running MapReduce without Hadoop
A MapReduce job can also be run without Hadoop. Although slower, this utility helps you debug faster and helps you isolate Hadoop errors from code errors.
cat path_to_dataset | python3 mapper.py [command line arguments] | sort -k 1,1 | python3 reducer.py [command line arguments] > output.txt
Starting Hadoop
If you are running Hadoop for the first time, run
hdfs namenode -format
Hadoop can be started using the following command.
$HADOOP_HOME/sbin/start-all.sh
You can view all the Java processes running on your system using jps. After running jps you should see the following processes running (in any order) along with their process IDs:
DataNode
SecondaryNameNode
Jps
ResourceManager
NameNode
NodeManager
HDFS Operations
The HDFS supports all file operations and is greatly similar to the file system commands available on Linux. You can access HDFS on command line using hdfs dfs and use the - prefix before the file system command to execute general Linux file system commands.
Loading a file into HDFS
A file can be loaded into HDFS using the following command.
hdfs dfs -put path_to_file /hdfs_directory_path
Listing files on HDFS
Files can be listed on HDFS using
hdfs dfs -ls /hdfs_directory_path
Similarly, HDFS also supports -mkdir, -rm and more.
Running a MapReduce Job
A MapReduce job can be run using the following command
hadoop jar path-to-streaming-jar-file \
-input path_to_input_folder_on_hdfs \
-output path_to_output_folder_on_hdfs \
-mapper absolute_path_to_mapper.py command_line_arguments \
-reducer absolute_path_to_reducer.py command_line_arguments