Steemit! - Assistance Bounty For Data Savvy People
Requesting Your Assistance & Advice
A co-worker and I have identified a niche market that is not currently being served and want to provide the service needed.
It is a book to cover the best places for a person in a particular (large) career field to relocate to. It will cover the best places in each state. But first, we need to find those places.
To do so requires a bit of data mining from mostly government sources.
He and I are not particularly skilled enough to do this easily, but I know that plenty of people here on Steemit are. I have a few questions for these Steemians.
What is the easiest way to gather and combine the useful bits of data to serve our needs?
How could we create (or have someone build) an excel program to rank this data using our methodology?
Where to find statistics and data for a book?
Here is what I found so far. The government sites provide raw data.
Demographics:
Housing:
Economy/Jobs:
- BLS.gov
- Local Chamber of Commerce
Crime:
State By State (or city) - Departments
- Find each states crime/education/statistics department data
Other Resources:
- Numbeo
- Neighborhood Scout
- City Rating
- City Data
- US News Best Places To Live - (probably the closest to what I want to write about)
Importance of categories:
How important are these categories (as a % out of 100) to you if you were to move?
Are there more categories that I should include, like weather or disasters?
- School Ratings
- Cost of living
- Quality of life
- Crime
- Traffic/Commute
- Economy/Jobs (Salary)
- Healthcare
- Entertainment Venues
Thank you
To those that provide useful replies or can help us further, I will give large or multiple upvotes to you.
For very good information I will directly send you STEEM.
If this is a job you might be interested in helping out with, let me know as well. The amount I could pay is probably more suited to someone in a less expensive country (as this niche market has not been tested).
Personally it's going to be easier to combine data in Ms Access, some learning curve but it will organize it better than excel and allow easier reports.
See what the us news and world report ranks on, usually they have a write up of how they rank.(but you probably know this) Could copy/modify it.
The simplest most rudimentary way to do this would be to download the data onto separate Excel sheets. Then, using the vlookup function after identifying key fields (unique fields across the sheets) you can bring the info that you want to compare into 1 sheet and then do comparisons / graphs there. This is obviously dependent on the volumes of data that you are comparing. Vlookups with too much data will kill most machines. If this is the case, you need to get someone to help build you an access database and some custom reports. Good luck with your project.
What you really want to find is online services that provide open APIs and downloadable datasets.
Here's a dashboard for some of the available information per department and how close they are to meeting their open data goals: https://labs.data.gov/dashboard/offices/qa
For example, first one is Department of Agriculture and here's one of their data sets. This is the raw information you can parse and search for useful information. Of course that takes a bit of work and some knowledge, so you might need to consult with a data scientist.
*ahem*PEW Research Center, widely respected mainstream analysts, have some public data sets here: http://www.pewresearch.org/download-datasets/
They also publish reports all the time that you can search. These are not raw data but interpreted reports on the data, so might be more useful if you're looking for high level information or less useful if you just want the data to analyse yourself.
There are a lot of other open data datasets that you can find too from many NGOs and think tanks, universities and other research institutes and companies.
As I said, aggregation is a bit of a challenge and you might want professional help for that but why not try to start yourself?
Let me do more research on it and I'll get back to you as soon as possible
Please do
Somebody will be able to help and give you some thoughts I am sure @getonthetrain