On focused dataframes in ArcMap

Q: Why can't I add a north arrow/legend/scale bar to my map?

A: Are you in the layout view (vs. the data view, often the default view)?

Q: Yes, of course. I didn't just install the software this year, you know.

A: OK, I can see I'm dealing with a non-novice here. Let's try something else. Is your layer focused?

Q: Um... it's clear to me, so I guess it's in focus?

A: No, I mean that if you're looking at the data view, is there a diagonal line border around it?

Q: Yes...

A: OK, that means you have that layer focused. It allows you to edit the layer in the layout view. See this thread for more info. To un-focus the layer, hit this button on your toolbar:

For me, it's located a few lines below the word "Selection", but as always the location of the buttons depends on where you have your toolbars anchored.

Metadata in ArcGIS10

The metadata file is embedded in the shapefile and can be accessed through ArcCatalog.

Open ArcCatalog 10 (not the shortcut through ArcMap10). Find your shapefile of interest in the table of contents on the left side of the screen. Click on the shapefile to open it. The main window on the right has 3 tabs, "Contents", "Preview", and "Description". Click on the "Description" tab (it may take a minute to load). Scroll down to the bottom of the screen to see if "ArcGIS Metadata" and "FGDC Metadata" is available. If they are, click on these titles to open the metadata. If they are not, or if you are looking for a particular style of metadata, you can change the metadata style by clicking on "Customize" on the very top of the page. Choose "Arc Catalog Options" and open the "Metadata" tab. You can choose the metadata style from the dropdown menu options here.

Identifying duplicate records in Excel, ArcGIS

Let me start by saying that I have no doubt there is an easier method. This is just the way I attacked this problem today, and it seemed to work, so I thought I'd share.

There was a question as to why I didn't just use Excel's highlight or delete duplicate action. This is a great addition to the Excel package (I believe it introduced with Excel 2007) and you can read about it here: http://www.lytebyte.com/2008/06/10/highlight-remove-delete-duplicates-excel-2007/. But since I didn't want to blindly delete duplicates, I needed to find a quicker way than eyeballing highlighted cells to investigate and ultimately delete duplicate records.

Problem: I have four layers of patient data, each with over 3000 records. An eyeball review suggests that each layer has duplicate records, defined as records with the same Patient ID and same diagnosis (Dx) date. How do I identify these duplicates so that I can delete them?

Roundabout solution: First, I exported the records for one layer (i.e. 2005 cases) into Excel. I deleted all fields except patient ID, dx date, and STFID. I translated all columns into numbers with no decimal places (select the cells, hit the exclamation point drop down and select "convert to number").

I created a concatenated column named "ID_and_Dx". By using the formula "=CONCATENATE(A2,C2)", I created a record number of patient ID (located in A2) and dx date in numerical form (located in C2).

The important next step: I sorted the data by the concatenated column (ID_and_Dx).

I created a column called "Dup_ID_Dx". This is my indicator variable for duplicate fields. I used the formula "@if(D3=D2,"Dup", "Not Dup"). Filling this formula down through the sheet will mark the first record in a duplicate set as "Dup".

The important next step: I copied the Dup_ID_Dx column, highlighted the next blank column, right click and selected "Paste special", and chose to paste values. This will create a column of "Dup" or "Not Dup" without the formula. I then deleted the formula version column.

One more problem: I noticed that some records have the same patient ID and dx date, but different STFID. I would like to be able to investigate these records later in ArcGIS. How can I denote these kind-of-duplicates in Excel before I bring them back into ArcGIS?

And back to the circuitous solutions: I created another concatenated column and named it "ID_and_Dx_and_STFID". I used the concatenate function as described above to mash up my previously created ID_and_Dx field and my STFID field. Note that if a records couldn't geocode and therefore didn't have an associated STFID, this ID_and_Dx_and_STFID cell only contains the ID_and_Dx value.

I then made another duplicate indicator column, called "ID_Dx_STFID_Dup" and used the formula "@if(F3=F2,"Dup", "Not Dup")" to fill down and identify duplicates. Note that duplicates in this column indicate records that have the same patient ID, Dx date, and STFID. I am actually interested in records that have the same Patient ID and Dx date but different STFID. I will get to these kind-of-duplicates later.

Then it was time to bring it in to ArcGIS10. I created a patient ID field type double and named it "Pat_ID_dbl" on the 2005 case layer on ArcGIS and used the field calculator to populate it with the patient ID. I did this because my duplicate investigation xls document has patient ID listed as a number.

I joined the 2005 layer to the duplicate investigation xls document on the patient ID field. I sorted by the Dup_ID_Dx_ field and selected all duplicate records (using "select by attribute") to calculate my % duplication. I then used "switch selection" to select the non-duplicate records and created a new layer from these selected features (right click on layer name -> data -> export data and make sure you're exporting "selected features"). This will be the layer I will use for my analysis because it contains no duplicate records.

Now, back to the issue of kind-of-duplicates (same patient ID, same Dx date, different STFIDs, indicating that something went quite wrong in the data entry). I used my original, joined layer to select duplicate records (Dup_ID_dx). I created a new layer from this selection (right click layer -> selection -> new layer from selected features). In this layer, I sorted by the kind-of-duplicate field (sort by "ID_Dx_STFID_Dup") so I can investigate the "Not Dup" within these records to try to figure out what's going on with these records. In my case, I noticed all of the patients with the same patient ID and Dx date had the same birthday (DOB). So even though the two kind-of-duplicate records have different home addresses and therefore different STFID's, they seem to indicate that they are true duplicates of the same patient and diagnosis.

STFID

From MIT's GIS Services website:

"The STFID column is made up of a combination of FIPS codes for the geographic area chosen. Each state has a unique 2 digit code, each county has a unique 3 digit code, each tract has 6 digits, each blockgroup has 1 digit and each block has 4 digits. The smaller the geographic component chosen the more longer the STFID column."

Adding X, Y coordinate data to a layer in ArcGIS 10

My point pattern data had been previously geocoded but did not include coordinate data, so I added it:

Opened attribute table. Added new field “X” (Type: Double). Right clicked new field, selected “Calculate Geometry” and selected "Property: X Coordinate of Point", "Coordinate System of data source (in this case, GCS: North American 1983), and Units "Decimal Degrees". Repeated the process for "Y" coordinate.

For some reason, the new fields initially populated with "1"'s. I closed the attribute table, removed the layer, re-added the layer, opened the attribute table, and the coordinates magically appeared. This only happened once; further additions of coordinate data to different layers populated correctly.

My unmatched records populated the coordinates with "1".

ArcGIS 10: Exporting part of an attribute table and renaming fields

Q: I want to create a dbf file containing a few columns from my layer's attribute table to send to a collaborator. I also want to rename some of the fields so she doesn't have to translate my shorthand. Please help!

A: Gladly!

First, you want to turn off the fields you are not interested in exporting: Open your attribute table, right click the field of (un)interest, and select "turn field off". This should remove it from view, but don't worry... you can undo this command by clicking the table options in the upper right corner of the attribute table (the icon looks like a bulleted paper)and selecting "Turn all fields on".

Back to the problem at hand... repeat turning off all of the fields you aren't interested in sharing. Now export the data as a dbf (check this post for info). Use Excel to open the dbf file.

Note that you can of course export your entire attribute table to Excel and remove unwanted fields by deleting the columns in Excel instead of hiding them in ArcGIS, but sometimes there are exportation restrictions on sensitive data, so I wanted to outline how to avoid importing prohibited data into a new program.

The file still has your original field names. We didn't change those in ArcGIS 10 because field aliases are lost when you export the data, so now we just change the column names in Excel.

And now for the silliest step... Excel 2007 won't save files as dbf,so we have to save the Excel file in the old .xls format (Save As Excel 97-2003 Workbook) and re-import it into ArcGIS (notes here) to then export it again as dBase (dbf). Horray for hoops!

Latitude, longitude, X, Y

Q: Is latitude the X coordinate or the Y coordinate?
A: Latitude, the distance from the equator, is the Y coordinate. Since the US is "above" the equator, your geocoded Y values should be positive.

Longitude is the distance from the prime meridian. Since the US lies to the west ("left") of the prime meridian, your geocoded X values should be negative.

ArcGIS fields: What's the difference between "address" and "ARC_Street"?

From Getting to Know ArcGIS Desktop, Ormsby et al:

"When you rematch unmatched addresses, you can edit the values in ARC_Street and ARC_Zone to make matches. You would do this, for example, if you found mistakes in the (original data) table. The ADDRESS and ZIP attributes, on the other hand, preserves the original information from the (original data) table."

Joining insanity in ArcGIS 10, Excel 2010

Problem: I haven't been able to join the Excel file that contains demographic data by census tract to the census tract shape file.

My specific issue is that when I open the Join window, the joinable field names are not selectable... if the CT shapefile field name is selected, the corresponding field name in the Excel file is not available to select, and vice versa.

I downloaded both sets of info from the census website... so why the technical trubs???

After renaming, resaving, repopulating, and refilling my beer, I finally thought to check the field properties for my problem fields in both layers (open attribute table, highlight column of interest, right click column heading, select "Properties", check "Type"). Surprise! My field of interest in my shapefile layer is type "string", while the corresponding field of interest in my Excel file layer is type "double".

Solution: I created a same-type field to complete the join.

Since I couldn't seem to add a string type field to the Excel layer in ArcGIS, I ended up adding a double type field to the shapefile layer: Open attribute table, select "Add field...", named it "GeoId10d" and type "double" leaving the default for precision and scale. Then I started Editor (button that looks like a pencil connecting the dots), selected the appropriate layer to edit, opened the attribute table, right clicked on my new GeoID10d column, selected "Field Calculator" and just double clicked the "GeoID10" column (telling ArcGIS that GeoID10d = [GeoId10]... in English, setting the new GeoID10d column to copy the values from the old GeoID10 column and store those values as type double).

Once I had a matching type field, the join worked! Horray!

Merging census data with Tiger shapefiles

The US census makes it easy with this nifty step-by-step guide:

http://www.census.gov/geo/www/tiger/wwtl/brochures/DownloadingAFFData.pdf

More helpful resources here:
http://www.census.gov/geo/www/tiger/wwtl/wwtl.html

Although I have to wonder why they didn't just integrate the datasets (shapefiles and demographic data) in a more user friendly format to begin with...

Looking up FIPS county codes

http://www.toolsformonkeys.com/countycodes.html

Entering data field by hand in Arc GIS

Problem: You've opened your attribute table and created a new field. It auto-populated with 0's, and no matter how many times you click those cells, it won't let you edit the attribute table.

Solution: Right click the layer and go to "Edit Features" -> "Start Editing". When you're done, go to the Editor toolbar (if it's not active, click the button around the top center that looks like a pencil connecting dots), then editor dropdown -> "Save Edits" then "Stop Editing".

It seems that you cannot add a field in the editor mode... curious.

Baltimore stats and maps

The Baltimore Neighborhood Indicators Alliance offers some great resources online.

Start with the neighborhood profiles at http://www.bniajfi.org/neighborhood_data. Of course, the 'hood you think you live in may not be the official name (i.e. there is no "East Baltimore" Community Statistical Area), so click "Choose your Community Statistical Area" under the map at the upper left of the page then and click a CSA on the resulting map. This will show you a map with the boundaries to that CSA. Repeat until you find your CSA of interest, then return to the neighborhood data page to get the stats.

The neighborhood data page has 3 sections. The first section lists the Community Statistical Area (CSA) Profiles. The individual CSAs link you to PDFs that include vital signs from the 2000 census, housing & community development stats, children and family stats, crime, workforce & economic development, sanitation facts (including the rate of rat incidents... RAT CZAR!), environment & transit data, education stats, and community action info like voting rates. Example here.

The second section on the neighborhood data page is a link to downloadable shapefiles (GIS Shapefiles page at http://www.bniajfi.org/map_gallery/gis_shapefiles). These shapefiles include Community Statistical Areas (CSAs), Neighborhood Statistical Areas (NSAs), Inner Harbor, Baltimore City Public Schools (2010-2011 School Year), Farmers' Markets, Enoch Pratt Libraries, Senior Centers, Recreation Centers, Block Groups, Census Tracts, and Zip Codes. This page also directs you to important links through the "Related Resources" right side bar, including US census maps and cartographic resources, GIS resources at the Maryland State Data Center, maps from the Baltimore City Planning Department, and National Atlas shapefiles.

The last section on the neighborhood data page include the Neighborhood Statistical Area (NSA) Profiles. The data varies for neighborhoods. For example, the Canton link simply returns a map of the Canton CSA (which includes the large section of Patterson Park but not, oddly, the small section... curious), while the link for East Baltimore Midway returns a PDF with stat summary data on demographic, social, economic, and housing characteristics from the 2000 census (check it).