Hudson QGIS

[jimrtex]

Part I

When you open QGIS, you should get a blank map area. Or on the menu select Project > New.

Drag the ".shp" file to the map area and it should look something like this:



In your browser, you can right click, and View Image to see a larger image.

When you add an edges layer, it picks a random color, so your lines will be a different color. There is probably a way to set a default display, but I've never bothered. If you don't like the color, it is easy enough to change.

Census Data is stored as Lat/Long pairs, and when you drag a census shapefile into the map area it will be displayed as Lat/Long values, which produces the horizontally stretched image you see.

[jimrtex]

Part 2

We'll now set a projection for the project.



First we will get the parameters for a particular projection that I like. If we were dealing with larger areas, we would just set some standard projection.

(1) Click on the the icon in the lower right, which brings up the menu.
(2) Checkmark "enable on the fly" transformation.
(3) Enter "Azimuthal" in the filter box. This is just a search filter. If you were producing a USA map you might enter "Albers" or "Lambert". For a Maine map you might enter "Maine" or "State" (Plane).
(4) Select "World Azimuthal Equidistant" which will cause its parameters to appear in the bottom box.
(5) Select and copy the text of the parameters (get a text insertion cursor in the box Ctrl-A Ctrl-C)
(6) Cancel. We are not setting the projection yet, just getting its parameter.

We will now add a user-defined projection. Before we do, we need some coordinates for the center of projection. Move the cursor in the map area, and choose a pair of coordinates. My habit is to round to a 1/4 degree. So I'll use 42.25 and -73.75. Note that western hemisphere longitude values are negative. Incidentally since 42.25 or 42°15' is right in the middle of Hudson, the north-south edge of 7.5-minute and 15-minute topo sheets cut Hudson. Since Hudson is laid out on a diagonal it is really annoying.



(1) On menu, select Settings > CRS Settings, and the box will appear.
(2) Click on the Green + sign to add a new user-defined projection.
(3) Enter the name of the projection.
(4) Paste the parameters for the projection, and edit the latitude and longitude.
(5) Click OK.

We have defined a new projection, but have not yet applied it.



(1) At bottom right click on icon to bring up box.
(2) Checkmark "Enable on Fly"
(3) Enter "Hudson" into search filter (or whatever you named the projection)
(4) Select the project.
(5) Click on Apply, and the map will be compressed horizontally.

[jimrtex]

If we want to use Google Satellite images as a layer, we need to enable their use, and then add a layer.

Open Google Earth or Bing as a Layer in QGIS (Youtube Video)

Adding the Google Layer will probably change the projection. You can restore it by repeating the final step of setting the projection.

We can rearrange the layers in the Layer Menu at the left. The Google Layer should ordinarily be last. The layers are drawn from bottom to top. Layers can be checkboxed on and off.

We might want to look at the satellite image without the lines interfering; or perhaps the lines without the image interfering. After working with a project for a while, you may have a couple dozen layers, most inactive at any time.

Your screen should now look something like this.

[Kevinstat]

Okay, I've done all that now.  The color (randomly?) chosen for my shapefile lines is yellow and the scale is 1:16,631 (I might have done something in the beginning that caused it to zoom out a bit), but otherwise my map looks like that image.

I saved my project after completing the last task and am ready for my next lesson, if there are any more to come.

[jimrtex]

Okay, I've done all that now.  The color (randomly?) chosen for my shapefile lines is yellow and the scale is 1:16,631 (I might have done something in the beginning that caused it to zoom out a bit), but otherwise my map looks like that image.

I saved my project after completing the last task and am ready for my next lesson, if there are any more to come.

The scale changes all the time when you zoom in or out. I might have changed it. You can also select a scale. You can either choose one of the fixed scales (I used 1:5,000 for most of my images). I'd only do that if you were creating a presentation. You can also enter a new value, for example you could edit "16,631" to "16,000"

From here:

2020 Census Program Phases

Download:

BBSP Using Your Own Software Participant Guide [PDF] 2.6MB

Partnership Shapefiles for New York state, Columbia County

and

Prototype 2020 Census  Blocks for New York state (state 36), Columbia County (County 021).

You have the final product, and it will help to explore how it was derived.

[jimrtex]

Part 4

We can change the display of a layer.



(1) Right Click on layer you want to change, and select Properties from the menu.
(2) Select Style
(3) Select Single Symbol
(4) Select Color. The menu allows you to select a standard color; compose a color; pick a color with an eyedropper, etc.
(5) Select width of color.
(6) Apply Changes.

Click on OK. If the color or width are not to your aesthetic tastes, change them. I thought the lines were a bit too wide, and the red a little too hot.

So under the color selection, I clicked on Choose Color, and then moved the vertical slider next to spectrum down a bit. I also reduced the width.

[jimrtex]

Part 5



This image shows the icons that I use on a regular basis. There is no action, just some numbers around the edge to identify the icon.

(1) There is a current layer. Sometimes you will switch this, or it will get switched, and you will try to do something in QGIS and it won't respond. You will be looking at a line, and using the cursor to select it, and nothing happens. This is usually because there is a different layer selected.

You can uncheck a box, and a layer won't be displayed. Uncheck the box next to changes shapefile, and the red lines disappear. Uncheck the box next to Google Satellite, and the background disappears.
Check the boxes, and they return.

(2) You can Save a project. A project does not contain the data files (shapefiles, etc.) but rather a reference to them, and also includes current settings, including presentation parameters. It is a good idea to save a project every so often. You will be prompted before exiting QGIS, but not before you lose electrical power, or even an orderly shutdown.

I save all projects (.qgs) files in a common directory. This makes it easy to switch between projects, since QGIS remembers directories.

(3) You can open a different project. If you have made changes to your current project, you will be prompted to save it, before the project is opened.

(4) You can create a new (blank) project.

(5) Use the hand, to switch to panning. Click and hold, to drag the map around.

(6) Use the magnifying glass with a + to Zoom In. You can use use a marquee zoom (click on one corner of area to zoom to, and hold as you move the cursor to the other corner of the zoom area. Or click on a point, and it will double the scale while zooming about that point.

(7) Use the magnifying glass with a - to Zoom Out. The larger area you select, the slower the zoom out will be. Select a real tiny area and you will zoom out a lot. Or you can click on a point and the scale will be halved.

(8) Use the magnifying glass with a bit of yellow, to Zoom to Selection - QGIS uses yellow to highlight selections, so yellow in an icon indicates that it is related to a selection.

(9) Use the magnifying glass with a white square to Zoom to layer. Sooner or later you will get lost. Click on this to zoom to the full extent of a layer.

Zoom in on a particular area of the map.  Click on Zoom to Layer to return to full extent of the changes shapefile (ie Hudson)

(10) Use the magnifying glass with a "<" to back up to previous displayed extent.

(11) Use the yellow icon with the lasso for Graphical Selection (more on this later)

(12) Use the yellow icon with the script E for "expression" to select by expression.

(13) Use the yellow icon with the red NO symbol to deselect everything.

(14) Use the icon that looks like the Monopoly card for Connecticut, Vermont, or Oriental avenues to to Display The Attribute table.

(15) Use the icon that looks like a ruler to measure distances, areas, and angles.

At the bottom:

(16) Current location. The center of our projection is at 42.25 degrees N, 73.75 degrees W. The coordinates are distances (in meters) from that point.

(17) Current scale. You can also use this to set a particular scale, either typing in a number or using the pulldown menu.

(18) Rotate map. You can rotate vectors (lines), but not rasters (satellite image) so it is of somewhat less utility here.

[Kevinstat]

Okay, I've done all that, except for 11.  And 12 technically, as I had no expressions I guess, but I did select all which is in that same "icon menu", plus tried the "invert selections" icon which was the same as select all when I had nothing selected, and the same as clearing all selections after I had selected everything, as one would expect.

[jimrtex]

Part 6

We are going to do a graphical selection. There are four types of graphical selection. One is a single click - more useful for selecting areas rather than lines; and three ways of selecting by area: polygon, circle, and free form. I find free form the most flexible, and can also be used for selecting multiple areas.



The pulldown menu lets us choose the selection method. Once selected, it remains the selection method. Once set, to make another selection click on the icon (without pulling down the menu).



With the cursor positioned, hold down the left mouse button, and lasso the area you want to select (in this case the orangeish area surrounding the Hudson Correctional Facility at the bottom of the map.
Release the mouse button and you should see:



Note in lower left the number of features you have selected. Selection is by intersection. You can select a single line by simply stroking with the cursor across it. However, what looks like a single line may be multiple lines connected end-to-end, so be sure to get all the parts of what you want.

Now we are going to look at the attributes of what we have selected.



(1) Click on the Vermont Avenue icon.
(2) At the top of the table click on the icon that has a yellow bar and a blue arrow. This brings the selected features to the top of the table - highlighted in blue. You can click on a column header and the features will be sorted. You can slide the columns about - which I did just to highlight the more important ones.
(3) More significant columns:
STATEFP: State code.

COUNTYFP: County code.

TLID: Permanent line ID (for the entire USA).

MTFCC: MAF/TIGER Feature Classification Code. MAF/TIGER stands for Master Address File/Topologically Integrated Geographic Encoding and Referencing, the census bureau's data base. S1400 is a neighborhood street.

FULLNAME: Name of feature (Hudson Correctional Facility is the "street name").

BBSPFLG: Block Boundary Suggestion Project Flag (for 2010). 4 = Hold as block boundary, 9 = do not hold.

CBBFLG: Census Bureau's intended block hold. 4 = hold, 9 = don't hold.

So comparing the two, you can identify features that the census bureau is deprecating.

BBSP_2020: This is my suggestion.  1 = hold, 2 = don't hold.

CHNG_TYPE: These are our suggested changes: DL = Delete Feature, CA Change attributes, AL = Add feature.  The Census Bureau would review the changes and decide whether to act on them.

JUSTIFY: Justification. This is free form text. The numbers were included as part of the text.

(4) The table windows stick around, so it is a good idea to delete them.

Looking at the whole map, with the prison roads selected, click on the Zoom to Select Icon (Magnifying Glass with Yellow). If you compare the satellite image to the mapped lines, you can see why I want to delete them.

Then click on the Zoom to Layer (magnifying glass with white box just to its right).

Then deselect features from all layers (Yellow Box with Red NO symbol). You can have features selected in multiple lines and it gets confusing.

[jimrtex]

Part 7

We can do more advanced displays:



(1) Right click on the layer and select Properties.
(2) Select style.
(3) Select Categorized.
(4) Set CHNG_TYPE as the field to classify on.
(5) Click on Classify.  It will produce a category for each unique value of CHNG_TYPE.
(6) Right click on each symbol (the line) and change color (red for DL, green for AL, blue for CL)
(7) Click on Apply



Once we have set up the separate categories, we can select individual categories.



(1) Uncheck all but 'AL'.  Selection is now limited to those categories. Do a graphical selection.

[Kevinstat]

The shapefiles for the Census Bureau's prototype blocks in Columbia County, as well as those for their 2010 blocks (tabblock2010, as well as the locally identical tabblock) don't show any of the Hudson Correctional Facility lines you want to delete as actual/planned block boundaries, while the "Vermont Avenue" table suggests that most of them were block boundaries in 2010, and many of them are boundaries of the Census Bureau's prototype blocks for 2020.  (I added those shapefiles to the project yesterday.)  Did the Census Bureau mess up with their shapefiles or something, or is there probably some error on my part?

Otherwise, I followed and understood everything in your two posts.

[jimrtex]

Part 8

It is also possible to do selection by expression.



(0) I have switched the Style for the layer back to red for all values. This is simply to show what is happening better, and has nothing to do with the selection per se.

(1) Click on the Selection by Expression icon (Yellow icon with script E for "expression"). This brings down an expression builder box. You will see this in other places, for example you can build expressions for controlling style or building labels.

You can type into the box or use the menu at the right to select pieces of expression. Since the left box is essentially text that is parsed on the fly, you might want to create complex expressions in something like Notepad or Excel, and then just cut and paste into the left box. Or if you like an expression, you can copy it somewhere else.

I generally only use the right box to find field names and sometimes search for functions.

(2) In the right box click on the > next to Fields and Values, which will open up the list of fields, scroll down to CHNG_TYPE and double click on it, which cause it to appear in the left box. References to fields are surrounded by double-quote marks.

You will notice that below the left box there is a sample expression value. CHNG_TYPE is a text value and one of the rows has a 'CA' in it.



(3) Continue to enter the rest of the expression:    = 'DL'

You could click on the "=" sign above the box, but I find it easier to use the keyboard. Literal values are enclosed in single-quote marks. As you type, the expression is continuously being parsed, so there will be an error until you enter the closing single-quote mark.

(4) With the pull down menu, click on Select.

This  will select features where   "CHNG_TYPE"   = 'DL'

That is, those features where the value of CHNG_TYPE equals 'DL'

We have selected the features to be deleted.



We can select within selections. Our current selection is features to be deleted. We want to find the railroads to be deleted. Railroads have an MTFCC that begins with 'R'.



(1) We Select by Expression.
(2) We want to use a string function, so we click on the  <  next to String
(3) We double click on  left  which copies it to the expression box, and adds an opening parentheses.

The left() function has two arguments, a string and a length. We want the first character so we want an expression of:    left("MTFCC", 1)

We continue to build our expression.



(4) Double click on MTFCC to add it to the expression.
(5) Type in the remainder of the expression.

The left box is ultimately a free-form text, so we can click on it to set a text entry cursor. For example, if we looked at the expression and realized that we wanted to select features that are not railroads, place the text cursor in front of the =  ; press the Delete key, and enter  <>; or place the text cursor after the  = and use the backspace.

(6) In the pull down menu, click on Select within Selection.   The other options of Add to Selection or Delete from Selection may also have applications.



This shows the two railroads that should be deleted: the railroad to the cement plant across South Bay, and a rail siding. The tracks have been physically removed. Their former presences lives on in the census database.

We could have also did the selection with a single expression:

[jimrtex]

Part 9

Let's load some more shapefiles:

_edges_
_block_ (prototype)
_faces_
_tabblock_
_tabblock2010_
_tracts2010_ (there is also curtracts, but in Columbia County it is the same)



Slide the layers around to match this order. Change the style for the area files. About half way down there is a layer transparency slider, slide it over about half way. All the way to the right is totally clear, all the way to the left is totally opaque. You may want to tone some of the colors down a bit.

We can still make kevinstat the current layer even thought it is not displayed. This lets us Pan to Layer which brings Hudson into focus.

Slide the edges layer below the faces layer, and set the style for the edges layer to be color red (leave the width minimal). Check box only the faces and edges file.

If you select another layer and then Pan to Layer, you will pan to Columbia County.



You can see where there are edges inside faces. Check off the box next to the edges file and then check it back on. This will help you see the edges that don't match faces.

Reverse the order of the edges and faces layers.

An edge is a polyline. A face is a closed polygon bounded by edges. Every face is bounded by edges. So when we put the edges on top, we can't see the faces.

Now put the edges layer back under the faces layer. Not every edge bounds a face, so we can see the edges that are not coincident with the faces.



Select the line shown (I've thickened the edges to make them more visible, and changed the color of the faces for more contrast). The line corresponds to State Street between N 3rd Street and N 4th Street.

(1) Be sure the edges layer is the current layer.
(2) Click on the pencil. This makes the layer editable. We aren't going to do any editing, but it necessary for the next step.
(3) Select the line. Notice the hatch marks. They correspond to the vertices of the polyline. Straight streets only need two vertices - this street has a third vertex near the middle of the block. Do a Control-C to copy the line to the clipboard.  Click on the pencil. If it asks about saving changes, say NO. If you didn't make any changes, you won't be prompted.
(4) Open up notepad and Control-V to save a copy of the line.

You must be logged in to read this quote.

Of particular interest are the vertices of the line. Census coordinates are stored as lat/long values with 6 digits to the right of the decimal point (storage is more precise than the actual measurement).

-73.787053 42.252381  
-73.787996 42.253074  
-73.789551 42.254225  

The first vertex is to the east and south, and the third vertex is to the west and north. So the line is encoded from N 4th Street to N 3rd Street. The second vertex is 38% of the distance from N 4th Street to N 3rd Street, so slightly closer to N 4th Street, which matches the graphical representation.

The TLID is new. TLID are assigned uniquely in the USA and are permanent. The original IDs in Hudson are around 4 million. New IDs are around 640 million. There used to be a (false) connection from Rope Alley to State Street. This was not used as a block boundary but did exist as a statistical line, and was removed. It would have forced State Street between 3rd and 4th Street to be encoded as two lines. Now that they can form a single line, the old TLIDs have been retired, and replaced by the new number.

Since the line has direction (from 4th Street to 3rd Street), it has a left and right side. If you open up the table, you will see that the line is indeed State St, and that the face to the left (TFIDL) is 218838193 and to the the right (TFIDR) is 261237232. There are also places for address ranges for the left and right sides of the street. The Census Bureau has not included them, but an update could include new addresses.

The file PVS_15_v2_addr_36021.dbf does have the address ranges. It can be opened in Excel. You can  also drag it to the map area - you can look at the parameters, but I have not figured a way to search it. I ordered it by TLID and then scrolled down to 640427671. In Excel, I can search for 640427671.

At any rate, addresses on the left side range from 399 to 301, and on the right side from 400 to 300, which is somewhat expected. Ordinarily 400 would be east of 4th Street, but 4th Street ends at State Street. 400 State Street is the former Hudson Area Library, and prior to that an orphanage and insane asylum. With the address ranges, we can convert from a street address of a census form, to a block number, so the people are enumerated in the correct block. The Census Bureau prefers to have streets for block boundaries, because they are then more likely to correctly map street addresses to blocks.

Not all edges form the boundary of a face, since they do not form part of a closed path. In those instances, the same face is on both the left and right sides.



In this map, the edges layers is classified based on:

   "TFIDL" = "TFIDR"

A value of one (true) indicates that the left face and right face are the same. A value of zero (false) indicates that the they are different. Thus the edges in red are in effect loose ends, streets, etc. that dead end, etc. that they don't form part of a closed path.

Incidentally, if you right click on a layer, and select Show Feature Count, it will show number of features for that layer.

Columbia County has 20,590 edges; 5,080 faces; and 3,878 census blocks (2010).  The prototype has 2680 blocks, a reduction of 31%.

We can also look at the faces.



Here we have selected the two faces on either side of State Street. As before, we click on the pencil to enable editing of the layer. Control-C to copy the contents of the selected faces, and paste into Notepad. Remember to click on the pencil to disable editing.

The  faces layer is displayed with a very transparent blue. That is why the haze clears as you travel over to Athens (across the Hudson River) in Greene County.

TFID 211838163, the face on the left side of State Street, also bounded by N 3rd Street, Long Alley, and N 4th Street has 6 vertices (with the first and last duplicated). TFID 261237232, the face on the right side has 20 vertices (19 unique). The northern edge is a statistical line. I suspect that the line was "created" in 1980 as a way to create a block in the area. In some Hudson GIS maps it is labelled as Clinton Street.

Census blocks are comprised of one or more faces. Thus the face could be a census block, but won't be because of the bogus statistical line.

Clicking on Vermont lets us see the attributes of the two faces. Census blocks are used as an atom for enumerating the census. Block groups, census tracts, places, MCD, school districts, legislative districts, urbanized areas, etc. are simply aggregations of blocks. The reason VTDs conform to block boundaries, is so that the census bureau can calculate the population of the VTD.

Since blocks are comprised of one or more faces, then a face can be part of every unit that a block is, and also part of the block. The face to the north of State Street is part of Census Tract 12, Block 1000.

I have added two additional shapefiles from the 2010 Census:

tl_2010_36021_edges
tl_2010_36021_faces

Tiger Products

Click on TIGER/Line Shapefiles, the first Product in the Green Table
Then 2010 in the blue table
Download, and then for the edges: Web Interface
Say 2010 again and "All Lines", which is toward the end of the list. Select the state and county, and download.

For the faces, back-up and select FTP Interface
Select FACES directory and then download the zip file based on the state and county codes of 36021 (36 = New York, 021 = Columbia County).

This compares the faces shapefiles for 2015 (partnership) and 2010 (Census)



You can see that in 2010 there were more faces than in 2015. You can see a similar result if you compare the two edges files (thicken the lines, and choose bright colors). For 2015, the box around the Hudson Correctional Facility was added, and the road to the east was more closely aligned with the satellite image. Otherwise, the 2015 file has some edges removed. These appear to be mainly pencil slips from the past. I suspect that before the 2010 Census, the Census Bureau started hunting for some "streets" and couldn't find them. They were converted to statistical lines, and then deleted after they weren't used.

This compares the 2010 faces to the 2010 blocks. There were faces that had been created by pencils that had slipped that weren't used as blocks, but there were also lots of blocks created by driveways and lanes in cemetery. And they didn't use the railroads as block boundaries as they had in 2000.



And finally we compare the prototype blocks to the 2015 faces.



In essence what we are suggesting is a counterproposal to the census bureau's prototype blocks.

(1) Delete edges corresponding to extraneous or non-existent features. This will eliminate some faces, which will cause some blocks to be merged. In other cases it will simply be removing digital debris.
(2) Correct placement of some streets. In two instances, they will form edge boundaries, and be usable as block boundaries. In other cases it makes the map match reality.
(3) Add missing features. In most cases these will be used as block boundaries.
(4) Indicated preferred must hold or must not hold which will modify block boundaries.

[cinyc]

The file PVS_15_v2_addr_36021.dbf does have the address ranges. It can be opened in Excel. You can  also drag it to the map area - you can look at the parameters, but I have not figured a way to search it. I ordered it by TLID and then scrolled down to 640427671. In Excel, I can search for 640427671.

There's still no way to save changes to a .dbf file in Excel, correct?  I always have to use OpenOffice to do the final step in merging my data when making an election map.  Can QGIS do something to merge tables (like, one .xls or .csv and the original Census .dbf)  that I haven't figured out how to do in MapWindow GIS?

[jimrtex]

The file PVS_15_v2_addr_36021.dbf does have the address ranges. It can be opened in Excel. You can  also drag it to the map area - you can look at the parameters, but I have not figured a way to search it. I ordered it by TLID and then scrolled down to 640427671. In Excel, I can search for 640427671.

There's still no way to save changes to a .dbf file in Excel, correct?  I always have to use OpenOffice to do the final step in merging my data when making an election map.  Can QGIS do something to merge tables (like, one .xls or .csv and the original Census .dbf)  that I haven't figured out how to do in MapWindow GIS?

I think Excel is part deliberate policy. If you want people to migrate to Office you provide one way conversion.

Yes, you can do a join.  That is how I did this map:



When I do redistricting, I do it in a spreadsheet. Starting with a population spreadsheet, with a row for each county, town, or block. I calculate the normalized population for each (share of a seat), and have a column for district number.

I use SUMIF to calculate the population of each district. So I just change the district number for a block, etc. to move it to a new district.  In olden days I would then use Paint to draw the map.

But now I save the parts of the Excel file as a CSV file, and join it to the census shapefile (actually a copy of it). For the map above, I added columns with the block population and the ward the block was assigned to. After the join, I used the block population for block label, and the assigned ward for the block color.

You can also do a merge operation. In a case like this, I would merge all the blocks with the same ward number to create a ward layer. I can then overlay the ward layer over the block layer:



In this case, I also put the total ward population in each block row so it carried over into the row for the (merged) ward. I probably could have calculated the total population in QGIS, but it was easy enough to do it in a spreadsheet.

[jimrtex]

Part 10

Since we are concentrating on Hudson, we can eliminate the rest of Columbia County.



(1) We will be using the census tracts layer for clipping. Hudson is entirely in Census Tracts 12 and 13, and Census Tracts 12 and 13 are entirely in Hudson. For simplicity display of all layers but the census tracts has been turned off.
(2) Select graphically by freehand.
(3) Select census tracts 12 and 13.

(4) Select Vector, Geoprocessing Tools
(5) Select Clip.

Clip keeps what is in the selection layer, Difference removes what is under the selection layer



(6) Select the input layer (when you click on the little arrow, you will get a list of layers to select from)
(7) Select clipping layer.
( Be sure Use Only Selected Features is checked. This is the default when there are selected features.
(9) Click on browse, make sure that you are in the correct directory, and enter the name of the output layer.
(10) Make sure add result to canvas. This isn't necessary, and if you were doing this on a batch basis you might turn it off. But it is a good idea to check.
(11) Click on OK.  The progress bar will move across as the processing takes a few seconds.



If you check your directory you will see a whole suite of new files (.shp, .dbf, etc.).

I did a few additional steps:

(12) Right click on the original layer, and select remove, and confirm when prompted. This will reduce the clutter, and avoid editing a source file. Even if I weren't clipping, I would do a Save As to make a copy. I might also do this if trying some new feature in QGIS.
(13) Right click on the Hudson-limited layer and select Show Feature Count. There are 2680 blocks in Columbia County, and 121 in Hudson.
(14) Pan to Layer, so that canvas is centered on Hudson.

Repeat the same process to produce

hudsonpartnertabblock2010 {faces, tabblock, edges}
hudsoncensusfaces {edges}

Note, the census tracts will remain selected unless you unselect them, so that part can be skipped.

[jimrtex]

Part 11

We can use complex expressions to control the display for analytical or presentation purposes:



(1) Right click on the layer and select Properties.
(2) Select Style.
(3) Select Categorized.
(4) Click on the Script E, to enter an expression. In this case we create a combination of the tract and block number. Census tract numbers are 6 digits, with an implied decimal point after the fourth digits. Conventionally, leading zeroes and fractions of .00 are suppressed.

001200 is displayed as "12"
001201 would be displayed as "12.01".

Since we know that the Hudson census tracts have no fraction, we can use a simplified expression to extract the tract number. We then append a hyphen (literal '-') and the block number. The || is the concatenation operator.

(5) We select a random color ramp.
(6) A click on classify. QGIS searches through the tables and creates a category for each unique value and assigns a color - because the color ramp is random, the colors assigned are random.
(7) We can also Delete All Categories. For example, if we have classified already, and then change our expression for classification, we can delete our current categories. Or if we change a value we are classifying on, we can simply click on the Classify, and any new classes will be added.
(8) Click on Apply.

We can also add labels.



(1) Right click on layer select Properties.
(2) Select Labels.
(3) Select Show Labels for this Layer
(4) Select Script E and enter expression for labels.
(5) Select Text
(6) Select Parameters for labels.

For our purposes right now, the default values are fine.

(7) Apply.

We can select/deselect which categories we want to display, with the check boxes.



In this case, our category is block number, so we are disabling display of certain blocks. One could imagine a map of Maine where towns were categorized by population. You could then choose to display towns with populations between 1000 and 2000, for example.

You can change the color of a category by right clicking on the color square.

In this case we selected the blocks that would be changed if our suggestions were accepted. We also changed the color of blocks that would be merged if the existing block boundaries were eliminated - either by deleting the line, or changing it to a not hold. For example, the three blocks in the Hudson Correction Facility can be merged. There are a total of six mergers, eliminating 7 blocks.



We can use complex expressions to control display.

You must be logged in to read this quote.

This expression combines values from three fields. JUSTIFY is a 150-character field to express justification for a suggested change. I included a two-digit number for each change.

CHNG_TYPE is either 'AL', 'CA', 'DL' for Add Line, Change Attributes, Delete Line.

BBSP_2020 is a 1 if he want to hold a line as a block boundary. If an added line is intended to be a block boundary we add a '+' to the category. If an existing line is to be held as a block boundary it is flagged with BBSP_2020 (in some cases, it is the flag that is the changed attribute).

Below the expression is an example output.

This shows some of the categories.



Here we do a little bit of label formatting.



(1) Right Click on layer, select Properties.
(2) Select Labels
(3) Select Show Labels for this Layer
(4) Click on Script E to enter Expression. In this case, we are displaying the leftmost two characters of JUSTIFY.
(5) Select Text
(6, 7, 8) Select Font, Select Bold, Select Size

For right now we just want some numbers to help us find the changes.

(9) Apply Changes.

This produces this rather raw looking map.



I then selected suggested items with a particular theme, in this case deleted lines which had no impact on block boundaries. Rather than doing a screen dump, I saved an image of the canvas:



(1) Select Project.
(2) Save as image.

This will save a .png image, as well as a .pngw file. The .pngw file describes the coordinates of the image, so that one could reload the image at a later time.

This image could be used as part of a presentation to those who would be reviewing the suggestions, whether local officials or the redistricting liaison. The numbers would provide a reference to text explaining what each feature represents, and why it is being removed.

[jimrtex]

Part 12

In this lesson we are going to edit the Prototype block boundaries. The Prototype file represents current Census Bureau intent for 2020. Our version is in essence our counter proposal incorporating all our changes.

We first make a copy of the prototype layer.



(1) Right Click on Prototype Layer, Select Save As.
(2) Select Browse then enter the file name of the layer (if you don't select Browse first and just enter a name, the files will end up somewhere you won't be able to find.
(3) OK.

We will be splitting blocks using lines from another layer. This explains how to do it.

QGIS lesson 26 – Splitting a polygon with lines from another layer (youtube)

The entire set of lessons can be found at:

Youtube Search: "Steven Bernard QGIS lesson"

The first should be:

"QGIS Uncovered"

And there is an item that says:

"View full playlist (31 videos)"

Click on that and you will have the full series available.

We first install the Digitizer Tools Plugin



(1) Click on Plugins
(2) Manage and Install Plugins



(3) Scroll down and select "Digitizing Tools"
(4) Install.

For reasons unknown, splitting a polygon with Digitizing Tools does not work when using projected coordinates. Both the census block and the edges are stored as LAT/LONG, but we have been projecting them because (1) LAT/LONG maps give me nausea; (2) We can't overlay them over the Google Map. The splitting works OK, but one of the resulting polygons is stored with projected coordinates. Since QGIS expects everything in the layer to be LAT/LONG the polygon is not displayed, and can not be graphically selected.



(1) Click on ICON in extreme lower right corner.
(2) Uncheck box for "Enable on Fly CRS transformation"
(3) Apply

We are now ready to begin editing.



(1) We have checked the box to display our blocks file. We have also set it to use a different color for each block, and to label with the block number.
(2) We have also checked the box to display our suggested lines.
(3) While the hudsonsuggestedblocks layer was selected, we clicked on the pencil icon To Toggle Editing. This enables editing of the layer - which is indicated by a small pencil on the icon next to the layer.  The pencil icon is a toggle. To disable editing, click on the pencil. You can also toggle editing for a layer by right clicking on the layer and click on Toggle Editing.

My practice is to do editing in small pieces. Toggle Editing on; Make the Edit; Toggle Editing off. You will be prompted to save the edit. If you have messed up, you can simply discard the edit.

(4) When you are editing, the icons for the Digitizing Tools are in color. When I installed the plugin, the icons were put on a 3rd row.  I dragged the vertical row of dots up to the second row.
(5) This is the icon for Splitting a Polygon.

Our first edit, will be to merge blocks 13-4003 and 13-4004. They are currently separated by a block boundary drawn through a church parking lot.



Select the blocks to be merged.



To merge the selected blocks



(1) In the Edit Menu
(2) Select Merge Selected Features

The following table will pop up.



(1) Select a block to take attributes for the merged block. It will be highlighted as orange.
(2) Click on Take Attributes from selected feature.
(3) Click on OK.

We are overloading Block Number. Is Block 4003 the original block or the merged block? Since we are mainly interested in display, it doesn't matter too much. I chose 4003 because it is the larger portion of the merged block, and also because it is a lower number.

The blocks have been merged.



The colored lines are from the kevinstat layer. You can confirm that the block is whole by selecting it.

Click on the Pencil to Toggle Editing for the layer.  You will be prompted to save the change.

[jimrtex]

Part 13

The next change is around the triangle formed by Columbia Street-Columbia Turnpike-Prospect Avenue



The tips of Prospect Avenue and Columbia Turnpike intersect just south of their respective intersections with Columbia Street. In addition, the census bureau has a bogus version of Columbia Turnpike connecting with Columbia Street on a north-south direct just to the east.

In 2010, there were two small census blocks. For 2020, the census bureau proposes adding the triangle to 13-4008 (prototype block number), and the bogus block to 13-4007.

Since we will delete the non-existent Columbia Turnpike connection, 13-4006 will extend further west. Columbia Turnpike is the ward boundary (and has been since 1855), so it is preferred that 13-4006 also contain the triangle.

We will first chop off the tips of 13-4008 and 13-4007, and then merge them with 14-4006.

We first select the line we wish to cut 13-4007 with in the Kevinstat layer.



We select the block (13-4007) that we wish to cut.



To actually split the block:



(1) Make sure the current layer is the one with the polygon to be split.
(2) Make sure editing is active for that layer, clicking on the pencil icon if necessary to toggle it.
(3) Click on the "Split selected features with selected line from another layer" tool.



(4) Make sure the correct splitter layer is chosen.
(5) Click OK

The block has been split.



The original block no longer exists - so it is not selected. The split line remains selected. The two new blocks take their attributes from the original block, so they are displayed with the same color, but are labeled separately (there are now two 13-4007 blocks).

We also need to split off the tip of 13-4008. We repeat the splitting process. This is what we see after the split has been completed.



We will now merge the two small blocks with 13-4006.



Select the blocks to be merged, to merge the blocks:



(1) Under Edit.
(2) Select Merge



(3) Select the block to take attributes from (it will be highlighted in orange)
(4) Click on Take Attributes From Selected Feature
(5) Click on OK



The two small areas have been split from their original blocks, and merged with 13-4006. Toggle editing to save the changes, and turn off all selections.

[jimrtex]

Part 14

Exercise: The Census Bureau has proposed attaching the triangle formed by Columbia Street, Green Street, and State Street to Block 13-5009. This would move the block boundary off Columbia Street, which is Hudson's longest continuous street, and also a ward boundary since 1855.



The preferred alternative is to attach the triangle to 13-5010. This would cut off the last piece of State Street.

Exercise: Split the triangle from 13-5009 and merge it with 13-5010 to produce:



Exercise: The street dividing 13-4008 and 13-4009 does not exist and should be deleted. This will result in the two blocks being merged.



Exercise: Merge 13-4008 and 13-4009 to produce:



Exercise: Rope Alley (green line) is missing between N 6th Street and N 7th Street. Adding it will split block 13-5003. The red line within the block is spurious, and will be deleted.



Exercise: Split 13-5003 on Rope Alley to produce.



Note that both new blocks are numbered 13-5003. We will correct that in the next part.

[jimrtex]

Part 15

In Part 14 we had split Block 13-5003 on Rope Alley producing two blocks 13-5003.



The Census Bureau may renumber census blocks between censuses. They did that between 2000 and 2010 and then produce equivalency files mapping blocks between the census. The Census Bureau may have a desire to have stable block numbers, but that may be unrealistic. For the 2020 prototype block files, they appear to have renumbered blocks.

Blocks are numbered within their block group, by a three digit number beginning with 000. Block Groups are numbered within their census tract by a single digit beginning with 1. Conventionally, the block group number and block number are combined so Block 5003 is Block 003 of Block Group 5.

The block numbers for Census Tract 13 are:
13-1000 through 13-1016
13-2000 through 13-2014
13-3000 through 13-3011
13-4000 through 13-4011
13-5000 through 13-5015

We could number one of the new blocks as 13-5016, and leave one of the split blocks as 13-5003. Instead we will number the blocks as 13-5103 and 13-5203. This indicates that the blocks are in block group 5, and derived from block (0)03. A more thorough change would be to add a field that could be suffixed to the block number (eg 13-5003A and 13-5003B). But since we are primarily interested in drawing a colored map of blocks, the more limited approach is sufficient.

To edit the attributes of a feature:



(1) Select the layer to be edited.
(2) Make sure editing is toggled on
(3) Select the feature (block) to be edited.
(4) Open the attribute table.
(5) Bring the selected feature(s) to the top of table.
(6) We can also toggle editing here.



(7) To change a cell value, we click on the cell and start typing. To finish the change click on another cell.

We also need to update the BLOCKID. It is a concatenation of the STATEFP, COUNTYFP, TRACTCE, BLOCKCE.

You can paste from the clipboard into a cell. Select the cell. Type a space, backspace, and then Ctrl-V to paste.

A feature does not need to be selected in order to edit its attribute, it is simply a lot easier. In addition if a column is sorted, the order will be updated when a cell is changed. Instead of seeing the updated feature, there may be other features.

Our edit complete, the northern part of the block is now 13-5103.



We repeat the editing process for the southern part of the block, renumbering it to 13-5203.



While the labels, are correct, the colors are the same.

We classified the blocks for coloring based on their BLOCKID. Since we changed the BLOCKID after we had done the classification, these blocks were assigned a default color for unclassified features.

To update the classification:



(1) Right Click on the Layer, Select Properties.
(2) Select Style
(3) Click on Classify.
(skip 4)
(5) Click on Apply

This will assign colors to the new BLOCKID's. When updating like this, QGIS doesn't seem to do a very good job of picking distinct colors. I scrolled down to the bottom of the classification, and changed the color of one of the BLOCKID's.

An alternative would be:

(4) Click on Delete All.
(3) Click on Classify.
(5) Click on Apply

In this case, the colors are arbitrary, and it may be simpler to use a new set.

We have now completed splitting the block, renaming the resulting blocks, and changing their colors.

[jimrtex]

Suggested Blocks



Census Prototype

[jimrtex]

Part 25

Beginning with PVS_15_v2_mcd_36021 we can create a simple political map of Columbia County.

(1) Drag shapefile to Canvas.
(2) Change style to classify on NAME.
(3) Change labels to display NAME

Done:



We can add additional data such as population or political data relatively easily. We first create the data in an Excel spreadsheet, and then output it as a .csv file. QGIS may be able to read an .xls or .xlsx file, but using a .csv file is easy enough. Simply copy the contents of an xlsx file to a blank worksheet and save as a .csv file.

Here is the .csv file that I am using. You should be able to copy it into Notepad and save it as a .csv file that may be read by either Excel or QGIS. Once you  have established an .xls or .xlsx file, you can easily produce new .csv files to input into QGIS.

You must be logged in to read this quote.

The first line of the file provides the names of the fields. This particular file was derived from the Geographic Identifiers product using American Fact Finder. I then stripped out a lot of the columns.

If we are going to join this to another layer, we need a common value. COUSUB is a census identifier for county subdivisions, and is also present in our original MCD layer. While you can match on names, you probably will have problems with spelling variations.

I also have fields for population, land area (in square meters), water area, and latitude and longitude.

We really don't need these latter fields, but I will use the area column to calculate population densities. If you had the right data you could also calculate things like population growth rate, political vote share, or swing. You might also choose to calculate such values in Excel and include them as columns in your .csv file.

The latitude and longitude permit us to create maps based on the .csv file alone.

You will also need a .csvt sidecar file describing the format of the fields. Hint: if you use the long census ID's, you will want to describe the field as a String rather than as Integer. Excel can handle large numbers, QGIS can not.

You must be logged in to read this quote.

We now load our .csv file into QGIS



(1) Click on comma icon to Add Delimited Text Layer (eg Comma Separated Values)
(2) Browse to find our CSV layer.
(3) Make sure CSV is selected.
(4) We have point coordinates so we select this radio button. Otherwise we would select on No Geometry. I think that QGIS scans for headers like X, Y, or LAT, LONG, and selects the default radio button based on that.
(5,6) We set our X and Y values to LONG and LAT (don't make a mistake of using LAT and LONG, unless you want to be transposed off the southern tip of Africa).
(7) Click OK.

Once we have selected the file, the table shows up in the bottom box, and we can verify that field types, etc. are OK.

If we don't have coordinate values, about all that needs to be done is browse for the file and click on OK. It takes about 15 seconds.

Since we have coordinate information, we also need to specify the coordinate system. All census data is NAD83.



(1) Select the  projection
(2) Select the  projection
(3) Click OK.

I added a .prj file. I don't know whether this selected the default values or not.

Our csv layer is now loaded. There is no relationship between our original MCD shapefile and our CSV file, other than they are geographically coincident. The  default presentation for the CSV file are the brown circles.



We can turn the MCD shapefile off.



And we can can add format the display. The circles are proportional to the Population, and the text is comprised of the town/city name and its population.



And here we have superimposed the two maps.

[cinyc]

Does QGIS automatically read the .csvt file alongside a .csv file of the same name?

Sometimes, you HAVE to define the census geography as String instead of Integer.  If the geography has leading zeroes, defining them as Integer will destroy the zeros and ruin the merge.  At least that's what I figured out when I didn't know how to define a column, and QGIS automatically read the GeoID columns as Integers.  Data for the alphabetically early state jurisdictions was missing.

What are the permissible .csvt fields?

[jimrtex]

Does QGIS automatically read the .csvt file alongside a .csv file of the same name?

Sometimes, you HAVE to define the census geography as String instead of Integer.  If the geography has leading zeroes, defining them as Integer will destroy the zeros and ruin the merge.  At least that's what I figured out when I didn't know how to define a column, and QGIS automatically read the GeoID columns as Integers.  Data for the alphabetically early state jurisdictions was missing.

What are the permissible .csvt fields?

I think that QGIS automatically checks for the .csvt sidecar file. I found a video that suggested that without it, that all fields will be interpreted as strings.

You are right about strings, I just was getting ready to explain how to do a join and about why my join didn't work.

Another related pitfall is when you are using numbers in expressions.

"COUNTYFP" = 021

Means that you want counties with a value equal to octal 21 (17 decimal).

How to Specify Data Types of CSV Columns for Use in QGIS

I've never used the formats. The length might be useful for strings. Maybe formats would be useful for outputting files.