Nate, small nit, but Garrison is an excellent commuter town with the train station that goes right into the city. West Point is right across the river, and the Garrison stop is the preferred mode for cadets on pass traveling to the city.
Adding to this, I haven't been to NYC in about a year, but last time I was there it seemed that Midtown and Downtown still basically only operated Tues-Thurs. I would think a long commute is far more manageable if it's 3x/week as opposed to 5. But I'm not sure how much this has changed.
I knew somebody who used to come from Pennsylvania to Manhattan (and the east side, at that) every day, as well somebody who came from Sullivan County, and another who would go from Suffolk County to Yonkers via LIRR and Metro North. Garrison is on the longer side for sure, but it's much less insane than those.
The Wisconsin-Minnesota difference is an interesting one, with Milwaukee’s demise versus the Twin Cities’ growth an interesting divergence correlating/causing very different political and economic fates.
Conversely, Madison is the only place in Wisconsin with really high population growth and is highly progressive, and may determine the future of Wisconsin politics if it continues on this path for the next 20-30 years.
Like Baltimore, or to a lesser extent, Philadelphia. Cities with an inferiority complex vis-a-vis the "big city" of their respective regions. Non-linearities at work here.
Every other row in these tables is illegible in the Substack app's default ("dark mode") view. The names of the odd-ranked states just appear as black bars (I'm assuming because they're using black text over transparency?)
Nate, have you heard of population-weighted (also called "perceived") density? It is trying to answer the question, "what is the average resident's experience of density"?
Mathematically, while conventional (area-weighted) density is [(density of tract A * area of tract A) + (density of tract B * area of tract B)] / total area, population-weighted is [(density of tract A * population of tract A) + (density of tract B * population of tract B)] / total population. See https://dx.doi.org/10.2139/ssrn.3119965 for more detail.
Looking at conventional vs. population-weighted density on the city level also explains why Las Vegas and Los Angeles metro areas are denser than New York's by the first definition but not the second. You can see population-weighted density for cities at https://luminocity3d.org/WorldPopDen, although I'm not sure if anyone has calculated it for US states.
He's basically taking the logarithm of the population-weighted density - but instead of weighting the density of each census block by the population of that block, he weights the density of the larger region around the block by the population of the block. If densities of blocks vary smoothly, then these are basically the same (though the highest peaks will be smoothed out under his measure) but there's a much bigger difference in a region consisting of an occasional block containing a dense town surrounded by empty space (so, something like a landscape full of traditional farming villages, or mining towns).
Not sure if Nate has heard of it, but yeah, I think population-weighted density is a meaningful improvement over area weighted density, but it fails to even attempt to capture the (highly relevant IMHO) substructure of where people work and do other things as distinguished by where their residence is located.
Very true. One attempt to synthesize the two is ambient density, which tries to estimate the average density of people's location across 24 hours (https://landscan.ornl.gov/about), but I don't know of any visualizations with great detail and accuracy for it. But I agree that employment density on its own is important, and the New York area being generally monocentric, with one of (if not *the*) world's densest job centers, goes a long way in making it feel like one big city in a way that a weakly centered city like Los Angeles does not.
Yeah, Landscan is a coarse move in the direction I was thinking of.
I'm not sure how you knew :) that my employment history has mostly been in greater Los Angeles, but I agree about it (Los Angeles) being very weakly centered, Los Angeles being something of the prototype for everyone going every which way all at once :) commuting pattern (see also in particular the DFW Metroplex aka Texas Triangle).
I was think also that greater NYC, impressionistically to me**, has a greater fraction of its total commute flux carried on public transport than even Boston (I don't have enough exposure to either Philly or Chicago to compare); commuting via public transport adds an component of urban experience that is missing from commuting via automobile.
LandScan USA theoretically includes dynamic data per their topline, but if you look at their metadata (at least for 2019, which I used for baseline purposes pre-Covid transients) there are no actual dynamic values there. A priori it seems like for example you could mine Google Maps traffic data looking for persistent, time of day based asymmetries and thereby derive some weak insights into commute patterns (obviously not so useful for those relative rare parts of the USA where public transit is still significant; to expand that dataset I'd be minimally willing to bet :) that the various public transit agencies have some datasets there that could be mined statistically, although probably without anonymized individual trip information).
** Data?? We don't need no steenkin' data when we have anecdata and emotionally loaded priors.
Yea the top 5 CSAs by public transit commuting (among those who aren't working from home) according to the 2022 ACS is New York (29.8%), SF (12.5%), Boston (11.4%), Chicago (9.8%), and DC (9.6%). Using traffic data is a good idea, I agree.
And there, as in so many ways, we see that NYC is at outlier. I'm a bit surprised that Philly came in below DC; a more heavily economically deadened downtown, but I had sort of expected more commuting along the Mainline. Philly the city contains a much larger fraction of its CSA population than Boston ...
As someone who browses on Zillow often the big thing I’ve noticed is sewer vs septic which could probably be a proxy for urban vs rural. I’ve always lived in an urban setting but the idea of a septic seems gross although obviously sewers are pretty gross too. ;)
I think septic is a great indicator of rural, but sewer not so much of urban. I've been to some fairly podunk places that still had sewer hookups, basically most towns over something like 10 or 20k?
I came to say exactly this. I used to use the natural logarithm, but I always found myself thinking “So how big is e^6 again?” over and over again. If the metric uses the common log, I see (e.g.) 6, and immediately know it’s 1M. The base of the log doesn’t affect the analysis or the trends, just the readability of the metric.
Population (and other exponential) growth is frequently modeled using the natural logarithm, but unless you’re doing calculus, the base of the log doesn’t matter because changing the base is a linear transformation. Any model you build will just have different coefficients, which end up canceling out the change of base.
Population growth is naturally modeled by exponential functions with some base or other. Base e is just a correction factor that gets applied when converting from compounding over a period to compounding continuously, but when you are basing your measure on observed data, you don't need to do either.
A largely missing vector from what I can tell in this discussion is putting the entire weight of urbanization on the location of residence. Someone living in Garrison, NY and commuting by train to Manhattan *experiences* far more urbanization than her next door neighbor in Garrison who drives a few miles north to her job at an exurban convenience store.
But we must go where the easily available data (e.g. from the Census) lead us.
Number of people living nearby -> Average across population in state -> Take logarithm
I feel like it should be:
Number of people living nearby -> Take logarithm -> Average across population in state
I'm not sure I can fully articulate a principled basis for thinking this. It think it's that the logarithm is capturing the feeling of the people living nearby, not the aggregation across the state. That way I could use the same UI metric at a different level of analysis than states - districts or regions or something.
I wonder how this would change the state ranking though.
I was definitely thinking this same thing! I'm trying to see what the big differences are.
Imagine a region with 10,000 people living at a regional density of 1,000 and 10,000 people living at a regional density of 10,000.
Under his order (take regional density, average, take log - I'll use log 10 rather than log e just for simplicity) you first get to an average of 5,500 and then take log to get about 3.3. Under the alternate order, you first take logs to get 3 and 4, and then average, to get 3.5.
If the town spread out to incorporate some suburbs, so that the city was less dense, we would now have 9,000 people living at a regional density of 1,000 and 11,000 people living at a regional density of 5,500. Under his order, you get the same 3.3. But under the alternate order, you would have 9,000 people at 3 and 11,000 people at about 3.3, and average to get about 3.15.
It seems right to me that doubling the region a town occupies should decrease overall density, so that seems like some motivation for this alternate order. But I haven't considered other configurations and transformations to see if his measure behaves more reasonably for them.
There is a spot on the Wilbur Cross Parkway, near New Haven, where if you are traveling southbound (toward NYC), you go through a tunnel and as soon as you come out of it you see a delicatessen and an NY style pizzeria in a strip mall by the side of the highway. This tunnel is the boundary between New England and the New York City metropolitan area.
Love this post, and I love that UI metric! Also, thank you for pointing out in your footnote about the absurdity of putting San Francisco and Silicon Valley in different metro areas. I live in San Mateo County which is technically in the SF metro, but feels every bit Silicon Valley. From a "vibe" feel, I've always considered "local" to be much larger. It doesn't speak much to the urban/rural divide that the article addresses, but I always loved the old Rand-McNally Major Trading Areas. Honestly, anyone from the Oregon border down to Big Sur and Visalia, and east even including Northwestern Nevada I consider "local". (i.e. I'd root for them if I saw someone in that entire range on a game show or in... say,... a poker tournament. :) )
It's interesting that Oregon and Washington are right next to each other in the %urban but Washington is well ahead of Oregon in UI. It really kind of makes intuitive sense where that comes from if you've spent time in both states--they tend to be seen as pretty similar, with most of the population concentrated in Portland/Seattle and a number of smaller cities interspersed elsewhere, with the eastern two thirds of both states being quite rural, but Seattle spreads out much more and the smaller cities and suburbs tend to be noticeably more urbanized than the smaller cities in Oregon, so there is a noticeable difference even though the groupings of "urban" vs. "rural" might show as similar by Census rules. Although that said one counterpoint may be that the Oregon coast tends to be more developed than Washington's coast, but that might be functionally undercounted by the UI because it exists in a narrow strip so a 5 mile radius along the coast would include a sizable chink of the Pacific ocean (plus the publicly owned lands immediately along the coast), as well as likely areas of the Coast Range that are quite sparsely populated. But then on the other other hand, Bend is really the only place in Eastern Oregon with any real population whereas Washington has Spokane, the Tri-Cities, and Yakima, which as smaller cities should decrease the UI in a way that you wouldn't see so much in Oregon. So it's interesting to think about. Obviously no method is perfect but yours seems a lot more functionally accurate than the Census'.
NY & NJ are going to be dark blue in 2024 unless Trump wins in an incredible landslide. But I think within 30 years they may be a bit purple because of the amazing growth rate of Orthodox Judaism. That's why being extremely urban may be a double edged sword to liberalism in future America.
I think both of those states have hit "peak blue" and will become lighter blue over time, but I have no idea if that means in 30 years they will be competitive. That's just way too far not to be prone to tail events or drastic political realignments that kind of "smash the Boggle grid" and make us start over. But in some limited areas we're starting to see the suburbs become almost as blue as the core cities.
I initially thought you were going to use population weighted density, though I like the straightforward interpretation of PWD maybe more (how dense does this state look, to the average person in the state?). I do not know exactly how you might compute that though: perhaps simply cutting the state into a grid might work. In any case, it probably would get you similar results, your approach is essentially the same.
I think his metric basically is the population weighted density, except that he's not weighting the density of each census block, but rather the density within a several mile radius of that census block. In a sense, his might even be a better measure of "how dense does this place look, to the average person", since the average person can see out a few miles from where they live, rather than just the immediate block.
Inspired by Nate's regression of vaccine regression I've been running regressions to try to get at the heart of public policy arguments. Here is my investigation of if urban areas (represented by democrats) increase the risk of violent crime. I was also inspired by rereading the rural violence described in Hillbilly Elegy. https://aarontesch.substack.com/p/show-me-the-data
Nate, small nit, but Garrison is an excellent commuter town with the train station that goes right into the city. West Point is right across the river, and the Garrison stop is the preferred mode for cadets on pass traveling to the city.
Added a footnote on this! It's great to have the option, but I'm not sure it's something for everyone.
All fair…thanks for the info. If you’re able, take in a football game at West Point this fall. Hugely underrated game day experience.
Adding to this, I haven't been to NYC in about a year, but last time I was there it seemed that Midtown and Downtown still basically only operated Tues-Thurs. I would think a long commute is far more manageable if it's 3x/week as opposed to 5. But I'm not sure how much this has changed.
I knew somebody who used to come from Pennsylvania to Manhattan (and the east side, at that) every day, as well somebody who came from Sullivan County, and another who would go from Suffolk County to Yonkers via LIRR and Metro North. Garrison is on the longer side for sure, but it's much less insane than those.
Seems right on the edge of commutable in my opinion. For the right job, and the right benefits, I could do it. Especially if it was hybrid.
The Wisconsin-Minnesota difference is an interesting one, with Milwaukee’s demise versus the Twin Cities’ growth an interesting divergence correlating/causing very different political and economic fates.
Conversely, Madison is the only place in Wisconsin with really high population growth and is highly progressive, and may determine the future of Wisconsin politics if it continues on this path for the next 20-30 years.
Milwaukee is in a tough spot, with Chicago and Minneapolis as regional centers and Milwaukee a virtual exurb of Chicago
Like Baltimore, or to a lesser extent, Philadelphia. Cities with an inferiority complex vis-a-vis the "big city" of their respective regions. Non-linearities at work here.
My initial guess upon reading the headline was Rhode Island because of the size of Providence compared to the rest of the state.
Every other row in these tables is illegible in the Substack app's default ("dark mode") view. The names of the odd-ranked states just appear as black bars (I'm assuming because they're using black text over transparency?)
Yeah, I was using a different process for uploading the graphics than usual and I guess it was producing some weird effects. Should be fixed now.
Yeah, looks like it's fixed! 🥳
Nate, have you heard of population-weighted (also called "perceived") density? It is trying to answer the question, "what is the average resident's experience of density"?
Mathematically, while conventional (area-weighted) density is [(density of tract A * area of tract A) + (density of tract B * area of tract B)] / total area, population-weighted is [(density of tract A * population of tract A) + (density of tract B * population of tract B)] / total population. See https://dx.doi.org/10.2139/ssrn.3119965 for more detail.
Looking at conventional vs. population-weighted density on the city level also explains why Las Vegas and Los Angeles metro areas are denser than New York's by the first definition but not the second. You can see population-weighted density for cities at https://luminocity3d.org/WorldPopDen, although I'm not sure if anyone has calculated it for US states.
He's basically taking the logarithm of the population-weighted density - but instead of weighting the density of each census block by the population of that block, he weights the density of the larger region around the block by the population of the block. If densities of blocks vary smoothly, then these are basically the same (though the highest peaks will be smoothed out under his measure) but there's a much bigger difference in a region consisting of an occasional block containing a dense town surrounded by empty space (so, something like a landscape full of traditional farming villages, or mining towns).
Not sure if Nate has heard of it, but yeah, I think population-weighted density is a meaningful improvement over area weighted density, but it fails to even attempt to capture the (highly relevant IMHO) substructure of where people work and do other things as distinguished by where their residence is located.
Very true. One attempt to synthesize the two is ambient density, which tries to estimate the average density of people's location across 24 hours (https://landscan.ornl.gov/about), but I don't know of any visualizations with great detail and accuracy for it. But I agree that employment density on its own is important, and the New York area being generally monocentric, with one of (if not *the*) world's densest job centers, goes a long way in making it feel like one big city in a way that a weakly centered city like Los Angeles does not.
Thanks, Jeb!
Yeah, Landscan is a coarse move in the direction I was thinking of.
I'm not sure how you knew :) that my employment history has mostly been in greater Los Angeles, but I agree about it (Los Angeles) being very weakly centered, Los Angeles being something of the prototype for everyone going every which way all at once :) commuting pattern (see also in particular the DFW Metroplex aka Texas Triangle).
I was think also that greater NYC, impressionistically to me**, has a greater fraction of its total commute flux carried on public transport than even Boston (I don't have enough exposure to either Philly or Chicago to compare); commuting via public transport adds an component of urban experience that is missing from commuting via automobile.
LandScan USA theoretically includes dynamic data per their topline, but if you look at their metadata (at least for 2019, which I used for baseline purposes pre-Covid transients) there are no actual dynamic values there. A priori it seems like for example you could mine Google Maps traffic data looking for persistent, time of day based asymmetries and thereby derive some weak insights into commute patterns (obviously not so useful for those relative rare parts of the USA where public transit is still significant; to expand that dataset I'd be minimally willing to bet :) that the various public transit agencies have some datasets there that could be mined statistically, although probably without anonymized individual trip information).
** Data?? We don't need no steenkin' data when we have anecdata and emotionally loaded priors.
Yea the top 5 CSAs by public transit commuting (among those who aren't working from home) according to the 2022 ACS is New York (29.8%), SF (12.5%), Boston (11.4%), Chicago (9.8%), and DC (9.6%). Using traffic data is a good idea, I agree.
And there, as in so many ways, we see that NYC is at outlier. I'm a bit surprised that Philly came in below DC; a more heavily economically deadened downtown, but I had sort of expected more commuting along the Mainline. Philly the city contains a much larger fraction of its CSA population than Boston ...
As someone who browses on Zillow often the big thing I’ve noticed is sewer vs septic which could probably be a proxy for urban vs rural. I’ve always lived in an urban setting but the idea of a septic seems gross although obviously sewers are pretty gross too. ;)
I think septic is a great indicator of rural, but sewer not so much of urban. I've been to some fairly podunk places that still had sewer hookups, basically most towns over something like 10 or 20k?
Great article! Sorta pedantic note, but I prefer log10 to log since you can at least have an idea of the order of magnitude without a calculator.
natural log of 10 is very close to 2.3 and natural log of 2 is very close to .7 if you're trying to do mental arithmetic
I came to say exactly this. I used to use the natural logarithm, but I always found myself thinking “So how big is e^6 again?” over and over again. If the metric uses the common log, I see (e.g.) 6, and immediately know it’s 1M. The base of the log doesn’t affect the analysis or the trends, just the readability of the metric.
I was wondering about this too, but apparently population growth is naturally modeled by exponential functions with base e.
Population (and other exponential) growth is frequently modeled using the natural logarithm, but unless you’re doing calculus, the base of the log doesn’t matter because changing the base is a linear transformation. Any model you build will just have different coefficients, which end up canceling out the change of base.
Population growth is naturally modeled by exponential functions with some base or other. Base e is just a correction factor that gets applied when converting from compounding over a period to compounding continuously, but when you are basing your measure on observed data, you don't need to do either.
Garrison absolutely is a NYC commuter town. It's maybe one-hour from Grand Central Station on the MetroNorth train line.
The authentically Appalachian part of New York State is actually the counties that comprise the Southern Tier.
A largely missing vector from what I can tell in this discussion is putting the entire weight of urbanization on the location of residence. Someone living in Garrison, NY and commuting by train to Manhattan *experiences* far more urbanization than her next door neighbor in Garrison who drives a few miles north to her job at an exurban convenience store.
But we must go where the easily available data (e.g. from the Census) lead us.
UI index uses:
Number of people living nearby -> Average across population in state -> Take logarithm
I feel like it should be:
Number of people living nearby -> Take logarithm -> Average across population in state
I'm not sure I can fully articulate a principled basis for thinking this. It think it's that the logarithm is capturing the feeling of the people living nearby, not the aggregation across the state. That way I could use the same UI metric at a different level of analysis than states - districts or regions or something.
I wonder how this would change the state ranking though.
I was definitely thinking this same thing! I'm trying to see what the big differences are.
Imagine a region with 10,000 people living at a regional density of 1,000 and 10,000 people living at a regional density of 10,000.
Under his order (take regional density, average, take log - I'll use log 10 rather than log e just for simplicity) you first get to an average of 5,500 and then take log to get about 3.3. Under the alternate order, you first take logs to get 3 and 4, and then average, to get 3.5.
If the town spread out to incorporate some suburbs, so that the city was less dense, we would now have 9,000 people living at a regional density of 1,000 and 11,000 people living at a regional density of 5,500. Under his order, you get the same 3.3. But under the alternate order, you would have 9,000 people at 3 and 11,000 people at about 3.3, and average to get about 3.15.
It seems right to me that doubling the region a town occupies should decrease overall density, so that seems like some motivation for this alternate order. But I haven't considered other configurations and transformations to see if his measure behaves more reasonably for them.
There is a spot on the Wilbur Cross Parkway, near New Haven, where if you are traveling southbound (toward NYC), you go through a tunnel and as soon as you come out of it you see a delicatessen and an NY style pizzeria in a strip mall by the side of the highway. This tunnel is the boundary between New England and the New York City metropolitan area.
Love this post, and I love that UI metric! Also, thank you for pointing out in your footnote about the absurdity of putting San Francisco and Silicon Valley in different metro areas. I live in San Mateo County which is technically in the SF metro, but feels every bit Silicon Valley. From a "vibe" feel, I've always considered "local" to be much larger. It doesn't speak much to the urban/rural divide that the article addresses, but I always loved the old Rand-McNally Major Trading Areas. Honestly, anyone from the Oregon border down to Big Sur and Visalia, and east even including Northwestern Nevada I consider "local". (i.e. I'd root for them if I saw someone in that entire range on a game show or in... say,... a poker tournament. :) )
It's interesting that Oregon and Washington are right next to each other in the %urban but Washington is well ahead of Oregon in UI. It really kind of makes intuitive sense where that comes from if you've spent time in both states--they tend to be seen as pretty similar, with most of the population concentrated in Portland/Seattle and a number of smaller cities interspersed elsewhere, with the eastern two thirds of both states being quite rural, but Seattle spreads out much more and the smaller cities and suburbs tend to be noticeably more urbanized than the smaller cities in Oregon, so there is a noticeable difference even though the groupings of "urban" vs. "rural" might show as similar by Census rules. Although that said one counterpoint may be that the Oregon coast tends to be more developed than Washington's coast, but that might be functionally undercounted by the UI because it exists in a narrow strip so a 5 mile radius along the coast would include a sizable chink of the Pacific ocean (plus the publicly owned lands immediately along the coast), as well as likely areas of the Coast Range that are quite sparsely populated. But then on the other other hand, Bend is really the only place in Eastern Oregon with any real population whereas Washington has Spokane, the Tri-Cities, and Yakima, which as smaller cities should decrease the UI in a way that you wouldn't see so much in Oregon. So it's interesting to think about. Obviously no method is perfect but yours seems a lot more functionally accurate than the Census'.
NY & NJ are going to be dark blue in 2024 unless Trump wins in an incredible landslide. But I think within 30 years they may be a bit purple because of the amazing growth rate of Orthodox Judaism. That's why being extremely urban may be a double edged sword to liberalism in future America.
I think both of those states have hit "peak blue" and will become lighter blue over time, but I have no idea if that means in 30 years they will be competitive. That's just way too far not to be prone to tail events or drastic political realignments that kind of "smash the Boggle grid" and make us start over. But in some limited areas we're starting to see the suburbs become almost as blue as the core cities.
I initially thought you were going to use population weighted density, though I like the straightforward interpretation of PWD maybe more (how dense does this state look, to the average person in the state?). I do not know exactly how you might compute that though: perhaps simply cutting the state into a grid might work. In any case, it probably would get you similar results, your approach is essentially the same.
I think his metric basically is the population weighted density, except that he's not weighting the density of each census block, but rather the density within a several mile radius of that census block. In a sense, his might even be a better measure of "how dense does this place look, to the average person", since the average person can see out a few miles from where they live, rather than just the immediate block.
Inspired by Nate's regression of vaccine regression I've been running regressions to try to get at the heart of public policy arguments. Here is my investigation of if urban areas (represented by democrats) increase the risk of violent crime. I was also inspired by rereading the rural violence described in Hillbilly Elegy. https://aarontesch.substack.com/p/show-me-the-data