The state cancer profiles website hosts visualization and data exploration tools for cancer incidence and mortality in the United States. For casual browsing, the website is great. However, if having access to the underlying data is the goal, the existing site does not have bulk downloads or an API. Therefore, we provide bulk data scraped from the website for data science applications.

Each release ships four gzipped CSVs and four Parquet files (same data, two formats):

• state_cancer_profiles_incidence.csv.gz / state_cancer_profiles_incidence.parquet — age-adjusted incidence rates (by cancer, age, sex, race, stage)
• state_cancer_profiles_mortality.csv.gz / state_cancer_profiles_mortality.parquet — age-adjusted mortality rates (same dimensions)
• state_cancer_profiles_risk.csv.gz / state_cancer_profiles_risk.parquet — BRFSS-derived screening and risk-factor prevalence (alcohol, smoking, mammograms, colonoscopy, HPV vaccination, etc.)
• state_cancer_profiles_demographics.csv.gz / state_cancer_profiles_demographics.parquet — ACS-derived demographics (crowding, education, poverty, SVI, etc.)

All four files cover both county-level and state-level rows in the same file (plus the national row at FIPS 00000), distinguished by the locale_type column.

Download from:

• https://github.com/seandavi/state-cancer-profile-scraper/releases/latest

A new release is generated every month with the latest data.

• State cancer profiles scraper
• Contents
• About the code
• About the data
• Using the data
  • Download the data
  • Data access in place
• Example rows from the data
  • Incidence data
  • Mortality data
• Local scraper usage
  • Running the scraper
• License
• Contributing

This is a simple web scraper that extracts cancer data from the State Cancer Profiles website. The data is extracted for all states and saved in two CSV files: one for cancer incidence and one for cancer mortality.

The data is extracted from the State Cancer Profiles website, which is a part of the National Cancer Institute. The website provides cancer statistics for all 50 states, the District of Columbia, and Puerto Rico. The data is available for cancer incidence and cancer mortality, and it is based on the SEER and NPCR programs.

The data is saved in four gzipped CSV files and four Parquet files: one each for cancer incidence, cancer mortality, demographic summary stats, and risk factor summary stats. The data is saved in a long format, with each row representing a single observation (i.e., a single cancer type in a single state in a single year).

The data are available for download from the releases page, with a link to the latest release here. To grab the latest release files programmatically, utilize the gh command line client:

gh release download -p "*" -D gh_latest -R seandavi/state-cancer-profile-scraper

Which will give you:

gh_latest
├── gh_hash.txt
├── scrape_catalog.jsonl
├── select_options.json
├── state_cancer_profiles_demographics.csv.gz
├── state_cancer_profiles_demographics.parquet
├── state_cancer_profiles_incidence.csv.gz
├── state_cancer_profiles_incidence.parquet
├── state_cancer_profiles_mortality.csv.gz
├── state_cancer_profiles_mortality.parquet
├── state_cancer_profiles_risk.csv.gz
└── state_cancer_profiles_risk.parquet

Note that R, python, and many other languages can read CSV files directly without the need for downloading the data which might a fast and easy way to access the data.

For those who simply want to query the data in place, both duckdb and clickhouse databases can query remote csv files directly. The same is true for R and Python. For example, using duckdb, you can run the following code to query the data:

install httpfs;
load
select 
  * 
from 
read_csv('https://github.com/seandavi/state-cancer-profile-scraper/releases/download/2025-02-10/state_cancer_profiles_incidence.csv.gz') 
limit 10;

Or, using clickhouse-local, start up clickhouse local and run the following query:

select 
  * 
from url('https://github.com/seandavi/state-cancer-profile-scraper/releases/download/2025-02-10/state_cancer_profiles_incidence.csv.gz') 
limit 10 
settings max_http_get_redirects = 10;

While most users will simply want to download the data, the scraper is available for those who want to run it themselves.

To install the required packages, run the following command:

pip install git+https://github.com/seandavi/state-cancer-profiles-scraper.git

After installation, a scps-scraper command is available:

# Scrape everything (incidence, mortality, risk, demographics)
scps-scraper scrape

# Pick a subset
scps-scraper scrape --datasets risk --datasets demographics

# Write to a specific directory
scps-scraper scrape --output-dir ./data

# See all options
scps-scraper scrape --help

The legacy module invocation is preserved and produces the same outputs:

python -m scps.scraper

The scraper writes gzipped CSVs and Parquet files to the current working directory (or --output-dir).

Each release ships a scrape_catalog.jsonl artifact alongside the CSVs. It records which (endpoint, dimension-combo) tuples returned non-empty data, so subsequent runs can skip the ~70-80% of the cartesian space that is suppressed or invalid (cervical cancer for males, pediatric cancers at adult ages, BRFSS small-cell suppression, etc.).

# Use ./scrape_catalog.jsonl if it exists (skip empty combos)
scps-scraper scrape

# Force full cartesian re-discovery and rewrite the catalog
scps-scraper scrape --refresh-catalog

# Point at a different catalog file
scps-scraper scrape --catalog-path /path/to/catalog.jsonl

CI does a full refresh quarterly (January, April, July, October) and uses the previous release's catalog the other eight months. After a catalog-driven run, the scraper probes the live select options and warns if upstream has added a new cancer / topic / risk / demo that the catalog doesn't know about.

This project is licensed under the MIT License - see the LICENSE file for details.

Please read CONTRIBUTING.md for details on our code of conduct, and the process for submitting pull requests to us.