GeoData Module

The Geodata module provides tools for managing and preprocessing geographic data, including administrative boundaries and census data. The functions in this module facilitate reading, filtering, and converting data into formats suitable for geographic analysis, such as GeoDataFrame and GeoPackage.

New code should import from istat_census_data. The reference below is generated from the implementation module so the full function documentation remains visible.

H3 census estimates

Use estimate_h3_census_indexes() after creating the finalized census_data.gpkg with finalize_census_data(). By default, the function reads layer census<year>, uses the packaged census catalogue to select additive indicators available in the layer, generates level-11 H3 cells for the AOI, and distributes source census-section values by area-weighted interpolation.

from pathlib import Path

import geopandas as gpd
from shapely.geometry import box

from istat_census_data import estimate_h3_census_indexes, generate_h3_grid

aoi = gpd.GeoDataFrame(
    geometry=[box(12.0, 41.0, 12.2, 41.2)],
    crs="EPSG:4326",
)

h3_estimates = estimate_h3_census_indexes(
    census_data_path=Path("census_data.gpkg"),
    year=2021,
    aoi=aoi,
    h3_resolution=11,
    indexes=["P1", "P2"],
    workers=4,
    chunk_size=5000,
    max_in_flight_chunks=4,
    output_path=Path("h3_census_2021.gpkg"),
)

The estimator enforces H3 resolution 11. The h3_resolution argument can be omitted because level 11 is the default, but any other value is rejected. H3 boundary cells are expanded by three neighbour rings by default through boundary_ring_size=3, which avoids dropping cells that intersect complex AOI edges.

generate_h3_grid() stores reuse metadata and a fingerprint in the returned GeoDataFrame attributes. The fingerprint includes the normalized AOI CRS, H3 resolution, boundary_ring_size, area_crs, grid CRS, AOI bounds, AOI area, and hashes of the validated unified AOI geometry. Advanced callers can pass that grid back through h3_grid to avoid regenerating identical cells:

h3_grid = generate_h3_grid(
    aoi=aoi,
    h3_resolution=11,
    boundary_ring_size=3,
)

h3_estimates = estimate_h3_census_indexes(
    census_data_path=Path("census_data.gpkg"),
    year=2021,
    aoi=aoi,
    indexes=["P1", "P2"],
    h3_grid=h3_grid,
)

The estimator validates the fingerprint before reuse. A different AOI geometry, even with the same bounding box, or a different grid-defining parameter causes a clear error instead of silently reusing stale cells.

Additive index columns in the returned H3 table and written GeoPackage are stored as integer counts. The estimator first validates the continuous area-weighted calculation in total_validation, then allocates integer cell values with a largest-remainder step and records post-allocation checks in integer_total_validation. Density columns remain floating-point values.

The same workflow is available as an editable example script in scripts/estimate_h3_census.py. Set the module-level values such as census_data_path, aoi_path, selected_year, selected_indexes, boundary_ring_size, workers, and chunk_size, then run the script. Use selected_indexes = [] to estimate every additive index from the catalogue that is present in the source layer.

H3 intermediate storage

H3 estimation uses Parquet/GeoParquet-backed intermediate storage by default. pyarrow is a required package dependency, so no extra install is needed for this mode. The estimator writes prepared census sections, prepared H3 grid chunks, and compact chunk result files to a temporary workspace, then reads the chunk results back for the final GeoPackage output.

This reduces peak RAM because workers do not need to return large GeoDataFrames to the parent process. It also avoids building many result columns one by one when selected_indexes = [] expands to all additive indicators.

The default Parquet path also clips the H3 overlay grid to the AOI one chunk at a time before estimation starts. Large AOIs therefore produce observable grid_chunks/ files instead of waiting for one full in-memory clipped H3 grid to finish. Retained intermediate directories no longer include a full h3_grid.parquet file.

Parallel chunk scheduling is bounded by max_in_flight_chunks. When the value is omitted, the estimator schedules at most the active worker count at one time. That default keeps Dask, thread, and process runs from queuing every AOI chunk in memory at once.

Use intermediate_dir when you want predictable intermediate locations. Use keep_intermediates=True to inspect retained files, and reuse_intermediates=True to reuse matching retained chunk result files from the same directory.

h3_estimates = estimate_h3_census_indexes(
    census_data_path=Path("census_data.gpkg"),
    year=2021,
    aoi=aoi,
    indexes=[],
    workers=4,
    chunk_size=5000,
    max_in_flight_chunks=4,
    intermediate_dir=Path("h3_intermediates"),
    keep_intermediates=True,
)

Set intermediate_storage="memory" only when you explicitly want the legacy in-memory path for small AOIs or debugging. The final user-facing output remains GeoPackage unless you omit output_path and work with the returned GeoDataFrame directly.

Parallel execution and benchmarking

H3 estimation is serial by default: workers=1. For larger AOIs, set workers above 1 and choose a chunk_size to split the H3 grid into deterministic cell-owned chunks. Each H3 cell is processed by exactly one chunk, so parallel results contain the same cells as the serial path and should match serial totals within the estimator tolerance.

h3_estimates = estimate_h3_census_indexes(
    census_data_path=Path("census_data.gpkg"),
    year=2021,
    aoi=aoi,
    indexes=["P1", "P2"],
    workers=4,
    chunk_size=5000,
    max_in_flight_chunks=4,
)

With the default Parquet/GeoParquet intermediate storage, thread, process, and Dask workers write one result file per chunk and return file paths to the parent process. By default, the parent performs the final merge after chunks complete. When stream_output=True is used with output_path, finalization streams those result chunks directly to the GeoPackage, creates the layer spatial index and a unique table index on h3_index, and returns a metadata-only GeoDataFrame with row_count and validation tables in attrs. Use parallel_backend="process" only after benchmarking on your machine: it can help CPU-bound overlays, but it adds worker startup and disk I/O overhead.

Use the optional Dask backend when you want Dask to schedule the same deterministic H3 chunks locally:

pip install "istat-census-data[dask]"

h3_estimates = estimate_h3_census_indexes(
    census_data_path=Path("census_data.gpkg"),
    year=2021,
    aoi=aoi,
    indexes=["P1", "P2"],
    workers=4,
    chunk_size=5000,
    parallel_backend="dask",
    max_in_flight_chunks=4,
)

The Dask backend currently uses Dask's local threaded scheduler and the same Parquet chunk contract as the standard backends. It is an optional runtime path, not a required dependency for normal package installs.

Start with workers no higher than the physical CPU cores available to the process. For H3 level 11, chunk_size values between 2,000 and 10,000 cells are a practical first range. Smaller chunks give earlier progress updates but create more scheduling overhead; larger chunks reduce overhead but can leave workers idle near the end of a run.

Keep max_in_flight_chunks equal to workers for the first large run. Increase it only after benchmarking shows scheduling overhead is more important than memory pressure. For large AOIs such as an entire region, prefer intermediate_storage="parquet" and avoid Dask settings that queue more chunks than the machine can comfortably hold in memory. In Parquet mode, chunk_size also controls how many H3 cells are projected and clipped to the AOI during each grid-preparation batch.

Use scripts/benchmark_h3_estimation.py on finalized data to compare settings before running a large production AOI. Edit worker_counts, chunk_sizes, parallel_backend, max_in_flight_chunks, boundary_ring_size, intermediate_storage, keep_intermediates, reuse_intermediates, and runs in the script. Keep selected_indexes = [] when you want the benchmark to estimate all additive catalogue indexes present in the source layer.

The benchmark output reports elapsed seconds, H3 cell count, and whether total validation passed for each worker/chunk setting. Records are logged and written to JSON after every completed benchmark run, so long executions have observable progress. Treat small AOI timings as a correctness and overhead check; medium or large AOIs are more useful for deciding whether parallel execution actually helps.

Configuring the benchmark script

scripts/benchmark_h3_estimation.py is configured by editing module-level variables in the file. It is not intended to be used through CLI arguments.

main_path = Path("/home/max/Desktop/census_test")
census_data_path = main_path / "census_data.gpkg"
aoi_path = main_path / "aoi.gpkg"
benchmark_output_path = main_path / "h3_benchmark.json"

selected_year = 2021
selected_indexes: list[str] = []
boundary_ring_size = DEFAULT_BOUNDARY_RING_SIZE

worker_counts = [1, 2, 4]
chunk_sizes = [DEFAULT_H3_CHUNK_SIZE]
parallel_backend = "thread"
max_in_flight_chunks = None

intermediate_storage = "parquet"
intermediate_dir = main_path / "h3_intermediates"
keep_intermediates = True
reuse_intermediates = False

runs = 1

Use worker_counts = [1] for a serial-only run. Include values greater than 1 to compare parallel execution. Start with parallel_backend = "thread"; test parallel_backend = "process" or parallel_backend = "dask" only after a baseline benchmark. The Dask backend requires the optional Dask install. Leave max_in_flight_chunks = None to schedule at most the active worker count, or set an explicit positive integer when you need repeatable batch sizing.

Keep boundary_ring_size = DEFAULT_BOUNDARY_RING_SIZE unless you are studying a specific AOI edge case. The current default is 3 and is included in benchmark JSON records so runs with different boundary coverage are comparable.

Set reuse_intermediates = False for a cold run that rebuilds intermediate files. Set it to True only for a later warm run that should reuse compatible retained chunks. Keep keep_intermediates = True when you want to inspect or reuse the Parquet/GeoParquet files; set it to False to remove them after the run.

End-to-end H3 workflow

Use run_h3_census_workflow() or scripts/run_h3_census_analysis.py when you want the selected census years to drive the whole pipeline. The workflow downloads only the requested years, preprocesses them, finalizes census_data.gpkg, and then writes one H3 level-11 output layer per year.

For same-AOI multi-year runs, the workflow generates the H3 grid once before the year loop and passes it to each yearly estimate. Census source preparation, overlay, validation, and layer writing still run independently for every year.

The editable example script scripts/run_h3_census_analysis.py exposes the same workflow through module-level settings. Set list_year, main_path, aoi_path, list_region, municipalities_code, and performance controls in the script. Use selected_indexes = [] to estimate all additive indicators available in each selected year. The H3 level is fixed at 11 and the special/fictitious section policy defaults to exclude.

The end-to-end workflow exposes the same PRP-012 performance controls as the finalized-data estimator. Edit workers, chunk_size, parallel_backend, and max_in_flight_chunks in the script to tune large jobs. After installing istat-census-data[dask], set parallel_backend = "dask" to use the optional local Dask backend.

For multi-year regional runs, set retain_results=False. With the default Parquet intermediate storage, the workflow then streams each yearly result layer from result_chunks/ to GeoPackage instead of materializing the full yearly GeoDataFrame. The returned workflow result still records row_counts, total_validations, and integer_total_validations for each year.

If the data already exists, skip completed stages by setting run_download, run_preprocess, or run_finalize to False in the script.

When --skip-finalize is used, <data-dir>/census_data.gpkg must already exist.

The same process is available from Python:

from pathlib import Path

import geopandas as gpd

from istat_census_data import run_h3_census_workflow

aoi = gpd.read_file("aoi.gpkg", layer="aoi")

result = run_h3_census_workflow(
    years=[2021],
    data_folder=Path("./census"),
    aoi=aoi,
    indexes=["P1", "P2"],
    municipality_codes=[58091],
    workers=4,
    chunk_size=5000,
    parallel_backend="thread",
    max_in_flight_chunks=4,
    retain_results=False,
)

print(result.output_path)
print(result.layers)

The method assumes uniform density inside each source census section. This is transparent and preserves additive totals against the clipped source-cell total, but it is not a dasymetric model and should not be interpreted as exact household or building-level placement.

Special and fictitious census sections

estimate_h3_census_indexes() flags source census sections whose section code contains one of these markers:

Marker	Meaning	Data delivered in those rows	H3 interpretation
`888888`	Fictitious sections for people without housing or without a fixed address who are registered in the municipal registry. ISTAT also documents their use for people registered through associations or reception facilities.	The rows can carry the same census variables released for section-level files. For these sections, population and person/family indicators may be meaningful, while dwelling or building indicators should not be read as physical conditions at the drawn polygon.	The geometry is conventional. ISTAT notes that these sections are drawn in uninhabited areas, preferably near the town hall. H3 densities over that polygon are therefore an allocation device, not a residential-location estimate.
`999999`	Fictitious sections related to contested zones claimed by two or more municipalities.	The rows can carry resident-individual counts for the municipality to which the contested geography was not assigned for census purposes. They can also flow through the normal section-level variable schema when present in the finalized layer.	Treat the result as an administrative allocation. Do not interpret the H3 cell as a precise location of residence unless the contested-zone geometry is analytically appropriate for your use case.
`7777777`	2011 Abruzzo-only fictitious sections for municipalities affected by the April 2009 earthquake.	ISTAT documents these sections as the placement for families that, at the census date, were temporarily domiciled in a municipality different from their registry municipality. Additive population/family indicators may therefore be present, but their geometry is not ordinary residence geography.	Use care in local density maps and overlays: the rows preserve census accounting, but the geometry encodes a post-earthquake administrative convention.

These markers are not treated as missing data, but H3 estimation excludes them by default because their conventional geometries can create misleading local densities. With special_section_policy="exclude", the estimator removes these source rows before the area-weighted overlay; totals in the result therefore exclude those records.

Use special_section_policy="include" only when you need full census accounting inside the AOI and accept that conventional polygons can receive people who are not physically distributed there. In that mode, the estimator includes their values in the area-weighted overlay and sets intersects_special_section=True for any H3 cell receiving area from one of them.

Use a stricter policy when that convention is not acceptable:

h3_estimates = estimate_h3_census_indexes(
    census_data_path=Path("census_data.gpkg"),
    year=2021,
    aoi=aoi,
    indexes=["P1"],
    special_section_policy="raise",
)

exclude: default for H3 processes; remove these source rows before estimating, so totals exclude those records.
include: keep the rows and flag intersecting H3 cells.
raise: stop with an error if any special or fictitious section intersects the AOI.

Inspect flagged output cells with:

flagged_cells = h3_estimates[h3_estimates["intersects_special_section"]]

The script example exposes the same policy through the special_section_policy argument passed to run().

Sources: ISTAT documents 888888x and 999999x in Dati per sezioni di censimento. ISTAT documents zones in contestation, sections for people without housing, and the 2011 Abruzzo 7777777 convention in Basi territoriali: anni 1991, 2001, 2011 e 2021.

preprocess_geodata(census_shp_folder, census_target_columns, census_tipo_loc_mapping, output_folder, census_layer_name, census_column_remapping=None, regions_file_path=None, regions_target_columns=None, regions_index_column=None, regions_column_remapping=None, provinces_file_path=None, provinces_target_columns=None, provinces_index_column=None, provinces_column_remapping=None, municipalities_file_path=None, municipalities_target_columns=None, municipalities_index_column=None, municipalities_column_remapping=None, municipalities_code=None)

Preprocess census geodata and administrative boundaries and save to GeoPackage.

This function executes the complete workflow for preparing geographic data for a census year, combining:

Reading and normalizing administrative boundaries (regions, provinces, municipalities).
Optionally correcting missing fields (e.g., COD_PROV for 2021).
Reading and preparing census data (sections) from shapefiles.
Joining sections with municipalities, provinces, and regions.
Optionally filtering for a subset of municipalities (municipalities_code).
Saving the final result to a GeoPackage.

PARAMETER	DESCRIPTION
`census_shp_folder`	Folder containing census shapefiles (sections). TYPE: `Path`
`census_target_columns`	Columns to select from census data (sections). TYPE: `list`
`census_tipo_loc_mapping`	Mapping for the `TIPO_LOC` field to derive the locality type description. TYPE: `dict`
`output_folder`	Folder where the resulting GeoPackage will be saved. TYPE: `Path`
`census_layer_name`	Name of the census layer (e.g., `census2011`), also used to derive the year from the suffix. TYPE: `str`
`census_column_remapping`	Optional mapping to rename census data columns. TYPE: `dict \| None` DEFAULT: `None`
`regions_file_path`	Optional path to the regional boundaries vector file. TYPE: `Path \| None` DEFAULT: `None`
`regions_target_columns`	Optional columns to select from regional data. TYPE: `list \| None` DEFAULT: `None`
`regions_index_column`	Optional column to use as index for regional data. TYPE: `str \| None` DEFAULT: `None`
`regions_column_remapping`	Optional mapping to rename regional data columns. TYPE: `dict \| None` DEFAULT: `None`
`provinces_file_path`	Optional path to the provincial boundaries vector file. TYPE: `Path \| None` DEFAULT: `None`
`provinces_target_columns`	Optional columns to select from provincial data. TYPE: `list \| None` DEFAULT: `None`
`provinces_index_column`	Optional column to use as index for provincial data. TYPE: `str \| None` DEFAULT: `None`
`provinces_column_remapping`	Optional mapping to rename provincial data columns. TYPE: `dict \| None` DEFAULT: `None`
`municipalities_file_path`	Optional path to the municipal boundaries vector file. TYPE: `Path \| None` DEFAULT: `None`
`municipalities_target_columns`	Optional columns to select from municipal data. TYPE: `list \| None` DEFAULT: `None`
`municipalities_index_column`	Optional column to use as index for municipal data. TYPE: `str \| None` DEFAULT: `None`
`municipalities_column_remapping`	Optional mapping to rename municipal data columns. TYPE: `dict \| None` DEFAULT: `None`
`municipalities_code`	Optional list of ISTAT municipality codes (`PRO_COM` field) to extract. If empty, all municipalities are kept. TYPE: `list[int] \| None` DEFAULT: `None`

RETURNS	DESCRIPTION
`Path`	Path to the generated GeoPackage containing the census layer enriched
`Path`	with administrative information.

Note

The census year is derived from the layer name census_layer_name[6:] (e.g., census2011 → 2011). For 2021, the COD_PROV column is manually reconstructed from PRO_COM_T (see repository issue #47). The GeoPackage is saved as {YEAR_GEODATA_NAME}.gpkg and the layer as {YEAR_GEODATA_NAME}{census_year}.

`read_administrative_boundaries(file_path, target_columns, index_column, column_remapping=None, output_folder=None, layer_name=None)`

Read administrative boundaries and return a DataFrame or GeoPackage.

This function reads an administrative boundary file (typically a shapefile), selects a subset of columns, and sets a column as the index. Depending on the provided parameters, it can:

Return a DataFrame without geometry, sorted and indexed; or
Save the data as a layer in a GeoPackage, preserving the geometry.

The encoding is derived from the .dbf file associated with the shapefile to avoid issues with accented characters or special symbols.

PARAMETER	DESCRIPTION
`file_path`	Path to the vector file (e.g., shapefile) containing administrative boundaries. TYPE: `Path`
`target_columns`	List of columns to select from the source dataset. The geometry column is added automatically. TYPE: `list`
`index_column`	Name of the column to use as the DataFrame index (e.g., ISTAT code). TYPE: `str`
`column_remapping`	Optional dictionary to rename columns (e.g., `{"DEN_REG": "REGIONE"}`). If None, original names are kept. TYPE: `dict \| None` DEFAULT: `None`
`output_folder`	Optional output folder where the GeoPackage will be saved. If None, the function returns a DataFrame (without geometry) instead of writing to disk. TYPE: `Path \| None` DEFAULT: `None`
`layer_name`	Optional name of the layer to use within the GeoPackage. Must be specified if `output_folder` is provided, to properly distinguish layers. TYPE: `str \| None` DEFAULT: `None`

RETURNS	DESCRIPTION
`DataFrame \| Path`	Either an indexed and sorted DataFrame without geometry column if
`DataFrame \| Path`	`output_folder` is None, or the path to the created GeoPackage if
`DataFrame \| Path`	`output_folder` is provided.

Note

The geometry column is automatically added to target_columns via the GEOMETRY_COLUMN_NAME constant. The GeoPackage is saved with a name based on the YEAR_GEODATA_NAME constant and contains the layer specified by layer_name.

`read_census(shp_folder, target_columns, tipo_loc_mapping, column_remapping=None, output_folder=None, layer_name=None)`

Read census data from shapefiles and return a GeoDataFrame or GeoPackage.

This function recursively searches for all shapefiles in a folder, reads their data, selects a subset of columns, corrects invalid geometries, adds the locality type description (derived from tipo_loc_mapping), and builds a unified GeoDataFrame with all census sections.

Depending on the parameters, it can:

Return the resulting GeoDataFrame directly; or
Save the data as a layer in a GeoPackage (YEAR_GEODATA_NAME.gpkg) and return the path to the created file.

PARAMETER	DESCRIPTION
`shp_folder`	Path to the folder containing census shapefiles (recursive reading via `rglob(".shp")`). TYPE:* `Path`
`target_columns`	List of columns to select from each shapefile (must include or be compatible with the geometry column). TYPE: `list`
`tipo_loc_mapping`	Mapping of locality codes for the `TIPO_LOC` field (e.g., `{1: "Centro abitato", 2: "Nucleo", ...}`), used to create the descriptive column `DEN_LOC`. TYPE: `dict`
`column_remapping`	Optional dictionary to rename selected columns (e.g., `{"PRO_COM": "PRO_COMUNE"}`). If None, original names are kept. TYPE: `dict \| None` DEFAULT: `None`
`output_folder`	Optional folder where the resulting GeoPackage will be saved. If None, the function does not write to disk and returns the GeoDataFrame directly. TYPE: `Path \| None` DEFAULT: `None`
`layer_name`	Optional name of the layer to use within the GeoPackage. Must be specified if `output_folder` is provided. TYPE: `str \| None` DEFAULT: `None`

RETURNS	DESCRIPTION
`GeoDataFrame \| Path`	Either a `GeoDataFrame` with census data and corrected geometries if
`GeoDataFrame \| Path`	`output_folder` is None, or the path to the created GeoPackage if
`GeoDataFrame \| Path`	`output_folder` is provided.

RAISES	DESCRIPTION
`ValueError`	If no shapefile is found in the specified folder.

Note

Geometries are validated with make_valid() to reduce issues caused by invalid polygons. An area_mq column containing the area in square meters is calculated. The GeoDataFrame index is set to the first column in df_columns (typically the census section code).

H3 area-weighted estimation for ISTAT census section indicators.

estimate_h3_census_indexes(census_data_path, year, aoi, h3_resolution=REQUIRED_H3_RESOLUTION, *, indexes='all_additive', aoi_crs=WGS84_CRS, area_crs=DEFAULT_AREA_CRS, layer_name=None, municipality_codes=None, special_section_policy=DEFAULT_SPECIAL_SECTION_POLICY, boundary_ring_size=DEFAULT_BOUNDARY_RING_SIZE, include_density=True, clip_output=False, output_crs=WGS84_CRS, output_path=None, output_layer=None, conservation_tolerance=1e-06, workers=DEFAULT_H3_WORKERS, chunk_size=DEFAULT_H3_CHUNK_SIZE, parallel_backend=DEFAULT_PARALLEL_BACKEND, intermediate_storage=DEFAULT_INTERMEDIATE_STORAGE, intermediate_dir=None, keep_intermediates=False, reuse_intermediates=False, max_in_flight_chunks=None, stream_output=False, h3_grid=None)

Estimate additive census indicators on an H3 grid.

Values are distributed from source census sections to H3 cells using uniform density inside each source section:

contribution = source_value * intersection_area / source_cell_area

PARAMETER	DESCRIPTION
`census_data_path`	Path to the finalized `census_data.gpkg` file. TYPE: `str \| Path`
`year`	Census year to read from layer `census<year>` unless `layer_name` is provided. TYPE: `int`
`aoi`	Area of interest as a Shapely geometry, GeoSeries, or GeoDataFrame. TYPE: `GeoInput`
`h3_resolution`	Required H3 resolution. This implementation supports only level 11. TYPE: `int` DEFAULT: `REQUIRED_H3_RESOLUTION`
`indexes`	Source index codes or canonical catalogue IDs to estimate. The default `"all_additive"` selects additive source codes present in the layer using the packaged census index catalogue. An empty iterable also selects all available additive indexes. TYPE: `str \| Iterable[str] \| None` DEFAULT: `'all_additive'`
`aoi_crs`	CRS to use when `aoi` is a plain Shapely geometry or lacks CRS. TYPE: `str \| int \| None` DEFAULT: `WGS84_CRS`
`area_crs`	Projected CRS used for area calculations. TYPE: `str \| int` DEFAULT: `DEFAULT_AREA_CRS`
`layer_name`	Optional source layer name. Defaults to `census<year>`. TYPE: `str \| None` DEFAULT: `None`
`municipality_codes`	Optional municipality codes used to filter source rows through the first available municipality-code column. TYPE: `Iterable[int \| str] \| None` DEFAULT: `None`
`special_section_policy`	How to handle special/fictitious sections. The default `"exclude"` removes them before estimation. Use `"include"` to keep and flag them, or `"raise"` to fail if any intersects the AOI. TYPE: `str` DEFAULT: `DEFAULT_SPECIAL_SECTION_POLICY`
`boundary_ring_size`	H3 neighbour rings added around filled cells before final AOI filtering. Defaults to 3 to avoid missing boundary cells in real administrative AOIs. TYPE: `int` DEFAULT: `DEFAULT_BOUNDARY_RING_SIZE`
`include_density`	If True, add `<index>_density_per_sq_km` columns based on the H3/AOI intersection area. TYPE: `bool` DEFAULT: `True`
`clip_output`	If True, output geometries are clipped to the AOI. Otherwise full H3 cell geometries are returned. TYPE: `bool` DEFAULT: `False`
`output_crs`	CRS for returned geometries. Defaults to EPSG:4326. TYPE: `str \| int` DEFAULT: `WGS84_CRS`
`output_path`	Optional GeoPackage path to write. TYPE: `str \| Path \| None` DEFAULT: `None`
`output_layer`	Optional output layer name. Defaults to `h3_census<year>_r<h3_resolution>`. TYPE: `str \| None` DEFAULT: `None`
`conservation_tolerance`	Relative tolerance stored in validation metadata. TYPE: `float` DEFAULT: `1e-06`
`workers`	Number of parallel workers used for H3-cell chunks. The default `1` uses the serial implementation. TYPE: `int` DEFAULT: `DEFAULT_H3_WORKERS`
`chunk_size`	Maximum number of H3 cells in each parallel chunk. In the default Parquet mode this also bounds H3 overlay grid preparation. TYPE: `int` DEFAULT: `DEFAULT_H3_CHUNK_SIZE`
`parallel_backend`	Standard-library executor backend for `workers > 1`. Use `"thread"` to avoid copying GeoDataFrames between processes, `"process"` for process-based execution when serialization overhead is acceptable, or `"dask"` after installing the optional Dask extra. TYPE: `str` DEFAULT: `DEFAULT_PARALLEL_BACKEND`
`intermediate_storage`	Intermediate execution mode. Use `"parquet"` to prepare the H3 overlay grid in bounded chunks, store prepared GeoParquet inputs, and write Parquet chunk outputs on disk. Use `"memory"` for the legacy in-memory path. Defaults to `"parquet"`. TYPE: `str` DEFAULT: `DEFAULT_INTERMEDIATE_STORAGE`
`intermediate_dir`	Optional directory used for H3 intermediate files. When omitted, a temporary directory is created automatically. TYPE: `str \| Path \| None` DEFAULT: `None`
`keep_intermediates`	If True, keep generated intermediate files after the run. Temporary files are removed by default. TYPE: `bool` DEFAULT: `False`
`reuse_intermediates`	If True, reuse matching chunk result files in `intermediate_dir` when they already exist. TYPE: `bool` DEFAULT: `False`
`max_in_flight_chunks`	Maximum number of chunks submitted or computed at once. Defaults to the active worker count, which bounds queue memory for thread, process, and Dask execution. Increase only after benchmarking if scheduling overhead dominates memory pressure. TYPE: `int \| None` DEFAULT: `None`
`stream_output`	If True, require Parquet intermediate storage and `output_path`, then stream finalized result chunks directly to the GeoPackage instead of materializing the full output GeoDataFrame. The returned GeoDataFrame is metadata-only; its `attrs` contain `row_count`, validation tables, and output information. TYPE: `bool` DEFAULT: `False`
`h3_grid`	Optional precomputed H3 grid returned by `generate_h3_grid()` for the same AOI, H3 resolution, `boundary_ring_size`, `area_crs`, and internal grid CRS. When supplied, the estimator validates the stored grid fingerprint before reusing it. TYPE: `GeoDataFrame \| None` DEFAULT: `None`

RETURNS	DESCRIPTION
`GeoDataFrame`	GeoDataFrame with one row per H3 cell, integer estimated index columns,
`GeoDataFrame`	density columns when requested, and coverage metadata. When
`GeoDataFrame`	`stream_output=True`, returns a metadata-only GeoDataFrame after writing
`GeoDataFrame`	the GeoPackage layer. The `attrs` dictionary contains continuous
`GeoDataFrame`	`source_totals`, `h3_totals`, and `total_validation`, plus
`GeoDataFrame`	`integer_total_validation` for the final integer output columns.

RAISES	DESCRIPTION
`FileNotFoundError`	If `census_data_path` does not exist.
`ValueError`	If CRS, AOI, requested indexes, parallel options, or source geometry are invalid.

`finalize_h3_result_chunks_to_geopackage(intermediate_workspace, output_path, *, output_layer=None, conservation_tolerance=1e-06, index_batch_size=H3_CHUNK_FINALIZATION_INDEX_BATCH_SIZE)`

Finalize retained H3 result chunks into a GeoPackage without full concatenation.

This function is intended for retained Parquet/GeoParquet workspaces created by estimate_h3_census_indexes(..., keep_intermediates=True). It streams the result_chunks/ files to the target GeoPackage, rebuilding the same validation and integer additive totals that the in-memory finalization path produces.

PARAMETER	DESCRIPTION
`intermediate_workspace`	Directory containing `metadata.json`, `prepared_source.parquet`, and `result_chunks/result_chunk_.parquet`. TYPE:* `str \| Path`
`output_path`	GeoPackage file to create or append to. TYPE: `str \| Path`
`output_layer`	GeoPackage layer name. Defaults to `h3_census<year>_r<h3_resolution>` from the retained metadata. TYPE: `str \| None` DEFAULT: `None`
`conservation_tolerance`	Relative tolerance used for continuous total validation. TYPE: `float` DEFAULT: `1e-06`
`index_batch_size`	Number of additive indexes to inspect together while planning integer apportionment. Lower values use less memory. TYPE: `int` DEFAULT: `H3_CHUNK_FINALIZATION_INDEX_BATCH_SIZE`

RETURNS	DESCRIPTION
`_H3ChunkExportResult`	Summary with output path, layer, row count, chunk count, and validation tables.

RAISES	DESCRIPTION
`FileNotFoundError`	If the retained workspace is incomplete.
`ValueError`	If metadata is invalid, no result chunk exists, or the output layer already exists.

`generate_h3_grid(aoi, h3_resolution=REQUIRED_H3_RESOLUTION, *, aoi_crs=WGS84_CRS, area_crs=DEFAULT_AREA_CRS, boundary_ring_size=DEFAULT_BOUNDARY_RING_SIZE, output_crs=WGS84_CRS)`

Generate H3 cell polygons intersecting an area of interest.

H3 polygon filling uses cell centres. This function expands the filled cells by neighbour rings and then keeps only cells that intersect the AOI, so boundary cells are retained for area-weighted overlays.

PARAMETER	DESCRIPTION
`aoi`	Area of interest as a Shapely geometry, GeoSeries, or GeoDataFrame. TYPE: `GeoInput`
`h3_resolution`	Required H3 resolution. This implementation supports only level 11. TYPE: `int` DEFAULT: `REQUIRED_H3_RESOLUTION`
`aoi_crs`	CRS to use when `aoi` is a plain Shapely geometry or when a GeoSeries/GeoDataFrame has no CRS. Defaults to EPSG:4326. TYPE: `str \| int \| None` DEFAULT: `WGS84_CRS`
`area_crs`	Projected CRS used to calculate `h3_cell_area_sqm`. TYPE: `str \| int` DEFAULT: `DEFAULT_AREA_CRS`
`boundary_ring_size`	Number of H3 neighbour rings added around filled cells before final AOI intersection filtering. Defaults to 3 to retain boundary cells reliably for real administrative AOIs. TYPE: `int` DEFAULT: `DEFAULT_BOUNDARY_RING_SIZE`
`output_crs`	CRS for returned cell geometries. Defaults to EPSG:4326. TYPE: `str \| int` DEFAULT: `WGS84_CRS`

RETURNS	DESCRIPTION
`GeoDataFrame`	GeoDataFrame with `h3_index`, `h3_resolution`, `h3_cell_area_sqm`, and
`GeoDataFrame`	polygon geometries. The `attrs` dictionary contains `h3_grid_metadata`
`GeoDataFrame`	and `h3_grid_fingerprint` for safe same-AOI reuse.

RAISES	DESCRIPTION
`TypeError`	If the AOI input type is unsupported.
`ValueError`	If the AOI is empty, CRS is missing, or the resolution is invalid.

End-to-end workflow for census download, preparation, and H3 estimation.

`H3CensusWorkflowResult` `dataclass`

Result metadata for an end-to-end H3 census workflow run.

run_h3_census_workflow(years, data_folder, aoi, *, indexes='all_additive', output_path=None, output_layer_prefix='h3_census', region_list=None, municipality_codes=None, special_section_policy=DEFAULT_SPECIAL_SECTION_POLICY, area_crs=DEFAULT_AREA_CRS, aoi_crs=None, clip_output=False, boundary_ring_size=DEFAULT_BOUNDARY_RING_SIZE, workers=DEFAULT_H3_WORKERS, chunk_size=DEFAULT_H3_CHUNK_SIZE, parallel_backend=DEFAULT_PARALLEL_BACKEND, intermediate_storage=DEFAULT_INTERMEDIATE_STORAGE, intermediate_dir=None, keep_intermediates=False, reuse_intermediates=False, max_in_flight_chunks=None, retain_results=True, run_download=True, run_preprocess=True, run_finalize=True, delete_download_folder=False, delete_preprocessed_data=False)

Run the full selected-year census pipeline before H3 estimation.

This workflow downloads and prepares only the selected census years, finalizes them into census_data.gpkg, and writes one fixed level-11 H3 output layer per year. The H3 grid for the workflow AOI is generated once and reused for every selected year after fingerprint validation.

PARAMETER	DESCRIPTION
`years`	One or more supported census years to download, process, finalize, and estimate. TYPE: `Iterable[int]`
`data_folder`	Root folder used for downloaded, preprocessed, finalized, and H3 output data. TYPE: `str \| Path`
`aoi`	Area of interest for H3 estimation. TYPE: `GeoInput`
`indexes`	Source index codes or canonical catalogue IDs to estimate. The default `"all_additive"` estimates every additive index present in each finalized source layer. An empty iterable has the same all-additive behavior. TYPE: `str \| Iterable[str] \| None` DEFAULT: `'all_additive'`
`output_path`	Optional H3 output GeoPackage. Defaults to `<data_folder>/h3_census_data.gpkg`. TYPE: `str \| Path \| None` DEFAULT: `None`
`output_layer_prefix`	Prefix for output layers. Layer names are `<prefix><year>_r11`. TYPE: `str` DEFAULT: `'h3_census'`
`region_list`	Optional region codes passed to the download stage. TYPE: `Iterable[int] \| None` DEFAULT: `None`
`municipality_codes`	Optional municipality codes passed to preprocessing and H3 estimation. TYPE: `Iterable[int \| str] \| None` DEFAULT: `None`
`special_section_policy`	Special/fictitious section policy. Defaults to `exclude` for H3 workflows. TYPE: `str` DEFAULT: `DEFAULT_SPECIAL_SECTION_POLICY`
`area_crs`	Projected CRS used for area calculations. TYPE: `str \| int` DEFAULT: `DEFAULT_AREA_CRS`
`aoi_crs`	CRS assigned to plain Shapely AOIs or AOIs without CRS. TYPE: `str \| int \| None` DEFAULT: `None`
`clip_output`	If True, output geometries are clipped to the AOI. TYPE: `bool` DEFAULT: `False`
`boundary_ring_size`	H3 neighbour rings added around filled cells before final AOI filtering. Defaults to the package's robust boundary setting. TYPE: `int` DEFAULT: `DEFAULT_BOUNDARY_RING_SIZE`
`workers`	Number of parallel H3 chunk workers. The default `1` keeps the serial path. TYPE: `int` DEFAULT: `DEFAULT_H3_WORKERS`
`chunk_size`	Maximum number of H3 cells per parallel chunk. TYPE: `int` DEFAULT: `DEFAULT_H3_CHUNK_SIZE`
`parallel_backend`	Backend used for `workers > 1`. Supported values are `thread`, `process`, and `dask`. The `dask` backend requires the optional `istat-census-data[dask]` package extra. TYPE: `str` DEFAULT: `DEFAULT_PARALLEL_BACKEND`
`intermediate_storage`	H3 intermediate execution mode. Defaults to Parquet/GeoParquet-backed processing. TYPE: `str` DEFAULT: `DEFAULT_INTERMEDIATE_STORAGE`
`intermediate_dir`	Optional folder used for H3 intermediate files. TYPE: `str \| Path \| None` DEFAULT: `None`
`keep_intermediates`	If True, retain intermediate files after each year. TYPE: `bool` DEFAULT: `False`
`reuse_intermediates`	If True, reuse matching retained chunk outputs from `intermediate_dir`. TYPE: `bool` DEFAULT: `False`
`max_in_flight_chunks`	Maximum H3 chunks submitted or computed at once. Defaults to the active worker count in the estimator. TYPE: `int \| None` DEFAULT: `None`
`retain_results`	If True, retain every yearly result GeoDataFrame in the returned metadata. Set to False for large output-writing workflows to stream Parquet-backed chunk finalization directly to GeoPackage and keep only row counts and validation metadata after each layer is written. TYPE: `bool` DEFAULT: `True`
`run_download`	If True, run the selected-year download stage. TYPE: `bool` DEFAULT: `True`
`run_preprocess`	If True, run the selected-year preprocessing stage. TYPE: `bool` DEFAULT: `True`
`run_finalize`	If True, run the selected-year finalization stage. TYPE: `bool` DEFAULT: `True`
`delete_download_folder`	Passed to `preprocess_census()`. TYPE: `bool` DEFAULT: `False`
`delete_preprocessed_data`	Passed to `finalize_census_data()`. TYPE: `bool` DEFAULT: `False`

RETURNS	DESCRIPTION
`H3CensusWorkflowResult`	Workflow metadata. When `retain_results` is True, `results` contains one
`H3CensusWorkflowResult`	GeoDataFrame per selected year. When it is False, `results` is empty and
`H3CensusWorkflowResult`	row counts plus validation tables are available in lightweight metadata
`H3CensusWorkflowResult`	dictionaries.

RAISES	DESCRIPTION
`ValueError`	If no year is supplied or an unsupported year is requested.
`FileNotFoundError`	If H3 estimation is requested before `census_data.gpkg` exists.

GeoData Module

H3 census estimates

H3 intermediate storage

Parallel execution and benchmarking

Configuring the benchmark script

End-to-end H3 workflow

Special and fictitious census sections

read_administrative_boundaries(file_path, target_columns, index_column, column_remapping=None, output_folder=None, layer_name=None)

read_census(shp_folder, target_columns, tipo_loc_mapping, column_remapping=None, output_folder=None, layer_name=None)

finalize_h3_result_chunks_to_geopackage(intermediate_workspace, output_path, *, output_layer=None, conservation_tolerance=1e-06, index_batch_size=H3_CHUNK_FINALIZATION_INDEX_BATCH_SIZE)

generate_h3_grid(aoi, h3_resolution=REQUIRED_H3_RESOLUTION, *, aoi_crs=WGS84_CRS, area_crs=DEFAULT_AREA_CRS, boundary_ring_size=DEFAULT_BOUNDARY_RING_SIZE, output_crs=WGS84_CRS)

H3CensusWorkflowResult dataclass

`read_administrative_boundaries(file_path, target_columns, index_column, column_remapping=None, output_folder=None, layer_name=None)`

`read_census(shp_folder, target_columns, tipo_loc_mapping, column_remapping=None, output_folder=None, layer_name=None)`

`finalize_h3_result_chunks_to_geopackage(intermediate_workspace, output_path, *, output_layer=None, conservation_tolerance=1e-06, index_batch_size=H3_CHUNK_FINALIZATION_INDEX_BATCH_SIZE)`

`generate_h3_grid(aoi, h3_resolution=REQUIRED_H3_RESOLUTION, *, aoi_crs=WGS84_CRS, area_crs=DEFAULT_AREA_CRS, boundary_ring_size=DEFAULT_BOUNDARY_RING_SIZE, output_crs=WGS84_CRS)`

`H3CensusWorkflowResult` `dataclass`