Data Module

This module handles non-geographic data.

New code should import from istat_census_data. The reference below is generated from the implementation module so the full function documentation remains visible.

`preprocess_data(data_folder, data_column_remapping=None, add_administrative_informations=None, regions_data_path=None, regions_target_columns=None, provinces_data_path=None, provinces_target_columns=None, municipalities_data_path=None, municipalities_target_columns=None, output_folder=None)`

Preprocess census CSV files and return aggregated data and trace record.

This function performs the following operations:

Searches for all CSV files in the specified folder.
Uses the last CSV file (in alphabetical order) as the trace record file.
Loads and concatenates all other CSV files into a single DataFrame.
Applies column name remapping if data_column_remapping is provided.
Adds administrative information (regions, provinces, municipalities) if requested.
Replaces any NaN values with 0.
Loads the trace record file into a dedicated DataFrame.
Returns either:
A dictionary containing census_data and trace DataFrames, or
Saves the resulting CSV files to output_folder and returns the path.

PARAMETER	DESCRIPTION
`data_folder`	Folder containing the CSV files to process. TYPE: `Path`
`data_column_remapping`	Optional dictionary for renaming census dataset columns (e.g., `{"pro_com": "PRO_COM"}`). TYPE: `dict \| None` DEFAULT: `None`
`add_administrative_informations`	If True, enriches data with administrative information (regions, provinces, municipalities) via `add_administrative_info()`. TYPE: `bool \| None` DEFAULT: `None`
`regions_data_path`	Optional path to the file containing region data. TYPE: `Path \| None` DEFAULT: `None`
`regions_target_columns`	Optional list of columns to extract/keep for region data. TYPE: `list \| None` DEFAULT: `None`
`provinces_data_path`	Optional path to the file containing province data. TYPE: `Path \| None` DEFAULT: `None`
`provinces_target_columns`	Optional list of columns to extract/keep for province data. TYPE: `list \| None` DEFAULT: `None`
`municipalities_data_path`	Optional path to the file containing municipality data. TYPE: `Path \| None` DEFAULT: `None`
`municipalities_target_columns`	Optional list of columns to extract/keep for municipality data. TYPE: `list \| None` DEFAULT: `None`
`output_folder`	Optional destination folder where the following files will be saved: - `census_data.csv` for concatenated data - `census_trace.csv` for the trace record If None, data is returned as a dictionary of DataFrames. TYPE: `Path \| None` DEFAULT: `None`

RETURNS	DESCRIPTION
`dict \| Path`	Either a dictionary with keys: - `"census_data"`: DataFrame containing concatenated census data - `"trace"`: DataFrame containing field trace record
`dict \| Path`	Or the path to `output_folder` if specified, where `census_data.csv` and
`dict \| Path`	`census_trace.csv` have been saved.

RAISES	DESCRIPTION
`ValueError`	If no CSV files are found in the specified folder.

Note

The trace record file is considered to be the last CSV in alphabetical order within data_folder. The check_encoding() function is used to determine the correct encoding for CSV files.

Catalogue ISTAT census indicators and curated cross-year comparability.

`get_census_index_catalog(year=None, trace_files=None)`

Return census indicator metadata for one year or all supported years.

The default catalogue is read from a packaged JSON resource generated from local ISTAT trace records and curated semantic mappings. Unknown fields are still listed, but they do not receive a cross-year comparable group.

PARAMETER	DESCRIPTION
`year`	Optional census year. If omitted, all supported years are returned. TYPE: `int \| None` DEFAULT: `None`
`trace_files`	Optional mapping from census year to trace CSV path. Values supplied here override the packaged JSON catalogue for those years. TYPE: `Mapping[int, str \| Path] \| None` DEFAULT: `None`

RETURNS	DESCRIPTION
`DataFrame`	DataFrame with one row per `(year, source_code)` and the public catalogue
`DataFrame`	columns documented in `CATALOG_COLUMNS`.

RAISES	DESCRIPTION
`ValueError`	If `year` is not supported or a trace/catalogue shape is invalid.
`FileNotFoundError`	If a selected trace file is missing.

`get_comparable_indexes(years, strict=True, trace_files=None)`

Return indexes with curated semantic comparability across selected years.

PARAMETER	DESCRIPTION
`years`	Census years to compare. TYPE: `Iterable[int]`
`strict`	If True, return only comparable groups present in every selected year. If False, return comparable groups present in at least one selected year and include missing-year metadata. TYPE: `bool` DEFAULT: `True`
`trace_files`	Optional mapping from census year to trace CSV path. Values supplied here override the packaged JSON catalogue for those years. TYPE: `Mapping[int, str \| Path] \| None` DEFAULT: `None`

RETURNS	DESCRIPTION
`DataFrame`	DataFrame containing catalogue rows for comparable groups, plus
`DataFrame`	`available_years`, `missing_years`, and `is_strict_match` columns.

RAISES	DESCRIPTION
`ValueError`	If `years` is empty or contains an unsupported census year.
`FileNotFoundError`	If a selected trace file is missing.

`list_census_indexes(year, trace_files=None)`

List available ISTAT source field codes for one supported census year.

PARAMETER	DESCRIPTION
`year`	Census year to inspect. Supported values are 1991, 2001, 2011, and 2021. TYPE: `int`
`trace_files`	Optional mapping from census year to trace CSV path. Values supplied here override the packaged JSON catalogue for those years. TYPE: `Mapping[int, str \| Path] \| None` DEFAULT: `None`

RETURNS	DESCRIPTION
`list[str]`	Source field codes in the same order as the trace record.

RAISES	DESCRIPTION
`ValueError`	If `year` is not supported or the trace/catalogue shape is invalid.
`FileNotFoundError`	If the selected trace file is missing.