Utilities Functions

Utilities Functions#

utils.py classes, methods, and modules

ogcore.utils#

class ogcore.utils.Inequality(dist, pop_weights, ability_weights, S, J)[source]#

A class with methods to compute different measures of inequality.

gini(type='overall')[source]#

Compute the Gini coefficient

Parameters:

None

Returns:

Gini coefficient

Return type:

gini_coeff (scalar)

pct(pct)[source]#

Returns value at given percentile

Parameters:

pct1 (scalar) – percentile to compute the value at, in (0, 1).

Returns:

value of variable at pct

Return type:

value (scalar)

ratio_pct1_pct2(pct1, pct2)[source]#

Compute the pct1/pct2 percentile ratio

Parameters:

  • pct1 (scalar) – percentile to compute the top pctile% for, in (0, 1).

  • pct2 (scalar) – percentile to compute the top pctile% for, in (0, 1)

Returns:

ratio of pct1 to pct2

Return type:

pct_ratio (scalar)

Notes

usually pct1 > pct 2

top_share(pctile)[source]#

Compute the top X% share

Parameters:

pctile (scalar) – percentile to compute the top pctile% for, in (0, 1).

Returns:

share of variable attributed to the

top pctile group

Return type:

pctile_share (scalar)

var_of_logs()[source]#

Compute the variance of logs

Parameters:

None

Returns:

variance of logs

Return type:

var_ln_dist (scalar)

class ogcore.utils.CustomHttpAdapter(ssl_context=None, **kwargs)[source]#

The UN Data Portal server doesn’t support “RFC 5746 secure renegotiation”. This causes and error when the client is using OpenSSL 3, which enforces that standard by default. The fix is to create a custom SSL context that allows for legacy connections. This defines a function get_legacy_session() that should be used instead of requests().

init_poolmanager(connections, maxsize, block=False)[source]#

Initializes a urllib3 PoolManager.

This method should not be called from user code, and is only exposed for use when subclassing the HTTPAdapter.

Parameters:

  • connections – The number of urllib3 connection pools to cache.

  • maxsize – The maximum number of connections to save in the pool.

  • block – Block when no free connections are available.

  • pool_kwargs – Extra keyword arguments used to initialize the Pool Manager.

ogcore.utils.avg_by_bin(x, y, weights=None, bins=10, eql_pctl=True)[source]#

For an x series and y series (vectors), this function computes vectors of weighted average values in specified bins.

Parameters:

  • x (array_like, either pd.Series or np.array) – x values

  • y (array_like, either pd.Series or np.array) – y values

  • weights (array_like, either pd.Series or np.array) – weights

  • bins (scalar, list, or np.array) – number of bins or specific bin edges

  • eql_pctl (boolean) – if True, bins are equal percentiles of x

Returns:

weighted average x values in bins y_binned (array_like, np.array): weighted average y values in bins weights_binned (array_like, np.array): total weights in bins

Return type:

x_binned(array_like, np.array)

ogcore.utils.comp_array(name, a, b, tol, unequal, exceptions={}, relative=False)[source]#

Compare two arrays in the L inifinity norm. Return True if | a - b | < tol, False otherwise. If not equal, add items to the unequal list name: the name of the value being compared

Parameters:

  • name (str) – name of variable being compared

  • a (array_like) – first array to compare

  • b (array_like) – second array to compare

  • tol (scalar) – tolerance used for comparison

  • unequal (dict) – dict of variables that are not equal

  • exceptions (dict) – exceptions

  • relative (bool) – whether comparison compares relative values

Returns:

whether two arrays are the same or not

Return type:

(bool)

ogcore.utils.comp_scalar(name, a, b, tol, unequal, exceptions={}, relative=False)[source]#

Compare two scalars in the L inifinity norm. Return True if abs(a - b) < tol, False otherwise. If not equal, add items to the unequal list.

Parameters:

  • name (str) – name of variable being compared

  • a (scalar) – first scalar to compare

  • b (scalra) – second scalar to compare

  • tol (scalar) – tolerance used for comparison

  • unequal (list) – list of variables that are not equal

  • exceptions (dict) – exceptions

  • relative (bool) – whether comparison compares relative values

Returns:

whether two arrays are the same or not

Return type:

(bool)

ogcore.utils.convex_combo(var1, var2, nu)[source]#

Takes the convex combination of two variables, where nu is in [0,1].

Parameters:

  • var1 (array_like) – any shape, variable 1

  • var2 (array_like) – same shape as var1, variable 2

  • nu (scalar) – weight on var1 in convex combination, in [0, 1]

Returns:

same shape as var1, convex combination of var1 and var2

Return type:

combo (array_like)

ogcore.utils.dict_compare(name1, dict1, name2, dict2, tol, verbose=False, exceptions={}, relative=False)[source]#

Compare two dictionaries. The values of each dict are either numpy arrays or else types that are comparable with the == operator. For arrays, they are considered the same if |x - y| < tol in the L_inf norm. For scalars, they are considered the same if x - y < tol.

Parameters:

  • name1 (str) – name of dictionary 1

  • dict1 (dict) – first dictionary to compare

  • name2 (str) – name of dictionary 2

  • dict2 (dict) – second dictionary to compare

  • tol (scalar) – tolerance used for comparison

  • verbose (bool) – whether print messages

  • exceptions (dict) – exceptions

  • relative (bool) – whether comparison compares relative values

Returns:

whether two dictionaries are the same or not

Return type:

(bool)

ogcore.utils.fetch_files_from_web(file_urls)[source]#

Fetches zip files from respective web addresses and saves them as temporary files. Prints progress bar as it downloads the files.

Parameters:

file_urls (list of strings) – list of URLs of respective data zip files

Functions called:

print_progress()

Objects created within function:

local_paths = list, local paths for teporary files iteration = int, the number of files that have been downloaded total = total, the total number of files to download f = temporary file of monthly CPS survey path = string, local path for temporary file zipped_file = ZipFile object, opened zipfile

Files created by this function:

.dta file for each year of SCF data

Returns:

local paths of temporary data

files

Return type:

local_paths (list of strings)

ogcore.utils.get_initial_path(x1, xT, p, shape)[source]#

This function generates a path from point x1 to point xT such that that the path x is a linear or quadratic function of time t.

  • linear: x = d*t + e

  • quadratic: x = a*t^2 + b*t + c

Parameters:

  • x1 (scalar) – initial value of the function x(t) at t=0

  • xT (scalar) – value of the function x(t) at t=T-1

  • T (int) – number of periods of the path, must be >= 3

  • shape (str) – shape of guess for time path, “linear”, “ratio”, or “quadratic”

Returns:

guess of variable over the time path

Return type:

xpath (Numpy array)

Notes

The identifying assumptions for quadratic are the following:

  1. x1 is the value at time t=0: x1 = c

  2. xT is the value at time t=T-1: xT = a*(T-1)^2 + b*(T-1) + c

  3. the slope of the path at t=T-1 is 0: 0 = 2*a*(T-1) + b

ogcore.utils.mkdirs(path)[source]#

Makes directories to save output.

Parameters:

path (str) – path name for new directory

Returns:

None

ogcore.utils.not_connected(url='http://www.google.com/', timeout=5)[source]#

Checks for internet connection status of machine.

Parameters:

  • url (string) – url used to check connectivity

  • timeout (float>0) – time to wait for timeout

Functions called: None

Returns:

=True if connection was made within timeout

Return type:

Boolean singleton

Raises:

ConnectionError – If no response from url withing timeout

ogcore.utils.params_to_json(p, path=None)[source]#

This function creates a JSON file with the model parameters of the format used for the default_parameters.json file.

Parameters:

  • p (OG-Core Specifications class) – model parameters object

  • path (string) – path to save JSON file to

Returns:

JSON on model parameters

Return type:

JSON (string)

ogcore.utils.pct_diff_func(simul, data)[source]#

Used to calculate the absolute percent difference between data moments and model moments.

Parameters:

  • simul (array_like) – any shape, model moments

  • data (array_like) – same shape as simul, data moments

Functions called: None

Returns:

percentage differences between model and

data moments

Return type:

output (array_like)

ogcore.utils.pickle_file_compare(fname1, fname2, tol=0.001, exceptions={}, relative=False)[source]#

Read two pickle files and unpickle each. We assume that each resulting object is a dictionary. The values of each dict are either numpy arrays or else types that are comparable with the == operator.

Parameters:

  • fname1 (str) – file name of file 1

  • fname2 (str) – file name of file 2

  • tol (scalar) – tolerance

  • exceptions (dict) – exceptions

  • relative (bool) – whether comparison compares relative values

Returns:

whether therea two dictionaries are the same

Return type:

comparison (bool)

ogcore.utils.print_progress(iteration, total, source_name='', prefix='Progress:', suffix='Complete', decimals=1, bar_length=50)[source]#

Prints a progress bar to the terminal when completing small tasks of a larger job.

Parameters:

  • iteration (int>=1) – which task the job is currently doing

  • total (int>=1) – how many tasks are in the job

  • source_name (string) – name of source data

  • prefix (string) – what to print before the progress bar

  • suffix (string) – what to print after the progress bar

  • decimals (int>=0) – how many decimals in the percentage

  • bar_length (int>=3) – how many boxes in the progress bar

Functions called: None

Objects created within function:

status (string): status of download str_format (string): string containing percentage completed percents (string): percentage completed filled_length (int): number of boxes in the progress bar to fill bar (string): progress bar

Returns: status

ogcore.utils.rate_conversion(annual_rate, start_age, end_age, S)[source]#

This function converts annual rates to model period ratesself.

Parameters:

  • annual_rate (array_like) – annualized rates

  • start_age (int) – age at which agents become economically active

  • end_age (int) – maximum age of agents

  • S (int) – number of model periods in agents life

Returns:

model period rates

Return type:

rate (array_like)

ogcore.utils.safe_read_pickle(file_path)[source]#

This function reads a pickle from Python 2 into Python 2 or Python 3

Parameters:

file_path (str) – path to pickle file

Returns:

object saved in pickle file

Return type:

obj (object)

ogcore.utils.save_return_table(table_df, output_type, path, precision=2)[source]#

Function to save or return a table of data.

Parameters:

  • table_df (Pandas DataFrame) – table

  • output_type (string) – specifies the type of file to save table to: ‘csv’, ‘tex’, ‘excel’, ‘json’, ‘md’

  • path (string) – specifies path to save file with table to

  • precision (integer) – number of significant digits to print. Defaults to 0.

Returns:

table

Return type:

table_df (Pandas DataFrame)

ogcore.utils.shift_bio_clock(param_in, initial_effect_period=1, final_effect_period=1, total_effect=1, min_age_effect_felt=20, bound_below=False, bound_above=False, use_spline=False)[source]#

This function takes an initial array of parameters that has a time dimension and an age dimension. It then applies a shift along the age dimension (i.e., a change in the “biological clock”) and phases this shift in over some period of time.

Parameters:

  • param_in (Numpy array) – initial parameter values, first two dimensions must be time then age, respectively

  • initial_effect_period (int) – first model period when transition to new parameter values occurs

  • final_effect_period (int) – model period when effect is fully phased in (so transition from param_in to param_out happens between model periods initial_effect_period and final_effect_period)

  • total_effect (float) – total number of model periods to shift the biological clock (allows for fractions of periods)

  • min_age_effect_felt (int) – minimum age at which the effect of the transition is felt in model periods

  • bound_below (bool) – whether param_out bounded below by param_in

  • bound_above (bool) – whether param_out bounded above by param_in

  • use_spline (bool) – whether to use a cubic spline to interpolate tail of param_out

Returns:

updated parameter values

Return type:

param_out (Numpy array)

ogcore.utils.to_timepath_shape(some_array)[source]#

This function takes an vector of length T and tiles it to fill a Tx1x1 array for time path computations.

Parameters:

some_array (Numpy array) – array to reshape

Returns:

reshaped array

Return type:

tp_array (Numpy array)

ogcore.utils.unstationarize_vars(var, tpi_vars, params)[source]#

This function takes a model variable and returns the unstationarized value by adjusting for underlying growth in labor productivity and population growth (where necessary).

Parameters:

  • var (string) – name of variable to unstationarize

  • tpi_vars (Numpy array) – time path of the output variables from the baseline run

  • params (Specifications object) – parameters from the baseline run

Returns:

unstationarized values for var

Return type:

pct_changes (array_like)