tradestat_multiprocessing_with_DB

tradestat_multiprocessing_with_DB is a Python program to generate meaningful reports of HK’s external merchandise trade statistics from numerical raw data issued by authorized department with object-oriented and multiprocessing concept, using SQLite as database.

link not valid

  • 4 types of reports are provided
  • HK’s external merchandise trade by currency (HKD, USD) and dollar units (thousand, million) with: Country (total number: 214 x 4 = 856 excel files in total)

Developing or suggested working environment:

  • Python version 3.8 or above
  • Window 10

Dependencies: suggest most updated version

  1. pandas
  2. NumPy
  3. xlsxwriter
  4. openpyxl
  5. multiprocessing
  6. sqlite3

Folders/Files description:

  1. C&SD_raw_data A few of periods of raw data in DAT format to demonstrate, and description file can be found

  2. Output Full completed reports in Excel format as examples can be downloaded as demo

  3. trades_metafiles_into_DB.py trades_metafiles_into_DB.py will insert geography, commodity, industry, product code, etc in a sqlite database by following functions: import_geography_code() import_sitctohs_code() import_hs_code() import_sitc_code() import_industry_code()

  4. trades_records_into_DB.py insert all the commodity trade records for HK with other countries into sqlite database in the folder merchandise_trades_DB

    the link not valid

  5. metadata stores the mapping table or metadata files

  6. GeneralTrades_getdata.py CommodityTrades_getdata.py These 2 files involve SQL query embedded into python modules to acquire data from SQLite database

  7. TradeReports_analysis_multiprocessing.py This module performs multiprocessing by apply_async function to export Excel Reports. It run around 230 seconds to export 856 excel reports (0.27 seconds used per one) by following codes:

    p = multiprocessing.Pool(processes = multiprocessing.cpu_count())

    for row in reports.acquire_countries_info().itertuples():   try:     p.apply_async(CountryReport,(all_figs, periods, 10, row.CODE)) p.close() p.join()

GitHub

View Github