historical_inputs modules
The module provides tools for accessing historical market data and preprocessing for compatibility with the SpotMarket class.
xml_cache
Classes:
|
Class for accessing data stored in AEMO's NEMDE output files. |
Exceptions:
Raise for unable to downloaded data from NEMWeb. |
- class nempy.historical_inputs.xml_cache.XMLCacheManager(cache_folder)
Class for accessing data stored in AEMO’s NEMDE output files.
Examples
A XMLCacheManager instance is created by providing the path to directory containing the cache of XML files.
>>> manager = XMLCacheManager('test_nemde_cache')
- Parameters:
cache_folder (str)
Methods:
populate(start_year, start_month, end_year, ...)Download data to the cache from the AEMO website.
populate_by_day(start_year, start_month, ...)Download data to the cache from the AEMO website.
load_interval(interval)Load the data for particular 5 min dispatch interval into memory.
Check if the cache contains the data for the loaded interval, primarily for debugging.
Get the file path to the currently loaded interval.
Get the filename of the currently loaded interval.
Get the initial conditions of units at the start of the dispatch interval.
Get the unit fast start dispatch inflexibility parameter values.
Get the unit volume bids
Get the unit volume bids
Get the unit unconstrained intermittent generation forecast.
Get the total volume violation of different constraint sets.
Get the price of violating different constraint sets.
Check if the interval currently loaded was subject to an intervention.
Get generic constraints rhs values.
Get generic constraints type.
Get generic constraints lhs term regional coefficients.
Get generic constraints lhs term unit coefficients.
Get generic constraints lhs term interconnector coefficients.
Get the bid availability of market interconnectors.
find_intervals_with_violations(limit, ...)Find the set of dispatch intervals where the non-intervention dispatch runs had constraint violations.
Get the energy market and FCAS prices by region.
- populate(start_year, start_month, end_year, end_month, verbose=True)
Download data to the cache from the AEMO website. Data downloaded is inclusive of the start and end month.
- populate_by_day(start_year, start_month, end_year, end_month, start_day, end_day, verbose=True)
Download data to the cache from the AEMO website. Data downloaded is inclusive of the start and end date.
- load_interval(interval)
Load the data for particular 5 min dispatch interval into memory.
If the file intervals data is not on disk then an attempt to download it from AEMO’s NEMweb portal is made.
Examples
>>> manager = XMLCacheManager('test_nemde_cache')
>>> manager.load_interval('2024/07/10 12:05:00')
- Parameters:
interval (str) – In the format ‘%Y/%m/%d %H:%M:%S’
- Raises:
MissingDataError – If the data for an interval is not in the cache and cannot be downloaded from NEMWeb.
- interval_inputs_in_cache()
Check if the cache contains the data for the loaded interval, primarily for debugging.
Examples
>>> manager = XMLCacheManager('test_nemde_cache')
>>> manager.load_interval('2024/07/10 12:05:00')
>>> manager.interval_inputs_in_cache() True
- Return type:
bool
- get_file_path()
Get the file path to the currently loaded interval.
Examples
>>> manager = XMLCacheManager('test_nemde_cache')
>>> manager.load_interval('2024/07/10 12:05:00')
>>> manager.get_file_path() PosixPath('test_nemde_cache/NEMSPDOutputs_2018123124000.loaded')
So the doctest runs on all Operating systems lets also look at the parts of the path.
>>> manager.get_file_path().parts ('test_nemde_cache', 'NEMSPDOutputs_2024071009700.loaded')
- get_file_name()
Get the filename of the currently loaded interval.
Examples
>>> manager = XMLCacheManager('test_nemde_cache')
>>> manager.load_interval('2024/07/10 12:05:00')
>>> manager.get_file_name() 'NEMSPDOutputs_2024071009700.loaded'
- get_unit_initial_conditions()
Get the initial conditions of units at the start of the dispatch interval.
Examples
>>> manager = XMLCacheManager('test_nemde_cache')
>>> manager.load_interval('2024/07/10 12:05:00')
>>> manager.get_unit_initial_conditions() DUID TRADERTYPE INITIALMW RAMPUPRATE RAMPDOWNRATE AGCSTATUS 0 ADPBA1G GENERATOR 0.00000 93.119938 93.119938 1.0 1 ADPBA1L LOAD 1.40400 93.119938 93.119938 1.0 2 ADPPV1 GENERATOR 10.90800 298.499937 298.499937 0.0 3 AGLHAL GENERATOR 0.00000 NaN NaN 0.0 4 AGLSOM GENERATOR 60.00000 NaN NaN 0.0 .. ... ... ... ... ... ... 492 YENDWF1 GENERATOR 6.75000 NaN NaN 0.0 493 YWPS1 GENERATOR 0.00000 180.000000 180.000000 0.0 494 YWPS2 GENERATOR 358.89621 176.624994 176.624994 1.0 495 YWPS3 GENERATOR 371.52658 181.124997 181.124997 1.0 496 YWPS4 GENERATOR 337.93546 180.000000 180.000000 1.0 [497 rows x 6 columns]
- Returns:
Columns:
Description:
DUID
unique identifier of a dispatch unit,
(as str)
INITIALMW
the output or consumption of the unit
at the start of the interval, in MW,
(as np.int64),
RAMPUPRATE
ramp up rate of unit as repoted by the
scada system at the start if the
interval, in MW/h, (as np.int64)
RAMPDOWNRATE
ramp down rate of unit as repoted by the
scada system at the start if the
interval, in MW/h, (as np.int64)
AGCSTATUS
flag to indicate whether the unit is
connected to the AGC system at the
start of the interval, 0.0 if not and
1.0 if it is, (as np.int64)
- Return type:
pd.DataFrame
- get_unit_fast_start_parameters()
Get the unit fast start dispatch inflexibility parameter values.
Examples
>>> manager = XMLCacheManager('test_nemde_cache')
>>> manager.load_interval('2024/07/10 12:05:00')
>>> manager.get_unit_fast_start_parameters() DUID MinLoadingMW CurrentMode CurrentModeTime T1 T2 T3 T4 0 AGLHAL 2 0 0 10 3 10 2 1 AGLSOM 12 4 2 20 2 35 2 2 BARRON-1 5 4 1 12 3 10 1 3 BARRON-2 5 0 0 12 3 10 1 4 BBTHREE1 17 0 0 8 4 1 1 .. ... ... ... ... .. .. .. .. 68 VPGS4 50 0 0 5 8 15 0 69 VPGS5 50 0 0 5 3 15 0 70 VPGS6 50 0 0 5 8 15 0 71 W/HOE#1 40 0 0 4 2 15 0 72 W/HOE#2 40 0 0 4 1 15 0 [73 rows x 8 columns]
- Returns:
Columns:
Description:
DUID
unique identifier of a dispatch unit,
(as str)
MinLoadingMW
in MW, (as np.int64)
CurrentMode
The dispatch mode if the unit at the
start of the interval, for mode
definitions
see AEMO doc,(as np.int64)
CurrentModeTime
The time already spent in the current
mode, in minutes, (as np.int64)
T1
The total length of mode 1, in minutes
(as np.int64)
T2
The total length of mode 2, in minutes
(as np.int64)
T3
The total length of mode 1, in minutes,
(as np.int64)
T4
The total length of mode 4, in minutes,
(as np.int64)
- Return type:
pd.DataFrame
- get_unit_volume_bids()
Get the unit volume bids
Examples
>>> manager = XMLCacheManager('test_nemde_cache')
>>> manager.load_interval('2024/07/10 12:05:00')
# >>> manager.load_interval(‘2024/08/01 02:15:00’)
>>> manager.get_unit_volume_bids() DUID BIDTYPE DIRECTION MAXAVAIL ENABLEMENTMIN ENABLEMENTMAX LOWBREAKPOINT HIGHBREAKPOINT BANDAVAIL1 BANDAVAIL2 BANDAVAIL3 BANDAVAIL4 BANDAVAIL5 BANDAVAIL6 BANDAVAIL7 BANDAVAIL8 BANDAVAIL9 BANDAVAIL10 RAMPDOWNRATE RAMPUPRATE 0 ADPBA1G ENERGY None 6.0 6.0 6.0 6.0 6.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 6.0 0.0 0.0 120.0 120.0 1 ADPBA1G LOWERREG None 6.0 0.0 6.0 6.0 6.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 6.0 0.0 0.0 0.0 0.0 2 ADPBA1G RAISE5MIN None 3.0 0.0 6.0 0.0 3.0 3.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 3 ADPBA1G RAISEREG None 6.0 0.0 6.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 6.0 0.0 0.0 0.0 0.0 4 ADPBA1G RAISE60SEC None 3.0 0.0 6.0 0.0 3.0 3.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... 1725 YWPS4 LOWER6SEC None 0.0 250.0 385.0 275.0 385.0 15.0 10.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1726 YWPS4 RAISE5MIN None 0.0 250.0 390.0 250.0 380.0 0.0 0.0 0.0 0.0 5.0 0.0 0.0 5.0 0.0 10.0 10.0 10.0 1727 YWPS4 RAISEREG None 0.0 250.0 385.0 250.0 370.0 0.0 0.0 0.0 0.0 0.0 0.0 5.0 10.0 0.0 5.0 5.0 5.0 1728 YWPS4 RAISE60SEC None 0.0 220.0 400.0 220.0 390.0 0.0 0.0 0.0 0.0 0.0 5.0 5.0 0.0 0.0 10.0 10.0 10.0 1729 YWPS4 RAISE6SEC None 0.0 220.0 405.0 220.0 390.0 0.0 0.0 0.0 5.0 5.0 5.0 5.0 0.0 0.0 5.0 5.0 5.0 [1730 rows x 20 columns]
- Returns:
Columns:
Description:
DUID
unique identifier of a dispatch unit,
(as str)
DIRECTION
”LOAD” or “GENERATOR”, (as str)
BIDTYPE
the service the bid applies to,
(as str)
MAXAVAIL
the bid in unit availablity, in MW, (as str)
ENABLEMENTMIN
in MW, (as np.float64)
ENABLEMENTMAX
in MW, (as np.float64)
LOWBREAKPOINT
in MW, (as np.float64)
HIGHBREAKPOINT
in MW, (as np.float64)
BANDAVAIL1
the volume bid in the first bid band, in MW, (as np.float64)
:
BANDAVAIL10
the volume bid in the tenth bid band, in MW, (as np.float64)
RAMPDOWNRATE
the bid in ramp down rate, in MW/h, (as np.int64)
RAMPUPRATE
the bid in ramp up rate, in MW/h, (as np.int64)
- Return type:
pd.DataFrame
- get_unit_price_bids()
Get the unit volume bids
Examples
>>> manager = XMLCacheManager('test_nemde_cache')
>>> manager.load_interval('2024/07/10 12:05:00')
>>> manager.get_unit_price_bids() DUID BIDTYPE DIRECTION PRICEBAND1 PRICEBAND2 PRICEBAND3 PRICEBAND4 PRICEBAND5 PRICEBAND6 PRICEBAND7 PRICEBAND8 PRICEBAND9 PRICEBAND10 0 ADPBA1G ENERGY GENERATOR -966.92 0.00 53.28 94.72 165.76 270.34 369.01 984.68 3945.63 9866.53 1 ADPBA1G LOWERREG GENERATOR 5.00 8.00 12.00 18.00 24.00 47.00 98.00 268.00 498.00 12000.00 2 ADPBA1G RAISE5MIN GENERATOR 0.00 1.00 2.00 3.00 4.00 5.00 6.00 100.00 1000.00 15000.00 3 ADPBA1G RAISEREG GENERATOR 5.00 8.00 12.00 18.00 24.00 47.00 98.00 268.00 498.00 12000.00 4 ADPBA1G RAISE60SEC GENERATOR 0.00 1.00 2.00 3.00 4.00 5.00 6.00 100.00 1000.00 15000.00 ... ... ... ... ... ... ... ... ... ... ... ... ... ... 1725 YWPS4 LOWER6SEC GENERATOR 0.03 0.05 0.16 0.30 1.90 25.04 30.04 99.00 4600.00 9899.00 1726 YWPS4 RAISE5MIN GENERATOR 0.36 0.71 1.41 4.33 19.88 28.88 46.88 97.88 558.88 12400.40 1727 YWPS4 RAISEREG GENERATOR 0.05 2.70 9.99 19.99 49.00 95.50 240.00 450.50 950.50 11900.00 1728 YWPS4 RAISE60SEC GENERATOR 0.36 0.84 1.41 4.78 19.88 28.88 46.88 97.88 558.88 11999.00 1729 YWPS4 RAISE6SEC GENERATOR 0.36 0.84 1.41 4.78 19.88 28.88 46.88 97.88 558.88 12299.00 [1730 rows x 13 columns]
- Returns:
Columns:
Description:
DUID
unique identifier of a dispatch unit,
(as str)
BIDTYPE
the service the bid applies to,
(as str)
DIRECTION
”LOAD” or “GENERATOR”
PRICEBAND1
the volume bid in the first bid band, in MW, (as np.float64)
:
PRICEBAND10
the volume bid in the tenth bid band, in MW, (as np.float64)
- Return type:
pd.DataFrame
- get_UIGF_values()
Get the unit unconstrained intermittent generation forecast.
Examples
>>> manager = XMLCacheManager('test_nemde_cache')
>>> manager.load_interval('2024/07/10 12:05:00')
>>> manager.get_UIGF_values() DUID UIGF 0 ADPPV1 10.90800 1 ARWF1 0.00000 2 AVLSF1 55.26000 3 BALDHWF1 59.81800 4 BANGOWF1 41.89800 .. ... ... 165 WSTWYSF1 49.90000 166 WYASF1 33.90909 167 YARANSF1 59.55000 168 YATSF1 20.00000 169 YENDWF1 7.00604 [170 rows x 2 columns]
- Returns:
Columns:
Description:
DUID
unique identifier of a dispatch unit,
(as str)
UGIF
the units generation forecast for end
of the inteval, in MW, (as np.float64)
- Return type:
pd.DataFrame
- get_violations()
Get the total volume violation of different constraint sets.
For more information on the constraint sets
see AMEO docsExamples
>>> manager = XMLCacheManager('test_nemde_cache')
>>> manager.load_interval('2024/07/10 12:05:00')
>>> manager.get_violations() {'regional_demand': 0.0, 'interocnnector': 0.0, 'generic_constraint': 0.0, 'ramp_rate': 0.416, 'unit_capacity': 0.3, 'energy_constraint': 0.0, 'energy_offer': 0.0, 'fcas_profile': 0.0, 'fast_start': 0.0, 'mnsp_ramp_rate': 0.0, 'msnp_offer': 0.0, 'mnsp_capacity': 0.0, 'ugif': 0.0}
- Return type:
dict
- get_constraint_violation_prices()
Get the price of violating different constraint sets.
For more information on the constraint sets
see AMEO docsExamples
>>> manager = XMLCacheManager('test_nemde_cache')
>>> manager.load_interval('2024/07/10 12:05:00')
>>> manager.get_constraint_violation_prices() {'regional_demand': 2625000.0, 'interocnnector': 20125000.0, 'generic_constraint': 525000.0, 'ramp_rate': 20212500.0, 'unit_capacity': 6475000.0, 'energy_offer': 19862500.0, 'fcas_profile': 2712500.0, 'fcas_max_avail': 2712500.0, 'fcas_enablement_min': 1225000.0, 'fcas_enablement_max': 1225000.0, 'fast_start': 19775000.0, 'mnsp_ramp_rate': 20212500.0, 'msnp_offer': 19862500.0, 'mnsp_capacity': 6387500.0, 'uigf': 6737500.0, 'voll': 17500.0, 'tiebreak': 1e-06}
- Return type:
dict
- is_intervention_period()
Check if the interval currently loaded was subject to an intervention.
Examples
>>> manager = XMLCacheManager('test_nemde_cache')
>>> manager.load_interval('2024/07/10 12:05:00')
>>> manager.is_intervention_period() False
- Return type:
bool
- get_constraint_rhs()
Get generic constraints rhs values.
Examples
>>> manager = XMLCacheManager('test_nemde_cache')
>>> manager.load_interval('2024/07/10 12:05:00')
>>> manager.get_constraint_rhs() set rhs 0 #BANGOWF2_E 82.800000 1 #BBATRYL1_E 50.000000 2 #BBATTERY_E 50.000000 3 #BBTHREE3_E 25.000000 4 #BOWWPV1_E 6.100000 ... ... ... 1107 V_T_NIL_BL1 -10125.000000 1108 V_T_NIL_FCSPS 493.111848 1109 V_WDR_NO_SCADA 95.000000 1110 V_WEMENSF_FLT_20 20.000000 1111 V_YATPSF_FLT_20 20.000000 [1112 rows x 2 columns]
- Returns:
Columns:
Description:
set
the unique identifier of the generic
constraint, (as str)
rhs
the rhs value of the constraint,
(as np.float64)
- Return type:
pd.DataFrame
- get_constraint_type()
Get generic constraints type.
Examples
>>> manager = XMLCacheManager('test_nemde_cache')
>>> manager.load_interval('2024/07/10 12:05:00')
>>> manager.get_constraint_type() set type cost 0 #BANGOWF2_E LE 6300000.0 1 #BBATRYL1_E LE 6300000.0 2 #BBATTERY_E LE 6300000.0 3 #BBTHREE3_E LE 6300000.0 4 #BOWWPV1_E LE 6300000.0 ... ... ... ... 1172 V_T_NIL_BL1 GE 6300000.0 1173 V_T_NIL_FCSPS LE 525000.0 1174 V_WDR_NO_SCADA LE 6300000.0 1175 V_WEMENSF_FLT_20 LE 612500.0 1176 V_YATPSF_FLT_20 LE 612500.0 [1177 rows x 3 columns]
- Returns:
Columns:
Description:
set
the unique identifier of the generic
constraint, (as str)
type
the type of constraint, i.e ‘=’, ‘<=’ or
’<=’, (as str)
cost
the cost of violating the constraint,
(as np.float64)
- Return type:
pd.DataFrame
- get_constraint_region_lhs()
Get generic constraints lhs term regional coefficients.
This is a compact way of describing constraints that apply to all units in a region. If a constraint set appears here and also in the unit specific lhs table then the coefficents used in the constraint is the sum of the two coefficients, this can be used to exclude particular units from otherwise region wide constraints.
Examples
>>> manager = XMLCacheManager('test_nemde_cache')
>>> manager.load_interval('2024/07/10 12:05:00')
>>> manager.get_constraint_region_lhs() set region service coefficient 0 D_I+BIP_ML2_L1 NSW1 L1SE 1.0 1 D_I+BIP_ML2_L1 QLD1 L1SE 1.0 2 D_I+BIP_ML2_L1 SA1 L1SE 1.0 3 D_I+BIP_ML2_L1 TAS1 L1SE 1.0 4 D_I+BIP_ML2_L1 VIC1 L1SE 1.0 .. ... ... ... ... 498 F_TASCAP_RREG_0220 NSW1 R5RE 1.0 499 F_TASCAP_RREG_0220 QLD1 R5RE 1.0 500 F_TASCAP_RREG_0220 SA1 R5RE 1.0 501 F_TASCAP_RREG_0220 VIC1 R5RE 1.0 502 F_T_NIL_MINP_R6 TAS1 R6SE 1.0 [503 rows x 4 columns]
- Returns:
Columns:
Description:
set
the unique identifier of the generic
constraint, (as str)
region
the regions the constraint applies in,
(as str)
service
the services the constraint applies too,
(as str)
coefficient
the coefficient of the terms on the lhs,
(as np.float64)
- Return type:
pd.DataFrame
- get_constraint_unit_lhs()
Get generic constraints lhs term unit coefficients.
If a constraint set appears here and also in the region lhs table then the coefficents used in the constraint is the sum of the two coefficients, this can be used to exclude particular units from otherwise region wide constraints.
Examples
>>> manager = XMLCacheManager('test_nemde_cache')
>>> manager.load_interval('2024/07/10 12:05:00')
>>> manager.get_constraint_unit_lhs() set unit service coefficient 0 #BANGOWF2_E BANGOWF2 ENOF 1.0 1 #BBATRYL1_E BBATRYL1 LDOF 1.0 2 #BBATTERY_E BBATTERY ENOF 1.0 3 #BBTHREE3_E BBTHREE3 ENOF 1.0 4 #BOWWPV1_E BOWWPV1 ENOF 1.0 ... ... ... ... ... 17032 V_WDR_NO_SCADA DRXVDX01 DROF 1.0 17033 V_WDR_NO_SCADA DRXVQP01 DROF 1.0 17034 V_WDR_NO_SCADA DRXVQX01 DROF 1.0 17035 V_WEMENSF_FLT_20 WEMENSF1 ENOF 1.0 17036 V_YATPSF_FLT_20 YATSF1 ENOF 1.0 [17037 rows x 4 columns]
- Returns:
Columns:
Description:
set
the unique identifier of the generic
constraint, (as str)
unit
the units the constraint applies in,
(as str)
service
the services the constraint applies too,
(as str)
coefficient
the coefficient of the terms on the lhs,
(as np.float64)
- Return type:
pd.DataFrame
- get_constraint_interconnector_lhs()
Get generic constraints lhs term interconnector coefficients.
Examples
>>> manager = XMLCacheManager('test_nemde_cache')
>>> manager.load_interval('2024/07/10 12:05:00')
>>> manager.get_constraint_interconnector_lhs() set interconnector coefficient 0 DATASNAP_DFS_LS N-Q-MNSP1 1.0 1 DATASNAP_DFS_NCAN N-Q-MNSP1 1.0 2 DATASNAP_DFS_NCWEST N-Q-MNSP1 1.0 3 DATASNAP_DFS_NNTH N-Q-MNSP1 1.0 4 DATASNAP_DFS_NSYD N-Q-MNSP1 1.0 .. ... ... ... 827 V_S_HEYWOOD_UFLS V-SA 1.0 828 V_S_NIL_ROCOF V-SA 1.0 829 V_T_FCSPS_DS T-V-MNSP1 -1.0 830 V_T_NIL_BL1 T-V-MNSP1 1.0 831 V_T_NIL_FCSPS T-V-MNSP1 -1.0 [832 rows x 3 columns]
- Returns:
Columns:
Description:
set
the unique identifier of the generic
constraint, (as str)
interconnector
the interconnector the constraint applies in,
(as str)
coefficient
the coefficient of the terms on the lhs,
(as np.float64)
- Return type:
pd.DataFrame
- get_market_interconnector_link_bid_availability()
Get the bid availability of market interconnectors.
Examples
>>> manager = XMLCacheManager('test_nemde_cache')
>>> manager.load_interval('2024/07/10 12:05:00')
>>> manager.get_market_interconnector_link_bid_availability() interconnector to_region availability 0 T-V-MNSP1 TAS1 478.0 1 T-V-MNSP1 VIC1 594.0
- Returns:
Columns:
Description:
interconnector
the interconnector the constraint applies in,
(as str)
to_region
the direction the bid availability applies to,
(as str)
availability
the availability as bid in by the
interconnector, (as str)
- Return type:
pd.DataFrame
- find_intervals_with_violations(limit, start_year, start_month, end_year, end_month)
Find the set of dispatch intervals where the non-intervention dispatch runs had constraint violations.
Examples
>>> manager = XMLCacheManager('test_nemde_cache')
>>> manager.load_interval('2024/07/10 12:05:00')
>>> manager.find_intervals_with_violations(limit=3, start_year=2019, start_month=1, end_year=2019, end_month=1) {'2019/01/01 00:00:00': ['unit_capacity'], '2019/01/01 00:05:00': ['unit_capacity'], '2019/01/01 00:10:00': ['unit_capacity']}
- Parameters:
limit (int) – number of intervals to find, finds first intervals in chronolgical order
start_year (int) – year to start search
start_month (int) – month to start search
end_year (int) – year to end search
end_month (int) – month to end search
- Return type:
dict
- get_service_prices()
Get the energy market and FCAS prices by region.
Examples
>>> manager = XMLCacheManager('test_nemde_cache')
>>> manager.load_interval('2024/07/10 12:05:00')
>>> manager.get_service_prices() region service price 0 NSW1 ENERGY 53.99972 1 NSW1 RAISE5MIN 0.25 2 NSW1 RAISE60SEC 0.25 3 NSW1 LOWER60SEC 3 4 NSW1 RAISE6SEC 0.38 5 NSW1 LOWER6SEC 1 6 NSW1 RAISE1SEC 0.94 7 NSW1 LOWER1SEC 0.01 8 QLD1 ENERGY -10.4 9 QLD1 RAISE5MIN 0.25 10 QLD1 RAISE60SEC 0.25 11 QLD1 LOWER60SEC 3 12 QLD1 RAISE6SEC 0.38 13 QLD1 LOWER6SEC 1 14 QLD1 RAISE1SEC 0.94 15 QLD1 LOWER1SEC 0.01 16 SA1 ENERGY -30 17 SA1 RAISE5MIN 0.25 18 SA1 RAISE60SEC 0.25 19 SA1 LOWER60SEC 3 20 SA1 RAISE6SEC 0.38 21 SA1 LOWER6SEC 1 22 SA1 RAISE1SEC 0.94 23 SA1 LOWER1SEC 0.01 24 TAS1 ENERGY 260.2 25 TAS1 RAISE5MIN 0.38 26 TAS1 RAISE60SEC 0.38 27 TAS1 LOWER60SEC 0.38 28 TAS1 RAISE6SEC 0.38 29 TAS1 LOWER6SEC 0.38 30 TAS1 RAISE1SEC 0.94 31 TAS1 LOWER1SEC 0 32 VIC1 ENERGY 202.07105 33 VIC1 RAISE5MIN 0.25 34 VIC1 RAISE60SEC 0.25 35 VIC1 LOWER60SEC 3 36 VIC1 RAISE6SEC 0.38 37 VIC1 LOWER6SEC 1 38 VIC1 RAISE1SEC 0.94 39 VIC1 LOWER1SEC 0.01
- Returns:
Columns:
Description:
region
the region (as str)
service
the services (as str), i.e. energy,
lower_1s, lower_5min, etc
price
the price of the service (as np.float64)
- Return type:
pd.DataFrame
mms_db
Classes:
|
Constructs and manages a sqlite database for accessing historical inputs for NEM spot market dispatch. |
|
Manages retrieving dispatch inputs by SETTLEMENTDATE. |
|
Manages retrieving dispatch inputs by INTERVAL_DATETIME. |
|
Manages retrieving dispatch inputs by SETTLEMENTDATE, where inputs are stored on a daily basis. |
|
Manages retrieving dispatch inputs by START_DATE and END_DATE. |
|
Manages retrieving dispatch inputs by matching against the DISPATCHCONSTRAINTS table |
|
Manages retrieving dispatch inputs by EFFECTTIVEDATE and VERSIONNO. |
|
Manages retrieving dispatch inputs by EFFECTTIVEDATE and VERSIONNO. |
|
Manages retrieving dispatch inputs where no filter is require. |
- class nempy.historical_inputs.mms_db.DBManager(connection)
Constructs and manages a sqlite database for accessing historical inputs for NEM spot market dispatch.
Constructs a database if none exists, otherwise connects to an existing database. Specific datasets can be added to the database from AEMO nemweb portal and inputs can be retrieved on a 5 min dispatch interval basis.
Examples
Create the database or connect to an existing one.
>>> import sqlite3 >>> import os
>>> con = sqlite3.connect('historical.db')
Create the database manager.
>>> historical = DBManager(con)
Create a set of default table in the database.
>>> historical.create_tables()
Add data from AEMO nemweb data portal. In this case we are adding data from the table DISPATCHREGIONSUM which contains a dispatch summary by region, the data comes in monthly chunks.
>>> historical.DISPATCHREGIONSUM.add_data(year=2020, month=1)
>>> historical.DISPATCHREGIONSUM.add_data(year=2020, month=2)
This table has an add_data method indicating that data provided by AEMO comes in monthly files that do not overlap. If you need data for multiple months then multiple add_data calls can be made.
Data for a specific 5 min dispatch interval can then be retrieved.
>>> print(historical.DISPATCHREGIONSUM.get_data('2020/01/10 12:35:00').head()) SETTLEMENTDATE REGIONID TOTALDEMAND DEMANDFORECAST INITIALSUPPLY 0 2020/01/10 12:35:00 NSW1 9938.01 34.23926 9902.79199 1 2020/01/10 12:35:00 QLD1 6918.63 26.47852 6899.76270 2 2020/01/10 12:35:00 SA1 1568.04 4.79657 1567.85864 3 2020/01/10 12:35:00 TAS1 1124.05 -3.43994 1109.36963 4 2020/01/10 12:35:00 VIC1 6633.45 37.05273 6570.15527
Some tables will have a set_data method instead of an add_data method, indicating that the most recent data file provided by AEMO contains all historical data for this table. In this case if multiple calls to the set_data method are made the new data replaces the old.
>>> historical.DUDETAILSUMMARY.set_data(year=2020, month=2)
Data for a specific 5 min dispatch interval can then be retrieved.
>>> print(historical.DUDETAILSUMMARY.get_data('2020/01/10 12:35:00').head()) DUID START_DATE END_DATE DISPATCHTYPE CONNECTIONPOINTID REGIONID TRANSMISSIONLOSSFACTOR DISTRIBUTIONLOSSFACTOR SCHEDULE_TYPE SECONDARY_TLF 5628 PLAYFB2 1998/10/25 00:00:00 1999/05/26 00:00:00 GENERATOR SPSD2 SA1 0.9580 1.0 SCHEDULED None 5629 PLAYFB3 1998/10/25 00:00:00 1999/05/26 00:00:00 GENERATOR SPSD3 SA1 0.9580 1.0 SCHEDULED None 5627 PLAYFB1 1998/10/25 00:00:00 1999/05/26 00:00:00 GENERATOR SPSD1 SA1 0.9580 1.0 SCHEDULED None 5630 PLAYFB4 1998/10/25 00:00:00 1999/05/26 00:00:00 GENERATOR SPSD4 SA1 0.9580 1.0 SCHEDULED None 1380 CLOVER1 1999/07/01 00:00:00 1999/10/14 00:00:00 GENERATOR VMBT1 VIC1 1.0244 1.0 SCHEDULED None
Clean up by deleting database created.
>>> con.close() >>> os.remove('historical.db')
- Parameters:
con (sqlite3.connection)
- BIDPEROFFER_D
Unit volume bids by 5 min dispatch intervals.
- Type:
- BIDDAYOFFER_D
Unit price bids by market day.
- Type:
- DISPATCHREGIONSUM
Regional demand terms by 5 min dispatch intervals.
- Type:
- DISPATCHLOAD
Unit operating conditions by 5 min dispatch intervals.
- Type:
- DUDETAILSUMMARY
Unit information by the start and end times of when the information is applicable.
- Type:
- DISPATCHCONSTRAINT
The generic constraints that were used in each 5 min interval dispatch.
- Type:
- GENCONDATA
The generic constraints information, their applicability to a particular dispatch interval is determined by reference to DISPATCHCONSTRAINT.
- SPDREGIONCONSTRAINT
The regional lhs terms in generic constraints, their applicability to a particular dispatch interval is determined by reference to DISPATCHCONSTRAINT.
- SPDCONNECTIONPOINTCONSTRAINT
The connection point lhs terms in generic constraints, their applicability to a particular dispatch interval is determined by reference to DISPATCHCONSTRAINT.
- SPDINTERCONNECTORCONSTRAINT
The interconnector lhs terms in generic constraints, their applicability to a particular dispatch interval is determined by reference to DISPATCHCONSTRAINT.
- INTERCONNECTOR
The the regions that each interconnector links.
- Type:
- INTERCONNECTORCONSTRAINT
Interconnector properties FROMREGIONLOSSSHARE, LOSSCONSTANT, LOSSFLOWCOEFFICIENT, MAXMWIN, MAXMWOUT by EFFECTIVEDATE and VERSIONNO.
- LOSSMODEL
Break points used in linearly interpolating interconnector loss funtctions by EFFECTIVEDATE and VERSIONNO.
- LOSSFACTORMODEL
Coefficients of demand terms in interconnector loss functions.
- DISPATCHINTERCONNECTORRES
Record of which interconnector were used in a particular dispatch interval.
- Type:
Methods:
Drops any existing default tables and creates new ones, this method is generally called a new database.
- create_tables()
Drops any existing default tables and creates new ones, this method is generally called a new database.
Examples
Create the database or connect to an existing one.
>>> import sqlite3 >>> import os
>>> con = sqlite3.connect('historical.db')
Create the database manager.
>>> historical = DBManager(con)
Create a set of default table in the database.
>>> historical.create_tables()
Default tables will now exist, but will be empty.
>>> print(pd.read_sql("Select * from DISPATCHREGIONSUM", con=con)) Empty DataFrame Columns: [SETTLEMENTDATE, REGIONID, TOTALDEMAND, DEMANDFORECAST, INITIALSUPPLY] Index: []
If you added data and then call create_tables again then any added data will be emptied.
>>> historical.DISPATCHREGIONSUM.add_data(year=2020, month=1)
>>> print(pd.read_sql("Select * from DISPATCHREGIONSUM limit 3", con=con)) SETTLEMENTDATE REGIONID TOTALDEMAND DEMANDFORECAST INITIALSUPPLY 0 2020/01/01 00:05:00 NSW1 7245.31 -26.35352 7284.32178 1 2020/01/01 00:05:00 QLD1 6095.75 -24.29639 6129.36279 2 2020/01/01 00:05:00 SA1 1466.53 1.47190 1452.25647
>>> historical.create_tables()
>>> print(pd.read_sql("Select * from DISPATCHREGIONSUM", con=con)) Empty DataFrame Columns: [SETTLEMENTDATE, REGIONID, TOTALDEMAND, DEMANDFORECAST, INITIALSUPPLY] Index: []
Clean up by deleting database created.
>>> con.close() >>> os.remove('historical.db')
- Return type:
None
- class nempy.historical_inputs.mms_db.InputsBySettlementDate(table_name, table_columns, table_primary_keys, con)
Manages retrieving dispatch inputs by SETTLEMENTDATE.
Methods:
get_data(date_time)Retrieves data for the specified date_time e.g. 2019/01/01 11:55:00".
add_data(year, month)"Download data for the given table and time, appends to any existing data.
Creates a table in the sqlite database that the object has a connection to.
- get_data(date_time)
Retrieves data for the specified date_time e.g. 2019/01/01 11:55:00”
Examples
>>> import sqlite3 >>> import os
Set up a database or connect to an existing one.
>>> con = sqlite3.connect('historical.db')
Create the table object.
>>> table = InputsBySettlementDate(table_name='EXAMPLE', table_columns=['SETTLEMENTDATE', 'INITIALMW'], ... table_primary_keys=['SETTLEMENTDATE'], con=con)
Create the table in the database.
>>> table.create_table_in_sqlite_db()
Normally you would use the add_data method to add historical data, but here we will add data directly to the database so some simple example data can be added.
>>> data = pd.DataFrame({ ... 'SETTLEMENTDATE': ['2019/01/01 11:55:00', '2019/01/01 12:00:00'], ... 'INITIALMW': [1.0, 2.0]})
>>> _ = data.to_sql('EXAMPLE', con=con, if_exists='append', index=False)
When we call get_data the output is filtered by SETTLEMENTDATE.
>>> print(table.get_data(date_time='2019/01/01 12:00:00')) SETTLEMENTDATE INITIALMW 0 2019/01/01 12:00:00 2.0
Clean up by closing the database and deleting if its no longer needed.
>>> con.close() >>> os.remove('historical.db')
- Parameters:
date_time (str) – Should be of format ‘%Y/%m/%d %H:%M:%S’, and always a round 5 min interval e.g. 2019/01/01 11:55:00.
- Return type:
pd.DataFrame
- add_data(year, month)
“Download data for the given table and time, appends to any existing data.
Note
This method and its documentation is inherited from the _MultiDataSource class.
Examples
>>> import sqlite3 >>> import os
Set up a database or connect to an existing one.
>>> con = sqlite3.connect('historical.db')
Create the table object.
>>> table = _MultiDataSource(table_name='DISPATCHREGIONSUM', ... table_columns=['SETTLEMENTDATE', 'REGIONID', 'TOTALDEMAND', ... 'DEMANDFORECAST', 'INITIALSUPPLY'], ... table_primary_keys=['SETTLEMENTDATE', 'REGIONID'], con=con)
Create the table in the database.
>>> table.create_table_in_sqlite_db()
Downloading data from http://nemweb.com.au/#mms-data-model into the table.
>>> table.add_data(year=2020, month=1)
Now the database should contain data for this table that is up to date as the end of Janurary.
>>> query = "Select * from DISPATCHREGIONSUM order by SETTLEMENTDATE DESC limit 1;"
>>> print(pd.read_sql_query(query, con=con)) SETTLEMENTDATE REGIONID TOTALDEMAND DEMANDFORECAST INITIALSUPPLY 0 2020/02/01 00:00:00 VIC1 5935.1 -15.9751 5961.77002
If we subsequently add data from an earlier month the old data remains in the table, in addition to the new data.
>>> table.add_data(year=2019, month=1)
>>> print(pd.read_sql_query(query, con=con)) SETTLEMENTDATE REGIONID TOTALDEMAND DEMANDFORECAST INITIALSUPPLY 0 2020/02/01 00:00:00 VIC1 5935.1 -15.9751 5961.77002
Clean up by closing the database and deleting if its no longer needed.
>>> con.close() >>> os.remove('historical.db')
- Parameters:
year (int) – The year to download data for.
month (int) – The month to download data for.
- Return type:
None
- create_table_in_sqlite_db()
Creates a table in the sqlite database that the object has a connection to.
Note
This method and its documentation is inherited from the _MMSTable class.
Examples
>>> import sqlite3 >>> import os
Set up a database or connect to an existing one.
>>> con = sqlite3.connect('historical.db')
Create the table object.
>>> table = _MMSTable(table_name='EXAMPLE', table_columns=['DUID', 'BIDTYPE'], table_primary_keys=['DUID'], ... con=con)
Create the corresponding table in the sqlite database, note this step many not be needed if you have connected to an existing database.
>>> table.create_table_in_sqlite_db()
Now a table exists in the database, but its empty.
>>> print(pd.read_sql("Select * from example", con=con)) Empty DataFrame Columns: [DUID, BIDTYPE] Index: []
Clean up by closing the database and deleting if its no longer needed.
>>> con.close() >>> os.remove('historical.db')
- class nempy.historical_inputs.mms_db.InputsByIntervalDateTime(table_name, table_columns, table_primary_keys, con)
Manages retrieving dispatch inputs by INTERVAL_DATETIME.
Methods:
get_data(date_time)Retrieves data for the specified date_time e.g. 2019/01/01 11:55:00".
add_data(year, month)"Download data for the given table and time, appends to any existing data.
Creates a table in the sqlite database that the object has a connection to.
- get_data(date_time)
Retrieves data for the specified date_time e.g. 2019/01/01 11:55:00”
Examples
>>> import sqlite3 >>> import os
Set up a database or connect to an existing one.
>>> con = sqlite3.connect('historical.db')
Create the table object.
>>> table = InputsByIntervalDateTime(table_name='EXAMPLE', table_columns=['INTERVAL_DATETIME', 'INITIALMW'], ... table_primary_keys=['INTERVAL_DATETIME'], con=con)
Create the table in the database.
>>> table.create_table_in_sqlite_db()
Normally you would use the add_data method to add historical data, but here we will add data directly to the database so some simple example data can be added.
>>> data = pd.DataFrame({ ... 'INTERVAL_DATETIME': ['2019/01/01 11:55:00', '2019/01/01 12:00:00'], ... 'INITIALMW': [1.0, 2.0]})
>>> _ = data.to_sql('EXAMPLE', con=con, if_exists='append', index=False)
When we call get_data the output is filtered by INTERVAL_DATETIME.
>>> print(table.get_data(date_time='2019/01/01 12:00:00')) INTERVAL_DATETIME INITIALMW 0 2019/01/01 12:00:00 2.0
Clean up by closing the database and deleting if its no longer needed.
>>> con.close() >>> os.remove('historical.db')
- Parameters:
date_time (str) – Should be of format ‘%Y/%m/%d %H:%M:%S’, and always a round 5 min interval e.g. 2019/01/01 11:55:00.
- Return type:
pd.DataFrame
- add_data(year, month)
“Download data for the given table and time, appends to any existing data.
Note
This method and its documentation is inherited from the _MultiDataSource class.
Examples
>>> import sqlite3 >>> import os
Set up a database or connect to an existing one.
>>> con = sqlite3.connect('historical.db')
Create the table object.
>>> table = _MultiDataSource(table_name='DISPATCHREGIONSUM', ... table_columns=['SETTLEMENTDATE', 'REGIONID', 'TOTALDEMAND', ... 'DEMANDFORECAST', 'INITIALSUPPLY'], ... table_primary_keys=['SETTLEMENTDATE', 'REGIONID'], con=con)
Create the table in the database.
>>> table.create_table_in_sqlite_db()
Downloading data from http://nemweb.com.au/#mms-data-model into the table.
>>> table.add_data(year=2020, month=1)
Now the database should contain data for this table that is up to date as the end of Janurary.
>>> query = "Select * from DISPATCHREGIONSUM order by SETTLEMENTDATE DESC limit 1;"
>>> print(pd.read_sql_query(query, con=con)) SETTLEMENTDATE REGIONID TOTALDEMAND DEMANDFORECAST INITIALSUPPLY 0 2020/02/01 00:00:00 VIC1 5935.1 -15.9751 5961.77002
If we subsequently add data from an earlier month the old data remains in the table, in addition to the new data.
>>> table.add_data(year=2019, month=1)
>>> print(pd.read_sql_query(query, con=con)) SETTLEMENTDATE REGIONID TOTALDEMAND DEMANDFORECAST INITIALSUPPLY 0 2020/02/01 00:00:00 VIC1 5935.1 -15.9751 5961.77002
Clean up by closing the database and deleting if its no longer needed.
>>> con.close() >>> os.remove('historical.db')
- Parameters:
year (int) – The year to download data for.
month (int) – The month to download data for.
- Return type:
None
- create_table_in_sqlite_db()
Creates a table in the sqlite database that the object has a connection to.
Note
This method and its documentation is inherited from the _MMSTable class.
Examples
>>> import sqlite3 >>> import os
Set up a database or connect to an existing one.
>>> con = sqlite3.connect('historical.db')
Create the table object.
>>> table = _MMSTable(table_name='EXAMPLE', table_columns=['DUID', 'BIDTYPE'], table_primary_keys=['DUID'], ... con=con)
Create the corresponding table in the sqlite database, note this step many not be needed if you have connected to an existing database.
>>> table.create_table_in_sqlite_db()
Now a table exists in the database, but its empty.
>>> print(pd.read_sql("Select * from example", con=con)) Empty DataFrame Columns: [DUID, BIDTYPE] Index: []
Clean up by closing the database and deleting if its no longer needed.
>>> con.close() >>> os.remove('historical.db')
- class nempy.historical_inputs.mms_db.InputsByDay(table_name, table_columns, table_primary_keys, con)
Manages retrieving dispatch inputs by SETTLEMENTDATE, where inputs are stored on a daily basis.
Methods:
get_data(date_time)Retrieves data for the specified date_time e.g. 2019/01/01 11:55:00, where inputs are stored on daily basis.
add_data(year, month)"Download data for the given table and time, appends to any existing data.
Creates a table in the sqlite database that the object has a connection to.
- get_data(date_time)
Retrieves data for the specified date_time e.g. 2019/01/01 11:55:00, where inputs are stored on daily basis.
Note that a market day begins with the first 5 min interval as 04:05:00, there for if and input date_time of 2019/01/01 04:05:00 is given inputs where the SETTLEMENDATE is 2019/01/01 00:00:00 will be retrieved and if a date_time of 2019/01/01 04:00:00 or earlier is given then inputs where the SETTLEMENDATE is 2018/12/31 00:00:00 will be retrieved.
Examples
>>> import sqlite3 >>> import os
Set up a database or connect to an existing one.
>>> con = sqlite3.connect('historical.db')
Create the table object.
>>> table = InputsByDay(table_name='EXAMPLE', table_columns=['SETTLEMENTDATE', 'INITIALMW'], ... table_primary_keys=['SETTLEMENTDATE'], con=con)
Create the table in the database.
>>> table.create_table_in_sqlite_db()
Normally you would use the add_data method to add historical data, but here we will add data directly to the database so some simple example data can be added.
>>> data = pd.DataFrame({ ... 'SETTLEMENTDATE': ['2019/01/01 00:00:00', '2019/01/02 00:00:00'], ... 'INITIALMW': [1.0, 2.0]})
>>> _ = data.to_sql('EXAMPLE', con=con, if_exists='append', index=False)
When we call get_data the output is filtered by SETTLEMENTDATE and the results from the appropriate market day starting at 04:05:00 are retrieved. In the results below note when the output changes
>>> print(table.get_data(date_time='2019/01/01 12:00:00')) SETTLEMENTDATE INITIALMW 0 2019/01/01 00:00:00 1.0
>>> print(table.get_data(date_time='2019/01/02 04:00:00')) SETTLEMENTDATE INITIALMW 0 2019/01/01 00:00:00 1.0
>>> print(table.get_data(date_time='2019/01/02 04:05:00')) SETTLEMENTDATE INITIALMW 0 2019/01/02 00:00:00 2.0
>>> print(table.get_data(date_time='2019/01/02 12:00:00')) SETTLEMENTDATE INITIALMW 0 2019/01/02 00:00:00 2.0
Clean up by closing the database and deleting if its no longer needed.
>>> con.close() >>> os.remove('historical.db')
- Parameters:
date_time (str) – Should be of format ‘%Y/%m/%d %H:%M:%S’, and always a round 5 min interval e.g. 2019/01/01 11:55:00.
- Return type:
pd.DataFrame
- add_data(year, month)
“Download data for the given table and time, appends to any existing data.
Note
This method and its documentation is inherited from the _MultiDataSource class.
Examples
>>> import sqlite3 >>> import os
Set up a database or connect to an existing one.
>>> con = sqlite3.connect('historical.db')
Create the table object.
>>> table = _MultiDataSource(table_name='DISPATCHREGIONSUM', ... table_columns=['SETTLEMENTDATE', 'REGIONID', 'TOTALDEMAND', ... 'DEMANDFORECAST', 'INITIALSUPPLY'], ... table_primary_keys=['SETTLEMENTDATE', 'REGIONID'], con=con)
Create the table in the database.
>>> table.create_table_in_sqlite_db()
Downloading data from http://nemweb.com.au/#mms-data-model into the table.
>>> table.add_data(year=2020, month=1)
Now the database should contain data for this table that is up to date as the end of Janurary.
>>> query = "Select * from DISPATCHREGIONSUM order by SETTLEMENTDATE DESC limit 1;"
>>> print(pd.read_sql_query(query, con=con)) SETTLEMENTDATE REGIONID TOTALDEMAND DEMANDFORECAST INITIALSUPPLY 0 2020/02/01 00:00:00 VIC1 5935.1 -15.9751 5961.77002
If we subsequently add data from an earlier month the old data remains in the table, in addition to the new data.
>>> table.add_data(year=2019, month=1)
>>> print(pd.read_sql_query(query, con=con)) SETTLEMENTDATE REGIONID TOTALDEMAND DEMANDFORECAST INITIALSUPPLY 0 2020/02/01 00:00:00 VIC1 5935.1 -15.9751 5961.77002
Clean up by closing the database and deleting if its no longer needed.
>>> con.close() >>> os.remove('historical.db')
- Parameters:
year (int) – The year to download data for.
month (int) – The month to download data for.
- Return type:
None
- create_table_in_sqlite_db()
Creates a table in the sqlite database that the object has a connection to.
Note
This method and its documentation is inherited from the _MMSTable class.
Examples
>>> import sqlite3 >>> import os
Set up a database or connect to an existing one.
>>> con = sqlite3.connect('historical.db')
Create the table object.
>>> table = _MMSTable(table_name='EXAMPLE', table_columns=['DUID', 'BIDTYPE'], table_primary_keys=['DUID'], ... con=con)
Create the corresponding table in the sqlite database, note this step many not be needed if you have connected to an existing database.
>>> table.create_table_in_sqlite_db()
Now a table exists in the database, but its empty.
>>> print(pd.read_sql("Select * from example", con=con)) Empty DataFrame Columns: [DUID, BIDTYPE] Index: []
Clean up by closing the database and deleting if its no longer needed.
>>> con.close() >>> os.remove('historical.db')
- class nempy.historical_inputs.mms_db.InputsStartAndEnd(table_name, table_columns, table_primary_keys, con)
Manages retrieving dispatch inputs by START_DATE and END_DATE.
Methods:
get_data(date_time)Retrieves data for the specified date_time by START_DATE and END_DATE.
Creates a table in the sqlite database that the object has a connection to.
set_data(year, month)"Download data for the given table and time, replace any existing data.
- get_data(date_time)
Retrieves data for the specified date_time by START_DATE and END_DATE.
Records with a START_DATE before or equal to the date_times and an END_DATE after the date_time will be returned.
Examples
>>> import sqlite3 >>> import os
Set up a database or connect to an existing one.
>>> con = sqlite3.connect('historical.db')
Create the table object.
>>> table = InputsStartAndEnd(table_name='EXAMPLE', table_columns=['DUID', 'START_DATE', 'END_DATE', 'INITIALMW'], ... table_primary_keys=['START_DATE'], con=con)
Create the table in the database.
>>> table.create_table_in_sqlite_db()
Normally you would use the add_data method to add historical data, but here we will add data directly to the database so some simple example data can be added.
>>> data = pd.DataFrame({ ... 'DUID': ['A', 'A'], ... 'START_DATE': ['2019/01/01 00:00:00', '2019/01/02 00:00:00'], ... 'END_DATE': ['2019/01/02 00:00:00', '2019/01/03 00:00:00'], ... 'INITIALMW': [1.0, 2.0]})
>>> _ = data.to_sql('EXAMPLE', con=con, if_exists='append', index=False)
When we call get_data the output is filtered by START_DATE and END_DATE.
>>> print(table.get_data(date_time='2019/01/01 00:00:00')) DUID START_DATE END_DATE INITIALMW 0 A 2019/01/01 00:00:00 2019/01/02 00:00:00 1.0
>>> print(table.get_data(date_time='2019/01/01 12:00:00')) DUID START_DATE END_DATE INITIALMW 0 A 2019/01/01 00:00:00 2019/01/02 00:00:00 1.0
>>> print(table.get_data(date_time='2019/01/02 00:00:00')) DUID START_DATE END_DATE INITIALMW 1 A 2019/01/02 00:00:00 2019/01/03 00:00:00 2.0
>>> print(table.get_data(date_time='2019/01/02 00:12:00')) DUID START_DATE END_DATE INITIALMW 1 A 2019/01/02 00:00:00 2019/01/03 00:00:00 2.0
Clean up by closing the database and deleting if its no longer needed.
>>> con.close() >>> os.remove('historical.db')
- Parameters:
date_time (str) – Should be of format ‘%Y/%m/%d %H:%M:%S’, and always a round 5 min interval e.g. 2019/01/01 11:55:00.
- Return type:
pd.DataFrame
- create_table_in_sqlite_db()
Creates a table in the sqlite database that the object has a connection to.
Note
This method and its documentation is inherited from the _MMSTable class.
Examples
>>> import sqlite3 >>> import os
Set up a database or connect to an existing one.
>>> con = sqlite3.connect('historical.db')
Create the table object.
>>> table = _MMSTable(table_name='EXAMPLE', table_columns=['DUID', 'BIDTYPE'], table_primary_keys=['DUID'], ... con=con)
Create the corresponding table in the sqlite database, note this step many not be needed if you have connected to an existing database.
>>> table.create_table_in_sqlite_db()
Now a table exists in the database, but its empty.
>>> print(pd.read_sql("Select * from example", con=con)) Empty DataFrame Columns: [DUID, BIDTYPE] Index: []
Clean up by closing the database and deleting if its no longer needed.
>>> con.close() >>> os.remove('historical.db')
- set_data(year, month)
“Download data for the given table and time, replace any existing data.
Note
This method and its documentation is inherited from the _SingleDataSource class.
Examples
>>> import sqlite3 >>> import os
Set up a database or connect to an existing one.
>>> con = sqlite3.connect('historical.db')
Create the table object.
>>> table = _SingleDataSource(table_name='DUDETAILSUMMARY', ... table_columns=['DUID', 'START_DATE', 'CONNECTIONPOINTID', 'REGIONID'], ... table_primary_keys=['START_DATE', 'DUID'], con=con)
Create the table in the database.
>>> table.create_table_in_sqlite_db()
Downloading data from http://nemweb.com.au/#mms-data-model into the table.
>>> table.set_data(year=2020, month=1)
Now the database should contain data for this table that is up to date as the end of Janurary.
>>> query = "Select * from DUDETAILSUMMARY order by START_DATE DESC limit 1;"
>>> print(pd.read_sql_query(query, con=con)) DUID START_DATE CONNECTIONPOINTID REGIONID 0 URANQ11 2020/02/04 00:00:00 NURQ1U NSW1
However if we subsequently set data from a previous date then any existing data will be replaced. Note the change in the most recent record in the data set below.
>>> table.set_data(year=2019, month=1)
>>> print(pd.read_sql_query(query, con=con)) DUID START_DATE CONNECTIONPOINTID REGIONID 0 WEMENSF1 2019/03/04 00:00:00 VWES2W VIC1
Clean up by closing the database and deleting if its no longer needed.
>>> con.close() >>> os.remove('historical.db')
- Parameters:
year (int) – The year to download data for.
month (int) – The month to download data for.
- Return type:
None
- class nempy.historical_inputs.mms_db.InputsByMatchDispatchConstraints(table_name, table_columns, table_primary_keys, con)
Manages retrieving dispatch inputs by matching against the DISPATCHCONSTRAINTS table
Methods:
get_data(date_time)Retrieves data for the specified date_time by matching against the DISPATCHCONSTRAINT table.
Creates a table in the sqlite database that the object has a connection to.
set_data(year, month)"Download data for the given table and time, replace any existing data.
- get_data(date_time)
Retrieves data for the specified date_time by matching against the DISPATCHCONSTRAINT table.
First the DISPATCHCONSTRAINT table is filtered by SETTLEMENTDATE and then the contents of the classes table is matched against that.
Examples
>>> import sqlite3 >>> import os
Set up a database or connect to an existing one.
>>> con = sqlite3.connect('historical.db')
Create the table object.
>>> table = InputsByMatchDispatchConstraints(table_name='EXAMPLE', ... table_columns=['GENCONID', 'EFFECTIVEDATE', 'VERSIONNO', 'RHS'], ... table_primary_keys=['GENCONID', 'EFFECTIVEDATE', 'VERSIONNO'], con=con)
Create the table in the database.
>>> table.create_table_in_sqlite_db()
Normally you would use the set_data method to add historical data, but here we will add data directly to the database so some simple example data can be added.
>>> data = pd.DataFrame({ ... 'GENCONID': ['X', 'X', 'Y', 'Y'], ... 'EFFECTIVEDATE': ['2019/01/02 00:00:00', '2019/01/03 00:00:00', '2019/01/01 00:00:00', ... '2019/01/03 00:00:00'], ... 'VERSIONNO': [1, 2, 2, 3], ... 'RHS': [1.0, 2.0, 2.0, 3.0]})
>>> _ = data.to_sql('EXAMPLE', con=con, if_exists='append', index=False)
>>> data = pd.DataFrame({ ... 'SETTLEMENTDATE' : ['2019/01/02 00:00:00', '2019/01/02 00:00:00', '2019/01/03 00:00:00', ... '2019/01/03 00:00:00'], ... 'CONSTRAINTID': ['X', 'Y', 'X', 'Y'], ... 'GENCONID_EFFECTIVEDATE': ['2019/01/02 00:00:00', '2019/01/01 00:00:00', '2019/01/03 00:00:00', ... '2019/01/03 00:00:00'], ... 'GENCONID_VERSIONNO': [1, 2, 2, 3]})
>>> _ = data.to_sql('DISPATCHCONSTRAINT', con=con, if_exists='append', index=False)
When we call get_data the output is filtered by the contents of DISPATCHCONSTRAINT.
>>> print(table.get_data(date_time='2019/01/02 00:00:00')) GENCONID EFFECTIVEDATE VERSIONNO RHS 0 X 2019/01/02 00:00:00 1 1.0 1 Y 2019/01/01 00:00:00 2 2.0
>>> print(table.get_data(date_time='2019/01/03 00:00:00')) GENCONID EFFECTIVEDATE VERSIONNO RHS 0 X 2019/01/03 00:00:00 2 2.0 1 Y 2019/01/03 00:00:00 3 3.0
Clean up by closing the database and deleting if its no longer needed.
>>> con.close() >>> os.remove('historical.db')
- Parameters:
date_time (str) – Should be of format ‘%Y/%m/%d %H:%M:%S’, and always a round 5 min interval e.g. 2019/01/01 11:55:00.
- Return type:
pd.DataFrame
- create_table_in_sqlite_db()
Creates a table in the sqlite database that the object has a connection to.
Note
This method and its documentation is inherited from the _MMSTable class.
Examples
>>> import sqlite3 >>> import os
Set up a database or connect to an existing one.
>>> con = sqlite3.connect('historical.db')
Create the table object.
>>> table = _MMSTable(table_name='EXAMPLE', table_columns=['DUID', 'BIDTYPE'], table_primary_keys=['DUID'], ... con=con)
Create the corresponding table in the sqlite database, note this step many not be needed if you have connected to an existing database.
>>> table.create_table_in_sqlite_db()
Now a table exists in the database, but its empty.
>>> print(pd.read_sql("Select * from example", con=con)) Empty DataFrame Columns: [DUID, BIDTYPE] Index: []
Clean up by closing the database and deleting if its no longer needed.
>>> con.close() >>> os.remove('historical.db')
- set_data(year, month)
“Download data for the given table and time, replace any existing data.
Note
This method and its documentation is inherited from the _SingleDataSource class.
Examples
>>> import sqlite3 >>> import os
Set up a database or connect to an existing one.
>>> con = sqlite3.connect('historical.db')
Create the table object.
>>> table = _SingleDataSource(table_name='DUDETAILSUMMARY', ... table_columns=['DUID', 'START_DATE', 'CONNECTIONPOINTID', 'REGIONID'], ... table_primary_keys=['START_DATE', 'DUID'], con=con)
Create the table in the database.
>>> table.create_table_in_sqlite_db()
Downloading data from http://nemweb.com.au/#mms-data-model into the table.
>>> table.set_data(year=2020, month=1)
Now the database should contain data for this table that is up to date as the end of Janurary.
>>> query = "Select * from DUDETAILSUMMARY order by START_DATE DESC limit 1;"
>>> print(pd.read_sql_query(query, con=con)) DUID START_DATE CONNECTIONPOINTID REGIONID 0 URANQ11 2020/02/04 00:00:00 NURQ1U NSW1
However if we subsequently set data from a previous date then any existing data will be replaced. Note the change in the most recent record in the data set below.
>>> table.set_data(year=2019, month=1)
>>> print(pd.read_sql_query(query, con=con)) DUID START_DATE CONNECTIONPOINTID REGIONID 0 WEMENSF1 2019/03/04 00:00:00 VWES2W VIC1
Clean up by closing the database and deleting if its no longer needed.
>>> con.close() >>> os.remove('historical.db')
- Parameters:
year (int) – The year to download data for.
month (int) – The month to download data for.
- Return type:
None
- class nempy.historical_inputs.mms_db.InputsByEffectiveDateVersionNoAndDispatchInterconnector(table_name, table_columns, table_primary_keys, con)
Manages retrieving dispatch inputs by EFFECTTIVEDATE and VERSIONNO.
Methods:
get_data(date_time)Retrieves data for the specified date_time by EFFECTTIVEDATE and VERSIONNO.
Creates a table in the sqlite database that the object has a connection to.
set_data(year, month)"Download data for the given table and time, replace any existing data.
- get_data(date_time)
Retrieves data for the specified date_time by EFFECTTIVEDATE and VERSIONNO.
For each unique record (by the remaining primary keys, not including EFFECTTIVEDATE and VERSIONNO) the record with the most recent EFFECTIVEDATE
Examples
>>> import sqlite3 >>> import os
Set up a database or connect to an existing one.
>>> con = sqlite3.connect('historical_inputs.db')
Create the table object.
>>> table = InputsByEffectiveDateVersionNoAndDispatchInterconnector(table_name='EXAMPLE', ... table_columns=['INTERCONNECTORID', 'EFFECTIVEDATE', 'VERSIONNO', 'INITIALMW'], ... table_primary_keys=['INTERCONNECTORID', 'EFFECTIVEDATE', 'VERSIONNO'], con=con)
Create the table in the database.
>>> table.create_table_in_sqlite_db()
Normally you would use the set_data method to add historical_inputs data, but here we will add data directly to the database so some simple example data can be added.
>>> data = pd.DataFrame({ ... 'INTERCONNECTORID': ['X', 'X', 'Y', 'Y'], ... 'EFFECTIVEDATE': ['2019/01/02 00:00:00', '2019/01/03 00:00:00', '2019/01/01 00:00:00', ... '2019/01/03 00:00:00'], ... 'VERSIONNO': [1, 2, 2, 3], ... 'INITIALMW': [1.0, 2.0, 2.0, 3.0]})
>>> _ = data.to_sql('EXAMPLE', con=con, if_exists='append', index=False)
We also need to add data to DISPATCHINTERCONNECTORRES because the results of the get_data method are filtered against this table
>>> data = pd.DataFrame({ ... 'INTERCONNECTORID': ['X', 'X', 'Y'], ... 'SETTLEMENTDATE': ['2019/01/02 00:00:00', '2019/01/03 00:00:00', '2019/01/02 00:00:00']})
>>> _ = data.to_sql('DISPATCHINTERCONNECTORRES', con=con, if_exists='append', index=False)
When we call get_data the output is filtered by the contents of DISPATCHCONSTRAINT.
>>> print(table.get_data(date_time='2019/01/02 00:00:00')) INTERCONNECTORID EFFECTIVEDATE VERSIONNO INITIALMW 0 X 2019/01/02 00:00:00 1 1.0 1 Y 2019/01/01 00:00:00 2 2.0
In the next interval interconnector Y is not present in DISPATCHINTERCONNECTORRES.
>>> print(table.get_data(date_time='2019/01/03 00:00:00')) INTERCONNECTORID EFFECTIVEDATE VERSIONNO INITIALMW 0 X 2019/01/03 00:00:00 2 2.0
Clean up by closing the database and deleting if its no longer needed.
>>> con.close() >>> os.remove('historical_inputs.db')
- Parameters:
date_time (str) – Should be of format ‘%Y/%m/%d %H:%M:%S’, and always a round 5 min interval e.g. 2019/01/01 11:55:00.
- Return type:
pd.DataFrame
- create_table_in_sqlite_db()
Creates a table in the sqlite database that the object has a connection to.
Note
This method and its documentation is inherited from the _MMSTable class.
Examples
>>> import sqlite3 >>> import os
Set up a database or connect to an existing one.
>>> con = sqlite3.connect('historical.db')
Create the table object.
>>> table = _MMSTable(table_name='EXAMPLE', table_columns=['DUID', 'BIDTYPE'], table_primary_keys=['DUID'], ... con=con)
Create the corresponding table in the sqlite database, note this step many not be needed if you have connected to an existing database.
>>> table.create_table_in_sqlite_db()
Now a table exists in the database, but its empty.
>>> print(pd.read_sql("Select * from example", con=con)) Empty DataFrame Columns: [DUID, BIDTYPE] Index: []
Clean up by closing the database and deleting if its no longer needed.
>>> con.close() >>> os.remove('historical.db')
- set_data(year, month)
“Download data for the given table and time, replace any existing data.
Note
This method and its documentation is inherited from the _SingleDataSource class.
Examples
>>> import sqlite3 >>> import os
Set up a database or connect to an existing one.
>>> con = sqlite3.connect('historical.db')
Create the table object.
>>> table = _SingleDataSource(table_name='DUDETAILSUMMARY', ... table_columns=['DUID', 'START_DATE', 'CONNECTIONPOINTID', 'REGIONID'], ... table_primary_keys=['START_DATE', 'DUID'], con=con)
Create the table in the database.
>>> table.create_table_in_sqlite_db()
Downloading data from http://nemweb.com.au/#mms-data-model into the table.
>>> table.set_data(year=2020, month=1)
Now the database should contain data for this table that is up to date as the end of Janurary.
>>> query = "Select * from DUDETAILSUMMARY order by START_DATE DESC limit 1;"
>>> print(pd.read_sql_query(query, con=con)) DUID START_DATE CONNECTIONPOINTID REGIONID 0 URANQ11 2020/02/04 00:00:00 NURQ1U NSW1
However if we subsequently set data from a previous date then any existing data will be replaced. Note the change in the most recent record in the data set below.
>>> table.set_data(year=2019, month=1)
>>> print(pd.read_sql_query(query, con=con)) DUID START_DATE CONNECTIONPOINTID REGIONID 0 WEMENSF1 2019/03/04 00:00:00 VWES2W VIC1
Clean up by closing the database and deleting if its no longer needed.
>>> con.close() >>> os.remove('historical.db')
- Parameters:
year (int) – The year to download data for.
month (int) – The month to download data for.
- Return type:
None
- class nempy.historical_inputs.mms_db.InputsByEffectiveDateVersionNo(table_name, table_columns, table_primary_keys, con)
Manages retrieving dispatch inputs by EFFECTTIVEDATE and VERSIONNO.
Methods:
get_data(date_time)Retrieves data for the specified date_time by EFFECTTIVEDATE and VERSIONNO.
Creates a table in the sqlite database that the object has a connection to.
set_data(year, month)"Download data for the given table and time, replace any existing data.
- get_data(date_time)
Retrieves data for the specified date_time by EFFECTTIVEDATE and VERSIONNO.
For each unique record (by the remaining primary keys, not including EFFECTTIVEDATE and VERSIONNO) the record with the most recent EFFECTIVEDATE
Examples
>>> import sqlite3 >>> import os
Set up a database or connect to an existing one.
>>> con = sqlite3.connect('historical.db')
Create the table object.
>>> table = InputsByEffectiveDateVersionNo(table_name='EXAMPLE', ... table_columns=['DUID', 'EFFECTIVEDATE', 'VERSIONNO', 'INITIALMW'], ... table_primary_keys=['DUID', 'EFFECTIVEDATE', 'VERSIONNO'], con=con)
Create the table in the database.
>>> table.create_table_in_sqlite_db()
Normally you would use the set_data method to add historical data, but here we will add data directly to the database so some simple example data can be added.
>>> data = pd.DataFrame({ ... 'DUID': ['X', 'X', 'Y', 'Y'], ... 'EFFECTIVEDATE': ['2019/01/02 00:00:00', '2019/01/03 00:00:00', '2019/01/01 00:00:00', ... '2019/01/03 00:00:00'], ... 'VERSIONNO': [1, 2, 2, 3], ... 'INITIALMW': [1.0, 2.0, 2.0, 3.0]})
>>> _ = data.to_sql('EXAMPLE', con=con, if_exists='append', index=False)
When we call get_data the output is filtered by most recent effective date and highest version no.
>>> print(table.get_data(date_time='2019/01/02 00:00:00')) DUID EFFECTIVEDATE VERSIONNO INITIALMW 0 X 2019/01/02 00:00:00 1 1.0 1 Y 2019/01/01 00:00:00 2 2.0
In the next interval interconnector Y is not present in DISPATCHINTERCONNECTORRES.
>>> print(table.get_data(date_time='2019/01/03 00:00:00')) DUID EFFECTIVEDATE VERSIONNO INITIALMW 0 X 2019/01/03 00:00:00 2 2.0 1 Y 2019/01/03 00:00:00 3 3.0
Clean up by closing the database and deleting if its no longer needed.
>>> con.close() >>> os.remove('historical.db')
- Parameters:
date_time (str) – Should be of format ‘%Y/%m/%d %H:%M:%S’, and always a round 5 min interval e.g. 2019/01/01 11:55:00.
- Return type:
pd.DataFrame
- create_table_in_sqlite_db()
Creates a table in the sqlite database that the object has a connection to.
Note
This method and its documentation is inherited from the _MMSTable class.
Examples
>>> import sqlite3 >>> import os
Set up a database or connect to an existing one.
>>> con = sqlite3.connect('historical.db')
Create the table object.
>>> table = _MMSTable(table_name='EXAMPLE', table_columns=['DUID', 'BIDTYPE'], table_primary_keys=['DUID'], ... con=con)
Create the corresponding table in the sqlite database, note this step many not be needed if you have connected to an existing database.
>>> table.create_table_in_sqlite_db()
Now a table exists in the database, but its empty.
>>> print(pd.read_sql("Select * from example", con=con)) Empty DataFrame Columns: [DUID, BIDTYPE] Index: []
Clean up by closing the database and deleting if its no longer needed.
>>> con.close() >>> os.remove('historical.db')
- set_data(year, month)
“Download data for the given table and time, replace any existing data.
Note
This method and its documentation is inherited from the _SingleDataSource class.
Examples
>>> import sqlite3 >>> import os
Set up a database or connect to an existing one.
>>> con = sqlite3.connect('historical.db')
Create the table object.
>>> table = _SingleDataSource(table_name='DUDETAILSUMMARY', ... table_columns=['DUID', 'START_DATE', 'CONNECTIONPOINTID', 'REGIONID'], ... table_primary_keys=['START_DATE', 'DUID'], con=con)
Create the table in the database.
>>> table.create_table_in_sqlite_db()
Downloading data from http://nemweb.com.au/#mms-data-model into the table.
>>> table.set_data(year=2020, month=1)
Now the database should contain data for this table that is up to date as the end of Janurary.
>>> query = "Select * from DUDETAILSUMMARY order by START_DATE DESC limit 1;"
>>> print(pd.read_sql_query(query, con=con)) DUID START_DATE CONNECTIONPOINTID REGIONID 0 URANQ11 2020/02/04 00:00:00 NURQ1U NSW1
However if we subsequently set data from a previous date then any existing data will be replaced. Note the change in the most recent record in the data set below.
>>> table.set_data(year=2019, month=1)
>>> print(pd.read_sql_query(query, con=con)) DUID START_DATE CONNECTIONPOINTID REGIONID 0 WEMENSF1 2019/03/04 00:00:00 VWES2W VIC1
Clean up by closing the database and deleting if its no longer needed.
>>> con.close() >>> os.remove('historical.db')
- Parameters:
year (int) – The year to download data for.
month (int) – The month to download data for.
- Return type:
None
- class nempy.historical_inputs.mms_db.InputsNoFilter(table_name, table_columns, table_primary_keys, con)
Manages retrieving dispatch inputs where no filter is require.
Methods:
get_data()Retrieves all data in the table.
Creates a table in the sqlite database that the object has a connection to.
set_data(year, month)"Download data for the given table and time, replace any existing data.
- get_data()
Retrieves all data in the table.
Examples
>>> import sqlite3 >>> import os
Set up a database or connect to an existing one.
>>> con = sqlite3.connect('historical_inputs.db')
Create the table object.
>>> table = InputsNoFilter(table_name='EXAMPLE', table_columns=['DUID', 'INITIALMW'], ... table_primary_keys=['DUID'], con=con)
Create the table in the database.
>>> table.create_table_in_sqlite_db()
Normally you would use the set_data method to add historical_inputs data, but here we will add data directly to the database so some simple example data can be added.
>>> data = pd.DataFrame({ ... 'DUID': ['X', 'Y'], ... 'INITIALMW': [1.0, 2.0]})
>>> _ = data.to_sql('EXAMPLE', con=con, if_exists='append', index=False)
When we call get_data all data in the table is returned.
>>> print(table.get_data()) DUID INITIALMW 0 X 1.0 1 Y 2.0
Clean up by closing the database and deleting if its no longer needed.
>>> con.close() >>> os.remove('historical_inputs.db')
- Return type:
pd.DataFrame
- create_table_in_sqlite_db()
Creates a table in the sqlite database that the object has a connection to.
Note
This method and its documentation is inherited from the _MMSTable class.
Examples
>>> import sqlite3 >>> import os
Set up a database or connect to an existing one.
>>> con = sqlite3.connect('historical.db')
Create the table object.
>>> table = _MMSTable(table_name='EXAMPLE', table_columns=['DUID', 'BIDTYPE'], table_primary_keys=['DUID'], ... con=con)
Create the corresponding table in the sqlite database, note this step many not be needed if you have connected to an existing database.
>>> table.create_table_in_sqlite_db()
Now a table exists in the database, but its empty.
>>> print(pd.read_sql("Select * from example", con=con)) Empty DataFrame Columns: [DUID, BIDTYPE] Index: []
Clean up by closing the database and deleting if its no longer needed.
>>> con.close() >>> os.remove('historical.db')
- set_data(year, month)
“Download data for the given table and time, replace any existing data.
Note
This method and its documentation is inherited from the _SingleDataSource class.
Examples
>>> import sqlite3 >>> import os
Set up a database or connect to an existing one.
>>> con = sqlite3.connect('historical.db')
Create the table object.
>>> table = _SingleDataSource(table_name='DUDETAILSUMMARY', ... table_columns=['DUID', 'START_DATE', 'CONNECTIONPOINTID', 'REGIONID'], ... table_primary_keys=['START_DATE', 'DUID'], con=con)
Create the table in the database.
>>> table.create_table_in_sqlite_db()
Downloading data from http://nemweb.com.au/#mms-data-model into the table.
>>> table.set_data(year=2020, month=1)
Now the database should contain data for this table that is up to date as the end of Janurary.
>>> query = "Select * from DUDETAILSUMMARY order by START_DATE DESC limit 1;"
>>> print(pd.read_sql_query(query, con=con)) DUID START_DATE CONNECTIONPOINTID REGIONID 0 URANQ11 2020/02/04 00:00:00 NURQ1U NSW1
However if we subsequently set data from a previous date then any existing data will be replaced. Note the change in the most recent record in the data set below.
>>> table.set_data(year=2019, month=1)
>>> print(pd.read_sql_query(query, con=con)) DUID START_DATE CONNECTIONPOINTID REGIONID 0 WEMENSF1 2019/03/04 00:00:00 VWES2W VIC1
Clean up by closing the database and deleting if its no longer needed.
>>> con.close() >>> os.remove('historical.db')
- Parameters:
year (int) – The year to download data for.
month (int) – The month to download data for.
- Return type:
None
loaders
Classes:
|
Provides single interface for accessing raw historical inputs. |
- class nempy.historical_inputs.loaders.RawInputsLoader(nemde_xml_cache_manager, market_management_system_database)
Provides single interface for accessing raw historical inputs.
Examples
>>> import sqlite3
>>> from nempy.historical_inputs import mms_db >>> from nempy.historical_inputs import xml_cache
For the RawInputsLoader to work we need to construct a database and inputs cache for it to load inputs from and then pass the interfaces to these to the inputs loader.
>>> con = sqlite3.connect('market_management_system.db') >>> mms_db_manager = mms_db.DBManager(connection=con) >>> xml_cache_manager = xml_cache.XMLCacheManager('test_nemde_cache')
In this example the database and cache have already been populated so the input loader can be created straight away.
>>> inputs_loader = RawInputsLoader(xml_cache_manager, mms_db_manager)
Then we set the dispatch interval that we want to load inputs from.
>>> inputs_loader.set_interval('2019/01/01 00:00:00')
And then we can load some inputs.
>>> inputs_loader.get_unit_volume_bids() DUID BIDTYPE DIRECTION MAXAVAIL ENABLEMENTMIN ENABLEMENTMAX LOWBREAKPOINT HIGHBREAKPOINT BANDAVAIL1 BANDAVAIL2 BANDAVAIL3 BANDAVAIL4 BANDAVAIL5 BANDAVAIL6 BANDAVAIL7 BANDAVAIL8 BANDAVAIL9 BANDAVAIL10 RAMPDOWNRATE RAMPUPRATE 0 AGLHAL ENERGY None 173.0 173.0 173.0 173.0 173.0 0.0 0.0 0.0 0.0 0.0 0.0 60.0 0.0 0.0 160.0 720.0 720.0 1 AGLSOM ENERGY None 160.0 160.0 160.0 160.0 160.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 170.0 480.0 480.0 2 ANGAST1 ENERGY None 43.0 43.0 43.0 43.0 43.0 0.0 0.0 0.0 0.0 0.0 50.0 0.0 0.0 0.0 50.0 840.0 840.0 3 APD01 LOWER5MIN None 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 300.0 300.0 300.0 4 APD01 LOWER60SEC None 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 300.0 300.0 300.0 ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... 1021 YWPS4 LOWER6SEC None 25.0 250.0 385.0 275.0 385.0 15.0 10.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1022 YWPS4 RAISE5MIN None 0.0 250.0 390.0 250.0 380.0 0.0 0.0 0.0 0.0 5.0 0.0 0.0 5.0 0.0 10.0 10.0 10.0 1023 YWPS4 RAISEREG None 15.0 250.0 385.0 250.0 370.0 0.0 0.0 0.0 0.0 0.0 0.0 5.0 10.0 0.0 5.0 5.0 5.0 1024 YWPS4 RAISE60SEC None 10.0 220.0 400.0 220.0 390.0 0.0 0.0 0.0 0.0 0.0 5.0 5.0 0.0 0.0 10.0 10.0 10.0 1025 YWPS4 RAISE6SEC None 15.0 220.0 405.0 220.0 390.0 0.0 0.0 0.0 10.0 5.0 0.0 0.0 0.0 0.0 10.0 10.0 10.0 [1026 rows x 20 columns]
Methods:
- set_interval(interval)
Set the interval to load inputs for.
Examples
For an example see the
class level documentation- Parameters:
interval (str) – In the format ‘%Y/%m/%d %H:%M:%S’
- get_unit_initial_conditions()
Direct interface to
nempy.historical_inputs.xml_cache.XMLCacheManager.get_unit_initial_conditions
- get_unit_volume_bids()
Direct interface to
nempy.historical_inputs.xml_cache.XMLCacheManager.get_unit_volume_bids
- get_unit_price_bids()
Direct interface to
nempy.historical_inputs.mms_db.DBManager.BIDDAYOFFER_D.get_data
- get_unit_details()
Direct interface to
nempy.historical_inputs.mms_db.DBManager.DUDETAILSUMMARY.get_data
- get_agc_enablement_limits()
Direct interface to
nempy.historical_inputs.mms_db.DBManager.DISPATCHLOAD.get_data
- get_UIGF_values()
Direct interface to
nempy.historical_inputs.xml_cache.XMLCacheManager.get_UIGF_values
- get_violations()
Direct interface to
nempy.historical_inputs.xml_cache.XMLCacheManager.get_violations
- get_constraint_violation_prices()
Direct interface to
nempy.historical_inputs.xml_cache.XMLCacheManager.get_constraint_violation_prices
- get_constraint_rhs()
Direct interface to
nempy.historical_inputs.xml_cache.XMLCacheManager.get_constraint_rhs
- get_constraint_type()
Direct interface to
nempy.historical_inputs.xml_cache.XMLCacheManager.get_constraint_type
- get_constraint_region_lhs()
Direct interface to
nempy.historical_inputs.xml_cache.XMLCacheManager.get_constraint_region_lhs
- get_constraint_unit_lhs()
Direct interface to
nempy.historical_inputs.xml_cache.XMLCacheManager.get_constraint_unit_lhs
- get_constraint_interconnector_lhs()
Direct interface to
nempy.historical_inputs.xml_cache.XMLCacheManager.get_constraint_interconnector_lhs
- get_market_interconnectors()
Direct interface to
nempy.historical_inputs.mms_db.DBManager.MNSP_INTERCONNECTOR.get_data
- get_market_interconnector_link_bid_availability()
Direct interface to
nempy.historical_inputs.xml_cache.XMLCacheManager.get_market_interconnector_link_bid_availability
- get_interconnector_constraint_parameters()
Direct interface to
nempy.historical_inputs.mms_db.DBManager.INTERCONNECTORCONSTRAINT.get_data
- get_interconnector_definitions()
Direct interface to
nempy.historical_inputs.mms_db.DBManager.INTERCONNECTOR.get_data
- get_regional_loads()
Direct interface to
nempy.historical_inputs.mms_db.DBManager.DISPATCHREGIONSUM.get_data
- get_interconnector_loss_segments()
Direct interface to
nempy.historical_inputs.mms_db.DBManager.LOSSMODEL.get_data
- get_interconnector_loss_parameters()
Direct interface to
nempy.historical_inputs.mms_db.DBManager.LOSSFACTORMODEL.get_data
- get_unit_fast_start_parameters()
Direct interface to
nempy.historical_inputs.xml_cache.XMLCacheManager.get_unit_fast_start_parameters
- is_over_constrained_dispatch_rerun()
Checks if the over constrained dispatch rerun process was used by AEMO to dispatch this interval.
Examples
>>> import sqlite3
>>> from nempy.historical_inputs import mms_db >>> from nempy.historical_inputs import xml_cache
For the RawInputsLoader to work we need to construct a database and inputs cache for it to load inputs from and then pass the interfaces to these to the inputs loader.
>>> con = sqlite3.connect('market_management_system.db') >>> mms_db_manager = mms_db.DBManager(connection=con) >>> xml_cache_manager = xml_cache.XMLCacheManager('test_nemde_cache')
In this example the database and cache have already been populated so the input loader can be created straight away.
>>> inputs_loader = RawInputsLoader(xml_cache_manager, mms_db_manager)
Then we set the dispatch interval that we want to load inputs from.
>>> inputs_loader.set_interval('2019/01/01 00:00:00')
And then we can load some inputs.
>>> inputs_loader.is_over_constrained_dispatch_rerun() False
- Return type:
bool
units
Exceptions:
Raise for calling methods in incompatible order. |
Classes:
|
Loads unit related raw inputs and preprocess them for compatibility with |
- exception nempy.historical_inputs.units.MethodCallOrderError
Raise for calling methods in incompatible order.
- class nempy.historical_inputs.units.UnitData(raw_input_loader)
Loads unit related raw inputs and preprocess them for compatibility with
nempy.markets.SpotMarketExamples
This example shows the setup used for the examples in the class methods.
>>> import sqlite3 >>> from nempy.historical_inputs import mms_db >>> from nempy.historical_inputs import xml_cache >>> from nempy.historical_inputs import loaders
The UnitData class requries a RawInputsLoader instance.
>>> con = sqlite3.connect('market_management_system.db') >>> mms_db_manager = mms_db.DBManager(connection=con) >>> xml_cache_manager = xml_cache.XMLCacheManager('test_nemde_cache') >>> inputs_loader = loaders.RawInputsLoader(xml_cache_manager, mms_db_manager) >>> inputs_loader.set_interval('2024/07/10 12:05:00')
Create the UnitData instance.
>>> unit_data = UnitData(inputs_loader)
>>> unit_data.get_unit_bid_availability() unit dispatch_type capacity 0 ADPBA1G generator 6.0 10 ADPBA1L load 6.0 12 ADPPV1 generator 19.0 13 AGLHAL generator 139.0 14 AGLSOM generator 128.0 ... ... ... ... 1713 YWPS4 generator 340.0 210 BHB1 generator 0.0 211 BHB1 load 0.0 1622 WANDB1 generator 0.0 1623 WANDB1 load 0.0 [446 rows x 3 columns]
Methods:
Get the bid in maximum availability for scheduled units.
Get the maximum availability predicted by the unconstrained intermittent generation forecast.
Get bid in ramp rates
get_scada_ramp_rates([inlude_initial_output])Get scada ramp rates
Get unit outputs at the start of the dispatch interval.
Get the parameters needed to construct the fast dispatch inflexibility profiles used for dispatch.
Get unit information.
Get processed unit bids.
Load the fcas trapezium constraints into the UnitData class so subsequent method calls can access them.
Get the unit bid maximum availability of each service.
Get the unit bid FCAS trapeziums for regulation services.
Get the unit bid FCAS trapeziums for contingency services.
- get_unit_bid_availability()
Get the bid in maximum availability for scheduled units.
Examples
>>> inputs_loader = _test_setup()
>>> unit_data = UnitData(inputs_loader)
>>> unit_data.get_unit_bid_availability() unit dispatch_type capacity 0 ADPBA1G generator 6.0 10 ADPBA1L load 6.0 12 ADPPV1 generator 19.0 13 AGLHAL generator 139.0 14 AGLSOM generator 128.0 ... ... ... ... 1713 YWPS4 generator 340.0 210 BHB1 generator 0.0 211 BHB1 load 0.0 1622 WANDB1 generator 0.0 1623 WANDB1 load 0.0 [446 rows x 3 columns]
- Returns:
Columns:
Description:
unit
unique identifier for units, (as str)
dispatch_type
”load” or “generator”, (as str)
capacity
unit bid in max availability, in MW,
(as str)
- Return type:
pd.DataFrame
- get_unit_uigf_limits()
Get the maximum availability predicted by the unconstrained intermittent generation forecast.
Examples
>>> inputs_loader = _test_setup()
>>> unit_data = UnitData(inputs_loader)
>>> unit_data.get_unit_uigf_limits() unit capacity 0 ADPPV1 10.90800 1 ARWF1 0.00000 2 AVLSF1 55.26000 3 BALDHWF1 59.81800 4 BANGOWF1 41.89800 .. ... ... 165 WSTWYSF1 49.90000 166 WYASF1 33.90909 167 YARANSF1 59.55000 168 YATSF1 20.00000 169 YENDWF1 7.00604 [170 rows x 2 columns]
- Returns:
Columns:
Description:
unit
unique identifier for units, (as str)
capacity
the forecast max availability, in MW,
(as str)
- Return type:
pd.DataFrame
- get_bid_ramp_rates()
Get bid in ramp rates
Examples
>>> inputs_loader = _test_setup()
>>> unit_data = UnitData(inputs_loader)
>>> unit_data.get_bid_ramp_rates() unit dispatch_type ramp_down_rate ramp_up_rate initial_output 0 ADPBA1G generator 120.0 120.0 0.00000 10 ADPBA1L load 120.0 120.0 1.40400 12 ADPPV1 generator 120.0 120.0 10.90800 13 AGLHAL generator 720.0 720.0 0.00000 14 AGLSOM generator 480.0 480.0 60.00000 ... ... ... ... ... ... 1713 YWPS4 generator 180.0 180.0 337.93546 1722 BHB1 generator 600.0 600.0 0.00000 1723 BHB1 load 600.0 600.0 0.00000 1724 WANDB1 generator 1200.0 1200.0 0.00000 1725 WANDB1 load 1200.0 1200.0 0.00000 [446 rows x 5 columns]
- Returns:
- Return type:
pd.DataFrame
- get_scada_ramp_rates(inlude_initial_output=False)
Get scada ramp rates
Examples
>>> inputs_loader = _test_setup()
>>> unit_data = UnitData(inputs_loader)
>>> unit_data.get_scada_ramp_rates(inlude_initial_output=True) unit scada_ramp_down_rate scada_ramp_up_rate initial_output 0 ADPBA1G 93.119938 93.119938 0.00000 1 ADPBA1L 93.119938 93.119938 1.40400 2 ADPPV1 298.499937 298.499937 10.90800 30 BALBG1 54000.000000 54000.000000 0.00000 31 BALBL1 54000.000000 54000.000000 0.00000 .. ... ... ... ... 481 WOOLGSF1 2112.000046 2112.000046 91.80000 493 YWPS1 180.000000 180.000000 0.00000 494 YWPS2 176.624994 176.624994 358.89621 495 YWPS3 181.124997 181.124997 371.52658 496 YWPS4 180.000000 180.000000 337.93546 [192 rows x 4 columns]
- Parameters:
inlude_initial_output – boolean specifying whether or not to
dataframe (inlcude the column initial_output in the returned)
False. (default)
- Returns:
- Return type:
pd.DataFrame
- get_initial_unit_output()
Get unit outputs at the start of the dispatch interval.
Examples
>>> inputs_loader = _test_setup()
>>> unit_data = UnitData(inputs_loader)
>>> unit_data.get_initial_unit_output() unit initial_output 0 ADPBA1G 0.00000 1 ADPBA1L 1.40400 2 ADPPV1 10.90800 3 AGLHAL 0.00000 4 AGLSOM 60.00000 .. ... ... 492 YENDWF1 6.75000 493 YWPS1 0.00000 494 YWPS2 358.89621 495 YWPS3 371.52658 496 YWPS4 337.93546 [497 rows x 2 columns]
- Returns:
Columns:
Description:
unit
unique identifier for units, (as str)
initial_output
the output/consumption of the unit at
the start of the dispatch interval,
in MW, (as np.float64)
- Return type:
pd.DataFrame
- get_fast_start_profiles_for_dispatch(unconstrained_dispatch=None, return_all_columns=False) pandas.DataFrame
Get the parameters needed to construct the fast dispatch inflexibility profiles used for dispatch.
If the results of a non-fast start constrained dispatch run are provided then these are used to commit fast start units starting the interval in mode zero, when they have a non-zero dispatch result.
For more info on fast start dispatch inflexibility profiles
see AEMO docs.Examples
>>> inputs_loader = _test_setup()
>>> unit_data = UnitData(inputs_loader)
>>> unit_data.get_fast_start_profiles_for_dispatch() unit current_mode 0 AGLHAL 0 1 AGLSOM 4 2 BARRON-1 4 3 BARRON-2 0 4 BBTHREE1 0 .. ... ... 68 VPGS4 0 69 VPGS5 0 70 VPGS6 0 71 W/HOE#1 0 72 W/HOE#2 0 [73 rows x 2 columns]
- Returns:
If unconstrained_dispatch is not provided, i.e. geting profiles of first run:
- Return type:
pd.DataFrame
dispatch_type “load” or “generator” (as str)
current_mode the fast start mode the unit starts the interval in
(as np.int64)
Columns: Description: unit unique identifier for units, (as str)
end_mode the fast start mode the unit will end
the dispatch interval in, (as np.int64)
time_in_end_mode the amount of time the unit will have
spend in the end mode at the end of the
dispatch interval, (as np.float64)
mode_two_length the length the units mode two, in minutes
(as np.float64)
mode_four_length the length the units mode four, in minutes
(as np.float64)
min_loading the mininum opperating level of the unit
during mode three, in MW, (as no.float64)
time_since_end_of_mode_two the time since the unit was last operating
in mode two in minutes , (as np.int64)
- get_unit_info()
Get unit information.
Examples
>>> inputs_loader = _test_setup()
>>> unit_data = UnitData(inputs_loader)
>>> unit_data.get_unit_info() unit dispatch_type region loss_factor 0 ADPBA1G generator SA1 1.013527 1 ADPBA1L load SA1 1.013527 2 ADPPV1 generator SA1 1.013527 3 AGLHAL generator SA1 0.956500 4 AGLSOM generator VIC1 0.979065 .. ... ... ... ... 494 YENDWF1 generator VIC1 0.930059 495 YWPS1 generator VIC1 0.962100 496 YWPS2 generator VIC1 0.960400 497 YWPS3 generator VIC1 0.960400 498 YWPS4 generator VIC1 0.960400 [499 rows x 4 columns]
- Returns:
Columns:
Description:
unit
unique identifier for units, (as str)
region
the market region in which the unit is
located, (as str)
dispatch_type
whether the unit is a ‘generator’ or
’load’, (as str)
loss_factor
the combined unit transmission and
distribution loss_factor, (as np.float64)
- Return type:
pd.DataFrame
- get_processed_bids()
Get processed unit bids.
The bids are processed by scaling for AGC enablement limits, scaling for scada ramp rates, scaling for the unconstrained intermittent generation forecast and enforcing the preconditions for enabling FCAS bids. For more info on these processes
see AEMO docs.Examples
>>> inputs_loader = _test_setup()
>>> unit_data = UnitData(inputs_loader)
>>> volume_bids, price_bids = unit_data.get_processed_bids()
>>> volume_bids unit service dispatch_type 1 2 3 4 5 6 7 8 9 10 0 ADPBA1G energy generator 0.0 0.0 0.0 0.0 0.0 0.0 0.0 6.0 0.0 0.0 10 ADPBA1L energy load 0.0 0.0 0.0 0.0 6.0 0.0 0.0 0.0 0.0 0.0 12 ADPPV1 energy generator 0.0 0.0 1.0 1.0 4.0 13.0 0.0 0.0 0.0 0.0 13 AGLHAL energy generator 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 255.0 14 AGLSOM energy generator 0.0 60.0 0.0 110.0 0.0 0.0 0.0 0.0 0.0 0.0 .. ... ... ... ... ... ... ... ... ... ... ... ... ... 619 KIAMSF1 lower_60s generator 0.0 37.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 620 KIAMSF1 lower_6s generator 0.0 37.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 621 WDGPH1 lower_5min generator 0.0 0.0 0.0 0.0 6.0 6.0 6.0 6.0 6.0 27.0 622 WDGPH1 lower_60s generator 0.0 0.0 0.0 0.0 6.0 6.0 6.0 6.0 6.0 27.0 623 WDGPH1 lower_6s generator 0.0 0.0 0.0 0.0 6.0 6.0 6.0 6.0 6.0 27.0 [1038 rows x 13 columns]
>>> price_bids unit service dispatch_type 1 2 3 4 5 6 7 8 9 10 0 ADPBA1G energy generator -980.00001 0.000000 54.000745 96.001325 168.002318 273.997024 374.001783 998.000259 3999.004510 9999.999485 1 ADPBA1L energy load -980.00001 -449.996075 -174.002401 -88.997850 19.003641 54.000745 133.998471 223.999713 348.998059 500.003522 2 ADPPV1 energy generator -1013.52750 -506.763750 -110.474497 -100.339222 -57.771067 -47.635792 0.000000 304.058250 3040.582500 17736.731250 3 AGLHAL energy generator -956.50000 0.000000 274.410285 363.460435 566.142785 956.385220 3808.677785 9469.235220 15112.585220 16738.750000 4 AGLSOM energy generator -979.06536 0.000000 109.635739 206.690488 278.054562 364.466871 454.296118 980.044425 13022.430866 17133.643800 ... ... ... ... ... ... ... ... ... ... ... ... ... ... 1033 WDGPH1 lower_6s generator 0.01000 0.130000 0.330000 0.850000 1.830000 4.870000 19.920000 97.790000 998.990000 17500.000000 1034 WKIEWA1 lower_60s generator 0.00000 0.030000 1.000000 2.000000 26.000000 98.900000 147.000000 300.000000 1199.000000 17500.000000 1035 WKIEWA1 lower_6s generator 0.00000 0.500000 1.000000 2.000000 45.000000 98.690000 144.000000 300.000000 1199.000000 17500.000000 1036 WKIEWA1 raise_60s generator 0.00000 0.600000 1.700000 9.500000 22.100000 99.600000 132.100000 240.100000 495.000000 17500.000000 1037 WKIEWA1 raise_6s generator 0.00000 0.600000 1.700000 9.500000 32.100000 99.600000 132.100000 240.100000 495.000000 17500.000000 [1038 rows x 13 columns]
Multiple Returns
volume_bids : pd.DataFrame
Columns:
Description:
unit
unique identifier for units, (as str)
service
the service the bid applies to, (as str)
dispatch_type
“load” or “generator”, (as str)
1
the volume bid the first bid band, in MW,
(as np.float64)
:
10
the volume in the tenth bid band, in MW,
(as np.float64)
price_bids : pd.DataFrame
Columns:
Description:
unit
unique identifier for units, (as str)
service
the service the bid applies to, (as str)
dispatch_type
“load” or “generator”, (as str)
1
the price of the first bid band, in MW,
(as np.float64)
:
10
the price of the the tenth bid band, in MW,
(as np.float64)
- add_fcas_trapezium_constraints()
Load the fcas trapezium constraints into the UnitData class so subsequent method calls can access them.
Examples
>>> inputs_loader = _test_setup() >>> unit_data = UnitData(inputs_loader)
If we try and call add_fcas_trapezium_constraints before calling get_processed_bids we get an error.
>>> unit_data.add_fcas_trapezium_constraints() Traceback (most recent call last): ... nempy.historical_inputs.units.MethodCallOrderError: Call get_processed_bids before add_fcas_trapezium_constraints.
After calling get_processed_bids it goes away.
>>> volume_bids, price_bids = unit_data.get_processed_bids()
>>> unit_data.add_fcas_trapezium_constraints()
If we try and access the trapezium constraints before calling this method we get an error.
>>> inputs_loader = _test_setup() >>> unit_data = UnitData(inputs_loader) >>> unit_data.get_fcas_max_availability() Traceback (most recent call last): ... nempy.historical_inputs.units.MethodCallOrderError: Call add_fcas_trapezium_constraints before get_fcas_max_availability.
After calling add_fcas_trapezium_constraints the error goes away.
>>> volume_bids, price_bids = unit_data.get_processed_bids() >>> unit_data.add_fcas_trapezium_constraints()
>>> unit_data.get_fcas_max_availability() unit service dispatch_type max_availability 0 ADPBA1G raise_5min generator 3.0 1 ADPBA1G raise_60s generator 3.0 2 ADPBA1G raise_6s generator 3.0 3 ADPBA1L lower_5min load 3.0 4 ADPBA1L lower_60s load 3.0 .. ... ... ... ... 619 KIAMSF1 lower_60s generator 37.0 620 KIAMSF1 lower_6s generator 37.0 621 WDGPH1 lower_5min generator 57.0 622 WDGPH1 lower_60s generator 57.0 623 WDGPH1 lower_6s generator 57.0 [592 rows x 4 columns]
- Return type:
None
- Raises:
MethodCallOrderError – if called before get_processed_bids
- get_fcas_max_availability()
Get the unit bid maximum availability of each service.
Examples
>>> inputs_loader = _test_setup() >>> unit_data = UnitData(inputs_loader)
Required calls before calling get_fcas_max_availability.
>>> volume_bids, price_bids = unit_data.get_processed_bids() >>> unit_data.add_fcas_trapezium_constraints()
Now facs max availibility can be accessed.
>>> unit_data.get_fcas_max_availability() unit service dispatch_type max_availability 0 ADPBA1G raise_5min generator 3.0 1 ADPBA1G raise_60s generator 3.0 2 ADPBA1G raise_6s generator 3.0 3 ADPBA1L lower_5min load 3.0 4 ADPBA1L lower_60s load 3.0 .. ... ... ... ... 619 KIAMSF1 lower_60s generator 37.0 620 KIAMSF1 lower_6s generator 37.0 621 WDGPH1 lower_5min generator 57.0 622 WDGPH1 lower_60s generator 57.0 623 WDGPH1 lower_6s generator 57.0 [592 rows x 4 columns]
- Returns:
Columns:
Description:
unit
unique identifier for units, (as str)
dispatch_type
”load” or “generator”, (as str)
service
the service the bid applies to, (as str)
max_availability
the unit bid maximum availability, in MW,
(as np.float64)
- Return type:
pd.DataFrame
- Raises:
MethodCallOrderError – if the method is called before add_fcas_trapezium_constraints.
- get_fcas_regulation_trapeziums()
Get the unit bid FCAS trapeziums for regulation services.
Examples
>>> inputs_loader = _test_setup() >>> unit_data = UnitData(inputs_loader)
Required calls before calling get_fcas_regulation_trapeziums.
>>> volume_bids, price_bids = unit_data.get_processed_bids() >>> unit_data.add_fcas_trapezium_constraints()
Now facs max availibility can be accessed.
>>> unit_data.get_fcas_regulation_trapeziums() unit service dispatch_type max_availability enablement_min low_break_point high_break_point enablement_max 474 ADPBA1G lower_reg generator 6.000000 0.0 6.0 6.000000 6.0 475 ADPBA1L lower_reg load 6.000000 0.0 0.0 0.000000 6.0 476 BALBG1 lower_reg generator 30.000000 0.0 30.0 30.000000 30.0 477 BALBL1 lower_reg load 30.000000 0.0 0.0 0.000000 30.0 478 BASTYAN lower_reg generator 63.000000 25.0 88.0 83.000000 83.0 .. ... ... ... ... ... ... ... ... 611 VP6 raise_reg generator 14.870144 250.0 250.0 535.129856 550.0 612 WALGRVG1 raise_reg generator 39.000000 0.0 0.0 0.000000 39.0 613 WALGRVL1 raise_reg load 35.000000 0.0 35.0 35.000000 35.0 614 WANDBG1 raise_reg generator 70.000000 0.0 0.0 30.000000 100.0 615 WANDBL1 raise_reg load 30.000000 0.0 30.0 75.000000 75.0 [132 rows x 8 columns]
- Returns:
- Return type:
pd.DataFrame
- Raises:
MethodCallOrderError – if the method is called before add_fcas_trapezium_constraints.
- get_contingency_services()
Get the unit bid FCAS trapeziums for contingency services.
Examples
>>> inputs_loader = _test_setup() >>> unit_data = UnitData(inputs_loader)
Required calls before calling get_contingency_services.
>>> volume_bids, price_bids = unit_data.get_processed_bids() >>> unit_data.add_fcas_trapezium_constraints()
Now facs max availibility can be accessed.
>>> unit_data.get_contingency_services() unit service dispatch_type max_availability enablement_min low_break_point high_break_point enablement_max 0 ADPBA1G raise_5min generator 3.0 0.0 0.0 3.00000 6.0000 1 ADPBA1G raise_60s generator 3.0 0.0 0.0 3.00000 6.0000 2 ADPBA1G raise_6s generator 3.0 0.0 0.0 3.00000 6.0000 3 ADPBA1L lower_5min load 3.0 0.0 0.0 3.00000 6.0000 4 ADPBA1L lower_60s load 3.0 0.0 0.0 3.00000 6.0000 .. ... ... ... ... ... ... ... ... 619 KIAMSF1 lower_60s generator 37.0 0.0 200.0 0.00000 0.0000 620 KIAMSF1 lower_6s generator 37.0 0.0 200.0 0.00000 0.0000 621 WDGPH1 lower_5min generator 57.0 59.0 116.0 276.82729 276.8273 622 WDGPH1 lower_60s generator 57.0 59.0 116.0 276.82729 276.8273 623 WDGPH1 lower_6s generator 57.0 59.0 116.0 276.82729 276.8273 [460 rows x 8 columns]
- Returns:
- Return type:
pd.DataFrame
- Raises:
MethodCallOrderError – if the method is called before add_fcas_trapezium_constraints.
interconnectors
Classes:
|
Loads interconnector related raw inputs and preprocess them for compatibility with |
Functions:
Creates a loss function for each interconnector. |
- class nempy.historical_inputs.interconnectors.InterconnectorData(raw_input_loader)
Loads interconnector related raw inputs and preprocess them for compatibility with
nempy.markets.SpotMarketExamples
This example shows the setup used for the examples in the class methods. This setup is used to create a RawInputsLoader by calling the function _test_setup.
>>> import sqlite3 >>> from nempy.historical_inputs import mms_db >>> from nempy.historical_inputs import xml_cache >>> from nempy.historical_inputs import loaders
The InterconnectorData class requries a RawInputsLoader instance.
>>> con = sqlite3.connect('market_management_system.db') >>> mms_db_manager = mms_db.DBManager(connection=con) >>> xml_cache_manager = xml_cache.XMLCacheManager('test_nemde_cache') >>> inputs_loader = loaders.RawInputsLoader(xml_cache_manager, mms_db_manager) >>> inputs_loader.set_interval('2024/07/10 12:05:00')
Create a InterconnectorData instance.
>>> interconnector_data = InterconnectorData(inputs_loader)
>>> interconnector_data.get_interconnector_definitions() interconnector from_region to_region min max link from_region_loss_factor to_region_loss_factor generic_constraint_factor 0 N-Q-MNSP1 NSW1 QLD1 -264.0 264.0 N-Q-MNSP1 1.0000 1.0000 1 1 NSW1-QLD1 NSW1 QLD1 -2478.0 2204.0 NSW1-QLD1 1.0000 1.0000 1 3 V-S-MNSP1 VIC1 SA1 -270.0 270.0 V-S-MNSP1 1.0000 1.0000 1 4 V-SA VIC1 SA1 -850.0 950.0 V-SA 1.0000 1.0000 1 5 VIC1-NSW1 VIC1 NSW1 -2299.0 2399.0 VIC1-NSW1 1.0000 1.0000 1 0 T-V-MNSP1 TAS1 VIC1 0.0 594.0 BLNKTAS 1.0000 0.9777 1 1 T-V-MNSP1 VIC1 TAS1 0.0 478.0 BLNKVIC 0.9852 1.0000 -1
- Parameters:
inputs_manager (historical_spot_market_inputs.DBManager)
Methods:
Returns inputs in the format needed to set interconnector losses in the SpotMarket class.
Returns inputs in the format needed to create interconnectors in the SpotMarket class.
- get_interconnector_loss_model()
Returns inputs in the format needed to set interconnector losses in the SpotMarket class.
Examples
>>> inputs_loader = _test_setup()
>>> interconnector_data = InterconnectorData(inputs_loader)
>>> loss_function, interpolation_break_points = interconnector_data.get_interconnector_loss_model()
>>> print(loss_function) interconnector link loss_function from_region_loss_share 0 N-Q-MNSP1 N-Q-MNSP1 <function InterconnectorData.get_interconnecto... 0.70 1 NSW1-QLD1 NSW1-QLD1 <function InterconnectorData.get_interconnecto... 0.63 2 V-S-MNSP1 V-S-MNSP1 <function InterconnectorData.get_interconnecto... 0.70 3 V-SA V-SA <function InterconnectorData.get_interconnecto... 0.67 4 VIC1-NSW1 VIC1-NSW1 <function InterconnectorData.get_interconnecto... 0.36 5 T-V-MNSP1 BLNKTAS <function InterconnectorData.get_interconnecto... 1.00 6 T-V-MNSP1 BLNKVIC <function InterconnectorData.get_interconnecto... 1.00
>>> print(interpolation_break_points) interconnector link loss_segment break_point 0 N-Q-MNSP1 N-Q-MNSP1 1 -265.0 1 N-Q-MNSP1 N-Q-MNSP1 2 -257.0 2 N-Q-MNSP1 N-Q-MNSP1 3 -249.0 3 N-Q-MNSP1 N-Q-MNSP1 4 -241.0 4 N-Q-MNSP1 N-Q-MNSP1 5 -233.0 .. ... ... ... ... 611 T-V-MNSP1 BLNKVIC -80 -546.0 612 T-V-MNSP1 BLNKVIC -81 -559.0 613 T-V-MNSP1 BLNKVIC -82 -571.0 614 T-V-MNSP1 BLNKVIC -83 -583.0 615 T-V-MNSP1 BLNKVIC -84 -595.0 [616 rows x 4 columns]
Multiple Returns
loss_functions : pd.DataFrame
Columns:
Description:
interconnector
unique identifier of a interconnector,
(as str)
from_region_loss_share
The fraction of loss occuring in
the from region, 0.0 to 1.0,
(as np.float64)
loss_function
A function that takes a flow,
in MW as a float and returns the
losses in MW, (as callable)
interpolation_break_points : pd.DataFrame
Columns:
Description:
interconnector
unique identifier of a interconnector,
(as str)
loss_segment
unique identifier of a loss segment on
an interconnector basis, (as np.float64)
break_point
points between which the loss function
will be linearly interpolated, in MW
(as np.float64)
- get_interconnector_definitions()
Returns inputs in the format needed to create interconnectors in the SpotMarket class.
Examples
>>> inputs_loader = _test_setup()
>>> interconnector_data = InterconnectorData(inputs_loader)
>>> interconnector_data.get_interconnector_definitions() interconnector from_region to_region min max link from_region_loss_factor to_region_loss_factor generic_constraint_factor 0 N-Q-MNSP1 NSW1 QLD1 -264.0 264.0 N-Q-MNSP1 1.0000 1.0000 1 1 NSW1-QLD1 NSW1 QLD1 -2478.0 2204.0 NSW1-QLD1 1.0000 1.0000 1 3 V-S-MNSP1 VIC1 SA1 -270.0 270.0 V-S-MNSP1 1.0000 1.0000 1 4 V-SA VIC1 SA1 -850.0 950.0 V-SA 1.0000 1.0000 1 5 VIC1-NSW1 VIC1 NSW1 -2299.0 2399.0 VIC1-NSW1 1.0000 1.0000 1 0 T-V-MNSP1 TAS1 VIC1 0.0 594.0 BLNKTAS 1.0000 0.9777 1 1 T-V-MNSP1 VIC1 TAS1 0.0 478.0 BLNKVIC 0.9852 1.0000 -1
- Returns:
Columns:
Description:
interconnector
unique identifier of a interconnector,
(as str)
to_region
the region that receives power
when flow is in the positive
direction, (as str)
from_region
the region that power is drawn
from when flow is in the
positive direction, (as str)
max
the maximum power flow on the
interconnector, in MW (as np.float64)
min
the minimum power flow on the
interconnector, if power can flow
neative direction then this will be
negative, in MW (as np.float64)
from_region_loss_factor
the loss factor between the from
end of the interconnector and the
regional reference node,
(as np.float)
to_region_loss_factor
the loss factor between the to
end of the interconnector and
the regional reference node,
(as np.float)
- Return type:
pd.DataFrame
- nempy.historical_inputs.interconnectors.create_loss_functions(interconnector_coefficients, demand_coefficients, demand)
Creates a loss function for each interconnector.
Transforms the dynamic demand dependendent interconnector loss functions into functions that only depend on interconnector flow. i.e takes the function f and creates g by pre-calculating the demand dependent terms.
f(inter_flow, flow_coefficient, nsw_demand, nsw_coefficient, qld_demand, qld_coefficient) = inter_losses
becomes
g(inter_flow) = inter_losses
The mathematics of the demand dependent loss functions is described in the
Marginal Loss Factors documentation section 3 to 5.Examples
>>> import pandas as pd
Some arbitrary regional demands.
>>> demand = pd.DataFrame({ ... 'region': ['VIC1', 'NSW1', 'QLD1', 'SA1'], ... 'loss_function_demand': [6000.0 , 7000.0, 5000.0, 3000.0]})
Loss model details from 2020 Jan NEM web LOSSFACTORMODEL file
>>> demand_coefficients = pd.DataFrame({ ... 'interconnector': ['NSW1-QLD1', 'NSW1-QLD1', 'VIC1-NSW1', ... 'VIC1-NSW1', 'VIC1-NSW1'], ... 'region': ['NSW1', 'QLD1', 'NSW1', 'VIC1', 'SA1'], ... 'demand_coefficient': [-0.00000035146, 0.000010044, ... 0.000021734, -0.000031523, ... -0.000065967]})
Loss model details from 2020 Jan NEM web INTERCONNECTORCONSTRAINT file
>>> interconnector_coefficients = pd.DataFrame({ ... 'interconnector': ['NSW1-QLD1', 'VIC1-NSW1'], ... 'loss_constant': [0.9529, 1.0657], ... 'flow_coefficient': [0.00019617, 0.00017027], ... 'from_region_loss_share': [0.5, 0.5]})
Create the loss functions
>>> loss_functions = create_loss_functions(interconnector_coefficients, ... demand_coefficients, demand)
Lets use one of the loss functions, first get the loss function of VIC1-NSW1 and call it g
>>> g = loss_functions[loss_functions['interconnector'] == 'VIC1-NSW1']['loss_function'].iloc[0]
Calculate the losses at 600 MW flow
>>> print(g(600.0)) -70.87199999999996
Now for NSW1-QLD1
>>> h = loss_functions[loss_functions['interconnector'] == 'NSW1-QLD1']['loss_function'].iloc[0]
>>> print(h(600.0)) 35.70646799999993
- Parameters:
interconnector_coefficients (pd.DataFrame) –
Columns:
Description:
interconnector
unique identifier of a interconnector,
(as str)
loss_constant
the constant term in the interconnector
loss factor equation, (as np.float64)
flow_coefficient
the coefficient of the interconnector
flow variable in the loss factor equation
(as np.float64)
from_region_loss_share
the proportion of loss attribute to the
from region, remainer are attributed to
the to region, (as np.float64)
demand_coefficients (pd.DataFrame) –
Columns:
Description:
interconnector
unique identifier of a interconnector,
(as str)
region
the market region whose demand the coefficient
applies too (as str)
demand_coefficient
the coefficient of regional demand variable
in the loss factor equation, (as np.float64)
demand (pd.DataFrame) –
Columns:
Description:
region
unique identifier of a region, (as str)
loss_function_demand
the estimated regional demand, as calculated
by initial supply + demand forecast,
in MW (as np.float64)
- Returns:
loss_functions
Columns:
Description:
interconnector
unique identifier of a interconnector, (as str)
loss_function
a function object that takes interconnector
flow (as float) an input and returns
interconnector losses (as float).
- Return type:
pd.DataFrame
demand
Classes:
|
Loads demand related raw data and preprocess it for complatibility with the SpotMarket class. |
- class nempy.historical_inputs.demand.DemandData(raw_inputs_loader)
Loads demand related raw data and preprocess it for complatibility with the SpotMarket class.
Examples
The DemandData class requries a RawInputsLoader instance.
>>> import sqlite3 >>> from nempy.historical_inputs import mms_db >>> from nempy.historical_inputs import xml_cache >>> from nempy.historical_inputs import loaders >>> con = sqlite3.connect('market_management_system.db') >>> mms_db_manager = mms_db.DBManager(connection=con) >>> xml_cache_manager = xml_cache.XMLCacheManager('test_nemde_cache') >>> inputs_loader = loaders.RawInputsLoader(xml_cache_manager, mms_db_manager) >>> inputs_loader.set_interval('2024/07/10 12:05:00')
>>> demand_data = DemandData(inputs_loader)
>>> demand_data.get_operational_demand() region demand 0 NSW1 6624.81 1 QLD1 4750.17 2 SA1 934.59 3 TAS1 1260.71 4 VIC1 5390.51
- Parameters:
raw_inputs_loader
Methods:
Get the operational demand used to determine the regional energy dispatch constraints.
- get_operational_demand()
Get the operational demand used to determine the regional energy dispatch constraints.
Examples
See class level example.
- Returns:
Columns:
Description:
region
unique identifier of a market region,
(as str)
demand
the non dispatchable demand the region,
in MW, (as np.float64)
loss_function_demand
the measure of demand used when creating
interconnector loss functions, in MW,
(as np.float64)
- Return type:
pd.DataFrame
constraints
Classes:
|
Loads generic constraint related raw inputs and preprocess them for compatibility with |
- class nempy.historical_inputs.constraints.ConstraintData(raw_inputs_loader)
Loads generic constraint related raw inputs and preprocess them for compatibility with
nempy.markets.SpotMarketExamples
This example shows the setup used for the examples in the class methods. This setup is used to create a RawInputsLoader by calling the function _test_setup.
>>> import sqlite3 >>> from nempy.historical_inputs import mms_db >>> from nempy.historical_inputs import xml_cache >>> from nempy.historical_inputs import loaders
The InterconnectorData class requries a RawInputsLoader instance.
>>> con = sqlite3.connect('market_management_system.db') >>> mms_db_manager = mms_db.DBManager(connection=con) >>> xml_cache_manager = xml_cache.XMLCacheManager('test_nemde_cache') >>> inputs_loader = loaders.RawInputsLoader(xml_cache_manager, mms_db_manager) >>> inputs_loader.set_interval('2024/07/10 12:05:00')
Create a InterconnectorData instance.
>>> constraint_data = ConstraintData(inputs_loader)
>>> constraint_data.get_rhs_and_type_excluding_regional_fcas_constraints() set rhs type 0 #BANGOWF2_E 82.800000 <= 1 #BBATRYL1_E 50.000000 <= 2 #BBATTERY_E 50.000000 <= 3 #BBTHREE3_E 25.000000 <= 4 #BOWWPV1_E 6.100000 <= ... ... ... ... 1107 V_T_NIL_BL1 -10125.000000 >= 1108 V_T_NIL_FCSPS 493.111848 <= 1109 V_WDR_NO_SCADA 95.000000 <= 1110 V_WEMENSF_FLT_20 20.000000 <= 1111 V_YATPSF_FLT_20 20.000000 <= [975 rows x 3 columns]
- Parameters:
inputs_manager (historical_spot_market_inputs.DBManager)
Methods:
Get the rhs values and types for generic constraints, excludes regional FCAS constraints.
Get the rhs values and types for generic constraints.
Get the lhs coefficients of units.
Get the lhs coefficients of interconnectors.
Get the lhs coefficients of regions.
Get constraint details needed for setting FCAS requirements.
Get the violation costs for generic constraints.
Get the violation costs of non-generic constraint groups.
Get a boolean indicating if the over constrained dispatch rerun process was used for this interval.
- get_rhs_and_type_excluding_regional_fcas_constraints()
Get the rhs values and types for generic constraints, excludes regional FCAS constraints.
Examples
>>> inputs_loader = _test_setup()
>>> unit_data = ConstraintData(inputs_loader)
>>> unit_data.get_rhs_and_type_excluding_regional_fcas_constraints() set rhs type 0 #BANGOWF2_E 82.800000 <= 1 #BBATRYL1_E 50.000000 <= 2 #BBATTERY_E 50.000000 <= 3 #BBTHREE3_E 25.000000 <= 4 #BOWWPV1_E 6.100000 <= ... ... ... ... 1107 V_T_NIL_BL1 -10125.000000 >= 1108 V_T_NIL_FCSPS 493.111848 <= 1109 V_WDR_NO_SCADA 95.000000 <= 1110 V_WEMENSF_FLT_20 20.000000 <= 1111 V_YATPSF_FLT_20 20.000000 <= [975 rows x 3 columns]
- Returns:
Columns:
Description:
set
the unique identifier of the constraint set,
(as str)
type
the direction of the constraint >=, <= or
=, (as str)
rhs
the right hand side value of the constraint,
(as np.float64)
- Return type:
pd.DataFrame
- get_rhs_and_type()
Get the rhs values and types for generic constraints.
Examples
>>> inputs_loader = _test_setup()
>>> unit_data = ConstraintData(inputs_loader)
>>> unit_data.get_rhs_and_type() set rhs type 0 #BANGOWF2_E 82.800000 <= 1 #BBATRYL1_E 50.000000 <= 2 #BBATTERY_E 50.000000 <= 3 #BBTHREE3_E 25.000000 <= 4 #BOWWPV1_E 6.100000 <= ... ... ... ... 1107 V_T_NIL_BL1 -10125.000000 >= 1108 V_T_NIL_FCSPS 493.111848 <= 1109 V_WDR_NO_SCADA 95.000000 <= 1110 V_WEMENSF_FLT_20 20.000000 <= 1111 V_YATPSF_FLT_20 20.000000 <= [1112 rows x 3 columns]
- Returns:
Columns:
Description:
set
the unique identifier of the constraint set,
(as str)
type
the direction of the constraint >=, <= or
=, (as str)
rhs
the right hand side value of the constraint,
(as np.float64)
- Return type:
pd.DataFrame
- get_unit_lhs()
Get the lhs coefficients of units.
Examples
>>> inputs_loader = _test_setup()
>>> unit_data = ConstraintData(inputs_loader)
>>> unit_data.get_unit_lhs() set unit service coefficient 0 #BANGOWF2_E BANGOWF2 energy 1.0 1 #BBATRYL1_E BBATRYL1 energy 1.0 2 #BBATTERY_E BBATTERY energy 1.0 3 #BBTHREE3_E BBTHREE3 energy 1.0 4 #BOWWPV1_E BOWWPV1 energy 1.0 ... ... ... ... ... 17032 V_WDR_NO_SCADA DRXVDX01 energy 1.0 17033 V_WDR_NO_SCADA DRXVQP01 energy 1.0 17034 V_WDR_NO_SCADA DRXVQX01 energy 1.0 17035 V_WEMENSF_FLT_20 WEMENSF1 energy 1.0 17036 V_YATPSF_FLT_20 YATSF1 energy 1.0 [17037 rows x 4 columns]
- Returns:
Columns:
Description:
set
the unique identifier of the constraint set
to map the lhs coefficients to, (as str)
unit
the unit whose variables will be mapped to
the lhs, (as str)
service
the service whose variables will be mapped to the lhs, (as str)
coefficient
the lhs coefficient (as np.float64)
- Return type:
pd.DataFrame
- get_interconnector_lhs()
Get the lhs coefficients of interconnectors.
Examples
>>> inputs_loader = _test_setup()
>>> unit_data = ConstraintData(inputs_loader)
>>> unit_data.get_interconnector_lhs() set interconnector coefficient 0 DATASNAP_DFS_LS N-Q-MNSP1 1.0 1 DATASNAP_DFS_NCAN N-Q-MNSP1 1.0 2 DATASNAP_DFS_NCWEST N-Q-MNSP1 1.0 3 DATASNAP_DFS_NNTH N-Q-MNSP1 1.0 4 DATASNAP_DFS_NSYD N-Q-MNSP1 1.0 .. ... ... ... 827 V_S_HEYWOOD_UFLS V-SA 1.0 828 V_S_NIL_ROCOF V-SA 1.0 829 V_T_FCSPS_DS T-V-MNSP1 -1.0 830 V_T_NIL_BL1 T-V-MNSP1 1.0 831 V_T_NIL_FCSPS T-V-MNSP1 -1.0 [832 rows x 3 columns]
- Returns:
Columns:
Description:
set
the unique identifier of the constraint set
to map the lhs coefficients to, (as str)
interconnetor
the interconnetor whose variables will be
mapped to the lhs, (as str)
coefficient
the lhs coefficient (as np.float64)
- Return type:
pd.DataFrame
- get_region_lhs()
Get the lhs coefficients of regions.
Examples
>>> inputs_loader = _test_setup()
>>> unit_data = ConstraintData(inputs_loader)
>>> unit_data.get_region_lhs() set region service coefficient 0 D_I+BIP_ML2_L1 NSW1 lower_1s 1.0 1 D_I+BIP_ML2_L1 QLD1 lower_1s 1.0 2 D_I+BIP_ML2_L1 SA1 lower_1s 1.0 3 D_I+BIP_ML2_L1 TAS1 lower_1s 1.0 4 D_I+BIP_ML2_L1 VIC1 lower_1s 1.0 .. ... ... ... ... 498 F_TASCAP_RREG_0220 NSW1 raise_reg 1.0 499 F_TASCAP_RREG_0220 QLD1 raise_reg 1.0 500 F_TASCAP_RREG_0220 SA1 raise_reg 1.0 501 F_TASCAP_RREG_0220 VIC1 raise_reg 1.0 502 F_T_NIL_MINP_R6 TAS1 raise_6s 1.0 [503 rows x 4 columns]
- Returns:
Columns:
Description:
set
the unique identifier of the constraint set
to map the lhs coefficients to, (as str)
region
the region whose variables will be mapped
to the lhs, (as str)
service
the service whose variables will be mapped
to the lhs, (as str)
coefficient
the lhs coefficient (as np.float64)
- Return type:
pd.DataFrame
- get_fcas_requirements()
Get constraint details needed for setting FCAS requirements.
Examples
>>> inputs_loader = _test_setup()
>>> unit_data = ConstraintData(inputs_loader)
>>> unit_data.get_fcas_requirements() set service region type volume 0 D_I+BIP_ML2_L1 lower_1s NSW1 >= -10000.000000 1 D_I+BIP_ML2_L1 lower_1s QLD1 >= -10000.000000 2 D_I+BIP_ML2_L1 lower_1s SA1 >= -10000.000000 3 D_I+BIP_ML2_L1 lower_1s TAS1 >= -10000.000000 4 D_I+BIP_ML2_L1 lower_1s VIC1 >= -10000.000000 .. ... ... ... ... ... 498 F_TASCAP_RREG_0220 raise_reg NSW1 >= 170.000000 499 F_TASCAP_RREG_0220 raise_reg QLD1 >= 170.000000 500 F_TASCAP_RREG_0220 raise_reg SA1 >= 170.000000 501 F_TASCAP_RREG_0220 raise_reg VIC1 >= 170.000000 502 F_T_NIL_MINP_R6 raise_6s TAS1 >= 34.040015 [503 rows x 5 columns]
- Returns:
Columns:
Description:
set
unique identifier of the requirement set,
(as str)
service
the service or services the requirement set
applies to (as str)
region
the regions that can contribute to meeting a
requirement, (as str)
volume
the amount of service required, in MW,
(as np.float64)
type
the direction of the constrain ‘=’, ‘>=’ or
’<=’, optional, a value of ‘=’ is assumed if
the column is missing (as str)
- Return type:
pd.DataFrame
- get_violation_costs()
Get the violation costs for generic constraints.
Examples
>>> inputs_loader = _test_setup()
>>> unit_data = ConstraintData(inputs_loader)
>>> unit_data.get_violation_costs() set cost 0 #BANGOWF2_E 6300000.0 1 #BBATRYL1_E 6300000.0 2 #BBATTERY_E 6300000.0 3 #BBTHREE3_E 6300000.0 4 #BOWWPV1_E 6300000.0 ... ... ... 1172 V_T_NIL_BL1 6300000.0 1173 V_T_NIL_FCSPS 525000.0 1174 V_WDR_NO_SCADA 6300000.0 1175 V_WEMENSF_FLT_20 612500.0 1176 V_YATPSF_FLT_20 612500.0 [1177 rows x 2 columns]
- Returns:
Columns:
Description:
set
the unique identifier of the constraint set
to map the lhs coefficients to, (as str)
cost
the cost to the objective function of
violating the constraint, (as np.float64)
- Return type:
pd.DataFrame
- get_constraint_violation_prices()
Get the violation costs of non-generic constraint groups.
Examples
>>> inputs_loader = _test_setup()
>>> unit_data = ConstraintData(inputs_loader)
>>> unit_data.get_constraint_violation_prices() {'regional_demand': 2625000.0, 'interocnnector': 20125000.0, 'generic_constraint': 525000.0, 'ramp_rate': 20212500.0, 'unit_capacity': 6475000.0, 'energy_offer': 19862500.0, 'fcas_profile': 2712500.0, 'fcas_max_avail': 2712500.0, 'fcas_enablement_min': 1225000.0, 'fcas_enablement_max': 1225000.0, 'fast_start': 19775000.0, 'mnsp_ramp_rate': 20212500.0, 'msnp_offer': 19862500.0, 'mnsp_capacity': 6387500.0, 'uigf': 6737500.0, 'voll': 17500.0, 'tiebreak': 1e-06}
- Return type:
dict
- is_over_constrained_dispatch_rerun()
Get a boolean indicating if the over constrained dispatch rerun process was used for this interval.
Examples
>>> inputs_loader = _test_setup()
>>> unit_data = ConstraintData(inputs_loader)
>>> unit_data.is_over_constrained_dispatch_rerun() False
- Return type:
bool
RHSCalc
Classes:
|
Engine for calculating generic constraint right hand side (RHS) values from scratch based on the equations provided in the NEMDE xml input files. |
- class nempy.historical_inputs.rhs_calculator.RHSCalc(xml_cache_manager)
Engine for calculating generic constraint right hand side (RHS) values from scratch based on the equations provided in the NEMDE xml input files.
AEMO publishes the RHS values used in dispatch, however, those values are dynamically calculated by NEMDE and depend on inputs such as transmission line flows, generator on statuses, and generator output levels. This class allows the user to update the input values which the RHS equations depend on and then recalulate RHS values. The primary reason for implementing this functionality is to allow the Bass link switch run to be implemented using Nempy, which requires that the RHS values of a number of constraints to be recalculated for the case where the bass link frequency controller is not active.
The methodology for the calculation is based on the description in the Constraint Implementation Guidelines published by AEMO,
see AEMO doc. The main limitation of the method implemented is that it does not allow for the calculation of constraints that use BRANCH operation. In 2013 there were three constraints using the branching operation (V^SML_NIL_3, V^SML_NSWRB_2, V^S_HYCP, Q^NIL_GC), and in 2023 it appears the branch operation is no longer in active use. While there are some difference between the RHS values produced, generally they are small,Examples
>>> xml_cache_manager = xml_cache.XMLCacheManager('test_nemde_cache') >>> xml_cache_manager.load_interval('2019/01/01 00:00:00') >>> rhs_calculator = RHSCalc(xml_cache_manager)
- Parameters:
xml_cache_manager (instance of nempy class XMLCacheManager)
Methods:
get_nemde_rhs(constraint_id)Get the RHS values of a constraints as calculated by NEMDE.
compute_constraint_rhs(constraint_id)Calculates the rhs values of the speficied constraint or list of constraints.
A helper method used to find the which constraints' RHS depend on a given input value.
update_spd_id_value(spd_id, type, value)Updates the value of one of the inputs which the RHS constraint equations depend on.
- get_nemde_rhs(constraint_id)
Get the RHS values of a constraints as calculated by NEMDE. This method is implemented primarily to assist with testing.
Examples
>>> xml_cache_manager = xml_cache.XMLCacheManager('test_nemde_cache') >>> xml_cache_manager.load_interval('2019/01/01 00:00:00') >>> rhs_calculator = RHSCalc(xml_cache_manager) >>> rhs_calculator.get_nemde_rhs("F_MAIN++NIL_BL_R60") -10290.279635
- Parameters:
constraint_id (str which is the unique ID of the constraint)
- Return type:
float
- compute_constraint_rhs(constraint_id)
Calculates the rhs values of the speficied constraint or list of constraints.
Examples
>>> xml_cache_manager = xml_cache.XMLCacheManager('test_nemde_cache') >>> xml_cache_manager.load_interval('2019/01/01 00:00:00') >>> rhs_calculator = RHSCalc(xml_cache_manager) >>> rhs_calculator.compute_constraint_rhs('F_MAIN++NIL_BL_R60') -10290.737541856766
>>> rhs_calculator.compute_constraint_rhs(['F_MAIN++NIL_BL_R60', 'F_MAIN++NIL_BL_R6']) set rhs 0 F_MAIN++NIL_BL_R60 -10290.737542 1 F_MAIN++NIL_BL_R6 -10581.475084
- Parameters:
constraint_id (str or list[str] which is the unique ID of the constraint or a list of the strings which are) – the constraint IDs
- Return type:
float or pandas DataFrame
- get_rhs_constraint_equations_that_depend_value(spd_id, type)
A helper method used to find the which constraints’ RHS depend on a given input value.
Examples
>>> xml_cache_manager = xml_cache.XMLCacheManager('nemde_cache_2014_12') >>> xml_cache_manager.load_interval('2014/12/05 00:00:00') >>> rhs_calculator = RHSCalc(xml_cache_manager) >>> rhs_calculator.get_rhs_constraint_equations_that_depend_value('BL_FREQ_ONSTATUS', 'W') ['F_MAIN++APD_TL_L5', 'F_MAIN++APD_TL_L6', 'F_MAIN++APD_TL_L60', 'F_MAIN++ML_L5_0400', 'F_MAIN++ML_L5_APD', 'F_MAIN++ML_L60_0400', 'F_MAIN++ML_L60_APD', 'F_MAIN++ML_L6_0400', 'F_MAIN++ML_L6_APD', 'F_MAIN++NIL_DYN_LREG', 'F_MAIN++NIL_DYN_RREG', 'F_MAIN++NIL_MG_R5', 'F_MAIN++NIL_MG_R6', 'F_MAIN++NIL_MG_R60', 'F_MAIN+APD_TL_L5', 'F_MAIN+APD_TL_L6', 'F_MAIN+APD_TL_L60', 'F_MAIN+ML_L5_0400', 'F_MAIN+ML_L5_APD', 'F_MAIN+ML_L60_0400', 'F_MAIN+ML_L60_APD', 'F_MAIN+ML_L6_0400', 'F_MAIN+ML_L6_APD', 'F_MAIN+NIL_DYN_LREG', 'F_MAIN+NIL_DYN_RREG', 'F_MAIN+NIL_MG_R5', 'F_MAIN+NIL_MG_R6', 'F_MAIN+NIL_MG_R60', 'F_T++LREG_0050', 'F_T++NIL_BB_TG_R5', 'F_T++NIL_BB_TG_R6', 'F_T++NIL_BB_TG_R60', 'F_T++NIL_MG_R5', 'F_T++NIL_MG_R6', 'F_T++NIL_MG_R60', 'F_T++NIL_ML_L5', 'F_T++NIL_ML_L6', 'F_T++NIL_ML_L60', 'F_T++NIL_TL_L5', 'F_T++NIL_TL_L6', 'F_T++NIL_TL_L60', 'F_T++NIL_WF_TG_R5', 'F_T++NIL_WF_TG_R6', 'F_T++NIL_WF_TG_R60', 'F_T++RREG_0050', 'F_T+LREG_0050', 'F_T+NIL_BB_TG_R5', 'F_T+NIL_BB_TG_R6', 'F_T+NIL_BB_TG_R60', 'F_T+NIL_MG_R5', 'F_T+NIL_MG_R6', 'F_T+NIL_MG_R60', 'F_T+NIL_ML_L5', 'F_T+NIL_ML_L6', 'F_T+NIL_ML_L60', 'F_T+NIL_TL_L5', 'F_T+NIL_TL_L6', 'F_T+NIL_TL_L60', 'F_T+NIL_WF_TG_R5', 'F_T+NIL_WF_TG_R6', 'F_T+NIL_WF_TG_R60', 'F_T+RREG_0050', 'T_V_NIL_BL1', 'V_T_NIL_BL1']
- Parameters:
spd_id (str, the ID of the value used in the NEMDE xml input file.)
type (str, the type of the value used in the NEMDE xml input file. See the Constraint Implementation Guidelines) – published by AEMO for more information on SPD types,
see AEMO doc
- Return type:
list[str] a list of strings detailing the constraits’ whose RHS equations depend on the specified value.
- update_spd_id_value(spd_id, type, value)
Updates the value of one of the inputs which the RHS constraint equations depend on.
Examples
>>> xml_cache_manager = xml_cache.XMLCacheManager('test_nemde_cache') >>> xml_cache_manager.load_interval('2019/01/01 00:00:00') >>> rhs_calculator = RHSCalc(xml_cache_manager) >>> rhs_calculator.update_spd_id_value('220_GEN_INERTIA', 'A', '100.0')
- Parameters:
spd_id (str, the ID of the value used in the NEMDE xml input file.)
type (str, the type of the value used in the NEMDE xml input file. See the Constraint Implementation Guidelines) – published by AEMO for more information on SPD types,
see AEMO docvalue (str (detailing a float number) the new value to set the input to.)