'(itUdZddlmZddlZddlZddlZddlmZddlZ ee  j dz Z dddd gd d d d ddgdd ddddgdd ddddgdd ddddgdd d d!d"d#gd$d d%d&d'd(gd)d d*d+d,d-d.d/gd d0d1d2d3gd4d d5d6d7dgd8d d9d:d;dgd<d g Zd=ed><gd?gd@gdAgdBgdCgdDgdEgdFgdGgdHgdIdJ ZdKedL<dMdNgdOgdPgdQdRgdSgdTdUgdVdWgdXgdYgdZgd[d\gd[gd] ZdKed^<ddcZdddZddeZdfdgdhdidjdkdldidmdndodigZd=edp<ddtZddxZdd|ZddZddZddZddZddZddZddZddZ e!dkr e dSdS)aGenerate sectors_data.json for the SPX sectors dashboard page. Data layers: - 11 GICS sectors (S&P 500 weights, 2026-Q1 snapshot) - For each sector: GICS industries (Level 3) with intra-sector weight - Forward P/E reference (Yardeni / FactSet style 2026-Q1 public values) - Trailing P/E live from yfinance per sector SPDR ETF Re-check quarterly: - Sector weights / industry breakdown: State Street SPDR fact sheets - Forward P/E: Yardeni "S&P 500 Sectors & Industries Forward P/Es" weekly ) annotationsN)Pathzsectors_data.jsonzInformation TechnologyXLKg>@g;@))Software@@)Semiconductorsg9@)z*Technology Hardware, Storage & Peripherals2@)#Semiconductor Materials & Equipment@)z IT Services @)zCommunications Equipment@)z.Electronic Equipment, Instruments & Componentsr )sectoretf weight_pct forward_pe industries FinancialsXLFgffffff,@g/@)),Financial Services (incl. BRK.B, V, MA, AXP)gG@)Banks (Diversified & Regional):@) Insurance1@)zCapital Marketsr )zConsumer Finance@z Health CareXLVg%@g1@))Pharmaceuticalsr) Health Care Equipment & Suppliesr) Biotechnology,@)zLife Sciences Tools & Services(@)z Health Care Providers & Servicesg&@)zHealth Care Technologyr zConsumer DiscretionaryXLYg$@8@))zBroadline Retail (AMZN-driven)r)zAutomobiles (TSLA-driven)r)zHotels, Restaurants & Leisure5@)zSpecialty Retail*@)z Textiles, Apparel & Luxury Goodsr )zHousehold Durablesr )zLeisure Products@) Distributorsr zCommunication ServicesXLCg333333#@g3@))*Interactive Media & Services (GOOGL, META)gH@)zEntertainment (NFLX, DIS, WBD)g3@)zWireless Telecom (TMUS, VZ, T)r")z&Diversified Telecommunication Servicesr )zMedia (advertising/cable)r IndustrialsXLIg333333!@6@) )Aerospace & Defenser ) Machineryr)z'Ground Transportation (Rail / Trucking)"@)zAir Freight & Logisticsr )zBuilding Productsr )zElectrical Equipmentr )z"Industrial Conglomerates (GE, HON)r.)zProfessional Servicesr.)zCommercial Services & Supplies@)z Trading Companies & Distributorsr )zConstruction & Engineeringr%)zPassenger Airlinesr%zConsumer StaplesXLPgffffff@r#))zBeverages (KO, PEP)r+)zHousehold Products (PG)r )zTobacco (PM, MO)r )z Food Productsr )z/Consumer Staples Merchandise Retail (COST, WMT)r")zFood & Staples Retailingr )zPersonal Care Productsr%EnergyXLEg333333 @g+@)Oil, Gas & Consumable FuelsgV@)Energy Equipment & Servicesr MaterialsXLBg@r ))#Chemicals (LIN, SHW, APD, ECL, DOW)gQ@)zMetals & Miningr$)zContainers & Packagingr )zConstruction Materialsr UtilitiesXLUg333333@))zElectric UtilitiesgO@)zMulti-Utilitiesr")z)Independent Power & Renewable Electricityr )z Gas Utilitiesr/)zWater Utilitiesrz Real EstateXLREg@) )-Specialized REITs (data center, self-storage)gB@)Telecom Tower REITs (AMT, CCI)r$)zIndustrial REITs (PLD)r$)z Retail REITsr )zResidential REITsg$@)zHealth Care REITsr )z Office REITsr)zHotel & Resort REITsr)z$Real Estate Management & Developmentg@ list[dict]SECTORS)NVDAAVGOAMDQCOMINTCAMATLRCXKLACMUTXNADIMRVLMCHPMPWRNXPIONASMLTERARM)MSFTORCLCRMSAPADBENOWINTUPLTRIBMCRWDPANWFTNTSNOWWDAYADSKTEAMDDOGZSNET)LLYJNJABBVMRKPFEBMYZTSVTRS) AMGNVRTXGILDREGNBIIBALNYMRNABMRNINCY)ISRGABTMDTSYKBSXEWBDXBAXZBHRMDPODDIDXXSTEDXCMRVTYWSTALGNGEHC)JPMBACWFCCUSBPNCTFCMTBFITBRFHBANKEYCFGCMA) zBRK-BVMAAXPPYPLFISFICOFGPNDFS)PGRCBAIGMETPRUALLAFLHIGTRVMMCAONAJGWTWAIZ) RTXBALMTGEGDNOCLHXTDGHEIHIIAXONTXT)GOOGLGOOGMETAPINSMTCH)XOMCVXCOPEOGOXYMPCPSXVLOHESFANGKMIWMBETDVN) rrrrrrrrr,r(r3zdict[str, list[str]]INDUSTRY_TICKERSSOXXSMHzSOXX (partial)IGVIBBXBIIHEKBEKREITAXARXOPXESIYMVPNSRVR) rr rrrrr,r3r4r7r;r< ETF_HINTStickerstrreturn float | Nonec tj|jpi}|d}|t |ndS#t $rYdSwxYw)N trailingPE)yfTickerinfogetfloat Exception)rrvs P/home/raptorx-1/project/yardeni/sentiment_indicator/web/generate_sectors_data.py_live_trailing_persey  %+ HH\ " "=uQxxxd2 ttsAA AAc tj|ddd}|jrgSdt |j|dDS#t $rgcYSwxYw)zjFull-history monthly close prices for an ETF, used for regime / NBER backtests. Returns [] on failure.max1moTperiodinterval auto_adjustcg|]U\}}|t|tr||k"|dtt|ddVS)N%Y-%m-%ddateclose isinstancerstrftimeround.0dtps r z2_live_etf_history_monthly_full..c   A}jE&:&:}Q[[,,uU1XXq7I7I J J@FCloserrhistoryemptyzipindexrrhists r_live_etf_history_monthly_fullr s y  ((SW(XX : I  TZg77     2A$A A)(A)c tj|ddd}|jrgSdt |j|dDS#t $rgcYSwxYw)z5y weekly close prices for an ETF, formatted for charting. Returns [] on failure. We use weekly to keep JSON size sane (~260 points).5y1wkTrcg|]U\}}|t|tr||k"|dtt|ddVS)Nrrrrs rrz/_live_etf_history_5y_weekly...rrrrr s r_live_etf_history_5y_weeklyr's y  ((uRV(WW : I  TZg77     r z 2001 dot-comz 2001-03-01z 2001-11-30)namestartendz2008 GFCz 2007-12-01z 2009-06-30z 2020 COVIDz 2020-02-01z 2020-04-30NBER_RECESSIONSseries start_dateend_datec|sdStfd|Dd}d}|D]}|d|kr |d}|||dkrdSt||z dz dzdS) uCumulative total return of an ETF time-series between two dates. Picks the first close ≥ start_date and the last close ≤ end_date. Returns None if either anchor is missing (ETF not yet listed).Nc3@K|]}|dk|dVdS)rrN)rrrs r z)_series_return_between..Fs5NNqai:6M6M' 6M6M6M6MNNrrrrdr)nextr)rrr start_close end_closers ` r_series_return_betweenr%@s tNNNNFNNNPTUUKI  V9 ' II i/;!3C3Ct )k)A-4a 8 88rdate_strmonthsintcddlm}|d\}}}t|t|}}|dz|dz z|z}t |d\}}|dd|dzddS) z:date_str + N months, returned as ISO date string (day=01).r)r- r 04d02dz-01)datetimersplitr(divmod) r&r'rym_totalnynms r _add_monthsr7RsnnS!!GAq! q663q66qA Vq1u  'E E2  FB & & &rAv & & & &&rweekly_historydict[str, list[dict]]sectorsc  |dg}|siS|dd}d|D}i}|D]L}||dg}|s!|dd|dd|Dd||d<Md |D}|D]} | d } | d} | |z } | d kri} |D]E\}}|d | }||d kr)||dz }|d|| z z| |<Ft| }|d kr| D]7\}}||| t ||z dzdd8|S)uApproximate per-sector SPX weight over 5y from ETF total-return ratios. Math: anchor at "today's known weights". For each historical date t: raw_i(t) = w_i(today) × (ETF_i_price(t) / ETF_i_today) / (SPY(t) / SPY_today) w_i(t) = raw_i(t) / Σ_j raw_j(t) × 100 The (ETF_i / SPY) ratio captures relative cumulative return → reverse-engineers each sector's market-cap fraction. Then renormalize so the 11 sectors sum to 100%. Caveat: assumes intra-sector share-count proportions are stable (no major IPO/buyback rebalancing). For 5y this is a decent approximation; for sector- weight time series of true accuracy, S&P / FactSet historical data needed. SPYrc,i|]}|d|dSrrrrs r z+_compute_weight_history..os"222qy!G*222rrrc,i|]}|d|dSrrr?s rr@z+_compute_weight_history..zs"AAAQ1V9ajAAAr) today_closew_today series_mapci|]}|gSrr)rrs rr@z+_compute_weight_history..}s!D!D!Dc#r!D!D!DrrrrDNrBrCr!)rr)ritemssumvaluesappendr)r8r:spy spy_todayspy_mapsector_anchorssroutrrspy_t spy_ratiorawrancetf_t etf_ratior4raw_ws r_compute_weight_historyrX\s   UB ' 'C  B I22c222G')N   ##AeHb11  !":g.AA&AAA$ $ qx  "E!D^!D!D!DC YYy' I% >> &,,.. @ @HC %))$//E} M 22I9~Y)>?CHHCJJLL!! A:: ))++ Y YJC HOOTuu}s?RTU9V9VWW X X X X Y Jrmonthly_historydictc T |dg}|siSg}tD]p}|d|ddd|d|dd|d |ddd |dt|dd dqd |Ddgz g |D]}|d |d|d|did}t ||d|d}||dd< D]j}|dkr t ||g|d|d} | |d|<| |t | |z dnd|d|d<k |d fd } dddd| dddd dd| d dg} | z ddS)u4For each NBER recession, compute per-sector total return during: - Recession period (peak → trough) - Early Recovery: 12 months after recession end And one regime spanning the entire sample (1999-12 → today) for "Full History" reference. Returns dict ready for JSON serialization.r<z Recession (r) recessionrr)regimephaserrzEarly Recovery (early_recoveryr+cg|] }|d SrrrrOs rrz*_build_sector_rotation..s***Q%***rr^r_)r^r_rrreturnsrdNr_alpharrrZc8i}D]fdD}|r-tt|t|z dnd|<dkrFfdD}|r-tt|t|z dnd|d<|S)Ncg|]D}|dk|d)|dES)r_rdrrrrr_s rrz?_build_sector_rotation.._avg_across..sTuuua1W:;N;NSTU^S_ScScdgShShStAiL$$S))StStStrrr<cg|]J}|dk|dd,|ddKS)r_rdrerhris rrz?_build_sector_rotation.._avg_across..sQQQa1W:Y^K^K^cdencocscswzuCuCuCdDdDdP9))S...99dPdPdPrre)rrHlen)r_rPvalsavalsrall_etfsrowss` @r _avg_acrossz+_build_sector_rotation.._avg_acrosss') [ [Cuuuuu4uuuD:>HuSYYT2A666DCHe||QQQQQ4QQQKP&ZeCJJU,CQ&G&G&GVZsNNN# ruAVG · Recessionu—uAVG · Early RecoveryuPer-sector cumulative total return during NBER recession peaks/troughs and the 12-month period immediately after each recession ('early recovery'). alpha = sector return − SPY return for that window. ETFs without coverage (e.g. XLC inception 2018, XLRE 2015) show n/a.)regimesetfsmethod)r_rrrZ)rrrJr7r%r)rYr:rKrqrecregrowspy_retrretrp compositernros @@r_build_sector_rotationrzs   eR ( (C  G   2CK222 \u:      7V777%Zs5z2..       +*'***eW4H D  XWG ]`af]gtvww(c'lCJGG 'Iu | |Ce||()<)>> * * C&kRW`er}r}JsKsK L L*6FRW`er}r}OsPsP Q QI t# S   rc tjdd}|jrdSt t |djddS#t$rYdSwxYw)z10Y Treasury yield (percent), used as the risk-free anchor for ERP. Source: yfinance ^TNX (CBOE 10Y Treasury Yield Index). Returns the most recent close in percent (e.g. 4.31 means 4.31%).z^TNX5d)rNrr=r)rrrrrrilocr)r s r_live_10y_treasury_yield_pctr~s|y  (((55 : 4U4=-b122A666 tts/A!-A!! A/.A/tickers list[str] dict | Nonec |sdSd}d}d}g}|D]} tj|jpi}n%#t$r||YAwxYw|d}|d}t |ttfrbt |ttfrF|dkr@|dkr:|t|z }|t|t|z z }|dz }|||dkrdSt||z dd||t|d z dt|d z dd S) u7Aggregate forward P/E for a sub-industry as a true portfolio P/E: Σ(market_cap_i) / Σ(market_cap_i / forward_PE_i) i.e. total portfolio market cap divided by total forward earnings — mathematically equivalent to a market-cap-weighted HARMONIC mean of P/E. This matches the Yardeni / FactSet / JPM "Guide to Markets" sector P/E convention. The naive arithmetic mcap-weighted PE is wrong because it over-weights high-PE outliers (e.g. ARM @ 109) whose earnings contribution is tiny. Returns None when no constituents have valid data.Nr forwardPE marketCapr ruGportfolio P/E (= Σ mcap / Σ forward earnings, harmonic mcap-weighted)geA)valuersconstituents_usedconstituents_skippedtotal_mcap_billion_usd"total_forward_earnings_billion_usd) rrrrrJrrr(rr) rsum_mcap sum_earningsusedskippedtrfpemcaps r_weighted_industry_fwd_pers tHL DG  9Q<<$*DD    NN1    H hh{##xx $$ cC< ( ( ZsEl-K-K PSVWPWPW\`cd\d\d d #H E$KK%**4 4L AIDD NN1    qtx,.22[! '"'3":":.3L34F.J.J   s/AAr trailing_pecr|r|r |dks|dkrdS||z dz dz}|dkrdSt||z dS)uImplied PEG = fwdPE / implied_growth%, where implied_growth% = (trailing_PE / forward_PE - 1) * 100. Returns None when growth ≤ 0 (PEG undefined / "infinitely expensive").rNr r!rFr)rr growth_pcts r _implied_pegrsa [J!OO{a?O?Ot *Q.#5JQt j(! , ,,rrIlist[float | None]cd|D}t|dkrdgt|zStj|tj|dkr d|DSfd|DS)zOPer-element z-score. Drops None from mean/std calculation; None passes through.cg|]}||SNrrrs rrz_zscore..s 0 0 01!-Q---rrNrcg|]}|dnd S)Nrrrs rrz_zscore..$s ???1q}$???rcDg|]}|t|z z dndS)NrFr)rrmusds rrz_zscore..%s4 O O O1q}E1r6R- # # #$ O O Or)rk statisticsmeanstdev)rIvalidrrs @@r_zscorers 0 0 0 0 0E 5zzA~~vF ##   B  % B Qww?????? O O O O O O O OOr ust10y_pct!tuple[float | None, float | None]c||dkrdSd|z }|t|ddfSt|dt||z dfS)zKEarnings Yield = 1/fwdPE (in %); ERP = EY - 10Y Treasury yield (both in %).Nr)NNY@rr)rreys r_ey_erpr(sYZ1__z  BR||T!! Q<<rJ22 22rct}dtDdgz}d|D}g}d}d}tD]}t|d}td|dD}g} |dD]\} } | | t |d| zd z d t | gd } tt| g} | 3| d | d <| | d<t| d |\}}|| d<|| d<| | t|d |\}}t|d |}| |d|d|d||d |||t |d| d ||dz }||d|d zz }|rt ||z dnd}t||\}}d\}}d|D}d|D}t|}t|}t|D]P\}}||||}}||d|d<||||dd|d<*t ||z||zzd |d<||||d|d<Qt||}t||}dt |d|||||||dddd d!d"d#d$d%d&d'd(d)d* |d+ S),Ncg|] }|d Srbrrcs rrz!build_payload..9s111Aqx111rr<c.i|]}|t|Sr)r )rrs rr@z!build_payload..:s#RRRQA5a88RRRrrrc3 K|] \}}|V dSrr)rr3ws rrz build_payload..Bs&66da666666rrrrrF)rweight_in_sector_pctweight_in_spx_pct etf_hintsrrforward_pe_metaearnings_yield_pcterp_pctrr) rrrrrrr implied_pegindustry_total_checkr)g333333?g?c8g|]}|dr d|dz ndS)rg?Nrrcs rrz!build_payload..ns1XXX1Am,<Fq'''$XXXrcg|] }|d S)rrrcs rrz!build_payload..os * * *QAiL * * *rcomposite_scoreuAPEG undefined (implied growth ≤ 0); composite score unavailable) z_inv_pegz_erpalphabetanotecomposite_score_breakdown)rrrrz2026-Q1z,S&P / State Street SPDR fact sheets, 2026-Q1z*State Street SPDR ETF fact sheets, 2026-Q1z0yfinance live (sector SPDR ETF .info.trailingPE)z)Yardeni / FactSet 2026-Q1 public releasesuyfinance live, market-cap-weighted aggregation across the industry's top constituents (≥80% AUM coverage). Industries not yet in INDUSTRY_TICKERS fall back to inheriting the parent sector's fwd P/E.z9yfinance ^TNX (CBOE 10Y Treasury Yield Index), live closeu:EY = 1 / forward P/E (盈利收益率, 类似债券 yield)ufERP = EY - 10Y Treasury yield (Equity Risk Premium 简化版, 越高股票相对债券越有吸引力)uTPEG = forward P/E / implied growth %, 其中 implied_growth% = (trail/fwd - 1) * 100uComposite Value Score (简化 2-factor): score = 0.6 × z(1/PEG) + 0.4 × z(ERP), z-score 在 11 个板块内部标准化. 越高 = 综合性价比越好. 完整 3-factor 版还需历史 PE z-score (需付费源 / 手工 Yardeni 数据).uwyfinance full-history monthly closes (auto-adjusted), 用于价格走势图. 覆盖范围: 9 核心 sector ETF (XLK/XLF/XLV/XLY/XLP/XLI/XLB/XLE/XLU) 1998-12 起, XLRE 2015-10 起, XLC 2018-06 起 (GICS 改版). 注: 这不是历史 PE 时间序列, 仅是价格走势; 因 yfinance 对 ETF 无历史 EPS, 真历史 PE 需要付费源 (FactSet/Bloomberg) 或 SimFin/AV 重建.uPer-sector approx SPX weight, monthly, 1998-12+. 重建公式: w_i(t) = w_i(today) × (ETF_i(t)/ETF_i(today)) ÷ (SPY(t)/SPY(today)), 再 normalize 使各时点 11 板块加和 = 100%. 假设 sector 内 share-count 比例稳定. 近似精度: 5y ±0.5pp, 27y 长期 ±2pp 量级 (有 GICS 改版导致结构性偏差, 主要是 2018 XLC 重组). 真实历史 SPX 板块权重需要 S&P/FactSet 付费源.uA板块在 NBER 衰退/复苏 regime 下的累计回报 + 相对 SPY 的 alpha. Recession 期 = NBER 顶到 NBER 底; Early Recovery = NBER 底 + 12 个月. 覆盖 2001 dot-com / 2008 GFC / 2020 COVID 三次衰退. XLC (inception 2018), XLRE (inception 2015) 仅覆盖 2020 COVID. ETF 来自 yfinance auto-adjusted 月线.) sector_weightsindustry_breakdownrforward_pe_sectorforward_pe_industry ust10y_yieldearnings_yielderprrhistorical_pricessector_weight_historysector_rotation) as_ofweight_total_checkspx_forward_peust10y_yield_pctspx_earnings_yield_pct spx_erp_pcthistorical_prices_max_monthly!sector_weight_history_max_monthlyr data_sourcesr:)r~r>rrHrrrrrrrJrr enumeraterzrX)ust10yhistory_tickers monthly_fullr: weight_totalfwd_pe_weightedrOr ind_totalrrpctind_objaggrrpegrspx_eyspx_erpALPHABETAinv_pegserpsrrizpzerweight_historys r build_payloadr2s ) + +F 21111UG;ORR/RRRLGLO &=&='%11 66a o66666  < ' 'ID#(+%*1\?S+@5+H!%L%L&]]444 G ,,<,@,@r,J,JKKC(+G  %-0)*!#g,77C02,-%( "   g & & & &!L/622C1\?K88H+x o* o&("(-i(;(;(    ,' 1\?Q|_<<AMWU?\91===SWNnf55OFG KE4XXPWXXXH * *' * * *D!!I DMME'""  11uQxB :#'A "ud[..A) * * $)dRi)?#C#CA "ud..A) * *-\7CCO,\7CCN#L!44(""()5-;*L"NM!L/ XZ{qe hM__annotations__rrrr rrr%r7rXrzr~rrrrrr__name__rrrrs   #"""""  h   ! ! (+> >+             +    "+         *%     1 1               ]_____R ))) '''555  333$$$Y0*0*0000puo,<+=U^w',en!5>$)7$)7,176;V_',g # #             &|LII |LII |LII9999$''''4444nAAAAH    ))))X - - - - P P P P3333GGGGT$ zDFFFFFr