Add OpenAlex ETL pipeline (Advanced level - source-agnostic import)#28
Draft
Buildspace081 wants to merge 7 commits into
Draft
Add OpenAlex ETL pipeline (Advanced level - source-agnostic import)#28Buildspace081 wants to merge 7 commits into
Buildspace081 wants to merge 7 commits into
Conversation
added 7 commits
July 18, 2026 15:20
- New www/services/openalex_client.py: search + cursor pagination, batch ID resolution for references, retry/backoff with Retry-After support - New www/services/openalex_mapper.py: OpenAlex -> 34-column WoS-style schema mapping (Dispatcher + declarative field mapping pattern) - New www/services/type_contracts.py: column type specs, coercion, and validation (STRING/INTEGER/STRING_LIST) - New www/services/etl_pipeline.py: end-to-end orchestrator (query -> extract -> map -> calculated fields -> validate -> DataFrame) - Patched get_historiograph.py, get_localcitedauthors.py, get_localciteddocuments.py to handle None from histNetwork() gracefully instead of crashing (pre-existing gap, not source-specific) - Fixed PY column type contract (STRING -> INTEGER): historical WoS pipeline only worked correctly by accident via pd.read_json's implicit type inference Tested end-to-end against 35/43 functions in functions/ with real OpenAlex API data.
Load Bibliometrix Data (.xlsx) previously did a bare pd.read_excel() with no handling for multi-value columns (AU, C1, CR, DE, etc). Excel cannot hold native Python lists, so a naive round-trip silently turned them into their Python repr() as a literal string, surviving undetected until an analysis function tried to treat it as a list (e.g. get_relevant_authors.py crashing with an opaque ValueError). Added _split_multivalue_cell() to re-split semicolon-joined values back into list[str] after loading, per the internal delimiter convention already specified in the project brief.
Replaces the 'under construction' placeholder in the API nav panel with a working query interface: text input + max_results, wired to the already-tested run_openalex_etl() pipeline. Follows the same UI/reactive pattern as the existing 'Import or Load' path (loading modal, error handling via notification_show, df.set() + reset_all_analyses(), results table via get_table()). Manually verified end-to-end in the live dashboard: query -> results table populated with real OpenAlex data -> CSV/Excel export working.
- app.py: sidebar menu wasn't appearing after API-page queries; toggle_sidebar() wasn't triggered by start_api_button (added it to the existing reactive.event trigger list) - openalex_client.py: requests' single timeout value only limits inter-chunk inactivity, not total request time; replaced with explicit (connect, read) tuple timeout, plus an overall time budget on search_works pagination - openalex_client.py/etl_pipeline.py: _request_with_retry honored Retry-After headers unconditionally, causing an ~8 hour sleep after hitting OpenAlex's rate limit during testing; capped the wait at MAX_RETRY_AFTER_WAIT_SECONDS=20, falls back to a normal OpenAlexRequestError if still failing after the capped wait
New donut chart showing OA status distribution (gold/green/hybrid/ bronze/diamond/closed/unknown), following the existing analysis page pattern (Plotly FigureWidget, Plot/Table tabs, Add in Report). This is an analysis not meaningfully available in the classic WoS schema - made possible specifically by using OpenAlex as the source. Live-verified: sidebar menu correctly shows all extended sections after loading data (also confirms the earlier toggle_sidebar fix works end-to-end), new page renders correctly with real data.
…ce issue - couplingmap.py::localCitations: same None-handling pattern as the 3 previous fixes (get_historiograph.py, get_localcitedauthors.py, get_localciteddocuments.py), adapted to this caller's expected return shape: returns LCS=0 for each document with correctly-shaped empty tables, instead of a bare None, since normalizeCitationScore downstream always expects a dict with a valid 'LCS' column. - get_clusteringcoupling.py: documented a known performance issue (Cluster by Coupling runs slowly on OpenAlex data) as out of scope, same pattern as the previously documented get_factorialanalysis.py limitation - root cause not identified in the coupling-matrix construction, not clearly related to our standardizer or CR format.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Source-agnostic ETL pipeline that converts OpenAlex API data into the standardized 34-column WoS-style schema used internally by bibliometrix-python (replicates R's
convert2df()). Advanced level: query-driven extraction (no manual downloads), pagination/retry/rate-limit handling, verified against existing analytical functions.Architecture (Dispatcher + Mapping + Type Contracts)
openalex_client.py— Extract: cursor-paginated search, batch reference-ID resolution, retry/backoff withRetry-Aftersupport.openalex_mapper.py— Transform: oneformat_XX_column()per schema field (declarative mapping, mirrors the existingformat_functions.pypattern), orchestrated bymap_work_to_record().type_contracts.py— Validation: per-column type contract (STRING/INTEGER/STRING_LIST), coercion, and validation — the final null-safety barrier required by the brief.etl_pipeline.py— Entry point:run_openalex_etl(query, max_results)chains the above end-to-end, no duplicated logic.Key design choices:
SRreuses the existingmetatagextraction.py::SR()instead of being reimplemented;CRresolves OpenAlex's reference IDs into readable "Author, Year, Journal" strings (bare ID as fallback);C1uses OpenAlex's structured institution/country data but formats it to match the WoS convention, so the existing country-extraction logic works unmodified.Debugging log (patches to existing code)
PYtype bug: historical pipeline only got numeric years by accident viapd.read_json's implicit type inference. Fixed by makingPYgenuinelyINTEGERin the type contract (was crashingget_annualproduction.py).histNetwork()returnsNoneuncaught for any non-WoS/Scopus source (pre-existing gap). Patchedget_historiograph.py,get_localcitedauthors.py,get_localciteddocuments.pyto degrade gracefully instead of crashing. Did not extendhistNetwork()itself — would require reintroducingSR_FULLand would produce near-always-empty results anyway.SN/JImapping bug (caught internally): ISSN was initially mapped to the wrong column. Fixed.Known limitation (out of scope)
get_factorialanalysis.pyhas a pre-existing, source-independent indexing bug (fails on WoS data under the same conditions too) — documented, not patched.Testing
35/43
functions/modules verified end-to-end against real OpenAlex API data. 8/43 require a live Shiny session (expected, covered by the dashboard demo). 1/43 open pre-existing bug (above).How to test
Or via dashboard:
shiny run app.py→ OpenAlex query option under Import Data.