Skip to content

[MCC-1515548] Python Lib - Data type of fetch_data() does not match d…#51

Merged
afieraru-mdsol merged 1 commit into
mainfrom
feat/MCC-1515548
Jun 16, 2026
Merged

[MCC-1515548] Python Lib - Data type of fetch_data() does not match d…#51
afieraru-mdsol merged 1 commit into
mainfrom
feat/MCC-1515548

Conversation

@butsyk-mdsol

@butsyk-mdsol butsyk-mdsol commented Jun 16, 2026

Copy link
Copy Markdown
Contributor

…ata type as shown in PWB

feature/bug/enhance/refactor/tech_debt/hotfix

  • Python Lib - Data type of fetch_data() does not match data type as shown in PWB
  • MCC-1507281

Checklist

  • Review the pull request to fix typos and ensure variable/function names are intuitive, etc.
  • Make sure the pull request does not have excessive number of unnecessary commits. Utilize the git commit --amend --no-edit command to reduce commit messages when making small file changes (like changing linespacing).
  • Make sure you have added unit tests for the code changes. Tests should be added in the tests/ folder.
  • Modify docs, if required. Add any new documentation in the doc/ folder.
  • Rebase on latest active development branch (develop/main).

Changes Summary

Clear schema metadata so types_mapper reliably maps Arrow types to ArrowDtype (embedded pandas metadata can override types_mapper, causing object dtypes).

Deployment Notes

In addition to IAC/code, add any notes for reviewer should know.

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR adjusts the Arrow→Pandas conversion used by fetch_data() to ensure Arrow-backed dtypes are consistently produced, matching the expected dtype behavior shown in PWB (avoiding unexpected object dtypes caused by embedded pandas schema metadata).

Changes:

  • Read the Arrow IPC stream into a pyarrow.Table before conversion.
  • Strip schema-level metadata from the Arrow table to prevent pandas metadata from overriding types_mapper.
  • Convert to pandas with types_mapper=pd.ArrowDtype after metadata removal.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment thread dataconnect/service/mappers.py
Comment thread dataconnect/service/mappers.py
@afieraru-mdsol afieraru-mdsol merged commit 95cbe5d into main Jun 16, 2026
4 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants