Skip to content

fix(spark): BatchFeatureView with TransformationMode.PYTHON now reads all source columns#6310

Merged
ntkathole merged 1 commit into
feast-dev:masterfrom
abhijeet-dhumal:fix/bfv-python-mode-column-info
Apr 22, 2026
Merged

fix(spark): BatchFeatureView with TransformationMode.PYTHON now reads all source columns#6310
ntkathole merged 1 commit into
feast-dev:masterfrom
abhijeet-dhumal:fix/bfv-python-mode-column-info

Conversation

@abhijeet-dhumal
Copy link
Copy Markdown
Contributor

What

FeatureBuilder.get_column_info() sets feature_cols=[] to signal "read all source columns" for transformation modes that compute output features from raw input. This signal was only applied for "ray" and "pandas" modes — "python" was missing.
TransformationMode.PYTHON is the default mode for @batch_feature_view. This means every default @batch_feature_view using SparkComputeEngine was broken out of the box.

Why it breaks

With feature_cols populated from the view's output schema (e.g. ["user_avg_rating", "user_review_count"]), the Spark offline store read step generates:

SELECT user_id, user_avg_rating, user_review_count, event_timestamp
FROM s3a://bucket/processed/reviews/
WHERE event_timestamp BETWEEN ...

These output feature names don't exist in the raw source data — the UDF is supposed to compute them. Spark's analyzer throws:

AnalysisException: [UNRESOLVED_COLUMN.WITH_SUGGESTION] A column or function
parameter with name 'user_avg_rating' cannot be resolved.

Fix

Added "python" to the existing mode guard in get_column_info()

[x] Verified end-to-end on OpenShift AI with Feast 0.62.0 + PySpark 3.5.3 + MinIO + Redis — 2M+ feature vectors materialized successfully after fix

@abhijeet-dhumal abhijeet-dhumal force-pushed the fix/bfv-python-mode-column-info branch from 4070205 to 14b0203 Compare April 22, 2026 15:27
@abhijeet-dhumal abhijeet-dhumal changed the title fix: batchFeatureView with TransformationMode.PYTHON now reads all so… fix(spark): BatchFeatureView with TransformationMode.PYTHON now reads all source columns Apr 22, 2026
@abhijeet-dhumal abhijeet-dhumal force-pushed the fix/bfv-python-mode-column-info branch from 14b0203 to 4614530 Compare April 22, 2026 15:30
@abhijeet-dhumal abhijeet-dhumal marked this pull request as ready for review April 22, 2026 15:30
@abhijeet-dhumal abhijeet-dhumal requested a review from a team as a code owner April 22, 2026 15:30
Copy link
Copy Markdown
Contributor

@devin-ai-integration devin-ai-integration Bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ Devin Review: No Issues Found

Devin Review analyzed this PR and found no potential bugs to report.

View in Devin Review to see 3 additional findings.

Open in Devin Review

@abhijeet-dhumal abhijeet-dhumal force-pushed the fix/bfv-python-mode-column-info branch from 4614530 to 76ac945 Compare April 22, 2026 16:17
devin-ai-integration[bot]

This comment was marked as resolved.

devin-ai-integration[bot]

This comment was marked as resolved.

@ntkathole
Copy link
Copy Markdown
Member

@abhijeet-dhumal seems pushed the different commits in this PR ?

@abhijeet-dhumal abhijeet-dhumal force-pushed the fix/bfv-python-mode-column-info branch 3 times, most recently from 2139cff to bd3d4a1 Compare April 22, 2026 17:27
… all source columns

Signed-off-by: abhijeet-dhumal <abhijeetdhumal652@gmail.com>
@abhijeet-dhumal abhijeet-dhumal force-pushed the fix/bfv-python-mode-column-info branch from bd3d4a1 to d956d09 Compare April 22, 2026 17:28
@ntkathole ntkathole merged commit a310eaf into feast-dev:master Apr 22, 2026
29 of 33 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants