Scaffolding: Federation gateway setup by klpoland · Pull Request #299 · spectrumx/sds-code

klpoland · 2026-06-19T15:19:47Z

Note

Medium Risk
Introduces a new internal export surface and changes global API-key auth/permissions; misconfiguration of CIDRs or operational flags could block sync or widen access, though export is off by default and sync keys are route-scoped.

Overview
Adds gateway-side federation scaffolding so a local federation-sync service can pull public metadata and react to changes, controlled by FEDERATION_ENABLED and related env settings (documented in example django.env).

Internal export API is registered under /api/.../federation/ with list/detail endpoints for public finalized datasets and public captures. Responses use new DatasetFederationSerializer / CaptureFederationSerializer payloads (including site_name from FEDERATION_SITE_NAME). Access requires a FederationSync API key, federation operational checks (sync health URL, Redis ping, presence of sync key), and client IP in FEDERATION_EXPORT_ALLOWED_CIDRS (defaults to private ranges).

Change notifications: post_save signals on datasets/captures publish created/updated/deleted events to Redis (FEDERATION_EVENTS_CHANNEL) when federation is operational and the record is (or ceases to be) exportable.

API key behavior: APIKeyAuthentication now sets request.auth to the UserAPIKey object. A global DisallowFederationSyncKey permission blocks sync keys from normal asset APIs; export routes use the inverse IsFederationSyncKey. New create_federation_sync_api_key management command and DB migration extend KeySources with federation_sync.

Also includes contract tests against sibling sds_federation Pydantic models, hardening/operational tests, and small dev-tool tweaks (fallow-cross-file-dupes.sh, create_test_files import paths).

^{Reviewed by Cursor Bugbot for commit 3393c0f. Bugbot is set up for automated code reviews on this repo. Configure here.}

semanticdiff-com · 2026-06-19T15:19:50Z

Review changes with

Changed Files

File	Status
gateway/sds_gateway/api_methods/views/dataset_endpoints.py	42% smaller
gateway/sds_gateway/api_methods/tests/test_authenticate.py	26% smaller
gateway/config/settings/base.py	7% smaller
gateway/.envs/example/django.env	Unsupported file format
gateway/config/api_router.py	0% smaller
gateway/pyproject.toml	Unsupported file format
gateway/scripts/fallow-cross-file-dupes.sh	Unsupported file format
gateway/sds_gateway/api_methods/apps.py	0% smaller
gateway/sds_gateway/api_methods/authentication.py	0% smaller
gateway/sds_gateway/api_methods/federation/__init__.py	0% smaller
gateway/sds_gateway/api_methods/federation/availability.py	0% smaller
gateway/sds_gateway/api_methods/federation/events.py	0% smaller
gateway/sds_gateway/api_methods/federation/export_contract.py	0% smaller
gateway/sds_gateway/api_methods/federation/permissions.py	0% smaller
gateway/sds_gateway/api_methods/federation/signals.py	0% smaller
gateway/sds_gateway/api_methods/helpers/compile_federated_data.py	0% smaller
gateway/sds_gateway/api_methods/models.py	0% smaller
gateway/sds_gateway/api_methods/permissions.py	0% smaller
gateway/sds_gateway/api_methods/serializers/capture_serializers.py	0% smaller
gateway/sds_gateway/api_methods/serializers/dataset_serializers.py	0% smaller
gateway/sds_gateway/api_methods/tests/test_federation_export.py	0% smaller
gateway/sds_gateway/api_methods/tests/test_federation_export_contract.py	0% smaller
gateway/sds_gateway/api_methods/tests/test_federation_hardening.py	0% smaller
gateway/sds_gateway/api_methods/views/capture_endpoints.py	0% smaller
gateway/sds_gateway/api_methods/views/federation_endpoints.py	0% smaller
gateway/sds_gateway/api_methods/views/file_endpoints.py	0% smaller
gateway/sds_gateway/users/management/commands/create_federation_sync_api_key.py	0% smaller
gateway/sds_gateway/users/management/commands/create_test_files.py	0% smaller
gateway/sds_gateway/users/migrations/0012_alter_userapikey_source_federation_sync.py	0% smaller
gateway/sds_gateway/users/migrations/max_migration.txt	Unsupported file format
gateway/sds_gateway/users/models.py	0% smaller

klpoland · 2026-06-19T15:23:52Z

Added some dedicated API endpoints, serializers, and signals for federation and configuration that defaults to disabling federation sync.

lucaspar · 2026-06-25T19:12:17Z

+# FEDERATION
+# ------------------------------------------------------------------------------
+# the below environment variables are used for site federation with peers.
+# By default, federation events are disabled.
+# If FEDERATION_ENABLED is TRUE, deployment will need to include
+# federation sync service configuration.
+# FEDERATION_ENABLED=true
+# FEDERATION_EVENTS_ENABLED=true
+# FEDERATION_EVENTS_CHANNEL=federation:events
+# FEDERATION_SYNC_USER_EMAIL=
+# FEDERATION_SITE_NAME=
+# FEDERATION_SYNC_HEALTH_URL=http://federation-sync:8000/sync/health
+# FEDERATION_EXPORT_INTERNAL_HEADER_SECRET=
+# FEDERATION_EXPORT_TRUST_X_FORWARDED_FOR=false
+


add a one line comment explaining each value; highlight that federation-sync is a docker service in the same network, defined in /federation

cursor · 2026-06-26T16:10:05Z


        # Create files
-        minio_client = MinioClient()
+        minio_client = get_minio_client()


Test command calls missing upload method

Low Severity

This commit switches the dev command to get_minio_client(), which returns ObjectStoreFacade with MinIO-compatible methods like fput_object, but the upload loop still calls upload_file. That method is not defined on the facade or delegated MinIO client, so the command fails at runtime when uploading fixtures.

^{Reviewed by Cursor Bugbot for commit 2c09fae. Configure here.}

cursor

Cursor Bugbot has reviewed your changes using high effort and found 1 potential issue.

There are 2 total unresolved issues (including 1 from previous review).

^{❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.}

^{Reviewed by Cursor Bugbot for commit 3393c0f. Configure here.}

cursor · 2026-06-26T16:12:58Z

+            user=user,
+            source=KeySources.FederationSync,
+            description="Federation sync service (export endpoints only)",
+        )


Command always rotates sync keys

Medium Severity

Each run of create_federation_sync_api_key deletes every existing FederationSync API key for the sync user before creating a new one. Re-running the command invalidates keys already configured in federation-sync without an explicit rotate flag.

^{Reviewed by Cursor Bugbot for commit 3393c0f. Configure here.}

lucaspar · 2026-07-01T19:36:09Z

+# FEDERATION
+# ------------------------------------------------------------------------------
+# Peer sync uses the federation-sync Docker service (same sds-network as gateway;
+# service definition lives under /federation). Bootstrap: enable federation, run
+# create_federation_sync_api_key, pass the key to federation-sync. Set FEDERATION_SITE_NAME
+# (e.g. crc) when enabling federation; use SDS_SITE_FQDN for the public host (RFC [site].fqdn).
+# FEDERATION_ENABLED=true  # Master switch for export APIs and Redis federation events.
+# FEDERATION_SITE_NAME=crc  # RFC [site].name (short peer id); set SDS_SITE_FQDN separately for [site].fqdn.
+# FEDERATION_EVENTS_CHANNEL=federation:events  # Redis pub/sub channel federation-sync subscribes to.
+# FEDERATION_SYNC_HEALTH_URL=http://federation-sync:8000/sync/health  # Health probe target (federation-sync service).
+# FEDERATION_SYNC_USER_EMAIL=federation-sync@internal.local  # Service user email for create_federation_sync_api_key.
+# FEDERATION_EXPORT_ALLOWED_CIDRS=  # Comma-separated CIDRs allowed to call export (default: private Docker ranges).
+


comments need to be in a new line for .env files; otherwise the # ... will unintentionally be part of the values set:

cat .env | grep SSH_KEY SSH_KEY= # this is not interpreted as a comment docker exec -it ubuntu bash -c 'env | grep SSH_KEY' SSH_KEY=# this is not interpreted as a comment

klpoland self-assigned this Jun 19, 2026

klpoland added feature New feature or request federation Federation related work (sync service, document storage, peer configuration, etc.) labels Jun 19, 2026

klpoland changed the title ~~gateway: federation export API, Redis events, sync API key~~ Scaffolding: Federation gateway setup Jun 19, 2026

klpoland added the gateway Gateway component label Jun 19, 2026

klpoland force-pushed the feature-kpoland-federation-gateway-scaffolding branch 2 times, most recently from 9f94dfd to 83d51d0 Compare June 25, 2026 19:02

klpoland marked this pull request as ready for review June 25, 2026 19:03

klpoland requested a review from lucaspar June 25, 2026 19:03

cursor Bot reviewed Jun 25, 2026

View reviewed changes

Comment thread gateway/sds_gateway/api_methods/views/dataset_endpoints.py Outdated

Comment thread gateway/sds_gateway/api_methods/federation/availability.py

klpoland mentioned this pull request Jun 25, 2026

Scaffolding: Federation sync app creation #298

Open

klpoland added 5 commits June 25, 2026 15:52

gateway: federation export API, Redis events, sync API key

78b03aa

federation config hardening

7b6f1e0

pre-commit fixes

9e3fc32

arg to kwarg

3b3ea8f

fix test errors

4bbcbed

klpoland force-pushed the feature-kpoland-federation-gateway-scaffolding branch from 83d51d0 to 4bbcbed Compare June 25, 2026 19:53

lucaspar requested changes Jun 25, 2026

View reviewed changes

lucaspar added the migrations Code changes that require data or schema migrations in the database. label Jun 25, 2026

clean up env variable settings, ip parsing, auth classes

2c09fae

cursor Bot reviewed Jun 26, 2026

View reviewed changes

gateway mirror for health check

3393c0f

cursor Bot reviewed Jun 26, 2026

View reviewed changes

lucaspar requested changes Jul 1, 2026

View reviewed changes

lucaspar reviewed Jul 1, 2026

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Scaffolding: Federation gateway setup#299

Scaffolding: Federation gateway setup#299
klpoland wants to merge 7 commits into
masterfrom
feature-kpoland-federation-gateway-scaffolding

klpoland commented Jun 19, 2026 •

edited by cursor Bot

Loading

Uh oh!

semanticdiff-com Bot commented Jun 19, 2026 •

edited

Loading

Uh oh!

klpoland commented Jun 19, 2026

Uh oh!

Uh oh!

Uh oh!

lucaspar Jun 25, 2026

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

cursor Bot Jun 26, 2026

Uh oh!

cursor Bot left a comment

Uh oh!

cursor Bot Jun 26, 2026

Uh oh!

lucaspar Jul 1, 2026 •

edited

Loading

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Uh oh!

Conversation

klpoland commented Jun 19, 2026 • edited by cursor Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

semanticdiff-com Bot commented Jun 19, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

klpoland commented Jun 19, 2026

Uh oh!

Uh oh!

Uh oh!

lucaspar Jun 25, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

cursor Bot Jun 26, 2026

Choose a reason for hiding this comment

Test command calls missing upload method

Uh oh!

cursor Bot left a comment

Choose a reason for hiding this comment

Uh oh!

cursor Bot Jun 26, 2026

Choose a reason for hiding this comment

Command always rotates sync keys

Uh oh!

lucaspar Jul 1, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

klpoland commented Jun 19, 2026 •

edited by cursor Bot

Loading

semanticdiff-com Bot commented Jun 19, 2026 •

edited

Loading

lucaspar Jul 1, 2026 •

edited

Loading