#2168 - proposal replace meilisearch elasticsearch with seal#2408
#2168 - proposal replace meilisearch elasticsearch with seal#2408MrHDOLEK wants to merge 45 commits into
Conversation
…ch-with-seal # Conflicts: # composer.json # composer.lock # phpstan.neon
Codecov Report❌ Patch coverage is Additional details and impacted files@@ Coverage Diff @@
## 1.x #2408 +/- ##
============================================
+ Coverage 84.94% 85.10% +0.15%
- Complexity 20648 21181 +533
============================================
Files 1570 1597 +27
Lines 63532 65358 +1826
============================================
+ Hits 53968 55623 +1655
- Misses 9564 9735 +171
🚀 New features to boost your workflow:
|
|
@MrHDOLEK as we discussed offline, please find some feedback and guidance below. Extractor / LoaderIn general I like the idea of having one extractor that accepts Seal EngineInterface which is aligned with how Doctrine Adapter works right now. Same goes for the loaders. It would also work very well with frameworks as seal provides a bundle for symfony that literally configures engine so to use it with flow it would be a simple matter of dependency injection. What to testI had to think about it a bit more but I dont think we should retest every backend that seal is testing. Doctrine adapter is testing MySQL, PostgreSql nad Sqlite because flow-php/docrtine-dbal-bulk which extends doctrine default behaviors that needs to be tested. At this time seal provides following adapters:
I would chose one of them (that requires the least configuration and is the easiest to setup in docker) and cover it with proper integration tests that would confirm that moving data from flow to seal works as expected. How to testSo in general we want to cover in tests that seal adapter can handle all types of flow Entries. You can achieve that by using one of the two extractors: They both expose static method SchemaConverterSchema converters are recursive algorithms that can convert schema in both directiosn Flow to Seal and Seal to Flow. You can find some inspirations here: It might be the easiest to use LLM to help you create one based on those 3 examples for Seal (it's a recursive brain damaging exercise that might not be worth spending time on). Of course schema converters would need a DSL method. Search Engine in TestsSo in this PR you are using traits to configure backends in tests, which is fine but there is a different pattern which I found cleaner and easier to maintain. Contexts. Here is a good example of DatabaseContext that is used in DatabaseTableListCommandTest If there will be more than one integration tests you can extract an abstract In case of any questions, you know where to find me 😁 |
…flow-php#2439) split DateTimeConverter into TimestampConverter and TimestampTzConverter - normalize TIMESTAMP values to UTC, tag timestamp reads as UTC - map DateTimeEntry/DateTimeType to timestamp by default
norberttech
left a comment
norberttech
left a comment
There was a problem hiding this comment.
Looks much better! I left some comments, nothing critical, but there is one gap related to Delete operation.
Please let me know if you have any questions about those comments!
| use Flow\ETL\Rows; | ||
| use Generator; | ||
|
|
||
| final readonly class RowsNormalizer |
There was a problem hiding this comment.
this one is missing unit tests but even more important, EntryNormalizer is missing them
| MYSQL_ROOT_PASSWORD: root | ||
| networks: | ||
| - flow-php | ||
| elasticsearch: |
There was a problem hiding this comment.
are we sure we dont wanna keep at least one real search engine in the repo? I mean I know we should be able to trust seal it works exactly same way but... Maybe keeping elasticsearch wouldnt be a bad idea and use it in integration tests keeping memory adapter for unit tests? (no strong opinion about this one yet, I never used seal so dont know how much in memory implementation is different from actual one)
…ime columns as timestamp instead of timestamptz (flow-php#2440)
…ow-php#2438) Bumps [symfony/polyfill-mbstring](https://github.com/symfony/polyfill-mbstring) from 1.38.1 to 1.38.2. - [Release notes](https://github.com/symfony/polyfill-mbstring/releases) - [Commits](symfony/polyfill-mbstring@v1.38.1...v1.38.2) --- updated-dependencies: - dependency-name: symfony/polyfill-mbstring dependency-version: 1.38.2 dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
) --- updated-dependencies: - dependency-name: phpstan/phpstan dependency-version: 2.2.2 dependency-type: direct:development update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
|
|
||
| enum Operation: string | ||
| { | ||
| case DELETE = 'delete'; |
There was a problem hiding this comment.
It's more replace than delete cause data will be available, no?
| return; | ||
| } | ||
|
|
||
| if (count($documents) < $this->pageSize) { |
There was a problem hiding this comment.
Instead of counting, add counter variable and increment it in loop?
| { | ||
| $engine = new Engine(new MemoryAdapter(), $schema); | ||
|
|
||
| foreach (array_keys($schema->indexes) as $index) { |
There was a problem hiding this comment.
| foreach (array_keys($schema->indexes) as $index) { | |
| foreach ($schema->indexes as $index => $value) { |
…utes (flow-php#2441) - add `resource_attributes` parameter (env, urldecoded key=value pairs) - merge attributes onto the telemetry resource, overriding defaults - percent-decode `OTEL_RESOURCE_ATTRIBUTES` keys/values in telemetry EnvironmentDetector (OTel spec), dropping backslash escaping
…flow-php#2442) * feat(flow-php/telemetry): expose log severity to the attribute filter - inject log.severity (number) and log.severity_name into signal attributes for matching * fix: remove coverage from test suits other than 8.3
…utdown SIGSEGV (grpc#38216) no longer fails CI (flow-php#2444) * fix: set experimental env var on grpc to avoid crashes * fix(flow-php/telemetry-otlp-bridge): isolate grpc tests from PHP 8.5 shutdown segfault - tag grpc-exercising tests with #[Group('grpc')] - split CI into main (--exclude-group grpc) and grpc steps - grpc step continue-on-error on 8.5 only (upstream grpc#38216)
…fault comparison (flow-php#2446) - model defaults as a `ColumnDefault` value object (literal, type, kind) - compare constant defaults type-aware so a wrong-typed stored cast is detected - add `ColumnType::isSameBaseType()` (base type, ignoring typmod)
…ndles (flow-php#2448) * feature: add Web Profiler integration for telemetry and postgresql bundles - telemetry data collector with spans, metrics and logs panels - postgresql data collector recording queries with EXPLAIN support - RecordingClient/QueryLog to capture executed queries - CompositeExporter to fan signals out to multiple exporters - profiler controllers, twig templates and DI wiring * chore: raise Web Profiler coverage and fix docs wording
…hp#2449) chore: Migrate Symfony XML usage to the PHP one
Bumps [google/protobuf](https://github.com/protocolbuffers/protobuf-php) from 5.35.0 to 5.35.1. - [Commits](protocolbuffers/protobuf-php@v5.35.0...v5.35.1) --- updated-dependencies: - dependency-name: google/protobuf dependency-version: 5.35.1 dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com>
…ogle/protobuf-5.35.1 build(deps): bump google/protobuf from 5.35.0 to 5.35.1
…h-elasticsearch-with-seal' into 2168-proposal-replace-meilisearch-elasticsearch-with-seal # Conflicts: # composer.lock
4 participants
The Elasticsearch and Meilisearch adapters are replaced by a single SEAL adapter
(
flow-php/etl-adapter-seal) built on top of SEAL— a PHP Search Engine Abstraction Layer. One adapter now covers Elasticsearch, OpenSearch,
Meilisearch, Algolia, Solr, Typesense, RediSearch and Loupe: the user builds a
CmsIg\Seal\EngineInterfaceand passes it to the DSL, exactly like the PostgreSQL/Doctrineadapters take a client. Tests are backend-agnostic (Memory adapter) and prove every Flow
Entry type survives a round-trip through the adapter.
Resolves: #2168
Change Log
Added
flow-php/etl-adapter-seal) — a single, strongly typed integration for SEAL search engines (Elasticsearch, OpenSearch, Meilisearch, Algolia, Solr, Typesense, RediSearch, Loupe)from_seal()extractor andto_seal()loader, working with any SEALEngineInterfaceto_seal_schema()andseal_schema_to_flow()DSL — recursive, bi-directional Flow Schema ↔ SEAL Schema conversion (nested structures, lists and maps)seal_create_index(),seal_drop_index(),seal_create_schema()andseal_drop_schema()DSL index-lifecycle helpersFixed
Changed
Removed
flow-php/etl-adapter-elasticsearch) — superseded by the SEAL adapter; use SEAL with the Elasticsearch backend (cmsig/seal-elasticsearch-adapter)Deprecated
Security