diff --git a/CHANGELOG.md b/CHANGELOG.md index 56fdef005a..636430ddf5 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -10,6 +10,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 ### Added - SAP HANA database driver plugin. (#1966) +- Column mappings remembered per table for CSV, JSON and Excel imports, plus Match by Name and Match by Position. (#3172) - Folders for tables and views in the sidebar. (#3167) - Shift JIS, EUC-JP, GB 18030, Big5, EUC-KR and UTF-16 options for CSV and SQL import. @@ -26,6 +27,11 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 - Explain Analyze running write statements on Read-Only connections and skipping the Alert and Safe Mode confirmation. - Remote deletions of connections, groups, tags, SSH profiles and table favorites applied with their sync category off. - **Local only** connections taking edits and deletions made on another device. +- Import sheet ignoring a CSV or Excel option change until the next edit, then resetting the column mapping. +- Import sheet showing an earlier table's columns after switching tables while the first was still loading. +- CSV and JSON imports failing on every row, or writing a skipped field, when two fields differ only by case. +- CSV and Excel imports reading a column under the wrong header when headers repeat or a blank one comes first. +- Import sheet's Try Again for an existing table discarding the column edits made for a new table. - Export dialog offering a SQL row scope on MongoDB, Redis and other engines without SQL. - CSV, XLSX and MQL exports ignoring a table's row filter, row limit and column choice. - MQL export rounding 64-bit integers past 2^53 and restoring whole doubles and small 64-bit integers as 32-bit ones. @@ -58,6 +64,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 - `EXPLAIN ANALYSE` treated as a plain `EXPLAIN` that does not run its statement. - Safe Mode level lost when importing a connection file from Mac to iOS or from iOS to Mac. - Undo and Redo in a tab with unsaved edits replaying another tab's changes against the wrong rows. +- Table Transfer emptying a destination table, then failing, when two source columns map to one column. - MongoDB, Elasticsearch, Typesense and SurrealDB saving a long array or object shortened for display as the cut text. - MongoDB refusing text like `[DRAFT] Chapter one...` as a value shortened for display. - Clipboard URL banner turning `sslmode=verify-full` or `verify-ca` into Required and ignoring `sslmode=disable`. diff --git a/Packages/TableProCore/Tests/TableProSyncTests/SyncRecordMapperTests.swift b/Packages/TableProCore/Tests/TableProSyncTests/SyncRecordMapperTests.swift index 0463cdcad4..2fa2afdfb4 100644 --- a/Packages/TableProCore/Tests/TableProSyncTests/SyncRecordMapperTests.swift +++ b/Packages/TableProCore/Tests/TableProSyncTests/SyncRecordMapperTests.swift @@ -86,4 +86,25 @@ struct SyncRecordMapperTests { let decoded = try #require(SyncRecordMapper.toConnection(record)) #expect(decoded.safeModeLevel == .confirmWrites) } + + @Test("An unrecognized wire value keeps the legacy read-only restriction") + func unknownWireValuePreservesReadOnly() throws { + let record = makeRawRecord(safeModeLevelRaw: "someFutureLevel", isReadOnly: true) + let decoded = try #require(SyncRecordMapper.toConnection(record)) + #expect(decoded.safeModeLevel == .readOnly) + } + + @Test("A rename preserves an unrecognized wire value and requires confirmation") + func renamePreservesUnknownWireValue() throws { + let record = makeRawRecord(safeModeLevelRaw: "someFutureLevel") + var connection = try #require(SyncRecordMapper.toConnection(record)) + connection.name = "Renamed" + + SyncRecordMapper.updateRecord(record, with: connection) + + #expect(record["safeModeLevel"] as? String == "someFutureLevel") + let decoded = try #require(SyncRecordMapper.toConnection(record)) + #expect(decoded.name == "Renamed") + #expect(decoded.safeModeLevel == .confirmWrites) + } } diff --git a/Plugins/BeancountDriverPlugin/Info.plist b/Plugins/BeancountDriverPlugin/Info.plist index 7eb91db53c..57fc985e56 100644 --- a/Plugins/BeancountDriverPlugin/Info.plist +++ b/Plugins/BeancountDriverPlugin/Info.plist @@ -3,7 +3,7 @@ TableProPluginKitVersion - 33 + 34 TableProProvidesDatabaseTypeIds Beancount diff --git a/Plugins/BigQueryDriverPlugin/Info.plist b/Plugins/BigQueryDriverPlugin/Info.plist index 786451d274..dc1215cdd8 100644 --- a/Plugins/BigQueryDriverPlugin/Info.plist +++ b/Plugins/BigQueryDriverPlugin/Info.plist @@ -5,6 +5,6 @@ TableProMinAppVersion 0.42.0 TableProPluginKitVersion - 33 + 34 diff --git a/Plugins/CSVExportPlugin/Info.plist b/Plugins/CSVExportPlugin/Info.plist index 94c80fac7e..5f65421016 100644 --- a/Plugins/CSVExportPlugin/Info.plist +++ b/Plugins/CSVExportPlugin/Info.plist @@ -3,7 +3,7 @@ TableProPluginKitVersion - 33 + 34 TableProProvidesExportFormatIds csv diff --git a/Plugins/CSVImportPlugin/CSVImportParsing.swift b/Plugins/CSVImportPlugin/CSVImportParsing.swift index 1897fef5fd..1a8a41c965 100644 --- a/Plugins/CSVImportPlugin/CSVImportParsing.swift +++ b/Plugins/CSVImportPlugin/CSVImportParsing.swift @@ -31,22 +31,10 @@ enum CSVImportParsing { } static func columnNames(header: [String]?, columnCount: Int) -> [String] { - var names: [String] = [] - names.reserveCapacity(columnCount) - var used = Set() - for index in 0.. PluginCellValue { diff --git a/Plugins/CSVImportPlugin/CSVImportPlugin.swift b/Plugins/CSVImportPlugin/CSVImportPlugin.swift index f8761cfffe..817d03e083 100644 --- a/Plugins/CSVImportPlugin/CSVImportPlugin.swift +++ b/Plugins/CSVImportPlugin/CSVImportPlugin.swift @@ -18,6 +18,7 @@ final class CSVImportPlugin: ObservableObject, ImportFormatPlugin, SettablePlugi static let acceptedFileExtensions = ["csv", "tsv"] static let iconName = "tablecells" static let requiresTargetTable = true + static let sourceFieldsFollowFileOrder = true typealias Settings = CSVImportOptions static let settingsStorageId = "csv-import" diff --git a/Plugins/CSVImportPlugin/Info.plist b/Plugins/CSVImportPlugin/Info.plist index 4f0d593ed7..9911a35c82 100644 --- a/Plugins/CSVImportPlugin/Info.plist +++ b/Plugins/CSVImportPlugin/Info.plist @@ -3,7 +3,7 @@ TableProPluginKitVersion - 33 + 34 TableProProvidesImportFormatIds csv diff --git a/Plugins/CassandraDriverPlugin/Info.plist b/Plugins/CassandraDriverPlugin/Info.plist index 446492c1ce..2dc8310456 100644 --- a/Plugins/CassandraDriverPlugin/Info.plist +++ b/Plugins/CassandraDriverPlugin/Info.plist @@ -21,6 +21,6 @@ NSPrincipalClass $(PRODUCT_MODULE_NAME).CassandraPlugin TableProPluginKitVersion - 33 + 34 diff --git a/Plugins/ClickHouseDriverPlugin/Info.plist b/Plugins/ClickHouseDriverPlugin/Info.plist index e1fadc2b29..ec88638929 100644 --- a/Plugins/ClickHouseDriverPlugin/Info.plist +++ b/Plugins/ClickHouseDriverPlugin/Info.plist @@ -3,7 +3,7 @@ TableProPluginKitVersion - 33 + 34 TableProProvidesDatabaseTypeIds ClickHouse diff --git a/Plugins/CloudflareD1DriverPlugin/Info.plist b/Plugins/CloudflareD1DriverPlugin/Info.plist index 786451d274..dc1215cdd8 100644 --- a/Plugins/CloudflareD1DriverPlugin/Info.plist +++ b/Plugins/CloudflareD1DriverPlugin/Info.plist @@ -5,6 +5,6 @@ TableProMinAppVersion 0.42.0 TableProPluginKitVersion - 33 + 34 diff --git a/Plugins/CloudflareR2SQLDriverPlugin/Info.plist b/Plugins/CloudflareR2SQLDriverPlugin/Info.plist index 9118d2c539..9718934655 100644 --- a/Plugins/CloudflareR2SQLDriverPlugin/Info.plist +++ b/Plugins/CloudflareR2SQLDriverPlugin/Info.plist @@ -3,7 +3,7 @@ TableProPluginKitVersion - 33 + 34 TableProProvidesDatabaseTypeIds Cloudflare R2 SQL diff --git a/Plugins/DamengDriverPlugin/Info.plist b/Plugins/DamengDriverPlugin/Info.plist index 5fa09c4a55..9491a9e2ff 100644 --- a/Plugins/DamengDriverPlugin/Info.plist +++ b/Plugins/DamengDriverPlugin/Info.plist @@ -3,7 +3,7 @@ TableProPluginKitVersion - 33 + 34 TableProProvidesDatabaseTypeIds Dameng diff --git a/Plugins/DuckDBDriverPlugin/Info.plist b/Plugins/DuckDBDriverPlugin/Info.plist index 7d62301f48..a018e00342 100644 --- a/Plugins/DuckDBDriverPlugin/Info.plist +++ b/Plugins/DuckDBDriverPlugin/Info.plist @@ -3,6 +3,6 @@ TableProPluginKitVersion - 33 + 34 diff --git a/Plugins/DynamoDBDriverPlugin/Info.plist b/Plugins/DynamoDBDriverPlugin/Info.plist index 786451d274..dc1215cdd8 100644 --- a/Plugins/DynamoDBDriverPlugin/Info.plist +++ b/Plugins/DynamoDBDriverPlugin/Info.plist @@ -5,6 +5,6 @@ TableProMinAppVersion 0.42.0 TableProPluginKitVersion - 33 + 34 diff --git a/Plugins/ElasticsearchDriverPlugin/Info.plist b/Plugins/ElasticsearchDriverPlugin/Info.plist index 6a9b0aa7b4..dd7ec70c4b 100644 --- a/Plugins/ElasticsearchDriverPlugin/Info.plist +++ b/Plugins/ElasticsearchDriverPlugin/Info.plist @@ -5,6 +5,6 @@ TableProMinAppVersion 0.53.0 TableProPluginKitVersion - 33 + 34 diff --git a/Plugins/EtcdDriverPlugin/Info.plist b/Plugins/EtcdDriverPlugin/Info.plist index 786451d274..dc1215cdd8 100644 --- a/Plugins/EtcdDriverPlugin/Info.plist +++ b/Plugins/EtcdDriverPlugin/Info.plist @@ -5,6 +5,6 @@ TableProMinAppVersion 0.42.0 TableProPluginKitVersion - 33 + 34 diff --git a/Plugins/HTMLExportPlugin/Info.plist b/Plugins/HTMLExportPlugin/Info.plist index a1afdcf010..b84b0c5cc2 100644 --- a/Plugins/HTMLExportPlugin/Info.plist +++ b/Plugins/HTMLExportPlugin/Info.plist @@ -3,7 +3,7 @@ TableProPluginKitVersion - 33 + 34 TableProProvidesExportFormatIds html diff --git a/Plugins/HanaDriverPlugin/Info.plist b/Plugins/HanaDriverPlugin/Info.plist index e7b11b65ef..1529e627de 100644 --- a/Plugins/HanaDriverPlugin/Info.plist +++ b/Plugins/HanaDriverPlugin/Info.plist @@ -3,7 +3,7 @@ TableProPluginKitVersion - 33 + 34 TableProProvidesDatabaseTypeIds SAP HANA diff --git a/Plugins/JSONExportPlugin/Info.plist b/Plugins/JSONExportPlugin/Info.plist index b599526ad4..4c76c61d16 100644 --- a/Plugins/JSONExportPlugin/Info.plist +++ b/Plugins/JSONExportPlugin/Info.plist @@ -3,7 +3,7 @@ TableProPluginKitVersion - 33 + 34 TableProProvidesExportFormatIds json diff --git a/Plugins/JSONImportPlugin/Info.plist b/Plugins/JSONImportPlugin/Info.plist index 3ddbd678c3..aaf6396d02 100644 --- a/Plugins/JSONImportPlugin/Info.plist +++ b/Plugins/JSONImportPlugin/Info.plist @@ -3,7 +3,7 @@ TableProPluginKitVersion - 33 + 34 TableProProvidesImportFormatIds json diff --git a/Plugins/KafkaDriverPlugin/Info.plist b/Plugins/KafkaDriverPlugin/Info.plist index 0bc50d5de2..b4670c3003 100644 --- a/Plugins/KafkaDriverPlugin/Info.plist +++ b/Plugins/KafkaDriverPlugin/Info.plist @@ -3,7 +3,7 @@ TableProPluginKitVersion - 33 + 34 TableProProvidesDatabaseTypeIds Kafka diff --git a/Plugins/LibSQLDriverPlugin/Info.plist b/Plugins/LibSQLDriverPlugin/Info.plist index 786451d274..dc1215cdd8 100644 --- a/Plugins/LibSQLDriverPlugin/Info.plist +++ b/Plugins/LibSQLDriverPlugin/Info.plist @@ -5,6 +5,6 @@ TableProMinAppVersion 0.42.0 TableProPluginKitVersion - 33 + 34 diff --git a/Plugins/MQLExportPlugin/Info.plist b/Plugins/MQLExportPlugin/Info.plist index cf8f012006..a691a12fc9 100644 --- a/Plugins/MQLExportPlugin/Info.plist +++ b/Plugins/MQLExportPlugin/Info.plist @@ -3,7 +3,7 @@ TableProPluginKitVersion - 33 + 34 TableProProvidesExportFormatIds mql diff --git a/Plugins/MSSQLDriverPlugin/Info.plist b/Plugins/MSSQLDriverPlugin/Info.plist index 7d62301f48..a018e00342 100644 --- a/Plugins/MSSQLDriverPlugin/Info.plist +++ b/Plugins/MSSQLDriverPlugin/Info.plist @@ -3,6 +3,6 @@ TableProPluginKitVersion - 33 + 34 diff --git a/Plugins/MarkdownExportPlugin/Info.plist b/Plugins/MarkdownExportPlugin/Info.plist index 9cd8d90acf..d9bf7476e9 100644 --- a/Plugins/MarkdownExportPlugin/Info.plist +++ b/Plugins/MarkdownExportPlugin/Info.plist @@ -3,7 +3,7 @@ TableProPluginKitVersion - 33 + 34 TableProProvidesExportFormatIds md diff --git a/Plugins/MongoDBDriverPlugin/Info.plist b/Plugins/MongoDBDriverPlugin/Info.plist index 7d62301f48..a018e00342 100644 --- a/Plugins/MongoDBDriverPlugin/Info.plist +++ b/Plugins/MongoDBDriverPlugin/Info.plist @@ -3,6 +3,6 @@ TableProPluginKitVersion - 33 + 34 diff --git a/Plugins/MySQLDriverPlugin/Info.plist b/Plugins/MySQLDriverPlugin/Info.plist index 549edc162f..91285512aa 100644 --- a/Plugins/MySQLDriverPlugin/Info.plist +++ b/Plugins/MySQLDriverPlugin/Info.plist @@ -3,7 +3,7 @@ TableProPluginKitVersion - 33 + 34 TableProProvidesDatabaseTypeIds MySQL diff --git a/Plugins/OracleDriverPlugin/Info.plist b/Plugins/OracleDriverPlugin/Info.plist index 7d62301f48..a018e00342 100644 --- a/Plugins/OracleDriverPlugin/Info.plist +++ b/Plugins/OracleDriverPlugin/Info.plist @@ -3,6 +3,6 @@ TableProPluginKitVersion - 33 + 34 diff --git a/Plugins/ParquetExportPlugin/Info.plist b/Plugins/ParquetExportPlugin/Info.plist index f3a58772f6..ca0902d9e0 100644 --- a/Plugins/ParquetExportPlugin/Info.plist +++ b/Plugins/ParquetExportPlugin/Info.plist @@ -3,7 +3,7 @@ TableProPluginKitVersion - 33 + 34 TableProProvidesExportFormatIds parquet diff --git a/Plugins/PostgreSQLDriverPlugin/Info.plist b/Plugins/PostgreSQLDriverPlugin/Info.plist index da9a028021..f8d121bc94 100644 --- a/Plugins/PostgreSQLDriverPlugin/Info.plist +++ b/Plugins/PostgreSQLDriverPlugin/Info.plist @@ -3,7 +3,7 @@ TableProPluginKitVersion - 33 + 34 TableProProvidesDatabaseTypeIds PostgreSQL diff --git a/Plugins/RedisDriverPlugin/Info.plist b/Plugins/RedisDriverPlugin/Info.plist index bc2ed99841..07cb86ad14 100644 --- a/Plugins/RedisDriverPlugin/Info.plist +++ b/Plugins/RedisDriverPlugin/Info.plist @@ -21,7 +21,7 @@ NSPrincipalClass $(PRODUCT_MODULE_NAME).RedisPlugin TableProPluginKitVersion - 33 + 34 TableProProvidesDatabaseTypeIds Redis diff --git a/Plugins/SQLExportPlugin/Info.plist b/Plugins/SQLExportPlugin/Info.plist index 2516ed6e7f..1029bf8322 100644 --- a/Plugins/SQLExportPlugin/Info.plist +++ b/Plugins/SQLExportPlugin/Info.plist @@ -3,7 +3,7 @@ TableProPluginKitVersion - 33 + 34 TableProProvidesExportFormatIds sql diff --git a/Plugins/SQLImportPlugin/Info.plist b/Plugins/SQLImportPlugin/Info.plist index 11c016144a..26bc3c95a6 100644 --- a/Plugins/SQLImportPlugin/Info.plist +++ b/Plugins/SQLImportPlugin/Info.plist @@ -3,7 +3,7 @@ TableProPluginKitVersion - 33 + 34 TableProProvidesImportFormatIds sql diff --git a/Plugins/SQLiteDriverPlugin/Info.plist b/Plugins/SQLiteDriverPlugin/Info.plist index 8b90c8d990..c6c1d9ef2f 100644 --- a/Plugins/SQLiteDriverPlugin/Info.plist +++ b/Plugins/SQLiteDriverPlugin/Info.plist @@ -3,7 +3,7 @@ TableProPluginKitVersion - 33 + 34 TableProProvidesDatabaseTypeIds SQLite diff --git a/Plugins/SnowflakeDriverPlugin/Info.plist b/Plugins/SnowflakeDriverPlugin/Info.plist index a830c0398c..a36d116fa0 100644 --- a/Plugins/SnowflakeDriverPlugin/Info.plist +++ b/Plugins/SnowflakeDriverPlugin/Info.plist @@ -5,6 +5,6 @@ TableProMinAppVersion 0.48.0 TableProPluginKitVersion - 33 + 34 diff --git a/Plugins/SpannerDriverPlugin/Info.plist b/Plugins/SpannerDriverPlugin/Info.plist index 018ec85028..90947b2004 100644 --- a/Plugins/SpannerDriverPlugin/Info.plist +++ b/Plugins/SpannerDriverPlugin/Info.plist @@ -5,7 +5,7 @@ TableProMinAppVersion 0.42.0 TableProPluginKitVersion - 33 + 34 TableProProvidesDatabaseTypeIds Spanner diff --git a/Plugins/SurrealDBDriverPlugin/Info.plist b/Plugins/SurrealDBDriverPlugin/Info.plist index cc4a1d9966..dcafba0519 100644 --- a/Plugins/SurrealDBDriverPlugin/Info.plist +++ b/Plugins/SurrealDBDriverPlugin/Info.plist @@ -3,7 +3,7 @@ TableProPluginKitVersion - 33 + 34 TableProProvidesDatabaseTypeIds SurrealDB diff --git a/Plugins/TableProPluginKit/ImportFieldNaming.swift b/Plugins/TableProPluginKit/ImportFieldNaming.swift new file mode 100644 index 0000000000..472ffac869 --- /dev/null +++ b/Plugins/TableProPluginKit/ImportFieldNaming.swift @@ -0,0 +1,33 @@ +// +// ImportFieldNaming.swift +// TableProPluginKit +// + +import Foundation + +/// Headers the file spells out keep their names first; blanks and repeats are named afterwards, so a +/// placeholder can never take a name a later header spells literally. +public enum ImportFieldNaming { + public static func uniqueNames(for header: [String?], placeholder: (Int) -> String) -> [String] { + let trimmed = header.map { ($0 ?? "").trimmingCharacters(in: .whitespacesAndNewlines) } + var used = Set() + var names = [String?](repeating: nil, count: trimmed.count) + + for (index, name) in trimmed.enumerated() where !name.isEmpty && used.insert(name).inserted { + names[index] = name + } + + for index in names.indices where names[index] == nil { + let base = trimmed[index].isEmpty ? placeholder(index) : trimmed[index] + var unique = base + var suffix = 2 + while !used.insert(unique).inserted { + unique = "\(base) \(suffix)" + suffix += 1 + } + names[index] = unique + } + + return names.map { $0 ?? "" } + } +} diff --git a/Plugins/TableProPluginKit/ImportFormatPlugin.swift b/Plugins/TableProPluginKit/ImportFormatPlugin.swift index 035857aa5f..7d7ebe9cbb 100644 --- a/Plugins/TableProPluginKit/ImportFormatPlugin.swift +++ b/Plugins/TableProPluginKit/ImportFormatPlugin.swift @@ -14,6 +14,7 @@ public protocol ImportFormatPlugin: TableProPlugin, Sendable { static var supportedDatabaseTypeIds: [String] { get } static var excludedDatabaseTypeIds: [String] { get } static var requiresTargetTable: Bool { get } + static var sourceFieldsFollowFileOrder: Bool { get } var fieldDetectionSignature: String { get } @@ -31,6 +32,7 @@ public extension ImportFormatPlugin { static var supportedDatabaseTypeIds: [String] { [] } static var excludedDatabaseTypeIds: [String] { [] } static var requiresTargetTable: Bool { false } + static var sourceFieldsFollowFileOrder: Bool { false } var fieldDetectionSignature: String { "" } diff --git a/Plugins/TeradataDriverPlugin/Info.plist b/Plugins/TeradataDriverPlugin/Info.plist index 7d62301f48..a018e00342 100644 --- a/Plugins/TeradataDriverPlugin/Info.plist +++ b/Plugins/TeradataDriverPlugin/Info.plist @@ -3,6 +3,6 @@ TableProPluginKitVersion - 33 + 34 diff --git a/Plugins/TrinoDriverPlugin/Info.plist b/Plugins/TrinoDriverPlugin/Info.plist index 7d62301f48..a018e00342 100644 --- a/Plugins/TrinoDriverPlugin/Info.plist +++ b/Plugins/TrinoDriverPlugin/Info.plist @@ -3,6 +3,6 @@ TableProPluginKitVersion - 33 + 34 diff --git a/Plugins/TypesenseDriverPlugin/Info.plist b/Plugins/TypesenseDriverPlugin/Info.plist index 815f9b9297..f1363fa04f 100644 --- a/Plugins/TypesenseDriverPlugin/Info.plist +++ b/Plugins/TypesenseDriverPlugin/Info.plist @@ -5,7 +5,7 @@ TableProMinAppVersion 0.73.0 TableProPluginKitVersion - 33 + 34 TableProProvidesDatabaseTypeIds Typesense diff --git a/Plugins/WeaviateDriverPlugin/Info.plist b/Plugins/WeaviateDriverPlugin/Info.plist index d7e105cad4..b5a0934955 100644 --- a/Plugins/WeaviateDriverPlugin/Info.plist +++ b/Plugins/WeaviateDriverPlugin/Info.plist @@ -5,7 +5,7 @@ TableProMinAppVersion 0.73.0 TableProPluginKitVersion - 33 + 34 TableProProvidesDatabaseTypeIds Weaviate diff --git a/Plugins/XLSXExportPlugin/Info.plist b/Plugins/XLSXExportPlugin/Info.plist index 42a6464d2e..a5f2acde68 100644 --- a/Plugins/XLSXExportPlugin/Info.plist +++ b/Plugins/XLSXExportPlugin/Info.plist @@ -3,7 +3,7 @@ TableProPluginKitVersion - 33 + 34 TableProProvidesExportFormatIds xlsx diff --git a/Plugins/XLSXImportPlugin/Info.plist b/Plugins/XLSXImportPlugin/Info.plist index 96e97930fc..d1ea7c12fe 100644 --- a/Plugins/XLSXImportPlugin/Info.plist +++ b/Plugins/XLSXImportPlugin/Info.plist @@ -3,7 +3,7 @@ TableProPluginKitVersion - 33 + 34 TableProProvidesImportFormatIds xlsx diff --git a/Plugins/XLSXImportPlugin/XLSXImportPlugin.swift b/Plugins/XLSXImportPlugin/XLSXImportPlugin.swift index 65465898da..510182d0ac 100644 --- a/Plugins/XLSXImportPlugin/XLSXImportPlugin.swift +++ b/Plugins/XLSXImportPlugin/XLSXImportPlugin.swift @@ -19,6 +19,7 @@ final class XLSXImportPlugin: ObservableObject, ImportFormatPlugin, SettablePlug static let acceptedFileExtensions = ["xlsx"] static let iconName = "tablecells" static let requiresTargetTable = true + static let sourceFieldsFollowFileOrder = true typealias Settings = XLSXImportOptions static let settingsStorageId = "xlsximport" @@ -154,17 +155,13 @@ final class XLSXImportPlugin: ObservableObject, ImportFormatPlugin, SettablePlug guard settings.hasHeaderRow else { let width = rows.first?.count ?? 0 return Sheet( - header: (0 ..< width).map { "column\($0 + 1)" }, + header: (0 ..< width).map(XLSXSheetParser.placeholderName), rows: rows ) } guard !rows.isEmpty else { return Sheet(header: nil, rows: []) } let headerRow = rows.removeFirst() - let header = headerRow.enumerated().map { index, value -> String in - let name = value?.trimmingCharacters(in: .whitespacesAndNewlines) ?? "" - return name.isEmpty ? "column\(index + 1)" : name - } - return Sheet(header: header, rows: rows) + return Sheet(header: XLSXSheetParser.headerNames(from: headerRow), rows: rows) } /// Every value arrives as text, so the type is inferred from what the values look like. The diff --git a/Plugins/XLSXImportPlugin/XLSXSheetParser.swift b/Plugins/XLSXImportPlugin/XLSXSheetParser.swift index d433d4c6b1..52e44af92b 100644 --- a/Plugins/XLSXImportPlugin/XLSXSheetParser.swift +++ b/Plugins/XLSXImportPlugin/XLSXSheetParser.swift @@ -25,6 +25,14 @@ enum XLSXSheetParser { } /// A cell's column index, from the letters in its reference. `A` is 0, `Z` 25, `AA` 26. + static func headerNames(from row: [String?]) -> [String] { + ImportFieldNaming.uniqueNames(for: row, placeholder: placeholderName) + } + + static func placeholderName(_ index: Int) -> String { + "column\(index + 1)" + } + static func columnIndex(fromReference reference: String) -> Int? { var index = 0 var sawLetter = false diff --git a/Plugins/XMLExportPlugin/Info.plist b/Plugins/XMLExportPlugin/Info.plist index c4a2862a55..4f3aaad64f 100644 --- a/Plugins/XMLExportPlugin/Info.plist +++ b/Plugins/XMLExportPlugin/Info.plist @@ -3,7 +3,7 @@ TableProPluginKitVersion - 33 + 34 TableProProvidesExportFormatIds xml diff --git a/TablePro/Core/Plugins/ImportDataSinkAdapter.swift b/TablePro/Core/Plugins/ImportDataSinkAdapter.swift index 1626b7b891..4b3613ab4b 100644 --- a/TablePro/Core/Plugins/ImportDataSinkAdapter.swift +++ b/TablePro/Core/Plugins/ImportDataSinkAdapter.swift @@ -15,7 +15,10 @@ final class ImportDataSinkAdapter: PluginImportDataSink, @unchecked Sendable { private let driver: DatabaseDriver private let databaseType: DatabaseType private let grammar: SQLLexicalGrammar - private let columnMapping: [String: String] + private let exactMapping: [String: String] + private let foldedMapping: [String: String] + private let sourceFields: Set + private let sourceFieldsPerFoldedName: [String: Int] private let rowGenerator: SQLStatementGenerator? /// Asked before every statement this sink sends, because one `insertRows` call is no longer one @@ -38,6 +41,7 @@ final class ImportDataSinkAdapter: PluginImportDataSink, @unchecked Sendable { databaseType: DatabaseType, targetTable: String? = nil, columnMapping: [String: String] = [:], + sourceFields: Set = [], isCancelled: @escaping @Sendable () -> Bool = { false } ) { self.isCancelled = isCancelled @@ -46,10 +50,11 @@ final class ImportDataSinkAdapter: PluginImportDataSink, @unchecked Sendable { self.grammar = databaseType.lexicalGrammar self.databaseTypeId = databaseType.rawValue self.targetTable = targetTable - self.columnMapping = Dictionary( - columnMapping.map { ($0.key.lowercased(), $0.value) }, - uniquingKeysWith: { _, last in last } - ) + self.exactMapping = columnMapping + self.sourceFields = sourceFields + self.sourceFieldsPerFoldedName = Dictionary(sourceFields.map { ($0.lowercased(), 1) }, uniquingKeysWith: +) + self.foldedMapping = Dictionary(grouping: columnMapping, by: { $0.key.lowercased() }) + .compactMapValues { $0.count == 1 ? $0.first?.value : nil } if let targetTable { self.rowGenerator = try? SQLStatementGenerator( tableName: targetTable, @@ -98,14 +103,7 @@ final class ImportDataSinkAdapter: PluginImportDataSink, @unchecked Sendable { throw PluginImportError.importFailed("Could not resolve SQL dialect for \(targetTable)") } - var columns: [String] = [] - var bindValues: [PluginCellValue] = [] - for (field, value) in values { - guard let column = columnMapping[field.lowercased()] else { continue } - columns.append(column) - bindValues.append(value) - } - + let (columns, bindValues) = mappedColumnsAndValues(values) guard !columns.isEmpty else { guard values.isEmpty else { throw PluginImportError.importFailed(Self.unmappedRowMessage) @@ -201,11 +199,23 @@ final class ImportDataSinkAdapter: PluginImportDataSink, @unchecked Sendable { String(localized: "No values in this row matched the column mapping") } - private func mappedColumnsAndValues(_ values: [String: PluginCellValue]) -> ([String], [PluginCellValue]) { + /// Exact names first. A case-insensitive match is only a fallback for a field the sheet never + /// listed, such as a JSON key respelled past the sample, and only when that spelling points at + /// one listed field, one mapping key and one field of the row, none of them already matched. + func mappedColumnsAndValues(_ values: [String: PluginCellValue]) -> ([String], [PluginCellValue]) { var pairs: [(column: String, value: PluginCellValue)] = [] + var matchedNames = Set() for (field, value) in values { - guard let column = columnMapping[field.lowercased()] else { continue } + guard let column = exactMapping[field] else { continue } pairs.append((column, value)) + matchedNames.insert(field.lowercased()) + } + let unmatched = values.filter { exactMapping[$0.key] == nil && !sourceFields.contains($0.key) } + for (folded, group) in Dictionary(grouping: unmatched, by: { $0.key.lowercased() }) { + guard group.count == 1, let entry = group.first, !matchedNames.contains(folded), + sourceFieldsPerFoldedName[folded, default: 0] <= 1, + let column = foldedMapping[folded] else { continue } + pairs.append((column, entry.value)) } pairs.sort { $0.column < $1.column } return (pairs.map(\.column), pairs.map(\.value)) diff --git a/TablePro/Core/Plugins/PluginManager.swift b/TablePro/Core/Plugins/PluginManager.swift index bd0c0c707e..e7d566b981 100644 --- a/TablePro/Core/Plugins/PluginManager.swift +++ b/TablePro/Core/Plugins/PluginManager.swift @@ -109,7 +109,7 @@ final class PluginManager: ObservableObject { /// learned about a table another connection changed. The defaults approve every save, find /// every save finished and keep nothing, so an already-built plugin keeps loading and saves as /// before. - nonisolated static let currentPluginKitVersion = 33 + nonisolated static let currentPluginKitVersion = 34 /// Still 19, so every plugin already published for the previous release keeps loading. nonisolated static let minimumCompatiblePluginKitVersion = 19 diff --git a/TablePro/Core/Services/Export/ImportColumnMatcher.swift b/TablePro/Core/Services/Export/ImportColumnMatcher.swift new file mode 100644 index 0000000000..7c9ca833a5 --- /dev/null +++ b/TablePro/Core/Services/Export/ImportColumnMatcher.swift @@ -0,0 +1,193 @@ +// +// ImportColumnMatcher.swift +// TablePro +// + +import Foundation + +/// A row can be unticked while still naming a column, so the two are kept apart. +internal struct ImportFieldChoice: Equatable, Sendable { + var include: Bool + var column: String? + + static let skipped = ImportFieldChoice(include: false, column: nil) + + var mappedColumn: String? { + include ? column : nil + } +} + +/// Only departures from the name match are stored, so a column the table gains later is still found +/// by name instead of staying skipped. +internal enum ImportMappingOverride: Codable, Equatable, Sendable { + case column(String) + case skip + + init(mappedColumn: String?) { + guard let mappedColumn else { + self = .skip + return + } + self = .column(mappedColumn) + } + + var choice: ImportFieldChoice { + switch self { + case .column(let column): + return ImportFieldChoice(include: true, column: column) + case .skip: + return .skipped + } + } +} + +/// Unlike `TableColumnMatcher`, exact matches come before case-insensitive ones and a column goes to +/// one field at most, because the sheet refuses two fields mapped to one column. +internal enum ImportColumnMatcher { + static func byName(fields: [String], columns: [String]) -> [ImportFieldChoice] { + resolve(fields: fields, columns: columns, preferring: []) + } + + static func byPosition(fields: [String], columns: [String]) -> [ImportFieldChoice] { + fields.indices.map { index in + guard index < columns.count else { return .skipped } + return ImportFieldChoice(include: true, column: columns[index]) + } + } + + /// Each tier is settled for every field before the next starts, so a higher tier wins a column + /// whatever the field order. A choice that can no longer be honoured falls through to the name match. + static func resolve( + fields: [String], + columns: [String], + preferring tiers: [[String: ImportFieldChoice]] + ) -> [ImportFieldChoice] { + var claimed = Set() + var choices = [ImportFieldChoice?](repeating: nil, count: fields.count) + + for tier in tiers { + for (index, field) in fields.enumerated() where choices[index] == nil { + guard let preferred = choice(for: field, among: fields, in: tier) else { continue } + guard let wanted = preferred.column else { + choices[index] = preferred + continue + } + let available = matchingColumn(wanted, in: columns, excluding: preferred.include ? claimed : []) + guard let column = available else { + if !preferred.include { + choices[index] = .skipped + } + continue + } + if preferred.include { + claimed.insert(column) + } + choices[index] = ImportFieldChoice(include: preferred.include, column: column) + } + } + + for (index, field) in fields.enumerated() where choices[index] == nil { + guard columns.contains(field), !claimed.contains(field) else { continue } + claimed.insert(field) + choices[index] = ImportFieldChoice(include: true, column: field) + } + + for (index, field) in fields.enumerated() where choices[index] == nil { + guard let column = matchingColumn(field, in: columns, excluding: claimed) else { + choices[index] = .skipped + continue + } + claimed.insert(column) + choices[index] = ImportFieldChoice(include: true, column: column) + } + + return choices.map { $0 ?? .skipped } + } + + static func applying( + _ overrides: [String: ImportMappingOverride], + fields: [String], + columns: [String] + ) -> [ImportFieldChoice] { + resolve(fields: fields, columns: columns, preferring: [overrides.mapValues(\.choice)]) + } + + /// The fields whose current choice is a remembered override the name match would not have made. + static func restoredFields( + _ overrides: [String: ImportMappingOverride], + fields: [String], + columns: [String], + choices: [ImportFieldChoice] + ) -> Set { + let automatic = byName(fields: fields, columns: columns) + var restored = Set() + for (index, field) in fields.enumerated() where index < choices.count { + let current = choices[index].mappedColumn + guard let override = choice(for: field, among: fields, in: overrides), + current.map(normalized) == override.choice.mappedColumn.map(normalized), + current != automatic[index].mappedColumn else { continue } + restored.insert(field) + } + return restored + } + + static func overrides( + fields: [String], + columns: [String], + choices: [ImportFieldChoice] + ) -> [String: ImportMappingOverride] { + let automatic = byName(fields: fields, columns: columns) + var overrides: [String: ImportMappingOverride] = [:] + for (index, field) in fields.enumerated() where index < choices.count { + let mapped = choices[index].mappedColumn + guard mapped != automatic[index].mappedColumn else { continue } + overrides[field] = ImportMappingOverride(mappedColumn: mapped) + } + return overrides + } + + /// Entries for fields this file does not hold are kept, so several layouts into one table all come + /// back. A header respelled only by case takes over its one differently-cased entry. + static func merging( + _ overrides: [String: ImportMappingOverride], + forFields fields: [String], + into saved: [String: ImportMappingOverride] + ) -> [String: ImportMappingOverride] { + let exactKeys = Set(fields).intersection(saved.keys) + var merged = saved.filter { !exactKeys.contains($0.key) } + for (folded, spellings) in Dictionary(grouping: fields, by: normalized) { + guard spellings.count == 1, let field = spellings.first, !exactKeys.contains(field) else { continue } + let variants = merged.keys.filter { normalized($0) == folded } + guard variants.count == 1, let variant = variants.first else { continue } + merged.removeValue(forKey: variant) + } + for (field, override) in overrides { + merged[field] = override + } + return merged + } + + /// A differently-cased entry is used only when that is unambiguous on both sides: `Email` and `email` + /// can be two fields of one file. + private static func choice(for field: String, among fields: [String], in prior: [String: Value]) -> Value? { + if let exact = prior[field] { return exact } + let folded = normalized(field) + guard fields.filter({ normalized($0) == folded }).count == 1 else { return nil } + let variants = prior.keys.filter { normalized($0) == folded } + guard variants.count == 1, let variant = variants.first else { return nil } + return prior[variant] + } + + private static func matchingColumn(_ name: String, in columns: [String], excluding claimed: Set) -> String? { + if columns.contains(name), !claimed.contains(name) { + return name + } + return columns.first { column in + !claimed.contains(column) && column.caseInsensitiveCompare(name) == .orderedSame + } + } + + private static func normalized(_ name: String) -> String { + name.lowercased() + } +} diff --git a/TablePro/Core/Services/Export/ImportService.swift b/TablePro/Core/Services/Export/ImportService.swift index bf90442a5a..22bc1c8e24 100644 --- a/TablePro/Core/Services/Export/ImportService.swift +++ b/TablePro/Core/Services/Export/ImportService.swift @@ -57,7 +57,8 @@ final class ImportService: ObservableObject { ownsDecompressedFile: Bool = false, knownStatementCount: Int? = nil, targetTable: String? = nil, - columnMapping: [String: String] = [:] + columnMapping: [String: String] = [:], + sourceFields: Set = [] ) async throws -> PluginImportResult { guard let plugin = PluginManager.shared.importPlugin(forFormat: formatId) else { throw PluginImportError.importFailed("Import format '\(formatId)' not found") @@ -139,7 +140,8 @@ final class ImportService: ObservableObject { source: source, progress: progress, targetTable: targetTable, - columnMapping: columnMapping + columnMapping: columnMapping, + sourceFields: sourceFields ) } } catch { @@ -209,13 +211,15 @@ final class ImportService: ObservableObject { source: any PluginImportSource, progress: PluginImportProgress, targetTable: String?, - columnMapping: [String: String] + columnMapping: [String: String], + sourceFields: Set ) async throws -> PluginImportResult { let sink = ImportDataSinkAdapter( driver: driver, databaseType: connection.type, targetTable: targetTable, columnMapping: columnMapping, + sourceFields: sourceFields, isCancelled: { progress.isCancelled } ) return try await plugin.performImport(source: source, sink: sink, progress: progress) diff --git a/TablePro/Core/Services/Export/TableColumnMatcher.swift b/TablePro/Core/Services/Export/TableColumnMatcher.swift index 642ae9daf6..d196c167cf 100644 --- a/TablePro/Core/Services/Export/TableColumnMatcher.swift +++ b/TablePro/Core/Services/Export/TableColumnMatcher.swift @@ -25,58 +25,90 @@ enum TableColumnMatcher { let unmatchedDestination: [String] var isEmpty: Bool { mapping.isEmpty } - } - /// Case-insensitive, because engines disagree about identifier folding and a transfer from a - /// case-folding engine to a case-preserving one would otherwise match nothing. - static func match(source: [String], destination: [String]) -> Match { - var destinationByFolded: [String: String] = [:] - for column in destination { - destinationByFolded[column.lowercased()] = column + /// Destination columns more than one source column is mapped to. The INSERT would name + /// each of them twice, which every engine refuses, so the transfer cannot run until the + /// user moves one. + var contestedDestinations: [String] { + TableColumnMatcher.contestedDestinations(in: mapping) } + } + /// Exact spelling first, then case-insensitive, because engines disagree about identifier + /// folding and a transfer from a case-folding engine to a case-preserving one would otherwise + /// match nothing. A destination column goes to one source column at most, so `Name` and `name` + /// on the source never both land on a lone `name`. + static func match(source: [String], destination: [String]) -> Match { var mapping: [String: String] = [:] - var unmatchedSource: [String] = [] - var claimed: Set = [] - for column in source { - guard let target = destinationByFolded[column.lowercased()] else { - unmatchedSource.append(column) - continue - } + for (column, target) in zip(source, pair(source, with: destination)) { + guard let target else { continue } mapping[column] = target - claimed.insert(target) } - return Match( - mapping: mapping, - unmatchedSource: unmatchedSource, - unmatchedDestination: destination.filter { !claimed.contains($0) } - ) + return resolved(mapping, source: source, destination: destination) } - /// Applies the user's overrides over an automatic match. An override to nil excludes the + /// The automatic match with the user's overrides laid over it. An override to nil excludes the /// column, which is how a source column with no destination is deliberately dropped rather - /// than failing the transfer. - static func applying( - overrides: [String: String?], - to match: Match, - destination: [String] + /// than failing the transfer. An override may point at a column another source column + /// already holds; that is kept and reported through `contestedDestinations`, not resolved + /// by quietly unmapping the other one. + static func match( + source: [String], + destination: [String], + overrides: [String: String?] ) -> Match { - var mapping = match.mapping - var unmatchedSource = Set(match.unmatchedSource) + var mapping = match(source: source, destination: destination).mapping for (sourceColumn, target) in overrides { guard let target, destination.contains(target) else { mapping.removeValue(forKey: sourceColumn) - unmatchedSource.insert(sourceColumn) continue } mapping[sourceColumn] = target - unmatchedSource.remove(sourceColumn) } + return resolved(mapping, source: source, destination: destination) + } + + /// Both sides' leftovers in their own table's column order, whichever way the mapping was made. + private static func resolved(_ mapping: [String: String], source: [String], destination: [String]) -> Match { let claimed = Set(mapping.values) return Match( mapping: mapping, - unmatchedSource: unmatchedSource.sorted(), + unmatchedSource: source.filter { mapping[$0] == nil }, unmatchedDestination: destination.filter { !claimed.contains($0) } ) } + + /// Destination columns named by more than one entry of `mapping`, sorted. + static func contestedDestinations(in mapping: [String: String]) -> [String] { + var sourceCount: [String: Int] = [:] + for target in mapping.values { + sourceCount[target, default: 0] += 1 + } + return sourceCount.filter { $0.value > 1 }.keys.sorted() + } + + /// Pairs each name with a candidate of the same spelling, then with a candidate left over that + /// differs only by case, earlier names first. A candidate is paired with one name at most. + private static func pair(_ names: [String], with candidates: [String]) -> [String?] { + var pairs = [String?](repeating: nil, count: names.count) + var claimed = Set() + let spelled = Set(candidates) + for (index, name) in names.enumerated() { + guard spelled.contains(name), !claimed.contains(name) else { continue } + pairs[index] = name + claimed.insert(name) + } + + var unclaimedByFolded: [String: [String]] = [:] + for candidate in candidates where !claimed.contains(candidate) { + unclaimedByFolded[candidate.lowercased(), default: []].append(candidate) + } + for (index, name) in names.enumerated() where pairs[index] == nil { + let folded = name.lowercased() + guard var remaining = unclaimedByFolded[folded], !remaining.isEmpty else { continue } + pairs[index] = remaining.removeFirst() + unclaimedByFolded[folded] = remaining + } + return pairs + } } diff --git a/TablePro/Core/Services/Export/TableTransferService.swift b/TablePro/Core/Services/Export/TableTransferService.swift index 993211a099..0177aeb37a 100644 --- a/TablePro/Core/Services/Export/TableTransferService.swift +++ b/TablePro/Core/Services/Export/TableTransferService.swift @@ -14,6 +14,7 @@ enum TableTransferError: LocalizedError { case sameConnectionAndContainer case targetMissing(table: String) case noMatchingColumns(table: String) + case contestedDestination(table: String, columns: [String]) case transferFailed(String) var errorDescription: String? { @@ -30,6 +31,11 @@ enum TableTransferError: LocalizedError { return String( format: String(localized: "No column of %@ matches a column on the destination table."), table) + case .contestedDestination(let table, let columns): + return String( + format: String(localized: "%1$@: more than one source column is mapped to %2$@."), + table, + columns.joined(separator: ", ")) case .transferFailed(let message): return String(format: String(localized: "Transfer failed: %@"), message) } @@ -137,6 +143,7 @@ final class TableTransferService: ObservableObject { ) async throws { let rowObjects = request.objects.filter { $0.kind.carriesRows } guard !rowObjects.isEmpty else { throw TableTransferError.noTablesSelected } + try Self.refuseContestedMappings(request.columnMapping, for: rowObjects) /// The flag is cleared on the way out, never on the way in. A Stop pressed while the sheet /// was still reading both sides' columns arrives before this line, and clearing it here @@ -161,6 +168,7 @@ final class TableTransferService: ObservableObject { databaseType: request.destinationType, targetTable: object.name, columnMapping: mapping, + sourceFields: Set(request.sourceColumns[object.name] ?? []), isCancelled: { [flag = cancellationFlag] in flag.isCancelled } ) try await transferOne(object: object, from: source, into: sink, request: request) @@ -168,6 +176,22 @@ final class TableTransferService: ObservableObject { state.warnings.append(contentsOf: source.cappedTableWarnings) } + /// Checked for every table before the first is written: the INSERT would name the contested + /// column twice, and the server refuses it only after "Delete existing rows first" has already + /// emptied the table, permanently when the table is not wrapped in a transaction. + nonisolated static func refuseContestedMappings( + _ mappings: [String: [String: String]], + for objects: [ExportObjectItem] + ) throws { + for object in objects { + guard let mapping = mappings[object.name] else { continue } + let contested = TableColumnMatcher.contestedDestinations(in: mapping) + guard contested.isEmpty else { + throw TableTransferError.contestedDestination(table: object.name, columns: contested) + } + } + } + /// The sink writes by column name and skips any field the mapping does not name, so an empty /// mapping writes nothing and reports every row as unmapped. A caller that supplies no mapping /// gets one matched by name, and a table whose columns match nothing is refused by name here diff --git a/TablePro/Core/Storage/ImportColumnMappingStore.swift b/TablePro/Core/Storage/ImportColumnMappingStore.swift new file mode 100644 index 0000000000..744b0d19be --- /dev/null +++ b/TablePro/Core/Storage/ImportColumnMappingStore.swift @@ -0,0 +1,78 @@ +// +// ImportColumnMappingStore.swift +// TablePro +// + +import Foundation + +/// Keyed by the table rather than the file, because the same export is downloaded again under new +/// names. Device-local, so it needs no CloudKit record type. +@MainActor +internal final class ImportColumnMappingStore: TableScopedSettingsStore { + static let shared = ImportColumnMappingStore() + + private static let keyPrefix = PreferenceKeys.importColumnMappingPrefix + + private let store: KeyValueStore + + init(defaults: KeyValueStore = AppStorageEnvironment.shared.defaults) { + store = defaults + } + + func overrides(for scope: TableScope) -> [String: ImportMappingOverride] { + guard let data = store.dataValue(forKey: PreferenceKeys.importColumnMapping(scope).name), + let overrides = try? JSONDecoder().decode([String: ImportMappingOverride].self, from: data) else { + return [:] + } + return overrides + } + + func remember(_ overrides: [String: ImportMappingOverride], forFields fields: [String], in scope: TableScope) { + let merged = ImportColumnMatcher.merging(overrides, forFields: fields, into: self.overrides(for: scope)) + let key = PreferenceKeys.importColumnMapping(scope).name + guard !merged.isEmpty, let data = try? JSONEncoder().encode(merged) else { + store.setDataValue(nil, forKey: key) + return + } + store.setDataValue(data, forKey: key) + } + + func renameTable(from oldScope: TableScope, to newScope: TableScope) { + store.moveValue( + fromKey: PreferenceKeys.importColumnMapping(oldScope).name, + toKey: PreferenceKeys.importColumnMapping(newScope).name + ) + } + + func renameContainer( + connectionId: UUID, + fromDatabase: String, + fromSchema: String?, + toDatabase: String, + toSchema: String? + ) { + store.moveValues( + withPrefix: Self.keyPrefix + + TableScope.storagePrefix(connectionId: connectionId, database: fromDatabase, schema: fromSchema), + toPrefix: Self.keyPrefix + + TableScope.storagePrefix(connectionId: connectionId, database: toDatabase, schema: toSchema) + ) + } + + func dropTable(_ scope: TableScope) { + store.setDataValue(nil, forKey: PreferenceKeys.importColumnMapping(scope).name) + } + + func dropContainer(connectionId: UUID, database: String, schema: String?) { + store.removeValues( + withPrefix: Self.keyPrefix + + TableScope.storagePrefix(connectionId: connectionId, database: database, schema: schema) + ) + } + + func purgeConnections(_ connectionIds: Set, leavesTombstones: Bool) { + for connectionId in connectionIds { + store.removeValues(withPrefix: Self.keyPrefix + TableScope.storagePrefix(connectionId: connectionId)) + } + } +} diff --git a/TablePro/Core/Storage/Preferences/PreferenceKeys.swift b/TablePro/Core/Storage/Preferences/PreferenceKeys.swift index cb66ac3d37..7b1dd0cc17 100644 --- a/TablePro/Core/Storage/Preferences/PreferenceKeys.swift +++ b/TablePro/Core/Storage/Preferences/PreferenceKeys.swift @@ -42,6 +42,7 @@ enum PreferenceKeys { static let columnDisplayFormatsPrefix = "com.TablePro.columns.displayFormat." static let foreignKeyLabelColumnPrefix = "com.TablePro.foreignKey.labelColumn." + static let importColumnMappingPrefix = "com.TablePro.import.columnMapping." static let tableFoldersPrefix = "com.TablePro.tableFolders." static func columnDisplayFormats(_ scope: TableScope) -> DefaultsKey<[String: ValueDisplayFormat]> { @@ -56,6 +57,10 @@ enum PreferenceKeys { DefaultsKey(foreignKeyLabelColumnPrefix + scope.storageComponent) } + static func importColumnMapping(_ scope: TableScope) -> DefaultsKey<[String: ImportMappingOverride]> { + DefaultsKey(importColumnMappingPrefix + scope.storageComponent) + } + static func tableFolders(connectionId: UUID) -> DefaultsKey { DefaultsKey(tableFoldersPrefix + connectionId.uuidString) } diff --git a/TablePro/Core/Storage/TableScopedSettingsStore.swift b/TablePro/Core/Storage/TableScopedSettingsStore.swift index 7363b85c0f..9e8d12ac0d 100644 --- a/TablePro/Core/Storage/TableScopedSettingsStore.swift +++ b/TablePro/Core/Storage/TableScopedSettingsStore.swift @@ -41,6 +41,7 @@ internal enum TableScopedSettingsRegistry { HighlightRuleStorage.shared, ValueDisplayFormatStorage.shared, ForeignKeyLabelColumnStore.shared, + ImportColumnMappingStore.shared, TableFolderStorage.shared ] } diff --git a/TablePro/Resources/Localizable.xcstrings b/TablePro/Resources/Localizable.xcstrings index 0535a15bcc..7cfa751493 100644 --- a/TablePro/Resources/Localizable.xcstrings +++ b/TablePro/Resources/Localizable.xcstrings @@ -349,6 +349,9 @@ } } } + }, + "Use Saved Mapping" : { + }, "Use Selection for Find" : { "localizations" : { @@ -3507,6 +3510,9 @@ }, "%@ loses its default" : { + }, + "%@ mapped more than once" : { + }, "%@ matches" : { @@ -5967,6 +5973,9 @@ }, "%1$@: %2$@ had no column of that name on the destination." : { + }, + "%1$@: more than one source column is mapped to %2$@." : { + }, "%1$@: only the first %2$lld rows were read, the most this database returns from one query." : { @@ -19887,6 +19896,9 @@ }, "Another profile already has this name." : { + }, + "Another source column is mapped to the same destination column." : { + }, "ANTHROPIC_API_KEY and ANTHROPIC_AUTH_TOKEN are removed from the tool's environment, so replies always draw on the subscription." : { "localizations" : { @@ -57274,6 +57286,9 @@ }, "Each database is written to its own file." : { + }, + "Each destination column can be mapped from only one source column." : { + }, "Each value is split at every match. Rows with fewer pieces get empty cells." : { @@ -90970,6 +90985,9 @@ } } } + }, + "Match by Position" : { + }, "Match case exactly" : { "localizations" : { @@ -91004,6 +91022,9 @@ } } } + }, + "Match Columns" : { + }, "Match System" : { "comment" : "Text for a sidebar row size preference that matches the system's sidebar row size.", @@ -128688,6 +128709,9 @@ } } } + }, + "Restored the mapping saved for %@." : { + }, "Restored “%@” from %@" : { "localizations" : { diff --git a/TablePro/Views/Export/TableTransferMappingEditor.swift b/TablePro/Views/Export/TableTransferMappingEditor.swift index cabf89e64b..8b4ae7898c 100644 --- a/TablePro/Views/Export/TableTransferMappingEditor.swift +++ b/TablePro/Views/Export/TableTransferMappingEditor.swift @@ -11,24 +11,45 @@ import SwiftUI /// the two schemas were renamed apart. Without this the only way to correct that would be to rename /// a column on one side. internal struct TableTransferMappingEditor: View { + internal static var contestedMappingMessage: String { + String(localized: "Each destination column can be mapped from only one source column.") + } + internal let tableName: String internal let sourceColumns: [String] internal let destinationColumns: [String] - @Binding internal var overrides: [String: String?] + internal let onChange: ([String: String?]) -> Void internal let dismiss: () -> Void - private var automatic: TableColumnMatcher.Match { - TableColumnMatcher.match(source: sourceColumns, destination: destinationColumns) + /// Held here rather than read back through the sheet: SwiftUI does not re-evaluate `.popover` + /// content when the presenting view re-renders, so a pick that only wrote the sheet's state + /// left this view drawing the mapping it opened with. + @State private var overrides: [String: String?] + + internal init( + tableName: String, + sourceColumns: [String], + destinationColumns: [String], + overrides: [String: String?], + onChange: @escaping ([String: String?]) -> Void, + dismiss: @escaping () -> Void + ) { + self.tableName = tableName + self.sourceColumns = sourceColumns + self.destinationColumns = destinationColumns + self.onChange = onChange + self.dismiss = dismiss + _overrides = State(initialValue: overrides) } private var resolved: TableColumnMatcher.Match { - overrides.isEmpty - ? automatic - : TableColumnMatcher.applying( - overrides: overrides, to: automatic, destination: destinationColumns) + TableColumnMatcher.match( + source: sourceColumns, destination: destinationColumns, overrides: overrides) } internal var body: some View { + let match = resolved + let contested = Set(match.contestedDestinations) VStack(alignment: .leading, spacing: 10) { Text(tableName) .font(.headline) @@ -42,36 +63,28 @@ internal struct TableTransferMappingEditor: View { ScrollView { VStack(alignment: .leading, spacing: 4) { ForEach(sourceColumns, id: \.self) { column in - HStack(spacing: 6) { - Text(column) - .lineLimit(1) - .truncationMode(.middle) - .frame(width: 130, alignment: .leading) - - Picker(String(format: String(localized: "Destination for %@"), column), - selection: binding(for: column)) { - Text("Skip").tag(String?.none) - ForEach(destinationColumns, id: \.self) { target in - Text(target).tag(String?.some(target)) - } - } - .labelsHidden() - .frame(width: 150) - } + row(for: column, isContested: match.mapping[column].map(contested.contains) ?? false) } } } .frame(height: 200) - if !resolved.unmatchedDestination.isEmpty { - Text(unmatchedDestinationLabel) + if !contested.isEmpty { + Text(Self.contestedMappingMessage) + .font(.caption) + .foregroundStyle(.red) + .fixedSize(horizontal: false, vertical: true) + } + + if !match.unmatchedDestination.isEmpty { + Text(unmatchedDestinationLabel(match.unmatchedDestination)) .font(.caption) .foregroundStyle(.secondary) .fixedSize(horizontal: false, vertical: true) } HStack { - Button("Match by Name") { overrides = [:] } + Button("Match by Name") { update([:]) } Spacer() Button("Done", action: dismiss) .keyboardShortcut(.defaultAction) @@ -81,19 +94,55 @@ internal struct TableTransferMappingEditor: View { .frame(width: 340) } + private func row(for column: String, isContested: Bool) -> some View { + HStack(spacing: 6) { + Text(column) + .lineLimit(1) + .truncationMode(.middle) + .frame(width: 130, alignment: .leading) + + Picker(String(format: String(localized: "Destination for %@"), column), + selection: binding(for: column)) { + Text("Skip").tag(String?.none) + ForEach(destinationColumns, id: \.self) { target in + Text(target).tag(String?.some(target)) + } + } + .labelsHidden() + .frame(width: 150) + + if isContested { + Image(systemName: "exclamationmark.triangle.fill") + .foregroundStyle(.red) + .help(String(localized: "Another source column is mapped to the same destination column.")) + .accessibilityLabel( + Text("Another source column is mapped to the same destination column.")) + } + } + } + /// A destination column nothing writes to takes its own default or null, which only fails when /// it is `NOT NULL` without one, so it is stated rather than blocked. - private var unmatchedDestinationLabel: String { + private func unmatchedDestinationLabel(_ columns: [String]) -> String { String( format: String(localized: "Not written: %@. Each takes its default or null."), - resolved.unmatchedDestination.joined(separator: ", ") + columns.joined(separator: ", ") ) } private func binding(for column: String) -> Binding { Binding( get: { resolved.mapping[column] }, - set: { overrides[column] = .some($0) } + set: { target in + var updated = overrides + updated[column] = .some(target) + update(updated) + } ) } + + private func update(_ updated: [String: String?]) { + overrides = updated + onChange(updated) + } } diff --git a/TablePro/Views/Export/TableTransferSheet.swift b/TablePro/Views/Export/TableTransferSheet.swift index 2c488a013c..652f17b728 100644 --- a/TablePro/Views/Export/TableTransferSheet.swift +++ b/TablePro/Views/Export/TableTransferSheet.swift @@ -55,7 +55,13 @@ struct TableTransferSheet: View { } private var canTransfer: Bool { - !isRunning && !selectedTables.isEmpty && destinationConnection != nil + !isRunning && !selectedTables.isEmpty && destinationConnection != nil && !hasContestedMapping + } + + /// Two source columns mapped to one destination column build an INSERT naming it twice, which + /// the server refuses only after "Delete existing rows first" has emptied the table. + private var hasContestedMapping: Bool { + selectedTables.contains { !resolvedMatch(for: $0.name).contestedDestinations.isEmpty } } var body: some View { @@ -177,6 +183,11 @@ struct TableTransferSheet: View { .foregroundStyle(.secondary) .lineLimit(1) .truncationMode(.middle) + } else if hasContestedMapping { + Text(TableTransferMappingEditor.contestedMappingMessage) + .font(.caption) + .foregroundStyle(.red) + .lineLimit(2) } } actions: { Button(isRunning ? String(localized: "Stop") : String(localized: "Cancel")) { @@ -213,6 +224,7 @@ struct TableTransferSheet: View { @ViewBuilder private func mappingSummary(for table: String) -> some View { let match = resolvedMatch(for: table) + let cannotTransfer = match.isEmpty || !match.contestedDestinations.isEmpty if isMatching, destinationColumns[table] == nil { ProgressView() .scaleEffect(0.5) @@ -228,7 +240,7 @@ struct TableTransferSheet: View { HStack(spacing: 3) { Text(mappingLabel(match)) .font(.caption) - .foregroundStyle(match.isEmpty ? Color.red : .secondary) + .foregroundStyle(cannotTransfer ? Color.red : .secondary) Image(systemName: "arrow.left.arrow.right") .font(.caption) } @@ -242,10 +254,8 @@ struct TableTransferSheet: View { tableName: table, sourceColumns: sourceColumns[table] ?? [], destinationColumns: destinationColumns[table] ?? [], - overrides: Binding( - get: { overrides[table] ?? [:] }, - set: { overrides[table] = $0 } - ), + overrides: overrides[table] ?? [:], + onChange: { overrides[table] = $0 }, dismiss: { inspectedTable = nil } ) } @@ -254,6 +264,12 @@ struct TableTransferSheet: View { private func mappingLabel(_ match: TableColumnMatcher.Match) -> String { guard !match.isEmpty else { return String(localized: "No columns match") } + let contested = match.contestedDestinations + guard contested.isEmpty else { + return String( + format: String(localized: "%@ mapped more than once"), + contested.joined(separator: ", ")) + } guard match.unmatchedSource.isEmpty else { return String( format: String(localized: "%1$lld mapped, %2$lld skipped"), @@ -264,12 +280,10 @@ struct TableTransferSheet: View { } private func resolvedMatch(for table: String) -> TableColumnMatcher.Match { - let destination = destinationColumns[table] ?? [] - let automatic = TableColumnMatcher.match( - source: sourceColumns[table] ?? [], destination: destination) - guard let tableOverrides = overrides[table], !tableOverrides.isEmpty else { return automatic } - return TableColumnMatcher.applying( - overrides: tableOverrides, to: automatic, destination: destination) + TableColumnMatcher.match( + source: sourceColumns[table] ?? [], + destination: destinationColumns[table] ?? [], + overrides: overrides[table] ?? [:]) } private func binding(for table: ExportObjectItem) -> Binding { diff --git a/TablePro/Views/Import/ImportDetectionSignatureObserver.swift b/TablePro/Views/Import/ImportDetectionSignatureObserver.swift new file mode 100644 index 0000000000..ac7f639766 --- /dev/null +++ b/TablePro/Views/Import/ImportDetectionSignatureObserver.swift @@ -0,0 +1,42 @@ +// +// ImportDetectionSignatureObserver.swift +// TablePro +// + +import Combine +import SwiftUI +import TableProPluginKit + +/// The options are edited in the plugin's own view, and the sheet around it observes only the plugin +/// manager, which publishes nothing when a plugin's settings change. +struct ImportDetectionSignatureObserver: View { + @ObservedObject var plugin: Plugin + let onChange: (String) -> Void + + var body: some View { + Color.clear + .onAppear { onChange(plugin.fieldDetectionSignature) } + .onChange(of: plugin.fieldDetectionSignature) { signature in + onChange(signature) + } + } +} + +enum ImportDetectionSignatureObservation { + @MainActor + static func observer( + for plugin: (any ImportFormatPlugin)?, + onChange: @escaping (String) -> Void + ) -> AnyView? { + guard let observable = plugin as? any ImportFormatPlugin & ObservableObject else { return nil } + return makeObserver(for: observable, onChange: onChange) + } + + @MainActor + private static func makeObserver( + for plugin: Plugin, + onChange: @escaping (String) -> Void + ) -> AnyView { + AnyView(ImportDetectionSignatureObserver(plugin: plugin, onChange: onChange)) + } +} diff --git a/TablePro/Views/Import/RowImportMapping.swift b/TablePro/Views/Import/RowImportMapping.swift new file mode 100644 index 0000000000..9ea8630370 --- /dev/null +++ b/TablePro/Views/Import/RowImportMapping.swift @@ -0,0 +1,158 @@ +// +// RowImportMapping.swift +// TablePro +// + +import Combine +import Foundation +import TableProPluginKit + +/// The mapping of a row import into an existing table, held outside the sheet so its rules are testable. +@MainActor +internal final class RowImportMapping: ObservableObject { + internal struct Row: Identifiable { + let field: PluginImportField + var choice: ImportFieldChoice + + var id: String { field.name } + } + + @Published internal private(set) var columns: [String] = [] + @Published internal var rows: [Row] = [] + @Published internal private(set) var table: TableScope? + + /// Set when a table is picked or the saved mapping is put back, never by a reload, so a failed + /// import does not announce the user's own mapping as restored. + @Published internal private(set) var showsSavedMapping = false + @Published private var saved: [String: ImportMappingOverride] = [:] + + /// Cleared with the rows, so picking a table again always reads it again. + internal private(set) var loadedRead: AnyHashable? + + private let store: ImportColumnMappingStore + + internal init(store: ImportColumnMappingStore = .shared) { + self.store = store + } + + internal var fields: [String] { + rows.map(\.field.name) + } + + internal var choices: [ImportFieldChoice] { + rows.map(\.choice) + } + + internal var columnMapping: [String: String] { + var mapping: [String: String] = [:] + for row in rows { + guard let column = row.choice.mappedColumn else { continue } + mapping[row.field.name] = column + } + return mapping + } + + internal var hasMappedField: Bool { + rows.contains { $0.choice.mappedColumn != nil } + } + + internal var mapsOneColumnTwice: Bool { + let targets = rows.compactMap { $0.choice.mappedColumn?.lowercased() } + return Set(targets).count != targets.count + } + + /// Only when the saved mapping covers a field of this file. + internal var canUseSavedMapping: Bool { + guard !saved.isEmpty else { return false } + let restored = savedChoices + guard restored != choices else { return false } + return !ImportColumnMatcher.restoredFields(saved, fields: fields, columns: columns, choices: restored).isEmpty + } + + internal func clear() { + table = nil + columns = [] + rows = [] + saved = [:] + showsSavedMapping = false + loadedRead = nil + } + + /// A reload of the same table prefers this sheet's choices, then the saved ones, then the name match. + internal func load( + fields: [PluginImportField], + columns: [String], + for scope: TableScope, + read: AnyHashable? = nil + ) { + let remembered = store.overrides(for: scope) + let reloading = table == scope + var tiers = [remembered.mapValues(\.choice)] + if reloading { + let session = Dictionary(rows.map { ($0.field.name, $0.choice) }, uniquingKeysWith: { first, _ in first }) + tiers.insert(session, at: 0) + } + table = scope + saved = remembered + self.columns = columns + let names = fields.map(\.name) + apply(ImportColumnMatcher.resolve(fields: names, columns: columns, preferring: tiers), to: fields) + showsSavedMapping = (reloading ? showsSavedMapping : true) && appliesSavedChoice + loadedRead = read + } + + internal func matchByName() { + apply(ImportColumnMatcher.byName(fields: fields, columns: columns)) + showsSavedMapping = false + } + + internal func matchByPosition() { + apply(ImportColumnMatcher.byPosition(fields: fields, columns: columns)) + showsSavedMapping = false + } + + internal func useSavedMapping() { + apply(savedChoices) + showsSavedMapping = appliesSavedChoice + } + + internal func setAllIncluded(_ include: Bool) { + for index in rows.indices { + rows[index].choice.include = include + } + } + + /// Called when an import starts, whether it succeeds or not, with the plan it runs: a mapping built + /// by hand is the costliest part of the sheet to redo. + internal func remember( + fields: [String], + columns: [String], + columnMapping: [String: String], + in scope: TableScope + ) { + let choices = fields.map { field in + columnMapping[field].map { ImportFieldChoice(include: true, column: $0) } ?? .skipped + } + let overrides = ImportColumnMatcher.overrides(fields: fields, columns: columns, choices: choices) + store.remember(overrides, forFields: fields, in: scope) + if table == scope { + saved = store.overrides(for: scope) + } + } + + private var appliesSavedChoice: Bool { + !ImportColumnMatcher.restoredFields(saved, fields: fields, columns: columns, choices: choices).isEmpty + } + + private var savedChoices: [ImportFieldChoice] { + ImportColumnMatcher.applying(saved, fields: fields, columns: columns) + } + + private func apply(_ choices: [ImportFieldChoice]) { + apply(choices, to: rows.map(\.field)) + } + + private func apply(_ choices: [ImportFieldChoice], to fields: [PluginImportField]) { + rows = zip(fields, choices).map { Row(field: $0, choice: $1) } + } +} diff --git a/TablePro/Views/Import/RowImportMappingToolbar.swift b/TablePro/Views/Import/RowImportMappingToolbar.swift new file mode 100644 index 0000000000..4a8d3320c0 --- /dev/null +++ b/TablePro/Views/Import/RowImportMappingToolbar.swift @@ -0,0 +1,36 @@ +// +// RowImportMappingToolbar.swift +// TablePro +// + +import SwiftUI + +/// One-shot commands rather than a mode, which would stop describing the mapping after one manual edit. +struct RowImportMappingToolbar: View { + @ObservedObject var mapping: RowImportMapping + let tableName: String + let fieldsFollowFileOrder: Bool + + var body: some View { + HStack(spacing: 8) { + if mapping.showsSavedMapping { + Text(String(format: String(localized: "Restored the mapping saved for %@."), tableName)) + .font(.caption) + .foregroundStyle(.secondary) + .lineLimit(1) + .truncationMode(.middle) + } + Spacer(minLength: 0) + Menu(String(localized: "Match Columns")) { + Button(String(localized: "Match by Name"), action: mapping.matchByName) + Button(String(localized: "Match by Position"), action: mapping.matchByPosition) + .disabled(!fieldsFollowFileOrder) + Divider() + Button(String(localized: "Use Saved Mapping"), action: mapping.useSavedMapping) + .disabled(!mapping.canUseSavedMapping) + } + .fixedSize() + .accessibilityIdentifier("row-import-match-columns") + } + } +} diff --git a/TablePro/Views/Import/RowImportSheet.swift b/TablePro/Views/Import/RowImportSheet.swift index 37c8f746b8..626a8fe13d 100644 --- a/TablePro/Views/Import/RowImportSheet.swift +++ b/TablePro/Views/Import/RowImportSheet.swift @@ -28,11 +28,21 @@ struct RowImportSheet: View { case newTable } - private struct FieldMapping: Identifiable { - let field: PluginImportField - var include: Bool - var targetColumn: String? - var id: String { field.name } + /// The identity of a read. `.task(id:)` cancels the read in flight when it changes. + private struct SourceRead: Hashable { + let destination: Destination + let scope: DatabaseScope? + let table: String? + let signature: String + let attempt: Int + } + + private struct ImportPlan { + let targetTable: String + let fields: [String] + let columns: [String] + let columnMapping: [String: String] + let newTable: PluginCreateTableDefinition? } private struct NewColumn: Identifiable { @@ -65,16 +75,19 @@ struct RowImportSheet: View { /// Concurrent callers wait for the one in flight, per the schema-loading invariant. @State private var isLoadingTables = false @State private var selectedTargetTable: String? - @State private var targetColumns: [String] = [] - @State private var mappings: [FieldMapping] = [] + @StateObject private var mapping = RowImportMapping() @State private var newTableName: String = "" /// The last name this sheet proposed, so a second pass can tell its own guess from what /// the user typed over it. @State private var proposedTableName: String = "" @State private var newColumns: [NewColumn] = [] - @State private var newColumnsLoaded = false @State private var isLoadingContext = false + + /// Copied from the plugin, whose options are edited in its own view that this sheet does not observe. + @State private var detectionSignature = "" + @State private var readAttempts: [Destination: Int] = [:] + @State private var lastNewColumnsRead: SourceRead? @State private var loadError: String? /// Moving focus here also selects the whole proposed name, measured rather than assumed: @@ -161,26 +174,26 @@ struct RowImportSheet: View { hostWindow = window } } + .background { detectionSignatureObserver } .task { settingsSnapshot = PluginSettingsSnapshot( plugins: [currentPlugin as? any SettablePluginDiscoverable].compactMap { $0 }) suggestNewTableName() await loadTables() - await loadNewColumns() + } + .task(id: sourceRead) { + await read(sourceRead) } .onChange(of: destination) { newValue in guard newValue == .newTable else { return } suggestNewTableName() newTableNameFocused = true } - .onChange(of: selectedTargetTable) { newValue in - mappings = [] - targetColumns = [] - guard destination == .existingTable, let table = newValue else { return } - Task { await loadExistingContext(table: table) } + .onChange(of: selectedTargetTable) { _ in + mapping.clear() } - .onChange(of: currentPlugin?.fieldDetectionSignature) { _ in - Task { await redetectFields() } + .onChange(of: currentPlugin?.fieldDetectionSignature) { newValue in + detectionSignature = newValue ?? "" } .onDisappear { importTask?.cancel() @@ -345,7 +358,7 @@ struct RowImportSheet: View { case .existingTable: if selectedTargetTable == nil { placeholder("Choose a destination table to map fields.") - } else if mappings.isEmpty { + } else if mapping.rows.isEmpty { placeholder("No fields found in the file.") } else { mappingTable @@ -374,19 +387,11 @@ struct RowImportSheet: View { } } + /// Per destination, so Try Again for one never reads the other again over its edits. @MainActor private func retryLoad() async { loadError = nil - newColumnsLoaded = false - newColumns = [] - mappings = [] - switch destination { - case .newTable: - await loadNewColumns() - case .existingTable: - guard let table = selectedTargetTable else { return } - await loadExistingContext(table: table) - } + readAttempts[destination, default: 0] += 1 } private func placeholder(_ message: String) -> some View { @@ -400,8 +405,30 @@ struct RowImportSheet: View { .frame(maxWidth: .infinity) } + @ViewBuilder + private var detectionSignatureObserver: some View { + if let observer = ImportDetectionSignatureObservation.observer( + for: currentPlugin, + onChange: { detectionSignature = $0 } + ) { + observer + } + } + + private var mappingToolbar: some View { + RowImportMappingToolbar( + mapping: mapping, + tableName: selectedTargetTable ?? "", + fieldsFollowFileOrder: currentPlugin.map { type(of: $0).sourceFieldsFollowFileOrder } ?? false + ) + } + private var mappingTable: some View { VStack(spacing: 0) { + mappingToolbar + .padding(.horizontal) + .padding(.vertical, 6) + Divider() HStack(spacing: 12) { Toggle(String(localized: "Import all fields"), isOn: allMappingsIncluded) .labelsHidden() @@ -423,7 +450,7 @@ struct RowImportSheet: View { ScrollView { VStack(spacing: 6) { - ForEach(mappings) { row in + ForEach(mapping.rows) { row in mappingRow(row) } } @@ -433,9 +460,9 @@ struct RowImportSheet: View { } } - private func mappingRow(_ row: FieldMapping) -> some View { + private func mappingRow(_ row: RowImportMapping.Row) -> some View { HStack(spacing: 12) { - Toggle(row.field.name, isOn: mappingBinding(row).include) + Toggle(row.field.name, isOn: mappingBinding(row).choice.include) .labelsHidden() .accessibilityLabel(Text(String(format: String(localized: "Import %@"), row.field.name))) .frame(width: 16) @@ -447,15 +474,15 @@ struct RowImportSheet: View { } .frame(maxWidth: .infinity, alignment: .leading) Picker(String(format: String(localized: "Column for %@"), row.field.name), - selection: mappingBinding(row).targetColumn) { + selection: mappingBinding(row).choice.column) { Text("Skip").tag(String?.none) - ForEach(targetColumns, id: \.self) { column in + ForEach(mapping.columns, id: \.self) { column in Text(column).tag(String?.some(column)) } } .labelsHidden() .frame(width: 240, alignment: .leading) - .disabled(!row.include) + .disabled(!row.choice.include) } } @@ -546,11 +573,11 @@ struct RowImportSheet: View { // MARK: - Bindings - private func mappingBinding(_ row: FieldMapping) -> Binding { - guard let index = mappings.firstIndex(where: { $0.id == row.id }) else { + private func mappingBinding(_ row: RowImportMapping.Row) -> Binding { + guard let index = mapping.rows.firstIndex(where: { $0.id == row.id }) else { return .constant(row) } - return $mappings[index] + return $mapping.rows[index] } private func columnBinding(_ row: NewColumn) -> Binding { @@ -562,8 +589,8 @@ struct RowImportSheet: View { private var allMappingsIncluded: Binding { Binding( - get: { !mappings.isEmpty && mappings.allSatisfy(\.include) }, - set: { value in for index in mappings.indices { mappings[index].include = value } } + get: { !mapping.rows.isEmpty && mapping.rows.allSatisfy(\.choice.include) }, + set: { mapping.setAllIncluded($0) } ) } @@ -577,8 +604,7 @@ struct RowImportSheet: View { private var validationMessage: String? { switch destination { case .existingTable: - let columns = mappings.filter { $0.include }.compactMap { $0.targetColumn?.lowercased() } - if Set(columns).count != columns.count { + if mapping.mapsOneColumnTwice { return String(localized: "Each column can be mapped from only one field.") } return nil @@ -650,16 +676,30 @@ struct RowImportSheet: View { } private var canImport: Bool { - guard !(importService?.state.isImporting ?? false), validationMessage == nil else { return false } + guard !(importService?.state.isImporting ?? false), validationMessage == nil, currentReadIsReady else { + return false + } switch destination { case .existingTable: - return selectedTargetTable != nil && mappings.contains { $0.include && $0.targetColumn != nil } + return selectedTargetTable != nil && mapping.hasMappedField case .newTable: return !newTableName.trimmingCharacters(in: .whitespaces).isEmpty && newColumns.contains { $0.include && !$0.name.trimmingCharacters(in: .whitespaces).isEmpty } } } + /// Import runs only what the rows show: a read of the file with the current options, into the current + /// table and database, that finished without an error. + private var currentReadIsReady: Bool { + guard !isLoadingContext, loadError == nil else { return false } + switch destination { + case .existingTable: + return mapping.loadedRead == AnyHashable(sourceRead) + case .newTable: + return lastNewColumnsRead == sourceRead + } + } + // MARK: - Loading /// Both failures used to leave an empty list and say nothing, so the destination picker offered @@ -729,14 +769,57 @@ struct RowImportSheet: View { }.value } + private var sourceRead: SourceRead { + let isExisting = destination == .existingTable + return SourceRead( + destination: destination, + scope: isExisting ? DatabaseManager.shared.browseScope(for: connection.id) : nil, + table: isExisting ? selectedTargetTable : nil, + signature: detectionSignature, + attempt: readAttempts[destination, default: 0] + ) + } + + /// A read that succeeded is not repeated when switching back to its destination, and it clears an + /// error the other destination left on screen. @MainActor - private func loadNewColumns() async { - guard !newColumnsLoaded, let plugin = currentPlugin else { return } + private func read(_ request: SourceRead) async { + switch request.destination { + case .existingTable: + guard let table = request.table else { + isLoadingContext = false + return + } + guard mapping.loadedRead != AnyHashable(request) else { + showCachedRead() + return + } + await loadExistingContext(table: table, for: request) + case .newTable: + guard lastNewColumnsRead != request else { + showCachedRead() + return + } + await loadNewColumns(for: request) + } + } + + private func showCachedRead() { + isLoadingContext = false + loadError = nil + } + + @MainActor + private func loadNewColumns(for request: SourceRead) async { + guard let plugin = currentPlugin else { + isLoadingContext = false + return + } isLoadingContext = true loadError = nil - defer { isLoadingContext = false } do { let fields = try await Self.detectFields(plugin: plugin, at: fileURL, targetTable: nil) + guard !Task.isCancelled else { return } let serverVersion = DatabaseManager.shared.driver(for: connection.id)?.serverVersion newColumns = fields.map { field in NewColumn( @@ -753,48 +836,41 @@ struct RowImportSheet: View { defaultValue: "" ) } - newColumnsLoaded = true + lastNewColumnsRead = request } catch { + guard !Task.isCancelled else { return } loadError = error.localizedDescription Self.logger.warning("Failed to read import fields: \(error.publicLogShape, privacy: .public)") } + isLoadingContext = false } @MainActor - private func loadExistingContext(table: String) async { - guard let plugin = currentPlugin, - DatabaseManager.shared.browseScope(for: connection.id) != nil else { return } + private func loadExistingContext(table: String, for request: SourceRead) async { + guard let plugin = currentPlugin, let scope = request.scope else { + isLoadingContext = false + return + } isLoadingContext = true loadError = nil - defer { isLoadingContext = false } do { - let columns = try await DatabaseManager.shared.withBrowseMetadataDriver( - connectionId: connection.id - ) { driver in + let columns = try await DatabaseManager.shared.withMetadataDriver(scope: scope) { driver in try await driver.fetchColumns(table: table) }.map(\.name) + guard !Task.isCancelled else { return } let fields = try await Self.detectFields(plugin: plugin, at: fileURL, targetTable: table) - targetColumns = columns - mappings = fields.map { field in - let match = columns.first { $0.caseInsensitiveCompare(field.name) == .orderedSame } - return FieldMapping(field: field, include: match != nil, targetColumn: match) - } + guard !Task.isCancelled else { return } + mapping.load(fields: fields, columns: columns, for: Self.tableScope(table, in: scope), read: request) } catch { + guard !Task.isCancelled else { return } loadError = error.localizedDescription Self.logger.warning("Failed to read import fields: \(error.publicLogShape, privacy: .public)") } + isLoadingContext = false } - @MainActor - private func redetectFields() async { - switch destination { - case .existingTable: - guard let table = selectedTargetTable else { return } - await loadExistingContext(table: table) - case .newTable: - newColumnsLoaded = false - await loadNewColumns() - } + private static func tableScope(_ table: String, in scope: DatabaseScope) -> TableScope { + TableScope(connectionId: scope.connectionId, database: scope.database, schema: scope.schema, table: table) } // MARK: - Import @@ -805,10 +881,25 @@ struct RowImportSheet: View { showErrorDialog = true return } + guard currentReadIsReady else { + if !isLoadingContext { + readAttempts[destination, default: 0] += 1 + } + return + } switch destination { case .existingTable: guard let table = selectedTargetTable else { return } - runImport(targetTable: table, mapping: existingMapping(), newTable: nil, scope: scope) + runImport( + ImportPlan( + targetTable: table, + fields: mapping.fields, + columns: mapping.columns, + columnMapping: mapping.columnMapping, + newTable: nil + ), + scope: scope + ) case .newTable: let name = newTableName.trimmingCharacters(in: .whitespaces) guard !name.isEmpty, let definition = newTableDefinition(tableName: name) else { @@ -816,7 +907,18 @@ struct RowImportSheet: View { showErrorDialog = true return } - runImport(targetTable: name, mapping: newTableMapping(), newTable: definition, scope: scope) + let columnMapping = newTableMapping() + let fields = newColumns.map(\.field.name) + runImport( + ImportPlan( + targetTable: name, + fields: fields, + columns: fields.compactMap { columnMapping[$0] }, + columnMapping: columnMapping, + newTable: definition + ), + scope: scope + ) } } @@ -827,16 +929,6 @@ struct RowImportSheet: View { ) } - private func existingMapping() -> [String: String] { - var mapping: [String: String] = [:] - for entry in mappings where entry.include { - if let column = entry.targetColumn { - mapping[entry.field.name] = column - } - } - return mapping - } - private func newTableMapping() -> [String: String] { var mapping: [String: String] = [:] for column in newColumns where column.include && !column.name.trimmingCharacters(in: .whitespaces).isEmpty { @@ -874,28 +966,31 @@ struct RowImportSheet: View { ) } - private func runImport( - targetTable: String, - mapping: [String: String], - newTable: PluginCreateTableDefinition?, - scope: DatabaseScope - ) { + private func runImport(_ plan: ImportPlan, scope: DatabaseScope) { let service = ImportService(connection: connection) importService = service showProgressDialog = true + let targetTable = plan.targetTable importTask = Task { do { - if let newTable { + if let newTable = plan.newTable { try await prepareTable(newTable, scope: scope) } + mapping.remember( + fields: plan.fields, + columns: plan.columns, + columnMapping: plan.columnMapping, + in: Self.tableScope(targetTable, in: scope) + ) let result = try await service.importFile( from: fileURL, formatId: formatId, encoding: .utf8, scope: scope, targetTable: targetTable, - columnMapping: mapping + columnMapping: plan.columnMapping, + sourceFields: Set(plan.fields) ) await MainActor.run { showProgressDialog = false diff --git a/TableProTests/Core/Export/TableColumnMatcherTests.swift b/TableProTests/Core/Export/TableColumnMatcherTests.swift new file mode 100644 index 0000000000..d89a0e092e --- /dev/null +++ b/TableProTests/Core/Export/TableColumnMatcherTests.swift @@ -0,0 +1,134 @@ +// +// TableColumnMatcherTests.swift +// TableProTests +// + +import Foundation +import Testing + +@testable import TablePro + +struct TableColumnMatcherTests { + @Test("Columns are matched by name, and the rest are reported on both sides") + func matchesByName() { + let match = TableColumnMatcher.match( + source: ["id", "name", "legacy"], destination: ["id", "name", "created_at"]) + + #expect(match.mapping == ["id": "id", "name": "name"]) + #expect(match.unmatchedSource == ["legacy"]) + #expect(match.unmatchedDestination == ["created_at"]) + #expect(match.contestedDestinations.isEmpty) + } + + @Test("A column spelled with another case still matches") + func matchesIgnoringCase() { + let match = TableColumnMatcher.match(source: ["ID", "Email"], destination: ["id", "email"]) + + #expect(match.mapping == ["ID": "id", "Email": "email"]) + #expect(match.unmatchedSource.isEmpty) + } + + /// Both used to land on `name`, and the INSERT named it twice. + @Test("The exact spelling claims a destination column before a twin that differs only by case") + func exactSpellingWinsTheColumn() { + let match = TableColumnMatcher.match(source: ["Name", "name"], destination: ["id", "name"]) + + #expect(match.mapping == ["name": "name"]) + #expect(match.unmatchedSource == ["Name"]) + #expect(match.contestedDestinations.isEmpty) + } + + @Test("Of two twins that both differ by case from the destination, only the first is matched") + func firstTwinWinsTheColumn() { + let match = TableColumnMatcher.match(source: ["NAME", "Name"], destination: ["name"]) + + #expect(match.mapping == ["NAME": "name"]) + #expect(match.unmatchedSource == ["Name"]) + } + + @Test("Twins on both sides each reach their own column") + func twinsReachTheirOwnColumns() { + let match = TableColumnMatcher.match(source: ["Name", "name"], destination: ["name", "Name"]) + + #expect(match.mapping == ["Name": "Name", "name": "name"]) + #expect(match.unmatchedDestination.isEmpty) + } + + @Test("A destination twin left over is matched ignoring case once the exact spelling is taken") + func leftoverTwinMatchesIgnoringCase() { + let match = TableColumnMatcher.match(source: ["NAME", "Name"], destination: ["Name", "name"]) + + #expect(match.mapping == ["Name": "Name", "NAME": "name"]) + } + + @Test("An override onto a column another source column holds is kept and reported as contested") + func overrideOntoAHeldColumnIsContested() { + let match = TableColumnMatcher.match( + source: ["first_name", "last_name"], + destination: ["first_name", "last_name"], + overrides: ["last_name": "first_name"] + ) + + #expect(match.mapping == ["first_name": "first_name", "last_name": "first_name"]) + #expect(match.contestedDestinations == ["first_name"]) + #expect(match.unmatchedDestination == ["last_name"]) + } + + @Test("Skipping one of the two source columns clears the contest") + func skippingOneClearsTheContest() { + let match = TableColumnMatcher.match( + source: ["first_name", "last_name"], + destination: ["first_name", "last_name"], + overrides: ["last_name": "first_name", "first_name": nil] + ) + + #expect(match.mapping == ["last_name": "first_name"]) + #expect(match.contestedDestinations.isEmpty) + #expect(match.unmatchedSource == ["first_name"]) + } + + @Test("Repointing a case twin onto the column its twin holds is contested") + func repointedTwinIsContested() { + let match = TableColumnMatcher.match( + source: ["Name", "name"], + destination: ["name"], + overrides: ["Name": "name"] + ) + + #expect(match.contestedDestinations == ["name"]) + } + + /// The leftovers are out of name order on both sides, so a path that sorted them, as the + /// override path once did, would not compare equal. + @Test("No overrides gives the automatic match") + func noOverridesIsTheAutomaticMatch() { + let source = ["zeta", "alpha", "id"] + let destination = ["id", "omega", "beta"] + + let automatic = TableColumnMatcher.match(source: source, destination: destination) + + #expect(automatic.unmatchedSource == ["zeta", "alpha"]) + #expect(automatic.unmatchedDestination == ["omega", "beta"]) + #expect(TableColumnMatcher.match(source: source, destination: destination, overrides: [:]) == automatic) + } + + @Test("An override leaves the unmatched source columns in the source table's order") + func overrideKeepsSourceOrder() { + let match = TableColumnMatcher.match( + source: ["zeta", "alpha", "id", "name"], + destination: ["id", "name"], + overrides: ["name": nil] + ) + + #expect(match.mapping == ["id": "id"]) + #expect(match.unmatchedSource == ["zeta", "alpha", "name"]) + } + + @Test("Only destination columns named more than once are contested, in name order") + func contestedDestinationsAreSorted() { + let contested = TableColumnMatcher.contestedDestinations( + in: ["a": "z", "b": "z", "c": "y", "d": "x", "e": "x", "f": "x"]) + + #expect(contested == ["x", "z"]) + } +} diff --git a/TableProTests/Core/Export/TableTransferServiceTests.swift b/TableProTests/Core/Export/TableTransferServiceTests.swift index 7372ddffc6..15bd693899 100644 --- a/TableProTests/Core/Export/TableTransferServiceTests.swift +++ b/TableProTests/Core/Export/TableTransferServiceTests.swift @@ -9,7 +9,83 @@ import Testing @testable import TablePro +/// Serves one table's rows as a source, and records what reaches it as a destination. +private final class TransferStubDriver: PluginDatabaseDriver, @unchecked Sendable { + let header: [String] + let rows: [PluginRow] + let columns: [String] + private(set) var executedQueries: [String] = [] + private(set) var executedParameters: [[PluginCellValue]] = [] + + init(header: [String] = [], rows: [PluginRow] = [], columns: [String] = []) { + self.header = header + self.rows = rows + self.columns = columns + } + + var insertStatements: [String] { + executedQueries.filter { $0.uppercased().hasPrefix("INSERT") } + } + + func defaultExportQuery(table: String, schema: String?) -> String? { + "SELECT * FROM \(table)" + } + + func streamRows(query: String) -> AsyncThrowingStream { + let header = PluginStreamHeader( + columns: header, columnTypeNames: header.map { _ in "TEXT" }) + let rows = rows + return AsyncThrowingStream { continuation in + continuation.yield(.header(header)) + continuation.yield(.rows(rows)) + continuation.finish() + } + } + + func connect() async throws {} + func disconnect() {} + + func execute(query: String) async throws -> PluginQueryResult { + executedQueries.append(query) + return PluginQueryResult(columns: [], columnTypeNames: [], rows: [], rowsAffected: 0, executionTime: 0) + } + + func executeParameterized(query: String, parameters: [PluginCellValue]) async throws -> PluginQueryResult { + executedQueries.append(query) + executedParameters.append(parameters) + return PluginQueryResult(columns: [], columnTypeNames: [], rows: [], rowsAffected: 0, executionTime: 0) + } + + func fetchTables(schema: String?) async throws -> [PluginTableInfo] { [] } + + func fetchColumns(table: String, schema: String?) async throws -> [PluginColumnInfo] { + columns.map { PluginColumnInfo(name: $0, dataType: "TEXT") } + } + + func fetchIndexes(table: String, schema: String?) async throws -> [PluginIndexInfo] { [] } + func fetchForeignKeys(table: String, schema: String?) async throws -> [PluginForeignKeyInfo] { [] } + func fetchTableDDL(table: String, schema: String?) async throws -> String { "" } + func fetchViewDefinition(view: String, schema: String?) async throws -> String { "" } + + func fetchTableMetadata(table: String, schema: String?) async throws -> PluginTableMetadata { + PluginTableMetadata(tableName: table) + } + + func fetchDatabases() async throws -> [String] { [] } + + func fetchDatabaseMetadata(_ database: String) async throws -> PluginDatabaseMetadata { + PluginDatabaseMetadata(name: database) + } +} + struct TableTransferServiceTests { + private func adapter(_ driver: TransferStubDriver, type: DatabaseType) -> PluginDriverAdapter { + PluginDriverAdapter(connection: DatabaseConnection(name: "Test", type: type), pluginDriver: driver) + } + + private func occurrences(of needle: String, in text: String) -> Int { + text.components(separatedBy: needle).count - 1 + } @Test("A row is keyed by its header's column names, in order") func rowIsKeyedByHeader() { @@ -100,4 +176,103 @@ struct TableTransferServiceTests { #expect(request.wrapInTransaction) #expect(!request.deleteExistingRows) } + + @Test("A mapping that sends two source columns to one destination column is refused") + func contestedMappingIsRefused() { + let objects = [ExportObjectItem(name: "people", kind: .table)] + #expect(throws: TableTransferError.self) { + try TableTransferService.refuseContestedMappings( + ["people": ["first_name": "name", "last_name": "name"]], for: objects) + } + #expect(throws: Never.self) { + try TableTransferService.refuseContestedMappings( + ["people": ["first_name": "first_name", "last_name": "last_name"]], for: objects) + } + } + + /// The server refused the INSERT only after "Delete existing rows first" had run, and with no + /// transaction around the table the deletion stood: the destination was left empty. + @MainActor @Test("A contested mapping deletes nothing on the destination") + func contestedMappingDeletesNothing() async { + let source = TransferStubDriver(header: ["first_name", "last_name"], rows: [[.text("Ada"), .text("Lovelace")]]) + let destination = TransferStubDriver(columns: ["name"]) + let request = TableTransferService.Request( + objects: [ExportObjectItem(name: "people", kind: .table, isSelected: true)], + sourceType: .postgresql, + destinationType: .mysql, + columnMapping: ["people": ["first_name": "name", "last_name": "name"]], + deleteExistingRows: true, + wrapInTransaction: false + ) + + do { + try await TableTransferService().transfer( + request: request, + sourceDriver: adapter(source, type: .postgresql), + destinationDriver: adapter(destination, type: .mysql) + ) + Issue.record("A mapping naming one destination column twice was transferred") + } catch TableTransferError.contestedDestination(let table, let columns) { + #expect(table == "people") + #expect(columns == ["name"]) + } catch { + Issue.record("Unexpected error: \(error)") + } + #expect(destination.executedQueries.isEmpty) + } + + /// `Name` and `name` both matched a lone `name`, and even with only one of them mapped the + /// sink folded the other onto it, so the INSERT named the column twice. + @MainActor @Test("A source column that differs only by case from a mapped one is not written twice") + func caseTwinIsNotWrittenTwice() async throws { + let source = TransferStubDriver( + header: ["id", "Name", "name"], rows: [[.text("1"), .text("Display"), .text("login")]]) + let destination = TransferStubDriver(columns: ["id", "name"]) + let request = TableTransferService.Request( + objects: [ExportObjectItem(name: "people", kind: .table, isSelected: true)], + sourceType: .postgresql, + destinationType: .mysql, + sourceColumns: ["people": ["id", "Name", "name"]], + deleteExistingRows: true, + wrapInTransaction: false + ) + + let service = TableTransferService() + try await service.transfer( + request: request, + sourceDriver: adapter(source, type: .postgresql), + destinationDriver: adapter(destination, type: .mysql) + ) + + let insert = try #require(destination.insertStatements.first) + #expect(destination.insertStatements.count == 1) + #expect(occurrences(of: "`name`", in: insert) == 1) + #expect(destination.executedParameters.last?.contains(.text("login")) == true) + #expect(destination.executedParameters.last?.contains(.text("Display")) == false) + #expect(service.state.transferredRows == 1) + #expect(service.state.warnings.contains { $0.contains("Name") }) + } + + @MainActor @Test("Twins mapped by the user to their own columns each reach their own column") + func mappedTwinsReachTheirOwnColumns() async throws { + let source = TransferStubDriver( + header: ["Name", "name"], rows: [[.text("Display"), .text("login")]]) + let destination = TransferStubDriver(columns: ["display_name", "login"]) + let request = TableTransferService.Request( + objects: [ExportObjectItem(name: "people", kind: .table, isSelected: true)], + sourceType: .postgresql, + destinationType: .mysql, + columnMapping: ["people": ["Name": "display_name", "name": "login"]] + ) + + try await TableTransferService().transfer( + request: request, + sourceDriver: adapter(source, type: .postgresql), + destinationDriver: adapter(destination, type: .mysql) + ) + + let insert = try #require(destination.insertStatements.first) + #expect(occurrences(of: "`display_name`", in: insert) == 1) + #expect(occurrences(of: "`login`", in: insert) == 1) + } } diff --git a/TableProTests/Core/Plugins/ImportDataSinkAdapterMappingTests.swift b/TableProTests/Core/Plugins/ImportDataSinkAdapterMappingTests.swift index 2bf3fb7b61..1a120f8a37 100644 --- a/TableProTests/Core/Plugins/ImportDataSinkAdapterMappingTests.swift +++ b/TableProTests/Core/Plugins/ImportDataSinkAdapterMappingTests.swift @@ -14,12 +14,13 @@ import Testing /// line, and the stop modes halt on a mapping that matches nothing. @MainActor struct ImportDataSinkAdapterMappingTests { - private func adapter(mapping: [String: String]) -> ImportDataSinkAdapter { + private func adapter(mapping: [String: String], sourceFields: Set = []) -> ImportDataSinkAdapter { ImportDataSinkAdapter( driver: MockDatabaseDriver(), databaseType: .mysql, targetTable: "people", - columnMapping: mapping + columnMapping: mapping, + sourceFields: sourceFields ) } @@ -48,14 +49,89 @@ struct ImportDataSinkAdapterMappingTests { try await sink.insertRow(["name": .text("Ada")]) } - /// The mapping is matched case-insensitively, so a header cased differently to the column still - /// reaches it rather than being refused. + /// A spelling the mapping was not made from, such as a JSON key cased differently in a row past + /// the sample, still reaches the one mapped field it matches ignoring case. @Test("Field matching ignores case") func fieldMatchingIgnoresCase() async throws { let sink = adapter(mapping: ["Name": "name"]) try await sink.insertRow(["NAME": .text("Ada")]) } + @Test("A field whose twin differing only by case is mapped exactly is left out") + func caseTwinOfAMappedFieldStaysUnmapped() { + let sink = adapter(mapping: ["Email": "email"]) + + let (columns, values) = sink.mappedColumnsAndValues(["Email": .text("work"), "email": .text("home")]) + + #expect(columns == ["email"]) + #expect(values == [.text("work")]) + } + + @Test("Two fields that differ only by case reach their own columns") + func caseTwinsReachTheirOwnColumns() { + let sink = adapter(mapping: ["Email": "work_email", "email": "home_email"]) + + let (columns, values) = sink.mappedColumnsAndValues(["Email": .text("work"), "email": .text("home")]) + + #expect(columns == ["home_email", "work_email"]) + #expect(values == [.text("home"), .text("work")]) + } + + @Test("A field the sheet listed and the user skipped never folds onto its twin's column") + func skippedFieldNeverFolds() { + let sink = adapter(mapping: ["Email": "email"], sourceFields: ["Email", "email"]) + + let (columns, _) = sink.mappedColumnsAndValues(["email": .text("home")]) + + #expect(columns.isEmpty) + } + + @Test("A new spelling of two listed fields that differ only by case reaches neither, mapped or skipped") + func newSpellingOfListedCaseTwinsDoesNotFold() { + let sink = adapter(mapping: ["Email": "email"], sourceFields: ["Email", "email"]) + + let (columns, _) = sink.mappedColumnsAndValues(["EMAIL": .text("x"), "id": .text("1")]) + + #expect(columns.isEmpty) + } + + @Test("A new spelling of one listed field still reaches its column") + func newSpellingOfOneListedFieldFolds() { + let sink = adapter(mapping: ["Email": "email"], sourceFields: ["Email"]) + + let (columns, values) = sink.mappedColumnsAndValues(["EMAIL": .text("x")]) + + #expect(columns == ["email"]) + #expect(values == [.text("x")]) + } + + @Test("An unknown spelling beside a skipped field of the same name reaches no column") + func unknownSpellingBesideASkippedTwinDoesNotFold() { + let sink = adapter(mapping: ["Name": "name"], sourceFields: ["Name", "NAME"]) + + let (columns, _) = sink.mappedColumnsAndValues(["NAME": .text("skipped"), "name": .text("unknown")]) + + #expect(columns.isEmpty) + } + + @Test("A third spelling of two mapped fields that differ only by case reaches neither column") + func ambiguousMappingKeysDoNotFold() { + let sink = adapter(mapping: ["Email": "work_email", "email": "home_email"]) + + let (columns, _) = sink.mappedColumnsAndValues(["EMAIL": .text("x")]) + + #expect(columns.isEmpty) + } + + @Test("Two unknown spellings of one mapped field in a row reach neither column") + func ambiguousRowKeysDoNotFold() { + let sink = adapter(mapping: ["Name": "name"]) + + let (columns, _) = sink.mappedColumnsAndValues(["NAME": .text("a"), "name": .text("b"), "id": .text("1")]) + + #expect(columns.isEmpty) + } + /// A row carrying nothing has nothing to lose, so it passes through. Only a row holding values /// that reach no column is worth stopping for, and conflating the two would turn an empty /// object in an NDJSON file into a failed import. diff --git a/TableProTests/Core/Services/ImportColumnMatcherTests.swift b/TableProTests/Core/Services/ImportColumnMatcherTests.swift new file mode 100644 index 0000000000..68f38c315d --- /dev/null +++ b/TableProTests/Core/Services/ImportColumnMatcherTests.swift @@ -0,0 +1,243 @@ +// +// ImportColumnMatcherTests.swift +// TableProTests +// + +@testable import TablePro +import Testing + +struct ImportColumnMatcherTests { + private let columns = ["id", "full_name", "email"] + + private func mapped(_ choices: [ImportFieldChoice]) -> [String?] { + choices.map(\.mappedColumn) + } + + @Test("A name match ignores case and leaves an unmatched field skipped") + func nameMatchIgnoresCase() { + let choices = ImportColumnMatcher.byName(fields: ["ID", "Name", "E-mail"], columns: columns) + + #expect(mapped(choices) == ["id", nil, nil]) + #expect(choices[1] == .skipped) + } + + @Test("Two fields that fold to one column do not both claim it") + func nameMatchClaimsEachColumnOnce() { + let choices = ImportColumnMatcher.byName(fields: ["email", "EMAIL"], columns: columns) + + #expect(mapped(choices) == ["email", nil]) + } + + @Test("An exact spelling wins over a case-insensitive one when the table holds both") + func exactSpellingWinsACaseCollision() { + let choices = ImportColumnMatcher.byName(fields: ["name", "Name"], columns: ["Name", "name"]) + + #expect(mapped(choices) == ["name", "Name"]) + } + + @Test("A field spelled exactly like the column wins it over an earlier field that differs only by case") + func exactSpellingWinsAcrossFields() { + let choices = ImportColumnMatcher.byName(fields: ["Email", "email"], columns: columns) + + #expect(mapped(choices) == [nil, "email"]) + } + + @Test("Position pairs field i with column i and skips the fields past the last column") + func positionPairsInOrder() { + let choices = ImportColumnMatcher.byPosition( + fields: ["Column 1", "Column 2", "Column 3", "Column 4"], + columns: columns + ) + + #expect(mapped(choices) == ["id", "full_name", "email", nil]) + } + + @Test("A file shorter than the table leaves the trailing columns unmapped") + func positionWithFewerFields() { + let choices = ImportColumnMatcher.byPosition(fields: ["a", "b"], columns: columns) + + #expect(mapped(choices) == ["id", "full_name"]) + } + + @Test("A remembered column is applied where the name match finds nothing") + func overrideAppliesToUnmatchedField() { + let choices = ImportColumnMatcher.applying( + ["Name": .column("full_name"), "E-mail": .column("email")], + fields: ["id", "Name", "E-mail"], + columns: columns + ) + + #expect(mapped(choices) == ["id", "full_name", "email"]) + } + + @Test("A remembered skip wins over a field whose name matches a column") + func rememberedSkipBeatsNameMatch() { + let choices = ImportColumnMatcher.applying(["id": .skip], fields: ["id", "email"], columns: columns) + + #expect(mapped(choices) == [nil, "email"]) + #expect(choices[0].include == false) + } + + @Test("A remembered column the table no longer has falls back to the name match") + func droppedColumnFallsBackToNameMatch() { + let choices = ImportColumnMatcher.applying( + ["email": .column("contact_email")], + fields: ["email"], + columns: columns + ) + + #expect(mapped(choices) == ["email"]) + } + + @Test("A remembered column resolves to the table's current spelling") + func overrideResolvesColumnCase() { + let choices = ImportColumnMatcher.applying(["Name": .column("FULL_NAME")], fields: ["Name"], columns: columns) + + #expect(mapped(choices) == ["full_name"]) + } + + @Test("A remembered field is found when the file spells its header in another case") + func overrideFieldLookupIgnoresCase() { + let choices = ImportColumnMatcher.applying(["name": .column("full_name")], fields: ["NAME"], columns: columns) + + #expect(mapped(choices) == ["full_name"]) + } + + @Test("Two remembered entries that differ only by case say nothing about a third spelling") + func ambiguousCaseVariantsAreIgnored() { + let choices = ImportColumnMatcher.applying( + ["Email": .skip, "email": .column("email")], + fields: ["EMAIL"], + columns: columns + ) + + #expect(mapped(choices) == ["email"]) + } + + @Test("A remembered column takes precedence over a name match that would claim it first") + func overrideClaimsBeforeNameMatch() { + let choices = ImportColumnMatcher.applying( + ["contact": .column("email")], + fields: ["email", "contact"], + columns: columns + ) + + #expect(mapped(choices) == [nil, "email"]) + } + + @Test("A choice switched off keeps its column without claiming it") + func excludedChoiceClaimsNothing() { + let choices = ImportColumnMatcher.resolve( + fields: ["email", "work_email"], + columns: columns, + preferring: [[ + "email": ImportFieldChoice(include: false, column: "email"), + "work_email": ImportFieldChoice(include: true, column: "email") + ]] + ) + + #expect(choices[0] == ImportFieldChoice(include: false, column: "email")) + #expect(choices[1] == ImportFieldChoice(include: true, column: "email")) + } + + @Test("A higher tier wins its column over a lower tier's choice for a field listed earlier") + func higherTierWinsAcrossFields() { + let choices = ImportColumnMatcher.resolve( + fields: ["legacy_email", "Column 2"], + columns: columns, + preferring: [ + ["Column 2": ImportFieldChoice(include: true, column: "email")], + ["legacy_email": ImportFieldChoice(include: true, column: "email")] + ] + ) + + #expect(mapped(choices) == [nil, "email"]) + } + + @Test("A skip saved for one of two fields that differ only by case leaves the other to the name match") + func savedSkipStaysWithItsOwnSpelling() { + let choices = ImportColumnMatcher.applying(["Email": .skip], fields: ["Email", "email"], columns: columns) + + #expect(mapped(choices) == [nil, "email"]) + } + + @Test("Merging keeps a case variant that belongs to another layout's field") + func mergeKeepsAnotherLayoutsCaseVariant() { + let merged = ImportColumnMatcher.merging( + [:], + forFields: ["Email"], + into: ["Email": .column("work_email"), "email": .column("home_email")] + ) + + #expect(merged == ["email": .column("home_email")]) + } + + @Test("Only departures from the name match are worth remembering") + func overridesKeepOnlyDepartures() { + let fields = ["id", "Name", "email", "notes"] + let choices = [ + ImportFieldChoice(include: true, column: "id"), + ImportFieldChoice(include: true, column: "full_name"), + ImportFieldChoice(include: false, column: "email"), + ImportFieldChoice.skipped + ] + + let overrides = ImportColumnMatcher.overrides(fields: fields, columns: columns, choices: choices) + + #expect(overrides == ["Name": .column("full_name"), "email": .skip]) + } + + @Test("A field left unmatched stores nothing, so a column added later is picked up by name") + func unmatchedFieldStoresNothing() { + let overrides = ImportColumnMatcher.overrides(fields: ["notes"], columns: columns, choices: [.skipped]) + let later = ImportColumnMatcher.applying(overrides, fields: ["notes"], columns: columns + ["notes"]) + + #expect(overrides.isEmpty) + #expect(mapped(later) == ["notes"]) + } + + @Test("Merging keeps the entries of fields this file does not hold") + func mergeKeepsOtherLayouts() { + let merged = ImportColumnMatcher.merging( + ["Name": .column("full_name")], + forFields: ["Name", "email"], + into: ["Customer": .column("full_name"), "email": .skip] + ) + + #expect(merged == ["Customer": .column("full_name"), "Name": .column("full_name")]) + } + + @Test("Merging replaces an entry whose name differs from a field of this file only by case") + func mergeReplacesCaseVariant() { + let merged = ImportColumnMatcher.merging( + ["Email": .column("email")], + forFields: ["Email"], + into: ["EMAIL": .skip] + ) + + #expect(merged == ["Email": .column("email")]) + } + + @Test("A field restored from a remembered choice is reported, a plain name match is not") + func restoredFieldsNameOnlyTheOverrides() { + let overrides: [String: ImportMappingOverride] = ["Name": .column("full_name"), "id": .column("id")] + let fields = ["id", "Name"] + let choices = ImportColumnMatcher.applying(overrides, fields: fields, columns: columns) + + let restored = ImportColumnMatcher.restoredFields(overrides, fields: fields, columns: columns, choices: choices) + + #expect(restored == ["Name"]) + } + + @Test("A restored field that the user changes again is no longer reported") + func restoredFieldsFollowTheCurrentChoice() { + let overrides: [String: ImportMappingOverride] = ["Name": .column("full_name")] + let fields = ["Name"] + + let restored = ImportColumnMatcher.restoredFields( + overrides, fields: fields, columns: columns, choices: [.skipped] + ) + + #expect(restored.isEmpty) + } +} diff --git a/TableProTests/Plugins/CSVImportPluginTests.swift b/TableProTests/Plugins/CSVImportPluginTests.swift index ebf21e1592..4cc2cf9be0 100644 --- a/TableProTests/Plugins/CSVImportPluginTests.swift +++ b/TableProTests/Plugins/CSVImportPluginTests.swift @@ -69,6 +69,12 @@ struct CSVImportPluginTests { #expect(names == ["x", "x 2", "x 3"]) } + @Test("A header spelled out keeps its name over a blank header's placeholder that comes first") + func testLiteralHeaderKeepsItsName() { + let names = CSVImportParsing.columnNames(header: ["", "Column 1"], columnCount: 2) + #expect(names == ["Column 1 2", "Column 1"]) + } + @Test("Without a header, names are synthesized positionally") func testColumnNamesSynthesized() { let names = CSVImportParsing.columnNames(header: nil, columnCount: 3) diff --git a/TableProTests/Plugins/XLSXImportTests.swift b/TableProTests/Plugins/XLSXImportTests.swift index 0964f78fcc..6deb1b93c0 100644 --- a/TableProTests/Plugins/XLSXImportTests.swift +++ b/TableProTests/Plugins/XLSXImportTests.swift @@ -20,6 +20,20 @@ struct XLSXSheetParserTests { #expect(XLSXSheetParser.columnIndex(fromReference: "BA10") == 52) } + @Test("A repeated or blank header gets a name of its own") + func headerNamesAreUnique() { + let names = XLSXSheetParser.headerNames(from: ["Notes", "Notes", nil, " Id "]) + + #expect(names == ["Notes", "Notes 2", "column3", "Id"]) + } + + @Test("A header spelled out keeps its name over a blank cell's placeholder that comes first") + func literalHeaderKeepsItsName() { + let names = XLSXSheetParser.headerNames(from: [nil, "column1", "Code", "Code", "Code 2"]) + + #expect(names == ["column1 2", "column1", "Code", "Code 3", "Code 2"]) + } + @Test("A reference with no letters is refused") func invalidReference() { #expect(XLSXSheetParser.columnIndex(fromReference: "1") == nil) diff --git a/TableProTests/Storage/ImportColumnMappingStoreTests.swift b/TableProTests/Storage/ImportColumnMappingStoreTests.swift new file mode 100644 index 0000000000..4fd23d874c --- /dev/null +++ b/TableProTests/Storage/ImportColumnMappingStoreTests.swift @@ -0,0 +1,134 @@ +// +// ImportColumnMappingStoreTests.swift +// TableProTests +// + +import Foundation +import Testing + +@testable import TablePro + +@MainActor +struct ImportColumnMappingStoreTests { + private func makeStore() throws -> ImportColumnMappingStore { + let defaults = try #require(UserDefaults(suiteName: "ImportColumnMappingStoreTests.\(UUID().uuidString)")) + return ImportColumnMappingStore(defaults: defaults) + } + + private func scope( + connectionId: UUID, + database: String? = "shop", + schema: String? = nil, + table: String = "people" + ) -> TableScope { + TableScope(connectionId: connectionId, database: database, schema: schema, table: table) + } + + @Test("Overrides remembered for a table come back for that table only") + func rememberedOverridesRoundTrip() throws { + let store = try makeStore() + let connectionId = UUID() + let people = scope(connectionId: connectionId) + store.remember(["Name": .column("full_name"), "id": .skip], forFields: ["id", "Name"], in: people) + + #expect(store.overrides(for: people) == ["Name": .column("full_name"), "id": .skip]) + #expect(store.overrides(for: scope(connectionId: connectionId, table: "orders")).isEmpty) + #expect(store.overrides(for: scope(connectionId: connectionId, schema: "archive")).isEmpty) + #expect(store.overrides(for: scope(connectionId: UUID())).isEmpty) + } + + @Test("A later import that falls back to the name match for every field forgets the table") + func importMatchingByNameClearsTheEntry() throws { + let store = try makeStore() + let people = scope(connectionId: UUID()) + store.remember(["Name": .column("full_name")], forFields: ["Name"], in: people) + + store.remember([:], forFields: ["Name"], in: people) + + #expect(store.overrides(for: people).isEmpty) + } + + @Test("A second file layout into the same table keeps the first layout's choices") + func secondLayoutKeepsTheFirst() throws { + let store = try makeStore() + let people = scope(connectionId: UUID()) + store.remember(["Name": .column("full_name")], forFields: ["Name"], in: people) + + store.remember(["Customer": .column("full_name")], forFields: ["Customer"], in: people) + + #expect(store.overrides(for: people) == ["Name": .column("full_name"), "Customer": .column("full_name")]) + } + + @Test("Renaming a table carries its choices to the new name") + func renameTableMovesTheEntry() throws { + let store = try makeStore() + let connectionId = UUID() + let old = scope(connectionId: connectionId, table: "people") + let new = scope(connectionId: connectionId, table: "customers") + store.remember(["Name": .column("full_name")], forFields: ["Name"], in: old) + + store.renameTable(from: old, to: new) + + #expect(store.overrides(for: old).isEmpty) + #expect(store.overrides(for: new) == ["Name": .column("full_name")]) + } + + @Test("Renaming a database carries every table's choices") + func renameContainerMovesEveryTable() throws { + let store = try makeStore() + let connectionId = UUID() + let people = scope(connectionId: connectionId, database: "shop", table: "people") + store.remember(["Name": .column("full_name")], forFields: ["Name"], in: people) + + store.renameContainer( + connectionId: connectionId, fromDatabase: "shop", fromSchema: nil, toDatabase: "store", toSchema: nil + ) + + #expect(store.overrides(for: people).isEmpty) + #expect(store.overrides(for: scope(connectionId: connectionId, database: "store")) == ["Name": .column("full_name")]) + } + + @Test("Dropping a table forgets its choices and leaves its siblings alone") + func dropTableForgetsOnlyThatTable() throws { + let store = try makeStore() + let connectionId = UUID() + let dropped = scope(connectionId: connectionId, table: "people") + let kept = scope(connectionId: connectionId, table: "orders") + store.remember(["Name": .column("full_name")], forFields: ["Name"], in: dropped) + store.remember(["Total": .column("amount")], forFields: ["Total"], in: kept) + + store.dropTable(dropped) + + #expect(store.overrides(for: dropped).isEmpty) + #expect(store.overrides(for: kept) == ["Total": .column("amount")]) + } + + @Test("Dropping a database forgets every table under it and nothing outside it") + func dropContainerForgetsTheWholeDatabase() throws { + let store = try makeStore() + let connectionId = UUID() + let inside = scope(connectionId: connectionId, database: "shop") + let outside = scope(connectionId: connectionId, database: "archive") + store.remember(["Name": .column("full_name")], forFields: ["Name"], in: inside) + store.remember(["Name": .column("full_name")], forFields: ["Name"], in: outside) + + store.dropContainer(connectionId: connectionId, database: "shop", schema: nil) + + #expect(store.overrides(for: inside).isEmpty) + #expect(store.overrides(for: outside) == ["Name": .column("full_name")]) + } + + @Test("Deleting a connection forgets its choices and nobody else's") + func purgeForgetsOnlyThatConnection() throws { + let store = try makeStore() + let deleted = scope(connectionId: UUID()) + let kept = scope(connectionId: UUID()) + store.remember(["Name": .column("full_name")], forFields: ["Name"], in: deleted) + store.remember(["Name": .column("full_name")], forFields: ["Name"], in: kept) + + store.purgeConnections([deleted.connectionId], leavesTombstones: true) + + #expect(store.overrides(for: deleted).isEmpty) + #expect(store.overrides(for: kept) == ["Name": .column("full_name")]) + } +} diff --git a/TableProTests/Views/Import/RowImportMappingTests.swift b/TableProTests/Views/Import/RowImportMappingTests.swift new file mode 100644 index 0000000000..927409b69e --- /dev/null +++ b/TableProTests/Views/Import/RowImportMappingTests.swift @@ -0,0 +1,227 @@ +// +// RowImportMappingTests.swift +// TableProTests +// + +import Foundation +import TableProPluginKit +import Testing + +@testable import TablePro + +@MainActor +struct RowImportMappingTests { + private let columns = ["id", "full_name", "email"] + private let people = TableScope(connectionId: UUID(), database: "shop", schema: nil, table: "people") + + private func makeStore() throws -> ImportColumnMappingStore { + let defaults = try #require(UserDefaults(suiteName: "RowImportMappingTests.\(UUID().uuidString)")) + return ImportColumnMappingStore(defaults: defaults) + } + + private func fields(_ names: [String]) -> [PluginImportField] { + names.map { PluginImportField(name: $0, sampleValue: nil, inferredType: .text) } + } + + private func mapped(_ mapping: RowImportMapping) -> [String?] { + mapping.rows.map(\.choice.mappedColumn) + } + + private func rememberShown(_ mapping: RowImportMapping, in scope: TableScope) { + mapping.remember( + fields: mapping.fields, + columns: mapping.columns, + columnMapping: mapping.columnMapping, + in: scope + ) + } + + private func pick(_ column: String?, for field: String, in mapping: RowImportMapping) throws { + let index = try #require(mapping.rows.firstIndex { $0.field.name == field }) + mapping.rows[index].choice = ImportFieldChoice(include: column != nil, column: column) + } + + @Test("The mapping picked for a table comes back the next time a file goes into it") + func pickedMappingComesBack() throws { + let store = try makeStore() + let first = RowImportMapping(store: store) + first.load(fields: fields(["id", "Name", "E-mail"]), columns: columns, for: people) + #expect(mapped(first) == ["id", nil, nil]) + #expect(first.showsSavedMapping == false) + + try pick("full_name", for: "Name", in: first) + try pick("email", for: "E-mail", in: first) + rememberShown(first, in: people) + + let second = RowImportMapping(store: store) + second.load(fields: fields(["id", "Name", "E-mail"]), columns: columns, for: people) + + #expect(mapped(second) == ["id", "full_name", "email"]) + #expect(second.showsSavedMapping) + } + + @Test("Reading the file again keeps the columns picked in this sheet") + func reloadKeepsSessionChoices() throws { + let mapping = RowImportMapping(store: try makeStore()) + mapping.load(fields: fields(["id", "Name"]), columns: columns, for: people) + try pick("full_name", for: "Name", in: mapping) + + mapping.load(fields: fields(["id", "Name"]), columns: columns, for: people) + + #expect(mapped(mapping) == ["id", "full_name"]) + } + + @Test("A choice made in this sheet wins over the remembered one on a reload") + func sessionChoiceBeatsRememberedOne() throws { + let store = try makeStore() + store.remember(["Name": .column("full_name")], forFields: ["Name"], in: people) + let mapping = RowImportMapping(store: store) + mapping.load(fields: fields(["Name"]), columns: columns, for: people) + try pick("email", for: "Name", in: mapping) + + mapping.load(fields: fields(["Name"]), columns: columns, for: people) + + #expect(mapped(mapping) == ["email"]) + } + + @Test("Another table starts from its own remembered mapping, not this one's") + func clearingForgetsTheSession() throws { + let mapping = RowImportMapping(store: try makeStore()) + mapping.load(fields: fields(["Name"]), columns: columns, for: people) + try pick("full_name", for: "Name", in: mapping) + let customers = TableScope(connectionId: people.connectionId, database: "shop", schema: nil, table: "customers") + + mapping.clear() + mapping.load(fields: fields(["Name"]), columns: columns, for: customers) + + #expect(mapped(mapping) == [nil]) + } + + @Test("Clearing the rows forgets what they were read for, so the same table is read again") + func clearingForgetsTheRead() throws { + let mapping = RowImportMapping(store: try makeStore()) + mapping.load(fields: fields(["id"]), columns: columns, for: people, read: AnyHashable("people")) + #expect(mapping.loadedRead == AnyHashable("people")) + + mapping.clear() + + #expect(mapping.loadedRead == nil) + #expect(mapping.rows.isEmpty) + } + + @Test("Match by Name drops the remembered choices and Use Saved Mapping brings them back") + func matchCommandsRewriteEveryRow() throws { + let store = try makeStore() + store.remember(["Name": .column("full_name")], forFields: ["Name"], in: people) + let mapping = RowImportMapping(store: store) + mapping.load(fields: fields(["id", "Name"]), columns: columns, for: people) + + mapping.matchByName() + #expect(mapped(mapping) == ["id", nil]) + #expect(mapping.showsSavedMapping == false) + #expect(mapping.canUseSavedMapping) + + mapping.useSavedMapping() + #expect(mapped(mapping) == ["id", "full_name"]) + #expect(mapping.showsSavedMapping) + #expect(mapping.canUseSavedMapping == false) + } + + @Test("A mapping saved by an import that failed is not announced as restored when the file is read again") + func ownMappingIsNotReportedAsRestored() throws { + let mapping = RowImportMapping(store: try makeStore()) + mapping.load(fields: fields(["id", "Name"]), columns: columns, for: people) + try pick("full_name", for: "Name", in: mapping) + rememberShown(mapping, in: people) + + mapping.load(fields: fields(["id", "Name"]), columns: columns, for: people) + + #expect(mapped(mapping) == ["id", "full_name"]) + #expect(mapping.showsSavedMapping == false) + #expect(mapping.canUseSavedMapping == false) + } + + @Test("A restored mapping stays announced when the file is read again") + func restoredMappingSurvivesAReload() throws { + let store = try makeStore() + store.remember(["Name": .column("full_name")], forFields: ["Name"], in: people) + let mapping = RowImportMapping(store: store) + mapping.load(fields: fields(["Name"]), columns: columns, for: people) + + mapping.load(fields: fields(["Name"]), columns: columns, for: people) + + #expect(mapping.showsSavedMapping) + } + + @Test("Use Saved Mapping is off for a file that shares no field with the saved mapping") + func savedMappingForAnotherLayoutIsNotOffered() throws { + let store = try makeStore() + store.remember(["Full Name": .column("full_name")], forFields: ["Full Name"], in: people) + let mapping = RowImportMapping(store: store) + mapping.load(fields: fields(["id", "email"]), columns: columns, for: people) + try pick(nil, for: "email", in: mapping) + + #expect(mapping.canUseSavedMapping == false) + } + + @Test("A column picked in this sheet wins over a saved choice for another field after a re-read") + func sessionChoiceWinsOverSavedChoiceForAnotherField() throws { + let store = try makeStore() + store.remember(["legacy_email": .column("email")], forFields: ["legacy_email"], in: people) + let mapping = RowImportMapping(store: store) + mapping.load(fields: fields(["Column 1", "Column 2"]), columns: columns, for: people) + try pick("email", for: "Column 2", in: mapping) + + mapping.load(fields: fields(["legacy_email", "Column 2"]), columns: columns, for: people) + + #expect(mapped(mapping) == [nil, "email"]) + } + + @Test("Match by Position pairs the fields with the columns in order") + func matchByPosition() throws { + let mapping = RowImportMapping(store: try makeStore()) + mapping.load(fields: fields(["Column 1", "Column 2", "Column 3"]), columns: columns, for: people) + + mapping.matchByPosition() + + #expect(mapped(mapping) == ["id", "full_name", "email"]) + #expect(mapping.columnMapping == ["Column 1": "id", "Column 2": "full_name", "Column 3": "email"]) + } + + @Test("Nothing is remembered for a table until an import into it starts") + func nothingIsSavedBeforeImport() throws { + let store = try makeStore() + let mapping = RowImportMapping(store: store) + mapping.load(fields: fields(["Name"]), columns: columns, for: people) + try pick("full_name", for: "Name", in: mapping) + + #expect(store.overrides(for: people).isEmpty) + #expect(mapping.canUseSavedMapping == false) + } + + @Test("A new table remembers only the fields its columns were renamed from or left out") + func createdTableRemembersRenames() throws { + let store = try makeStore() + let mapping = RowImportMapping(store: store) + + mapping.remember( + fields: ["id", "E-mail", "notes"], + columns: ["id", "email"], + columnMapping: ["id": "id", "E-mail": "email"], + in: people + ) + + #expect(store.overrides(for: people) == ["E-mail": .column("email")]) + } + + @Test("Two fields sent to one column are reported") + func duplicateTargetIsReported() throws { + let mapping = RowImportMapping(store: try makeStore()) + mapping.load(fields: fields(["email", "work_email"]), columns: columns, for: people) + #expect(mapping.mapsOneColumnTwice == false) + + try pick("email", for: "work_email", in: mapping) + + #expect(mapping.mapsOneColumnTwice) + } +} diff --git a/TableProUITests/RowImportMappingMemoryUITests.swift b/TableProUITests/RowImportMappingMemoryUITests.swift new file mode 100644 index 0000000000..437fea63a6 --- /dev/null +++ b/TableProUITests/RowImportMappingMemoryUITests.swift @@ -0,0 +1,67 @@ +import XCTest + +/// A CSV whose headers do not all match the table's columns is mapped by hand once, imported, and +/// the next import into the same table opens on that mapping. +/// +/// Every query is rooted at a window or its sheet rather than at the application, because the +/// sample database's grid publishes thousands of elements and an application-wide search walks +/// all of them. +final class RowImportMappingMemoryUITests: UITestCase { + private let fileName = "genres.csv" + private let genres = Data("GenreId,Title\n9001,Imported genre\n".utf8) + + func testTheMappingPickedForATableComesBackOnTheNextImport() throws { + let app = try launchWithSampleDatabase(andDataFile: fileName, contents: genres) + let windows = rowImportWindows(in: app, dataFileTitle: fileName) + + let first = openSheetIntoGenre(windows, in: app) + let title = element(first.popUpButtons, labelEndingWith: "Column for Title") + XCTAssertTrue(title.waitToExist(timeout: 20), "The sheet must list the Title field") + XCTAssertEqual(title.value as? String, "Skip", "Title matches no Genre column by name") + element(first.checkBoxes, labelEndingWith: "Import Title").click() + choose("Name", from: title) + first.buttons["Import"].firstMatch.click() + + let done = windows.connection.sheets.buttons["Done"].firstMatch + XCTAssertTrue(done.waitToExist(timeout: 30), "The import must finish and report success") + done.click() + XCTAssertTrue( + waitForPredicate(timeout: 10) { !windows.connection.sheets.firstMatch.exists }, + "The import sheet must close after the import" + ) + + let second = openSheetIntoGenre(windows, in: app) + let restored = element(second.popUpButtons, labelEndingWith: "Column for Title") + XCTAssertTrue(restored.waitToExist(timeout: 20), "The sheet must list the Title field again") + XCTAssertEqual(restored.value as? String, "Name", "Title must come back mapped to Name") + let caption = second.staticTexts + .matching(NSPredicate(format: "value == %@", "Restored the mapping saved for Genre.")) + .firstMatch + XCTAssertTrue(caption.waitToExist(timeout: 5), "The sheet must say the mapping was restored") + second.buttons["Cancel"].firstMatch.click() + } + + private func openSheetIntoGenre( + _ windows: (dataFile: XCUIElement, connection: XCUIElement), + in app: XCUIApplication + ) -> XCUIElement { + let sheet = openRowImportSheet( + from: windows.dataFile, + titled: fileName, + into: windows.connection, + in: app + ) + let destination = element(sheet.popUpButtons, labelEndingWith: "Import into") + XCTAssertTrue(waitUntilHittable(destination, timeout: 15), "The sheet must offer the destination tables") + choose("Genre", from: destination) + return sheet + } + + private func choose(_ title: String, from popUp: XCUIElement) { + XCTAssertTrue(waitUntilHittable(popUp, timeout: 10), "\(popUp) must be clickable") + popUp.click() + let item = popUp.menuItems[title].firstMatch + XCTAssertTrue(item.waitToExist(timeout: 10), "The menu must offer \(title)") + item.click() + } +} diff --git a/TableProUITests/Support/UITestCase+RowImport.swift b/TableProUITests/Support/UITestCase+RowImport.swift new file mode 100644 index 0000000000..1cd14057a6 --- /dev/null +++ b/TableProUITests/Support/UITestCase+RowImport.swift @@ -0,0 +1,84 @@ +import XCTest + +/// Reaches the row import sheet through the data file window's Import into Table, the one route to +/// the sheet that needs no open panel. +/// +/// On a 1024pt runner screen the data file window and the connection window open on the same frame, +/// so a click aimed at the data file's grid lands on whichever window is in front, which after an +/// import is the connection window. The data file window is brought forward through the Window menu +/// instead, by its title. +extension UITestCase { + internal func launchWithSampleDatabase(andDataFile name: String, contents: Data) throws -> XCUIApplication { + let root = try XCTUnwrap(sandboxRoot, "setUpWithError did not prepare a sandbox") + let fileURL = root.appendingPathComponent(name) + try contents.write(to: fileURL) + return try launchApp(environment: [ + "TABLEPRO_UI_TEST_OPEN_SAMPLE": "1", + "TABLEPRO_UI_TEST_OPEN_FILE": fileURL.path + ]) + } + + internal func rowImportWindows( + in app: XCUIApplication, + dataFileTitle: String + ) -> (dataFile: XCUIElement, connection: XCUIElement) { + let dataWindow = app.windows.matching(identifier: "main-data-file").firstMatch + XCTAssertTrue(dataWindow.waitToExist(timeout: 30), "\(dataFileTitle) produced no data file window") + let connectionWindow = app.windows + .matching(NSPredicate(format: "identifier != %@", "main-data-file")) + .firstMatch + XCTAssertTrue( + waitForPredicate(timeout: 30) { + objectBrowser(in: connectionWindow).descendants(matching: .staticText).firstMatch.exists + }, + "The sample database never finished opening" + ) + return (dataWindow, connectionWindow) + } + + internal func openRowImportSheet( + from dataWindow: XCUIElement, + titled dataFileTitle: String, + into connectionWindow: XCUIElement, + in app: XCUIApplication + ) -> XCUIElement { + let grid = dataWindow.tables.matching(identifier: "data-grid").firstMatch + XCTAssertTrue(grid.waitToExist(timeout: 30), "The data file window has no grid") + XCTAssertTrue(waitForClickableRows(in: grid), "The data file must load rows") + + let menuBar = app.menuBars.firstMatch + menuBar.menuBarItems["Window"].click() + let windowItem = menuBar.menuBarItems["Window"].menus.menuItems + .matching(NSPredicate(format: "title == %@", dataFileTitle)) + .firstMatch + XCTAssertTrue(windowItem.waitToExist(timeout: 10), "The Window menu must list \(dataFileTitle)") + windowItem.click() + + let edit = menuBar.menuBarItems["Edit"] + edit.click() + let data = edit.menus.menuItems["Data"].firstMatch + XCTAssertTrue(data.waitToExist(timeout: 5), "Edit must carry the Data submenu") + data.hover() + let importItem = data.menus.menuItems["Import into Table…"].firstMatch + XCTAssertTrue(importItem.waitToExist(timeout: 5), "Data must offer Import into Table") + importItem.click() + + let proceed = dataWindow.sheets.buttons["Continue"].firstMatch + XCTAssertTrue(proceed.waitToExist(timeout: 15), "Import into Table must ask for a connection") + proceed.click() + + let sheet = connectionWindow.sheets.firstMatch + XCTAssertTrue(sheet.waitToExist(timeout: 30), "The import sheet must open in the connection window") + return sheet + } + + /// A SwiftUI control with `labelsHidden()` and its own accessibility label publishes both, + /// joined, as `"Title, Import Title"` on the runner, so a subscript by the accessibility label + /// alone finds nothing there. + internal func element( + _ query: XCUIElementQuery, + labelEndingWith label: String + ) -> XCUIElement { + query.matching(NSPredicate(format: "label ENDSWITH %@", label)).firstMatch + } +} diff --git a/docs/development/plugin-development.mdx b/docs/development/plugin-development.mdx index b6214fcf6b..b7165c8b10 100644 --- a/docs/development/plugin-development.mdx +++ b/docs/development/plugin-development.mdx @@ -13,7 +13,7 @@ A plugin is a macOS loadable bundle target with `WRAPPER_EXTENSION = tableplugin | Key | Type | Required | Purpose | |-----|------|----------|---------| -| `TableProPluginKitVersion` | integer | Yes | The PluginKit ABI the plugin was built against. Current value: 33 | +| `TableProPluginKitVersion` | integer | Yes | The PluginKit ABI the plugin was built against. Current value: 34 | | `TableProProvidesDatabaseTypeIds` | array of strings | Recommended | Database type IDs the plugin serves, which is what makes lazy loading possible | | `CFBundleShortVersionString` | string | Yes | Plugin version, read by registry update checks | | `TableProMinAppVersion` | string | No | The loader rejects the plugin on an older app | diff --git a/docs/development/plugin-registry.mdx b/docs/development/plugin-registry.mdx index 3664c0377a..4f4836a180 100644 --- a/docs/development/plugin-registry.mdx +++ b/docs/development/plugin-registry.mdx @@ -68,13 +68,13 @@ Themes carry no native code, so they match on architecture alone. "binaries": [ { "architecture": "arm64", - "pluginKitVersion": 33, + "pluginKitVersion": 34, "downloadURL": "https://github.com/TableProApp/TablePro/releases/download/plugin-oracle-v1.0.26/OracleDriver-arm64.zip", "sha256": "" }, { "architecture": "x86_64", - "pluginKitVersion": 33, + "pluginKitVersion": 34, "downloadURL": "https://github.com/TableProApp/TablePro/releases/download/plugin-oracle-v1.0.26/OracleDriver-x86_64.zip", "sha256": "" } diff --git a/docs/features/import-export.mdx b/docs/features/import-export.mdx index 4f4efa886a..497c4000a6 100644 --- a/docs/features/import-export.mdx +++ b/docs/features/import-export.mdx @@ -247,12 +247,18 @@ Right-click tables in the sidebar and choose **Transfer To…** to copy their ro -Rows only. The destination table has to exist and its column names have to match, because inventing DDL that crosses from one engine to another would create tables whose types quietly disagree with the data landing in them. A per-table row filter set in the export tree is not carried over; narrow the transfer by transferring fewer tables. +Rows only, into a destination table that already exists. A per-table row filter set in the export tree is not carried over; narrow the transfer by transferring fewer tables. **Delete existing rows first** empties each destination table before writing. There is no undo. +### Column mapping + +Each ticked table shows how many of its columns map. Columns match by name, the same spelling first and then the same name in another case. Click the count to point a column at a different destination column, or pick **Skip** to leave it out. **Match by Name** puts every column back. + +A destination column takes one source column. Map a second column onto it and **Transfer** stays off until one of the two moves or is skipped. + ## Clipboard paste (CSV/TSV) Select a row in the data grid and press `Cmd+V` to paste tabular data straight in. Tabs parse as TSV, commas as CSV. @@ -322,7 +328,7 @@ A server that rejects the statement stops the import with that error, so clear t The sheet accepts an array of objects `[{…}, {…}]`, newline-delimited JSON streamed a line at a time, and TablePro's own export shape `{ "table": [ {…} ] }`, so an export round-trips. Then choose a destination: -- **Existing table**: map each JSON field to a column. Fields auto-match by name; switch one off to skip it. A column with no matching field keeps its default or NULL. The list holds tables and partitioned tables, and the server routes each row to the right partition. +- **Existing table**: map each JSON field to a column, as [Map fields to columns](#map-fields-to-columns) describes. A column with no field mapped to it keeps its default or NULL. The list holds tables and partitioned tables, and the server routes each row to the right partition. - **New table**: the name field opens on a name derived from the file, already selected, so one keystroke replaces it. Review the inferred columns underneath. Name, type, primary key, nullable flag, and default are all editable before the table is created. The proposed name drops the extension, turns spaces and punctuation into underscores, and lowercases the result. Letters from any script are kept as they are. On Oracle the name comes through in upper case instead, and is cut to 30 bytes rather than 63. A name an existing table or view already holds gains a numeric suffix, so re-importing `users.csv` next to a `users` table proposes `users_2`. Whatever you type over it is held to the same rules: a name already taken, one longer than the engine allows, or one starting with a prefix the engine keeps for itself is reported in the sheet, with **Import** off until it changes. @@ -346,7 +352,7 @@ The workbook is read whole rather than streamed, because a sheet's rows refer ba ### Import CSV -CSV and TSV open the same sheet as JSON, with parsing options in front of the mapping. The delimiter and encoding are detected from the file; change any option and the mapping re-reads it. +CSV and TSV open the same sheet as JSON, with parsing options in front of the mapping. The delimiter and encoding are detected from the file. Changing any option reads the file again: a field that is still there keeps the column it was mapped to, and a new table's column list is rebuilt. | Option | What it does | Default | |--------|-------------|---------| @@ -360,6 +366,22 @@ CSV and TSV open the same sheet as JSON, with parsing options in front of the ma Quoted fields keep embedded commas and newlines (RFC 4180), and a doubled quote (`""`) decodes to one. A line the encoding cannot read stops the import before any row is deleted or inserted, and the error names the line. +### Map fields to columns + +Every field in the file gets a row with a checkbox and a column menu. A field named like a column, ignoring case, starts mapped to it. The rest start on **Skip**. + +**Match Columns** above the rows fills them all at once: + +| Item | Result | +|---|---| +| **Match by Name** | Each field goes to the column of the same name, ignoring case. Fields with no such column are skipped | +| **Match by Position** | The first field goes to the first column, the second to the second, and so on. Dimmed for JSON, whose fields are listed alphabetically | +| **Use Saved Mapping** | Puts back the mapping saved for this table | + +Clicking **Import** saves each choice that differs from Match by Name for that table, whether the import succeeds or not. The next file imported into the same table starts from those choices, and the sheet reads **Restored the mapping saved for** and the table name. A saved column the table no longer has falls back to the name match. Importing into a new table saves any column you renamed, so the same file lands on the renamed columns next time. + +Saved mappings stay on this Mac. They follow a table renamed from the sidebar, and are forgotten when the table is dropped from the sidebar or the connection is deleted. A `DROP TABLE` run in the SQL editor leaves them in place, so a table created again under the same name starts from the old mapping. + ### Row imports CSV and JSON insert in batches, and the on-error and transaction options behave as they do for SQL. They add **Delete existing rows before import**, which clears the target table inside the import transaction, so a failure in the default Stop and Rollback mode brings the rows back. A row where no field matches a mapped column writes nothing and is reported rather than counted.